Automated Testframework for Contracts

RolandRoland South Tyrol, ItalyMember Posts: 26 ✭✭
Hi, this seems much needed. In modern software development you have automated tests that act like a specification to test your contract against. While developing you would like to know if the most crucial behaviors of your contract is still fullfilled. You want a "fixture" where you can load a clean state and test each scenario automatically.
My favorite is Behavior Driven Development and the formulation of test cases in Given/When/Then prases.
For the Egalitarian Contract for example: "GIVEN a 5 members of the DAO, and 2 voted for a contract change. WHEN a third member votes for the contract change, THEN the change should be applied".

Also great for contract audits, you see what is covered and what not by the audit.
First barrier: do I need to instantiate a whole testnet each time I want to run a test?


  • JasperJasper Eindhoven, the NetherlandsMember Posts: 514 ✭✭✭
    There is cll-sim, the tests are coded, not such statements. Also it runs CLL/HLL turned into python, which seems perfectly fine for now. Hopefully in the future the ethereum VM will run.(defaultly)

    Probably functions could be added that get pretty close to specifying the behavior as you talk about, if len(dao.members()) == 5 and dao.votes_contract_change() == 3: dao.check_changes_applied(), but all those functions have to be implemented. The test doesnt know how the contract stores that information.

    If things are done standardly, there aught to be a way to do less reimplementing. But i am not sure how to do that. (Not entirely sure if that is better, even)

    I also like the idea of 'transaction firewalls'. Where you specify what sort of transactions a contract can send. So anything outside that is automaticly a bug. These would also be useful when programs generate transactions for you, as they can then make claims, limiting what they can do maliciously, or at least giving any malicious behavior more chance of being visible.
  • RolandRoland South Tyrol, ItalyMember Posts: 26 ✭✭
    edited April 2014
    This comes pretty close to what I have in mind @Jasper?. Ideally we have a Domain Specific Language that goes like

    Given (size of daoMembers > 5) and (dao votes for newContractFeature == 3)
    when tx == newVote
    then dao contract featueres include newContractFeature
  • RolandRoland South Tyrol, ItalyMember Posts: 26 ✭✭
    And I also like the idea of tx firewalls, preconditions, postconditions, invariants!
  • RolandRoland South Tyrol, ItalyMember Posts: 26 ✭✭
    I realized this is more important than expected:
    In real life Ethereum Contracts would be both the contract and the person in charge to execute the contract. If we divide these roles we could increase security and transparency of contracts. If the GivenWhenThen specification is seen as the contract and the Ethereum Contract is the mandatee that executes the contract, then people could know what application of the mandatee are safe to execute. Eventually a pattern language of GivenWhenThen contracts will emerge, people will demand certain properties to be fullfilled. People could rate certain aspects of a contract and demand improvements. Yes people comprehend examples not algorithms so user experience of Ethereum can improve.
    Any thoughts? @Jasper? @Stephan_Tual? ?
  • JasperJasper Eindhoven, the NetherlandsMember Posts: 514 ✭✭✭
    @Roland: you mean that 'if when then' would just be used as something to put feedback in aswel as for testing? Latter sounds like not a bad idea.

    To be honest i am not entirely sure how behavioral testing is done, but it seems to be at least in part what we're infact already doing. In this context, the difference between `given` and `when` is that `given` is a requirement on the state of the contract, and `when` is a requirement on the transaction?

    Having been a lisp enthousiast, I am a bit in the code=data(or rather, a continuum) camp; i.e. i'd rather have the programming language itself suiting the test.

    @mids106 might be interested in this discussion aswel.
  • RolandRoland South Tyrol, ItalyMember Posts: 26 ✭✭
    This is how I feel: if you split the concept of a contract in two parts (1) the specification and (2) the execution you gain transparency and security. In real life a contract is not executable, it describes expectations a party must fulfill. Likewise the tests could be a specification that describe what a contract must execute to be valid. It can do other things as well, but must do what is written in the specification.
    Reaching to the stars:
    Can these specifications be written general enough to cover for example a Security? What are the minimum requirements a security must fulfill? Could we have a library of financial specifications?
  • chris613chris613 Member Posts: 93 ✭✭
    @gavofyork? Nice to see you posting :) As you can see, people are interested in getting serious about contract coding. I noticed your YellowPaper and think it's a huge step forward in formal clarity on how the protocol and VM operates. Thanks!

    If people want to move ahead with related projects, like stabilizing the existing compilers (which are all over the map!), setting up testing frameworks, etc, would you be able to give us a gut feeling for the level of stability that can be expected in the VM semantics at this point? Are we at 90% of what you want to go live with, or still in the 40-50% range with ongoing experimentation? I'm asking specifically about opcodes, memory/storage implementation details, and LLL language features (I saw the recent patches for this, will there be more?).

    I imagine if you're trying to get some of the UX and IDE framework as well as a usable mining algo in place for December, that work on the protocol fundamentals must be tapering off soon?

    @Roland I like the idea of spec based programming. I feel it leads to a bit more bloat, but it's usually worth it in exchange for actually delivering what is desired. Also, for most lay people, a declarative spec for a smart contract would read much closer to a conventional legal contract than would an HLL algorithm. Most people CAN read a bit of code if they put their mind to it, but it doesn't read as easily as a short paragraph of declarative specification.
Sign In or Register to comment.