Depending on the scope and complexity of your tests, it may be necessary to modify the default settings for how property-based tests are run.
The mechanism for doing this is thehypothesis.settingsobject. You can set up a @given based test to use this using a settings decorator:
from brownie.test import given from hypothesis settings
@given(strategy('uint256'))
@settings(max_examples=500) def test_this_thoroughly(x):
pass
You can also affect the settings permanently by adding a hypothesis field to your project’s brownie-config.
yamlfile:
Brownie Documentation, Release v1.6.2
hypothesis:
max_examples: 500
16.4.1 Available Settings
See the Hypothesissettings documentationfor a complete list of available settings. This section only lists settings where the default value has been changed.
deadline
The number of milliseconds that each individual example within a test is allowed to run. Tests that take longer than this time will be considered to have failed.
Because Brownie test times can vary widely, this property has been disabled by default.
default-value: None max_examples
The maximum number of times a test will be run before considering it to have passed.
For tests involving many complex transactions you may wish to reduce this value.
default-value: 50 stateful_step_count
The maximum number of rules to execute in a stateful program before ending the run and considering it to have passed.
For more complex state machines you may wish to increase this value - however you should keep in mind that this can result in siginificantly longer execution times.
default-value: 10
16.4. Settings 65
CHAPTER 17
Stateful Testing
Stateful testingis a more advanced method ofpropery-based testingused to test complex systems. In a stateful test you define a number of actions that can be combined together in different ways, and Hypothesis attempts to find a sequence of those actions that results in a failure. This is useful for testing complex contracts or contract-to-contract interactions where there are many possible states.
Brownie utilizes the hypothesis framework to allow for stateful testing.
Much of the content in this section is based on the officialhypothesis.workswebsite. To learn more about stateful testing, you may wish to read the following articles:
• Rule Based Stateful Testingby David R. MacIver
• Solving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesisby Nicholas Chammas
• Hypothesis Documentationon stateful testing
Warning: This functionality is still under development and should be considered experimental. Use common sense when evaluating the results, and if you encounter any problems pleaseopen an issueon Github.
17.1 Rule-based State Machines
A state machine is a class used within stateful testing. It defines the initial test state, a number of actions outlining the structure that the test will execute in, and invariants that should not be violated during execution.
Note: Unlike regular Hypothesis state machines, Brownie state machines should not subclass RuleBasedStateMachine.
67
17.1.1 Rules
At the core of every state machine are one or more rules. Rules are class methods that are very similar to @given based tests; they receive values drawn from strategies and pass them to a user defined test function. The key difference is that where @given based tests run independently, rules can be chained together - a single stateful test run may involve multiple rule invocations, which may interact in various ways.
Any state machine method named rule or begining with rule_ is treated as a rule.
class StateMachine:
def rule_one(self):
# performs a test action
def rule_two(self):
# performs another, different test action
17.1.2 Initializers
There is also a special type of rule known as an initializer. These are rules that are guaranteed to be executed at most one time at the beginning of a run (i.e. before any normal rule is called). They may be called in any order, or not at all, and the order will vary from run to run.
Any state machine method named initialize or beginning with initialize_ is treated as an initializer.
class StateMachine:
def initialize(self):
# this method may or may not be called prior to rule_two
def rule(self):
# once this method is called, initialize will not be called during the test
˓→run
17.1.3 Strategies
A state machine should contain one or moreStrategies, in order to provide data to it’s rules.
Strategies must be defined at the class level, typically before the first function. They can be given any name.
Similar to how fixtures work within pytest tests, state machine rules receive strategies by referencing them within their arguments. This is shown in the following example:
class StateMachine:
st_uint = strategy('uint256') st_bytes32 = strategy('bytes32') def initialize(self, st_uint):
# this method draws from the uint256 strategy
def rule(self, st_uint, st_bytes32):
# this method draws from both strategies
def rule_two(self, value="st_uint", othervalue="st_uint"):
# this method draws from the same strategy twice
Brownie Documentation, Release v1.6.2
17.1.4 Invariants
Along with rules, a state machine often defines invariants. These are properties that should remain unchanged, re-gardless of any actions performed by the rules. After each rule is executed, every invariant method is always called to ensure that the test has not failed.
Any state machine method named invariant or beginning with invariant_ is treated as an invariant. Invariants are meant for verifying correctness of state; they cannot receive strategies.
class StateMachine:
# assertions in this method should always pass regardless
# of actions in both rule_one and rule_two
17.1.5 Setup and Teardown
A state machine may optionally include setup and teardown procedures. Similar to pytest fixtures, setup and teardown methods are available to execute logic on a per-test and per-run basis.
classmethod StateMachine.__init__(cls, *args)
This method is called once, prior to the chain snapshot taken before the first test run. It is run as a class method - changes made to the state machine will persist through every run of the test.
__init__is the only method that can be used to pass external data into the state machine. In the following example, we use it to pass theaccountsfixture, and a deployed instance of a token contract:
class StateMachine:
def __init__(cls, accounts, token):
cls.accounts = accounts cls.token = token
def test_stateful(Token, accounts, state_machine):
token = Token.deploy("Test Token", "TST", 18, 1e23, {'from': accounts[0]})
# state_machine forwards all the arguments to StateMachine.__init__
state_machine(StateMachine, accounts, token) classmethod StateMachine.setup(self )
This method is called at the beginning of each test run, immediately after chain is reverted to the snapshot.
Changes applied during setup will only have an effect for the upcoming run.
classmethod StateMachine.teardown(self )
This method is called at the end of each successful test run, prior to the chain revert. teardown is not called if the run fails.
classmethod StateMachine.teardown_final(cls)
This method is called after the final test run has completed and the chain has been reverted. teardown_final is called regardless of whether the test passed or failed.
17.1. Rule-based State Machines 69