* Add an optional `targets` field to cases.
This PR adds an optional `targets` field to cases which takes presence
over that same field in the `Metadata`. The hope from this is to allow
us to limit specific tests so that they only run on specific platforms
only.
* Update the resolc tests submodule
* Update the default resolc version to use
* Update the default heap-size and stack-size in the cli
* Update the report processor
* Add a report processing tool
* Add expectations tests to the CI action
* Fix an issue with CI
* Fix CI
* Fix the path of the workdir in CI
* Fix CI issue with the paths
* Update the format of the expectations file
* Update report processor to only include failures
* Add a report processing tool
* Add expectations tests to the CI action
* Fix an issue with CI
* Fix CI
* Fix the path of the workdir in CI
* Fix CI issue with the paths
* Update the format of the expectations file
* Add a `ReceiptRetryLayer` for providers
* Fix the retry layer
* Rename the retry layer
* Remove outdated polling function
* Remoe unneeded dependencies
* Add code to disable the fallback gas filler
* Allow benchmarks driver to await tx receipts
* Improve the transaction submission logic
* Update Python Script to process Geth benchmarks
* Move all scripts to a single directory
* Switch to cargo-make
* Remove the polkadot-sdk from the submodules
* WIP: update the CI
* Add other jobs to CI
* Overhaul the polkadot-sdk caching step
* Add a testing step
* Fix the CI
* Install clang and llvm dependencies
* Update the version of clang
* Install llvm on macos
* Fix ci
* Fix ci
* Use 1.90.0 version of rust for the polkadot-sdk
* Fix CI
* Fix CI
* Fix CI
* Fix CI
* Fix CI
* Allow warnings
* Update runners
* Update runners
* Simplify CI
* Update MacOS runner
* Fix zombienet tests
* Make cache step faster
* Bump the version of resolc compiler tests
* Bump the version of resolc compiler tests
* Reduce the timeout for transactions to 2 minutes
* Bump resolc compiler tests
* Update the commit hash of resolc compiler tests
* Fix an issue with file errors in substrate export-chainspec
* Update the resolc compiler tests
* Fix the working directory canonicalization
* Remove references to kitchensink
* Update the ci for the revive-dev-node
* Update references to the substrate node
* Add the step path to the failure logs
* Update the CI
* fix machete
* Update tests
* Update the commit hash of the polkadot sdk
* Ignore the tx mine test
* Modify the structure of the `MinedBlockInformation`
* Report the step path to the watcher
* Make report format more benchmark friendly
* make report more benchmarks friendly
* Add more models to the report
* Remove corpus from the report
* Add step information to the benchmark report
* Include the contract information in the report
* Add the block information to the report
* compute metrics in each report
* Cleanup watcher from temp code
* Allow for genesis to be exported by the tool
* Allow for substrate-based nodes to be managed by the user
* Rename the commandline argument
* Rename the commandline argument
* Move existing rpc option to revive-dev-node
* Remove unneeded test
* Remove un-required function in cached compiler
* Change the default concurrency limit
* Update the default number of threads
* Update readme
* Remove accidentally comitted dir
* Update the readme
* Update the readme
* Remove the revive network
* Add a provider method to the `EthereumNode`
* Report the ref time and proof size for substrate chains in block information
* Remove un-needed dependency
* Minor zombienet cleanups
* Remove un-necessary trace call from the benchmark driver
* Improve the benchmarks driver
* Ignore the lighthouse tests
* Allow for the consensus to be specified for the revive dev node
* Ignore the zombienet tests for the time being
* Implement a solution for the pre-fund account limit
* Update the account pre-funding handling
* Fix the lighthouse node tracing issue
* refactor existing dt infra
* Implement the platform driver
* Wire up the cleaned up driver implementation
* Implement the core benchmarking components
* Remove some debug logging
* Fix issues in the benchmarks driver
* Implement a global concurrency limit on provider requests
* Update the concurrency limit
* Update the concurrency limit
* Cleanups
* Update the lighthouse ports
* Ignore certain tests
* Update the new geth test
* Add a lighthouse node implementation
* Implement production geth using kurtosis
* Connect the lighthouse node with the platforms
* Update the ci to include cargo fmt
* Add rustfmt to ci
* Add formatting component for macos
* Fix CI
* Add the cargo clippy component
* Install kurtosis in cli
* fix ci
* Skip lighthouse tests in MacOS in CI
* Increase the wait duration of kurtosis
* Support repetitions in the tool
* Add support for account allocations
* Update the JSON schema
* Support virtual repeats
* Add a step path
* Update the schema
* Generate schema for the metadata file
* Groundwork for dyn traits
* Make the ethereum node trait object compatible
* Allow for compilers to be created in the dyn trait
* Add more identifiers to the platform
* Implement the dyn compiler trait for compilers
* Support the dyn compiler in the builder pattern
* Introduce a geth platform
* Provide a common node implementation for substrate chains
* Add all of the platforms that we support
* Add a way to convert platform identifier into a platform
* Replace infra with the dyn infra
* Remoe all references to leader and follower
* Remove the old traits
* Remove an un-needed dependency
* Update the default values for the platforms
* Final set of renames
* Update the default values of the cli
* Update tests
* Add leader and follower node assignment to test
* Update the compilers interface
* Fix Cargo machete
* Add reporting back to the compilers
* Remove the static testing target from the report
* Uncomment instrument macro
* Switch to a for loop when reporting cases
* Update compilers to use interior caching
* Update tests stream func
* Fix tests
* Add a quick run script
* Add more context to errors
* Fix the issue with corpus directory canonicalization
* Update the quick run script
* Edit the runner script
* Support specifying the path of the polkadot sdk
* Configure kitchensink to use devnode by default
* Update the kitchensink tests
* Fix the logic for finding the ABI in resolc
* Edit how CLI reporter prints
* Remove the old reporting infra
* Use the Test struct more in the code
* Implement the initial set of reporter events
* Add more runner events to the reporter and refine the structure
* Add reporting infra for reporting ignored tests
* Update report to use better map data structures
* Add case status information to the report
* Integrate the reporting infrastructure with the
CLI reporter used by the program.
* Include contract compilation information in report
* Cleanup report model
* Add information on the deployed contracts
* Fix the OS FD error
* Cache the compiler versions
* Allow for auto display impl in declare wrapper type macro
* Better logging and fix concurrency issues
* Fix tests
* Format
* Make the code even more concurrent
* Make metadata serializable
* Refactor tests to use steps
* Add a balance assertion test step
* Test balance deserialization
* Box the test steps
* Permit size difference in step output
* Added a resolver tied to a specific block
* Increase the number of private keys
* Increase kitchensink wait time to 60 seconds
* Add a case ignore flag
* Parallelize over cases
* Rename the state and driver
* Parallelize execution
* Update the default config of the tool
* Make codebase async
* Fix machete
* Fix tests & clear node directories before startup
* Cleanup the cleanup logic
* Rename geth node
* Allow for downloader to use version requirements.
We will soon add support for the compiler version requirement from the
metadata files to be honored. The compiler version is specified in the
solc modes section of the file and its specified as a `VersionReq` and
not as a version.
Therefore, we need to have the ability to honor this version requirement
and find the best version that satisfies the requirement.
* Request `VersionOrRequirement` in compiler interface
* Honor the compiler version requirement in metadata
This commit honors the compiler version requirement listed in the solc
modes of the metadata file. If this version requirement is provided then
it overrides what was passed in the CLI. Otherwise, the CLI version will
be used.
* Make compiler IO completely generic.
Before this commit, the types that were used for the compiler input and
output were the resolc compiler types which was a leaky abstraction as
we have traits to abstract the compilers away but we expose their
internal types out to other crates.
This commit did the following:
1. Made the compiler IO types fully generic so that all of the logic for
constructing the map of compiled contracts is all done by the
compiler implementation and not by the consuming code.
2. Changed the input types used for Solc to be the forge standard JSON
types for Solc instead of resolc.
* Fix machete
* Add resolc to CI
* Add resolc to CI
* Add resolc to CI
* Add resolc to CI
* Allow for files to be specified in the corpus file
* Attempt to improve the geth tx indexing issue.
We're facing an issue where Geth transaction indexing can sometimes stall
on some of the nodes we're running. The logs show that for all transactions
we always need 1 second of waiting time. However, during certain runs we
sometimes run into an issue with some of the nodes where it seems like
their transaction indexer fails (either at the start or after some amount
of time) which leads us to never get the receipts back from these specific
nodes.
This is not a load issue as it appears like all of the other nodes handle
it just fine. However, it looks like once a node gets into this state it
can not get out of it and its bricked for the entire run.
This commit adds some more command line arguments to the geth command in
hopes of improving this issue.
* Re-order the input file.
This commit reorders the input file such that we have a definitions
section and an implementations section and such that the the order of
the items in both sections is the same.
* Implement a reverse polish calculator for calldata arithmetic
* Add a barebones common crate
* Refactor some code into the common crate
* Add a `ResolverApi` interface.
This commit adds a `ResolverApi` trait to the `format` crate that can be
implemented by any type that can act as a resolver. A resolver is able
to provide information on the chain state. This chain state could be
fresh or it could be cached (which is something that we will do in a
future PR).
This cleans up our crate graph so that `format` is not depending on the
node interactions crate for the `EthereumNode` trait.
* Cleanup the blocking executor
* Pre-seed accounts with more ETH.
This commit fixes and solves some issues around how much ETH we seed an
account with in genesis. Currently, any account that the node has keys
to sign for will be seeded with u128::MAX WEI in genesis. This also
includes the default signer account.
* Bump commit hash of polkadot SDK
* Change how the cache key is computed
* Revert "Change how the cache key is computed"
This reverts commit 75afdd9cfd.
* Revert "Bump commit hash of polkadot SDK"
This reverts commit 8aaa69780e.
* Add extra comments
* Revert "Add extra comments"
This reverts commit bd4de2c83d.
* Update the initial balance
* Add support for wrapper types
* Move `FilesWithExtensionIterator` to `core::common`
* Remove unneeded use of two `HashMap`s
* Make metadata structs more typed
* Impl new_from for wrapper types
* Implement the new input handling logic
* Fix edge-case in input handling
* Ignore macro doc comment tests
* Correct comment
* Fix edge-case in deployment order
* Handle calldata better
* Allow for the use of function signatures
* Add support for exceptions
* Cached nonce allocator
* Fix tests
* Add support for address replacement
* Cleanup implementation
* Cleanup mutability
* Wire up address replacement with rest of code
* Implement caller replacement
* Switch to callframe trace for exceptions
* Add a way to skip tests if they don't match the target
* Handle values from the metadata files
* Remove address replacement
* Correct the arguments
* Remove empty impl
* Remove address replacement
* Correct the arguments
* Remove empty impl
* Fix size_requirement underflow
* Add support for wildcards in exceptions
* Fix calldata construction of single calldata
* Better handling for length in equivalency checks
* Make initial balance a constant
* Fix size_requirement underflow
* Add support for wildcards in exceptions
* Fix calldata construction of single calldata
* Better handling for length in equivalency checks
* Fix tests
* Add support for wrapper types
* Move `FilesWithExtensionIterator` to `core::common`
* Remove unneeded use of two `HashMap`s
* Make metadata structs more typed
* Impl new_from for wrapper types
* Implement the new input handling logic
* Fix edge-case in input handling
* Ignore macro doc comment tests
* Correct comment
* Fix edge-case in deployment order
* Handle calldata better
* Remove todo
* Add support for wrapper types
* Move `FilesWithExtensionIterator` to `core::common`
* Remove unneeded use of two `HashMap`s
* Make metadata structs more typed
* Impl new_from for wrapper types
* Implement the new input handling logic
* Fix edge-case in input handling
* Ignore macro doc comment tests
* Correct comment
* Fix edge-case in deployment order
Omar