Commit Graph

111 Commits

Author SHA1 Message Date
Omar d02152b565 Update version of tests (#202)
* Update the commit hash of resolc compiler tests

* Update the version of tests
compilation-caches-v1.0
2025-11-02 23:48:23 +00:00
Omar 75159229df Update the commit hash of resolc compiler tests (#201) 2025-11-02 21:27:26 +00:00
Omar 2af1a62319 Supply the revert reason in the logs (#200) 2025-11-02 17:54:11 +00:00
Omar e09be4f3fa Remove references to kitchensink (#199)
* Remove references to kitchensink

* Update the ci for the revive-dev-node

* Update references to the substrate node

* Add the step path to the failure logs

* Update the CI

* fix machete

* Update tests

* Update the commit hash of the polkadot sdk

* Ignore the tx mine test
2025-11-01 05:30:43 +00:00
Omar 33b5faca45 Add revert reason to the assertion logs (#198) 2025-11-01 01:50:58 +00:00
Omar 172fb4700f Cleanup benchmarks (#197)
* Require test argument

* Increase tx timeout and channel limits

* Add default arguments for tests

* Fix tests

* Fix tests

* Cleanup benchmarks
2025-10-30 01:32:23 +00:00
Omar fefea17c8e Require test argument (#196)
* Require test argument

* Increase tx timeout and channel limits

* Add default arguments for tests

* Fix tests

* Fix tests
2025-10-27 00:17:47 +00:00
Omar b71445b632 Wire up reporting to benchmarks (#195)
* Modify the structure of the `MinedBlockInformation`

* Report the step path to the watcher

* Make report format more benchmark friendly

* make report more benchmarks friendly

* Add more models to the report

* Remove corpus from the report

* Add step information to the benchmark report

* Include the contract information in the report

* Add the block information to the report

* compute metrics in each report

* Cleanup watcher from temp code
2025-10-24 02:15:29 +00:00
Omar f1a911545e Update the commit hash of the test suite (#194) 2025-10-22 08:32:47 +00:00
Omar 48e7d69158 Drop the read lock in the remaining tasks logger (#193)
* Drop the read lock in the remaining tasks logger

* Increase substrate's timeout
2025-10-21 10:28:51 +00:00
Omar 260ac5d98e Add support for profiles (#192)
* Add support for profiles

* Set a default workdir for debug profile
2025-10-20 10:27:51 +00:00
Omar 94f116f843 Make tests a submodule of the repo (#191) 2025-10-16 00:11:06 +00:00
Omar 0d7a87a728 Get rid of corpus files (#190)
* Get rid of corpus files

* Update the readme
2025-10-15 22:40:09 +00:00
Omar 29bf5304ec User Managed Nodes (#189)
* Allow for genesis to be exported by the tool

* Allow for substrate-based nodes to be managed by the user

* Rename the commandline argument

* Rename the commandline argument

* Move existing rpc option to revive-dev-node

* Remove unneeded test

* Remove un-required function in cached compiler

* Change the default concurrency limit

* Update the default number of threads

* Update readme

* Remove accidentally comitted dir

* Update the readme

* Update the readme
2025-10-15 16:32:20 +00:00
Omar 491c23efb3 Remove the revive network (#188)
* Remove the revive network

* Add a provider method to the `EthereumNode`

* Report the ref time and proof size for substrate chains in block information

* Remove un-needed dependency
2025-10-14 13:50:36 +00:00
Omar 3c86cbb7ef Make output format deserializable (#187)
* Make output format deserializable

* Flush the buffer after writing the entire file output
2025-10-09 15:41:26 +00:00
Omar fde07b7c0d Allow for succeeding tests to be ignored (#186) 2025-10-09 14:35:09 +00:00
Omar ebc24a588b Add different output formats (#185)
* Add different output formats

* Add the mode to the output
2025-10-09 14:24:14 +00:00
Omar 21e25f09e6 Zombienet & Benchmarks Cleanups (#184)
* Minor zombienet cleanups

* Remove un-necessary trace call from the benchmark driver

* Improve the benchmarks driver

* Ignore the lighthouse tests

* Allow for the consensus to be specified for the revive dev node

* Ignore the zombienet tests for the time being
2025-10-09 11:41:01 +00:00
Marios 8c412dc924 Use absolute path for zombienet to avoid conflicts (#178) 2025-10-07 16:03:18 +00:00
Omar 6da3172581 Fix/fallback gas for substrate (#177)
* Update compiler semaphore

* Fix the fallback gas for substrate chains

* Fix the concurrency & the substrate gas limit fallback value
2025-10-07 02:25:37 +00:00
Marios c6eb04b04e Support zombienet (#172)
* Basic zombie node definition

* [WIP] - Impl EthereumNode for zombie node

* Remove unused imports

* [WIP] Support substrate node in zombienet network

* Impl zombie node resolver && node for zombie node

* Spawn eth-rpc on top of collator node

* Implement ZombienetPlatform and integrate zombie node with eth rpc

* Add wourkaround to run tests

* Add few comments

* fmt

* Replace default transaction request in test

* Merge - Fix conficts with main

* fmt

* Clippy fix

* Add polkadot and parachain node to ci

* CI - Fetch polkadot binaries from releases

* Fix unit test assertion

* Minor doc improvements

* Change names from ZombieNet to Zombienet and switch from Command to Process

* Refactor ZombieNode to cache provider using OnceCell

* CI: Cache polkadot binaries and use them if available

* Fix conficts with main

* Refactor shared_node to return static reference and add shared_state for context access

* fmt

* Rename ZombienetConfiguration to PolkadotParachainConfiguration and update related usage
2025-10-07 02:24:20 +00:00
Omar e5114d31dc Update compiler semaphore (#176) 2025-10-05 22:28:57 +00:00
Omar 74fdeb4a2e Core Benchmarking Infra (#175)
* Implement a solution for the pre-fund account limit

* Update the account pre-funding handling

* Fix the lighthouse node tracing issue

* refactor existing dt infra

* Implement the platform driver

* Wire up the cleaned up driver implementation

* Implement the core benchmarking components

* Remove some debug logging

* Fix issues in the benchmarks driver

* Implement a global concurrency limit on provider requests

* Update the concurrency limit

* Update the concurrency limit

* Cleanups

* Update the lighthouse ports

* Ignore certain tests

* Update the new geth test
2025-10-05 15:09:01 +00:00
Omar f9dc362c03 Lighthouse Node (#173)
* Add a lighthouse node implementation

* Implement production geth using kurtosis

* Connect the lighthouse node with the platforms

* Update the ci to include cargo fmt

* Add rustfmt to ci

* Add formatting component for macos

* Fix CI

* Add the cargo clippy component

* Install kurtosis in cli

* fix ci

* Skip lighthouse tests in MacOS in CI

* Increase the wait duration of kurtosis
2025-09-28 12:44:19 +00:00
Omar c2ba2cfed6 Cleanup Processes (#171)
* Clean up the process flow for nodes

* Cleanup some of the node tests to use shared nodes

* Commit dev-genesis
2025-09-24 02:47:36 +00:00
Omar 3dda739cef Add step paths (#163)
* Support repetitions in the tool

* Add support for account allocations

* Update the JSON schema

* Support virtual repeats

* Add a step path

* Update the schema
2025-09-22 03:53:16 +00:00
Omar 97e3f8bbff Support virtual repeats for benchmarks (#162)
* Support repetitions in the tool

* Add support for account allocations

* Update the JSON schema

* Support virtual repeats
2025-09-22 03:35:31 +00:00
Omar 7189361a58 Allow account allocations (#161)
* Support repetitions in the tool

* Add support for account allocations

* Update the JSON schema
2025-09-22 03:19:55 +00:00
Omar 9b700bfec2 Support repetitions in the tool (#160) 2025-09-22 03:03:59 +00:00
Omar 98b62d705f Make our traits object safe and implement the ReviveDevNodePolkaVMResolc target. (#159)
* Generate schema for the metadata file

* Groundwork for dyn traits

* Make the ethereum node trait object compatible

* Allow for compilers to be created in the dyn trait

* Add more identifiers to the platform

* Implement the dyn compiler trait for compilers

* Support the dyn compiler in the builder pattern

* Introduce a geth platform

* Provide a common node implementation for substrate chains

* Add all of the platforms that we support

* Add a way to convert platform identifier into a platform

* Replace infra with the dyn infra

* Remoe all references to leader and follower

* Remove the old traits

* Remove an un-needed dependency

* Update the default values for the platforms

* Final set of renames

* Update the default values of the cli

* Update tests
2025-09-19 21:59:28 +00:00
Omar 1a894f791a Generate schema for the metadata file (#158) 2025-09-08 16:52:57 +00:00
Omar c2526e48e7 Refactor the Global Configuration & Context (#157)
* Cleanup the config

* Update usage guides

* Update the run script

* Fix tests

* Use kitchensink in tests

* Use shared node more often in tests
2025-09-04 14:25:05 +00:00
Omar 7878f68c26 Better Compiler Interface & Shared Compiler Objects (#156)
* Add leader and follower node assignment to test

* Update the compilers interface

* Fix Cargo machete

* Add reporting back to the compilers

* Remove the static testing target from the report

* Uncomment instrument macro

* Switch to a for loop when reporting cases

* Update compilers to use interior caching

* Update tests stream func

* Fix tests
2025-08-28 15:03:45 +00:00
Omar 8b1afc36a3 Better errors in report (#155) 2025-08-26 15:35:19 +00:00
Omar 60328cd493 Add a Quick Run Script (#152)
* Add a quick run script

* Add more context to errors

* Fix the issue with corpus directory canonicalization

* Update the quick run script

* Edit the runner script

* Support specifying the path of the polkadot sdk
2025-08-25 21:03:28 +00:00
Omar eb264fcc7b feature/fix abi finding resolc (#154)
* Configure kitchensink to use devnode by default

* Update the kitchensink tests

* Fix the logic for finding the ABI in resolc

* Edit how CLI reporter prints
2025-08-25 20:47:29 +00:00
Omar 84b139d3b4 Configure kitchensink to use devnode by default (#153)
* Configure kitchensink to use devnode by default

* Update the kitchensink tests
2025-08-25 15:46:06 +00:00
Omar d93824d973 Updated Reporting Infrastructure (#151)
* Remove the old reporting infra

* Use the Test struct more in the code

* Implement the initial set of reporter events

* Add more runner events to the reporter and refine the structure

* Add reporting infra for reporting ignored tests

* Update report to use better map data structures

* Add case status information to the report

* Integrate the reporting infrastructure with the
CLI reporter used by the program.

* Include contract compilation information in report

* Cleanup report model

* Add information on the deployed contracts
2025-08-25 11:16:09 +00:00
Omar bec5a7e390 Increase Kitchensink maximum http connections (#148)
* Throttle the Kitchensink requests

* Increase max connections limit for kitchensink
2025-08-20 22:25:17 +00:00
Omar 85033cfead Update the readme (#145) 2025-08-19 17:41:26 +00:00
Omar 76d6a154c1 Fix concurrency issues (#142)
* Fix the OS FD error

* Cache the compiler versions

* Allow for auto display impl in declare wrapper type macro

* Better logging and fix concurrency issues

* Fix tests

* Format

* Make the code even more concurrent
2025-08-19 06:47:36 +00:00
Omar c58551803d Allow multiple files in corpus (#144) 2025-08-16 16:04:17 +00:00
Omar 185edcfad9 Cached compiler artifacts (#143)
* WIP compilation cache

* Implement a persistent compilation cache

* Correct the key and value encoding for the cache
2025-08-16 16:04:13 +00:00
James Wilson 09d56f5177 Redo how we parse and use modes (#125)
* WIP redo how we parse and use modes

* test expanding, too

* WIP integrate new Mode/ParsedMode into rest of code

* First pass integrated new mode bits

* fmt

* clippy

* Remove mode we no longer support from test metadata

* Address nits

* Add ability for compiler to opt out if it can't work with some Mode/version

* Elide viaIR input if compiler does not support it

* Improve test output a little; string modes and list ignored tests

* Move Mode to common crate

* constants.mod, and Display for CaseIdx to use it

* fmt

* Rename ModePipeline::E/Y

* Re-arrange Mode things; ParsedMode in format and Mode etc in common

* Move compile check to prepare_tests

* Remove now-unused deps

* clippy nits

* Update fallback tx weights to avoid out of gas errors

* Update kitchensink weights too and fmt

* Bump default geth timeout to 10s

* 30s timeout

* Improve geth stdout logging on failure

* fix line logging

* remove --networkid and arg, back to 5s timeout for geth
2025-08-16 11:38:17 +00:00
Omar a59e287fa1 Add a cached fs abstraction (#141) 2025-08-14 15:21:05 +00:00
Omar f2045db0e9 Add compiler directives to metadata (#139) 2025-08-14 07:38:56 +00:00
Omar 5a11f44673 Misc features/improvements (#138)
* Implement various needed features and improvements

* Reorder the metadata struct

* Format comments
2025-08-13 13:50:06 +00:00
James Wilson 46aea0890d Split reporter and case runner, use channels to pass test reports (#137)
* Use channels to send data to reporting thread and avoid hangs / mutex / duration. Limit max concurrent tasks to avoid too many open files

* More appropriate name for dirver/reporter task fns

* Back to parallelise individual cases, report individual cases, address grumbles

* newline before 'Failures' title in report
2025-08-13 13:10:26 +00:00
Omar 9b40c9b9e3 Add an EVM version filter (#136)
* Add an EVM version filter

* Update naming
2025-08-12 10:19:59 +00:00