* function generation with _name working, need to modify signature
* WIP
* support custom BenchmarkResult<T> type
* full support for BenchmarkResult<T> on benchmark function defs
* support () return type for benchmark function defs that don't use ?
* uncomment
* fix where clause handling
* fix benchmark function call bodies
* proper parsing of return type
* add UI tests for bad return type
* fix detection of missing last_stmt with defined return type
* UI tests covering missing last_stmt
* properly detect and complain about empty benchmark function defs
* fix missing Comma in Result<T, BenchmarkError> parsing + test
* add additional UI test
* allow complex path for BenchmarkResult and BenchmarkError in fn defs
* add UI tests covering complex path for BenchmarkResult, BenchmarkError
* retain doc comments and attributes
* also add attributes to struct
* add docs for benchmark function definition support
* fix imports on benchmark example
* fix issue with unused variables in extrinsic call fn def
* fix up docs
* remove support for v2::BenchmarkResult because it was confusing
* fix typo
* remove ability to use custom T for Result<T, BenchmarkError> in v2
* use missing call error instead of empty_fn()
* remove unneeded match statement
* Add a proper QED
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* fix other QED
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* cargo fmt
* add an explicit error for non TypePath as return type
* tweak error warning and add a UI test for non TypePath return
* remove comment
* add docs about T and I generic params
* improve docs referring to section "below"
* pull out return type checking logic into its own function
* pull out params parsing into its own function
* pull out call_def parsing into its own function
* add doc comment for missing_call()
* replace spaces with tabs
* add a result-based example to the benchmarking examples
---------
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* add stub for new benchmark macro
* benchmark syntax
* add #[extrinsic call] separator
* parse #[benchmark] item as a function
* proper emission of error when #[extrinsic_call] annotation is missing
* clean up
* enclosing module via benchmarks! { } working
* use an attribute macro on the module instead of benchmarks! { }
* cargo fmt
* working component implementation
* WIP
* working
* add syntax for Linear<A, B>
* parsing of param ranges (still need to build tuple though)
* params parsing WIP
* clean up (don't need extrinsic call name)
* use proper Result syntax for BenchmarkDef parsing
* proper parsing of Linear<0, 1> style args
* successfully parse and make use of linear component ranges 💥
* rename support variable => home because eventually will be moved
* compile-time check that param range types implement ParamRange
* switch to using balances as example, failing on instance pallet
* successfully set up __origin and __call with balances 💥
* clean up
* use a module
* don't need a variable for transfer
* rename benchmark_transfer -> transfer because no longer conflicts
* clean up
* working with transfer_increasing_users as well 💥
* re-add BareBlock
* add comments for undocumented structs+functions+traits
* refactor in preparation for removing module requirements
* switch to a block instead of a module
* use the outer macro pattern to to enable #[benchmarks] aggregation
* successfully generate SelectedBenchmark 💥
* implement components for SelectedBenchmark
* implement instance for SelectedBenchmark
* properly track #[extra]
* working impl for fn benchmarks()
* run_benchmarks WIP
* finish run_benchmark! impl 💥
* import balances transfer_best_case benchmark
* import transfer_keep_alive balances pallet benchmark
* import set_balance_creating balances pallet benchmark
* import set_balance_killing balances pallet benchmark
* import force_transfer balances pallet benchmark
* add #[extra] annotation and docs to transfer_increasing_users
* import transfer_all balances pallet benchmark
* import force_unreserve balances pallet benchmark
* prepare to implement impl_benchmark_test_suite!
* ensure tests cover #[extra] before and after #[benchmark] tag
* refactor
* clean up
* fix
* move to outer
* switch to benchmarks/instance_benchmarks
* test impl almost done, strange compiler error
* benchmark test suites working 💥
* clean up
* add stub and basic parsing for where_clause
* working except where clause and extrinsic calls containing method chains
* assume option (2) for now wrt https://github.com/paritytech/substrate/pull/12924#issuecomment-1372938718
* clean up
* switch to attribute-style
* properly handle where clauses
* fix subtle missing where clause, now just MessageQueue issues
* fix block formatting in message-queue pallet
* switch to block vs non-block parsing of extrinsic call
* working now but some benchmark tests failing
* message-queue tests working (run order issue fixed) 🎉
* add comments and internal docs for fame_support_procedural::benchmark
* fix license years
* docs for lib.rs
* add docs to new support procedural macros
* don't allow #[benchmark] outside of benchmarking module
* add docs
* use benchmark(extra, skip_meta) style args
* update docs accordingly
* appease clippy
* bump ci
* add notes about `extra` and `skip_meta`
* fix doc tests
* re-run CI
* use `ignore` instead of `no_run` on doc examples
* bump CI
* replace some if-lets with if-elses
* more refactoring of if-let statements
* fix remaining if-lets in BenchmarkDef::from()
* fix if-lets in benchmarks()
* fix remaining if-lets, use nested find_map for extrinsic call
* switch to use #[extrinsic_call] or #[block] situationally
* refactor ExtrinsicCallDef => BenchmarkCallDef
* update docs with info about #[block]
* add macro stub for #[extrinsic_call]
* fix docs and add stub for #[block] as well
* remove unused extern crate line
* fix clippy nits
* Use V2 bench syntax in pallet-example-basic
Just testing the dev-ex...
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* carry over comment
* use curly-brace style for impl_benchmark_test_suite!
* remove unneeded parenthesis
* proper handling of _() extrinsic call style
* add docs for _() syntax
* fix crate access
* simplify keyword access
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* simplify module content destructuring
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* fix crate access "frame_benchmarking" => "frame-benchmarking", compiles
* use _() extrinsic call syntax where possible in balances
* simplify attr.path.segments.last()
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* fix compile error being suppressed
* simplify extrinsic call keyword parsing
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* use ? operator instead of return None
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* rename generics => type_use_generics
rename full_generics => type_impl_generics
* simplify extrinsic call extraction with transpose
* bump CI
* nit
* proper handling of too many + too few block/extrinsic call annotations
* change to B >= A
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* remove unneeded ignore
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* remove another ignore
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* add ui tests
* use _() style extrinsic call on accumulate_dummy
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* add range check to ParamRange
* ui test for bad param ranges
* fix failing example
* add ignore back to other failing example
* tweak expr_call span
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* fix typo
* eliminate a match
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* change pub fn benchmarks to return Result<TokenStream>
* fix origin error span
* more informative error for invalid benchmark parameter name
* fix spans on a few benchmark errors
* remove unneeded clone
* refactor inner loop of benchmark function parsing
* preserve mod attributes
* refactor outer loop of benchmark def parsing code, greatly simplified
* simplify to use a ? operator when parsing benchmark attr path
* fix another ? operator
* further simplify benchmark function attr parsing with more ? ops
* refactor extrinsic call handling to use if let rather than match
* replace is_ok => is_err
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* re-use name during expansion of benchmark def
* remove unneeded clone
* fix span for origin missing error
* fix missing semi
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: parity-processbot <>
* Generate one #[test] fn per bench case.
* Update benchmark macro syntax in frame pallets.
* Explain new benchmark macro syntax in example pallet.
* support with and without a semicolon
* update pallets to use individual tests
* migrate staking too
* migrate more pallets
* fix up democracy and use individual tests
* Fix comment
* Put println message in panic
* Remove `another_set_dummy` from doc
`another_set_dummy` is not present in the benchmarking.rs (anymore).
* Update doc for benchmarks macro
* Update doc for impl_benchmark_test_suite macro
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Use 'Pallet' struct in construct_runtime.
* Fix genesis and metadata macro.
* Fix 'Pallet' type alias.
* Replace 'Module' with 'Pallet' for all construct_runtime use cases.
* Replace more deprecated 'Module' struct.
* Bring back AllModules and AllPalletsWithSystem type, but deprecate them.
* Replace deprecated 'Module' struct from merge master.
* Minor fix.
* Fix UI tests.
* Revert UI override in derive_no_bound.
* Fix more deprecated 'Module' use from master branch.
* Fix more deprecated 'Module' use from master branch.
* Create a macro which automates creation of benchmark test suites.
* bump impl_version
* allow unused on test_bench_by_name
* use proper doctest ignore attribute
* Explicitly hand the Module to the test suite
Much better practice than depending on it showing up implicitly in
the namespace.
* explicitly import what we need into `mod tests`
* bench_module is `ident` not `tt`
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* allow end users to specify arguments for new_test_ext
This turned out to be surprisingly easy. On reflection, it turns out
that of course the compiler can't eagerly evaluate the function call,
but needs to paste it in everywhere desired.
* enable explicitly specifying the path to the benchmarks invocation
also enable optional trailing commas
* Revert "bump impl_version"
This reverts commit 0209e4de33fd43873f8cfc6875815d0fd6151e63.
* list failing benchmark tests and the errors which caused the failure
* harden benchmark tests against internal panics
* suppress warning about ignored profiles
unfortunately, setting the profile here doesn't do anything; we'd
need to set it in every leaf package anyway. However, as this was
just making the default explicit anyway, I think it's safe enough
to remove entirely.
* impl_benchmark_test_suite for assets
* impl_benchmark_test_suite for balances
* impl_benchmark_test_suite for bounties
* impl_benchmark_test_suite for Collective
* impl_benchmark_test_suite for Contracts
* impl_benchmark_test_suite for Democracy
* don't impl_benchmark_test_suite for Elections-Phragmen
* impl_benchmark_test_suite for Identity
Note that Identity tests currently fail. They failed in an identical
way before this change, so as far as I'm concerned, the status quo is
good enough for now.
* impl_benchmark_test_suite for ImOnline
* impl_benchmark_test_suite for indices
For this crate also, the test suite fails identically with and without
this change, so we can say that this change is not the cause of the
tests' failure to compile.
* impl_benchmark_test_suite for lottery
* impl_benchmark_test_suite for merkle-mountain-range
* impl_benchmark_test_suite for Multisig
These tests fail identically with and without the change, so the change
seems unlikely to be the origin of the failures.
* impl_benchmark_test_suite for offences
* impl_benchmark_test_suite for Proxy
Fails identically with and without this change.
* impl_benchmark_test_suite for scheduler
* impl_benchmark_test_suite for session
It turns out to be important to be able to exclude items marked
`#[extra]` sometimes. Who knew?
* impl_benchmark_test_suite for staking
* impl_benchmark_test_suite for system
* impl_benchmark_test_suite for timestamp
* impl_benchmark_test_suite for tips
* impl_benchmark_test_suite for treasury
* impl_benchmark_test_suite for utility
Note that benchmark tests fail identically before and after this change.
* impl_benchmark_test_suite for vesting
* fix wrong module name in impl_benchmark_test_suite in Offences
* address line length nits
* enable optional keyword argument: exec_name
Took a _lot_ of macro-wrangling to get the functionality that I want,
but now you have the option to pass in
```rust
impl_benchmark_test_suite!(
Elections,
crate::tests::ExtBuilder::default().desired_members(13).desired_runners_up(7),
crate::tests::Test,
exec_name = build_and_execute,
);
```
and have it expand out properly. A selected fragment of the expansion:
```rust
fn test_benchmarks() {
crate::tests::ExtBuilder::default()
.desired_members(13)
.desired_runners_up(7)
.build_and_execute(|| {
```
* get rid of dead code
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* commented use of common
* hack to pass tests
* another hack
* remove all commented code
* fix the easy tests
* temp hack
* follow through comma hack until better solution
* patch macro
* missed one
* update benchmarks
* update docs
* fix docs
* removed too much
* fix changes
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Update balance benchmarks
* Update weight functions
* Remove user component
* make componentless
* Add support for `#[extra]` tag on benchmarks
* Update balances completely
* Apply suggestions from code review
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
* Fix some tests
* Maybe fix to test. Need approval from @tomusdrw this is okay
* Make test better
* keep weights conservative
* Update macro for merge master
* Add headers
* Apply suggestions from code review
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* Update to latest staking
* generate tests for benchmarking
* add tests, fix warnings
* starting on democracy
* impl_benchmark_tests
* Way more readable
* add test feature flag (does this work?)
* Fix `successful_origin` impl
* democracry benchmark tests
* Fix example benchmarks, add tests
* identity benchmark tests
* Update im-online benchmark tests
* try to add session benchmarking tests (problem with mock)
* staking and timestamp
* add test for treasury, issue with dynamic contains
* utility
* Vesting
* test instead of check
* hide until we figure out what is wrong
* add docs
* close code
* Create custom mock for session-pallet-benchmarking
* Use refcell pattern
* make un-pub
* test-linux-stable includes `runtime-benchmarks` feature
* Revert "test-linux-stable includes `runtime-benchmarks` feature"
This reverts commit a2dab38abd18ac3eb8a6220e4a00e687740bd38c.
* run tests in `--release`
* undo balance change
* build wasm
* Benchmark pallet
* Add a bunch more benchmarks
* do nothing test
* new benchmarks
* Clean up extra tests
* Encode and Decode Vec<T::AccountId>
* Starting to migrate benchmarks to macro
* Use macro
* Remove call and storage
* Update Cargo.toml
* Add storage recalc benchmark
* Add support for custom functions in benchmark! macro
* Reset DB for storage recalc
* Feedback from review
* Add more comments
* Remove benchmark pallet from node
* Fix cargo files
* Fix comments
* Change `crate` to `super`
* missed one
* Use results of benchmark encode/decode
* Pass generic to extra functions
* reset macro to master
* Update lib.rs
* Update to use standard syntax
This prs cleans up some of the frame benchmarking stuff:
- Move CLI into `frame-benchmarking-cli`. No frame related CLI should
exists in the default Substrate CLI.
- Move all traits and types related to frame benchmarking into the
`frame-benchmarking` trait. Frame types should be isolated in Frame.
* Initial transfer bench
* Add best case
* Transfer keep alive
* Set balance benchmarks
* Bump impl
* Fix text
Co-authored-by: Gavin Wood <github@gavwood.com>
Gavin Wood