new proc-macro-based benchmarking syntax (#12924)

* add stub for new benchmark macro * benchmark syntax * add #[extrinsic call] separator * parse #[benchmark] item as a function * proper emission of error when #[extrinsic_call] annotation is missing * clean up * enclosing module via benchmarks! { } working * use an attribute macro on the module instead of benchmarks! { } * cargo fmt * working component implementation * WIP * working * add syntax for Linear<A, B> * parsing of param ranges (still need to build tuple though) * params parsing WIP * clean up (don't need extrinsic call name) * use proper Result syntax for BenchmarkDef parsing * proper parsing of Linear<0, 1> style args * successfully parse and make use of linear component ranges 💥 * rename support variable => home because eventually will be moved * compile-time check that param range types implement ParamRange * switch to using balances as example, failing on instance pallet * successfully set up __origin and __call with balances 💥 * clean up * use a module * don't need a variable for transfer * rename benchmark_transfer -> transfer because no longer conflicts * clean up * working with transfer_increasing_users as well 💥 * re-add BareBlock * add comments for undocumented structs+functions+traits * refactor in preparation for removing module requirements * switch to a block instead of a module * use the outer macro pattern to to enable #[benchmarks] aggregation * successfully generate SelectedBenchmark 💥 * implement components for SelectedBenchmark * implement instance for SelectedBenchmark * properly track #[extra] * working impl for fn benchmarks() * run_benchmarks WIP * finish run_benchmark! impl 💥 * import balances transfer_best_case benchmark * import transfer_keep_alive balances pallet benchmark * import set_balance_creating balances pallet benchmark * import set_balance_killing balances pallet benchmark * import force_transfer balances pallet benchmark * add #[extra] annotation and docs to transfer_increasing_users * import transfer_all balances pallet benchmark * import force_unreserve balances pallet benchmark * prepare to implement impl_benchmark_test_suite! * ensure tests cover #[extra] before and after #[benchmark] tag * refactor * clean up * fix * move to outer * switch to benchmarks/instance_benchmarks * test impl almost done, strange compiler error * benchmark test suites working 💥 * clean up * add stub and basic parsing for where_clause * working except where clause and extrinsic calls containing method chains * assume option (2) for now wrt https://github.com/paritytech/substrate/pull/12924#issuecomment-1372938718 * clean up * switch to attribute-style * properly handle where clauses * fix subtle missing where clause, now just MessageQueue issues * fix block formatting in message-queue pallet * switch to block vs non-block parsing of extrinsic call * working now but some benchmark tests failing * message-queue tests working (run order issue fixed) 🎉 * add comments and internal docs for fame_support_procedural::benchmark * fix license years * docs for lib.rs * add docs to new support procedural macros * don't allow #[benchmark] outside of benchmarking module * add docs * use benchmark(extra, skip_meta) style args * update docs accordingly * appease clippy * bump ci * add notes about `extra` and `skip_meta` * fix doc tests * re-run CI * use `ignore` instead of `no_run` on doc examples * bump CI * replace some if-lets with if-elses * more refactoring of if-let statements * fix remaining if-lets in BenchmarkDef::from() * fix if-lets in benchmarks() * fix remaining if-lets, use nested find_map for extrinsic call * switch to use #[extrinsic_call] or #[block] situationally * refactor ExtrinsicCallDef => BenchmarkCallDef * update docs with info about #[block] * add macro stub for #[extrinsic_call] * fix docs and add stub for #[block] as well * remove unused extern crate line * fix clippy nits * Use V2 bench syntax in pallet-example-basic Just testing the dev-ex... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * carry over comment * use curly-brace style for impl_benchmark_test_suite! * remove unneeded parenthesis * proper handling of _() extrinsic call style * add docs for _() syntax * fix crate access * simplify keyword access Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * simplify module content destructuring Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix crate access "frame_benchmarking" => "frame-benchmarking", compiles * use _() extrinsic call syntax where possible in balances * simplify attr.path.segments.last() Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix compile error being suppressed * simplify extrinsic call keyword parsing Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * use ? operator instead of return None Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * rename generics => type_use_generics rename full_generics => type_impl_generics * simplify extrinsic call extraction with transpose * bump CI * nit * proper handling of too many + too few block/extrinsic call annotations * change to B >= A Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * remove unneeded ignore Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * remove another ignore Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add ui tests * use _() style extrinsic call on accumulate_dummy Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add range check to ParamRange * ui test for bad param ranges * fix failing example * add ignore back to other failing example * tweak expr_call span Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix typo * eliminate a match Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * change pub fn benchmarks to return Result<TokenStream> * fix origin error span * more informative error for invalid benchmark parameter name * fix spans on a few benchmark errors * remove unneeded clone * refactor inner loop of benchmark function parsing * preserve mod attributes * refactor outer loop of benchmark def parsing code, greatly simplified * simplify to use a ? operator when parsing benchmark attr path * fix another ? operator * further simplify benchmark function attr parsing with more ? ops * refactor extrinsic call handling to use if let rather than match * replace is_ok => is_err Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * re-use name during expansion of benchmark def * remove unneeded clone * fix span for origin missing error * fix missing semi Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> Co-authored-by: parity-processbot <>
2026-04-26 21:37:56 +00:00 · 2023-01-23 02:07:48 -05:00
parent 82075c1d26
commit 42e5c27c84
38 changed files with 1761 additions and 135 deletions
@@ -0,0 +1,36 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2020-2023 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 	http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[rustversion::attr(not(stable), ignore)]
+#[cfg(not(feature = "disable-ui-tests"))]
+#[test]
+fn benchmark_ui() {
+	// Only run the ui tests when `RUN_UI_TESTS` is set.
+	if std::env::var("RUN_UI_TESTS").is_err() {
+		return
+	}
+
+	// As trybuild is using `cargo check`, we don't need the real WASM binaries.
+	std::env::set_var("SKIP_WASM_BUILD", "1");
+
+	// Deny all warnings since we emit warnings as part of a Pallet's UI.
+	std::env::set_var("RUSTFLAGS", "--deny warnings");
+
+	let t = trybuild::TestCases::new();
+	t.compile_fail("tests/benchmark_ui/*.rs");
+	t.pass("tests/benchmark_ui/pass/*.rs");
+}
@@ -0,0 +1,18 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench(winton: Linear<1, 2>) {
+		let a = 2 + 2;
+		#[block]
+		{}
+		assert_eq!(a, 4);
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Benchmark parameter names must consist of a single lowercase letter (a-z) and no other characters.
+  --> tests/benchmark_ui/bad_param_name.rs:10:11
+   |
+10 |     fn bench(winton: Linear<1, 2>) {
+   |              ^^^^^^
@@ -0,0 +1,14 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	#[benchmark]
+	fn bench(xx: Linear<1, 2>) {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Benchmark parameter names must consist of a single lowercase letter (a-z) and no other characters.
+ --> tests/benchmark_ui/bad_param_name_too_long.rs:8:11
+  |
+8 |     fn bench(xx: Linear<1, 2>) {
+  |              ^^
@@ -0,0 +1,14 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	#[benchmark]
+	fn bench(D: Linear<1, 2>) {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Benchmark parameter names must consist of a single lowercase letter (a-z) and no other characters.
+ --> tests/benchmark_ui/bad_param_name_upper_case.rs:8:11
+  |
+8 |     fn bench(D: Linear<1, 2>) {
+  |              ^
@@ -0,0 +1,16 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench(x: Linear<3, 1>) {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: The start of a `ParamRange` must be less than or equal to the end
+  --> tests/benchmark_ui/bad_param_range.rs:10:21
+   |
+10 |     fn bench(x: Linear<3, 1>) {
+   |                        ^
@@ -0,0 +1,18 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench(y: Linear<1, 2>, x: u32) {
+		let a = 2 + 2;
+		#[block]
+		{}
+		assert_eq!(a, 4);
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Invalid benchmark function param. A valid example would be `x: Linear<5, 10>`.
+  --> tests/benchmark_ui/bad_params.rs:10:31
+   |
+10 |     fn bench(y: Linear<1, 2>, x: u32) {
+   |                                  ^^^
@@ -0,0 +1,20 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench() {
+		let a = 2 + 2;
+		#[block]
+		{}
+		#[block]
+		{}
+		assert_eq!(a, 4);
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Only one #[extrinsic_call] or #[block] attribute is allowed per benchmark.
+  --> tests/benchmark_ui/dup_block.rs:14:3
+   |
+14 |         #[block]
+   |         ^
@@ -0,0 +1,20 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench() {
+		let a = 2 + 2;
+		#[extrinsic_call]
+		_(stuff);
+		#[extrinsic_call]
+		_(other_stuff);
+		assert_eq!(a, 4);
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Only one #[extrinsic_call] or #[block] attribute is allowed per benchmark.
+  --> tests/benchmark_ui/dup_extrinsic_call.rs:14:3
+   |
+14 |         #[extrinsic_call]
+   |         ^
@@ -0,0 +1,16 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark(extra, extra)]
+	fn bench() {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: unexpected end of input, `extra` can only be specified once
+ --> tests/benchmark_ui/extra_extra.rs:9:26
+  |
+9 |     #[benchmark(extra, extra)]
+  |                             ^
@@ -0,0 +1,16 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark(skip_meta, skip_meta)]
+	fn bench() {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: unexpected end of input, `skip_meta` can only be specified once
+ --> tests/benchmark_ui/extra_skip_meta.rs:9:34
+  |
+9 |     #[benchmark(skip_meta, skip_meta)]
+  |                                     ^
@@ -0,0 +1,6 @@
+use frame_support::benchmarking::*;
+
+#[extrinsic_call]
+mod stuff {}
+
+fn main() {}
@@ -0,0 +1,7 @@
+error: `#[extrinsic_call]` must be in a benchmark function definition labeled with `#[benchmark]`.
+ --> tests/benchmark_ui/extrinsic_call_out_of_fn.rs:3:1
+  |
+3 | #[extrinsic_call]
+  | ^^^^^^^^^^^^^^^^^
+  |
+  = note: this error originates in the attribute macro `extrinsic_call` (in Nightly builds, run with -Z macro-backtrace for more info)
@@ -0,0 +1,13 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench() {}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: No valid #[extrinsic_call] or #[block] annotation could be found in benchmark function body.
+  --> tests/benchmark_ui/missing_call.rs:10:13
+   |
+10 |     fn bench() {}
+   |                ^^
@@ -0,0 +1,16 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark]
+	fn bench() {
+		#[extrinsic_call]
+		thing();
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: Single-item extrinsic calls must specify their origin as the first argument.
+  --> tests/benchmark_ui/missing_origin.rs:12:3
+   |
+12 |         thing();
+   |         ^^^^^
@@ -0,0 +1,17 @@
+use frame_support::benchmarking::*;
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark(skip_meta, extra)]
+	fn bench() {
+		let a = 2 + 2;
+		#[block]
+		{}
+		assert_eq!(a, 4);
+	}
+}
+
+fn main() {}
@@ -0,0 +1,16 @@
+use frame_support::benchmarking::*;
+#[allow(unused_imports)]
+use frame_support_test::Config;
+
+#[benchmarks]
+mod benches {
+	use super::*;
+
+	#[benchmark(skip_meta, extra, bad)]
+	fn bench() {
+		#[block]
+		{}
+	}
+}
+
+fn main() {}
@@ -0,0 +1,5 @@
+error: expected `extra` or `skip_meta`
+ --> tests/benchmark_ui/unrecognized_option.rs:9:32
+  |
+9 |     #[benchmark(skip_meta, extra, bad)]
+  |                                   ^^^