new proc-macro-based benchmarking syntax (#12924)

* add stub for new benchmark macro * benchmark syntax * add #[extrinsic call] separator * parse #[benchmark] item as a function * proper emission of error when #[extrinsic_call] annotation is missing * clean up * enclosing module via benchmarks! { } working * use an attribute macro on the module instead of benchmarks! { } * cargo fmt * working component implementation * WIP * working * add syntax for Linear<A, B> * parsing of param ranges (still need to build tuple though) * params parsing WIP * clean up (don't need extrinsic call name) * use proper Result syntax for BenchmarkDef parsing * proper parsing of Linear<0, 1> style args * successfully parse and make use of linear component ranges 💥 * rename support variable => home because eventually will be moved * compile-time check that param range types implement ParamRange * switch to using balances as example, failing on instance pallet * successfully set up __origin and __call with balances 💥 * clean up * use a module * don't need a variable for transfer * rename benchmark_transfer -> transfer because no longer conflicts * clean up * working with transfer_increasing_users as well 💥 * re-add BareBlock * add comments for undocumented structs+functions+traits * refactor in preparation for removing module requirements * switch to a block instead of a module * use the outer macro pattern to to enable #[benchmarks] aggregation * successfully generate SelectedBenchmark 💥 * implement components for SelectedBenchmark * implement instance for SelectedBenchmark * properly track #[extra] * working impl for fn benchmarks() * run_benchmarks WIP * finish run_benchmark! impl 💥 * import balances transfer_best_case benchmark * import transfer_keep_alive balances pallet benchmark * import set_balance_creating balances pallet benchmark * import set_balance_killing balances pallet benchmark * import force_transfer balances pallet benchmark * add #[extra] annotation and docs to transfer_increasing_users * import transfer_all balances pallet benchmark * import force_unreserve balances pallet benchmark * prepare to implement impl_benchmark_test_suite! * ensure tests cover #[extra] before and after #[benchmark] tag * refactor * clean up * fix * move to outer * switch to benchmarks/instance_benchmarks * test impl almost done, strange compiler error * benchmark test suites working 💥 * clean up * add stub and basic parsing for where_clause * working except where clause and extrinsic calls containing method chains * assume option (2) for now wrt https://github.com/paritytech/substrate/pull/12924#issuecomment-1372938718 * clean up * switch to attribute-style * properly handle where clauses * fix subtle missing where clause, now just MessageQueue issues * fix block formatting in message-queue pallet * switch to block vs non-block parsing of extrinsic call * working now but some benchmark tests failing * message-queue tests working (run order issue fixed) 🎉 * add comments and internal docs for fame_support_procedural::benchmark * fix license years * docs for lib.rs * add docs to new support procedural macros * don't allow #[benchmark] outside of benchmarking module * add docs * use benchmark(extra, skip_meta) style args * update docs accordingly * appease clippy * bump ci * add notes about `extra` and `skip_meta` * fix doc tests * re-run CI * use `ignore` instead of `no_run` on doc examples * bump CI * replace some if-lets with if-elses * more refactoring of if-let statements * fix remaining if-lets in BenchmarkDef::from() * fix if-lets in benchmarks() * fix remaining if-lets, use nested find_map for extrinsic call * switch to use #[extrinsic_call] or #[block] situationally * refactor ExtrinsicCallDef => BenchmarkCallDef * update docs with info about #[block] * add macro stub for #[extrinsic_call] * fix docs and add stub for #[block] as well * remove unused extern crate line * fix clippy nits * Use V2 bench syntax in pallet-example-basic Just testing the dev-ex... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * carry over comment * use curly-brace style for impl_benchmark_test_suite! * remove unneeded parenthesis * proper handling of _() extrinsic call style * add docs for _() syntax * fix crate access * simplify keyword access Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * simplify module content destructuring Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix crate access "frame_benchmarking" => "frame-benchmarking", compiles * use _() extrinsic call syntax where possible in balances * simplify attr.path.segments.last() Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix compile error being suppressed * simplify extrinsic call keyword parsing Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * use ? operator instead of return None Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * rename generics => type_use_generics rename full_generics => type_impl_generics * simplify extrinsic call extraction with transpose * bump CI * nit * proper handling of too many + too few block/extrinsic call annotations * change to B >= A Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * remove unneeded ignore Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * remove another ignore Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add ui tests * use _() style extrinsic call on accumulate_dummy Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add range check to ParamRange * ui test for bad param ranges * fix failing example * add ignore back to other failing example * tweak expr_call span Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fix typo * eliminate a match Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * change pub fn benchmarks to return Result<TokenStream> * fix origin error span * more informative error for invalid benchmark parameter name * fix spans on a few benchmark errors * remove unneeded clone * refactor inner loop of benchmark function parsing * preserve mod attributes * refactor outer loop of benchmark def parsing code, greatly simplified * simplify to use a ? operator when parsing benchmark attr path * fix another ? operator * further simplify benchmark function attr parsing with more ? ops * refactor extrinsic call handling to use if let rather than match * replace is_ok => is_err Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * re-use name during expansion of benchmark def * remove unneeded clone * fix span for origin missing error * fix missing semi Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> Co-authored-by: parity-processbot <>
2026-04-26 19:17:58 +00:00 · 2023-01-23 02:07:48 -05:00
parent 82075c1d26
commit 42e5c27c84
38 changed files with 1761 additions and 135 deletions
@@ -2745,5 +2745,228 @@ pub mod pallet_macros {
 	};
 }

+/// Contains macros, structs, and traits associated with v2 of the pallet benchmarking syntax.
+/// This module contains macros, structs, and traits associated with v2 of the pallet
+/// benchmarking syntax.
+///
+/// The [`benchmarking::benchmarks`] and [`benchmarking::instance_benchmarks`] macros can be
+/// used to designate a module as a benchmarking module that can contain benchmarks and
+/// benchmark tests. The `#[benchmarks]` variant will set up a regular, non-instance
+/// benchmarking module, and the `#[instance_benchmarks]` variant will set up the module in
+/// instance benchmarking mode.
+///
+/// Benchmarking modules should be gated behind a `#[cfg(feature = "runtime-benchmarks")]`
+/// feature gate to ensure benchmarking code that is only compiled when the
+/// `runtime-benchmarks` feature is enabled is not referenced.
+///
+/// The following is the general syntax for a benchmarks (or instance benchmarks) module:
+///
+/// ## General Syntax
+///
+/// ```ignore
+/// #![cfg(feature = "runtime-benchmarks")]
+///
+/// use super::{mock_helpers::*, Pallet as MyPallet};
+/// use frame_support::benchmarking::*;
+/// use frame_benchmarking::whitelisted_caller;
+///
+/// #[benchmarks]
+/// mod benchmarks {
+/// 	use super::*;
+///
+/// 	#[benchmark]
+/// 	fn bench_name_1(x: Linear<7, 1_000>, y: Linear<1_000, 100_0000>) {
+/// 		// setup code
+/// 		let z = x + y;
+/// 		let caller = whitelisted_caller();
+///
+/// 		#[extrinsic_call]
+/// 		extrinsic_name(SystemOrigin::Signed(caller), other, arguments);
+///
+/// 		// verification code
+/// 		assert_eq!(MyPallet::<T>::my_var(), z);
+/// 	}
+///
+/// 	#[benchmark]
+/// 	fn bench_name_2() {
+/// 		// setup code
+/// 		let caller = whitelisted_caller();
+///
+/// 		#[block]
+/// 		{
+/// 			something(some, thing);
+/// 			my_extrinsic(RawOrigin::Signed(caller), some, argument);
+/// 			something_else(foo, bar);
+/// 		}
+///
+/// 		// verification code
+/// 		assert_eq!(MyPallet::<T>::something(), 37);
+/// 	}
+/// }
+/// ```
+///
+/// ## Benchmark Definitions
+///
+/// Within a `#[benchmarks]` or `#[instance_benchmarks]` module, you can define individual
+/// benchmarks using the `#[benchmark]` attribute, as shown in the example above.
+///
+/// The `#[benchmark]` attribute expects a function definition with a blank return type and
+/// zero or more arguments whose names are valid `frame_benchmarking::BenchmarkParamater`
+/// parameters, such as `x`, `y`, `a`, `b`, etc., and whose param types must implement
+/// [`benchmarking::ParamRange`]. At the moment the only valid type that implements
+/// [`benchmarking::ParamRange`] is [`benchmarking::Linear`].
+///
+/// The valid syntax for defining a `Linear` is `Linear<A, B>` where `A`, and `B` are
+/// valid integer literals (that fit in a `u32`), such that `B` >= `A`.
+///
+/// Note that the benchmark function definition does not actually expand as a function
+/// definition, but rather is used to automatically create a number of impls and structs
+/// required by the benchmarking engine. For this reason, the visibility of the function
+/// definition as well as the return type are not used for any purpose and are discarded by the
+/// expansion code.
+///
+/// Also note that the `// setup code` and `// verification code` comments shown above are not
+/// required and are included simply for demonstration purposes.
+///
+/// ### `#[extrinsic_call]` and `#[block]`
+///
+/// Within the benchmark function body, either an `#[extrinsic_call]` or a `#[block]`
+/// annotation is required. These attributes should be attached to a block (shown in
+/// `bench_name_2` above) or a one-line function call (shown in `bench_name_1` above, in `syn`
+/// parlance this should be an `ExprCall`), respectively.
+///
+/// The `#[block]` syntax is broad and will benchmark any code contained within the block the
+/// attribute is attached to. If `#[block]` is attached to something other than a block, a
+/// compiler error will be emitted.
+///
+/// The one-line `#[extrinsic_call]` syntax must consist of a function call to an extrinsic,
+/// where the first argument is the origin. If `#[extrinsic_call]` is attached to an item that
+/// doesn't meet these requirements, a compiler error will be emitted.
+///
+/// As a short-hand, you may substitute the name of the extrinsic call with `_`, such as the
+/// following:
+///
+/// ```ignore
+/// 	#[extrinsic_call]
+/// _(RawOrigin::Signed(whitelisted_caller()), 0u32.into(), 0);
+/// ```
+///
+/// The underscore will be substituted with the name of the benchmark  (i.e. the name of the
+/// function in the benchmark function definition).
+///
+/// Regardless of whether `#[extrinsic_call]` or `#[block]` is used, this attribute also serves
+/// the purpose of designating the boundary between the setup code portion of the benchmark
+/// (everything before the `#[extrinsic_call]` or `#[block]` attribute) and the verification
+/// stage (everything after the item that the `#[extrinsic_call]` or `#[block]` attribute is
+/// attached to). The setup code section should contain any code that needs to execute before
+/// the measured portion of the benchmark executes. The verification section is where you can
+/// perform assertions to verify that the extrinsic call (or whatever is happening in your
+/// block, if you used the `#[block]` syntax) executed successfully.
+///
+/// Note that neither `#[extrinsic_call]` nor `#[block]` are real attribute macros and are
+/// instead consumed by the outer macro pattern as part of the enclosing benchmark function
+/// definition. This is why we are able to use `#[extrinsic_call]` and `#[block]` within a
+/// function definition even though this behavior has not been stabilized
+/// yet—`#[extrinsic_call]` and `#[block]` are parsed and consumed as part of the benchmark
+/// definition parsing code, so they never expand as their own attribute macros.
+///
+/// ### Optional Attributes
+///
+/// The keywords `extra` and `skip_meta` can be provided as optional arguments to the
+/// `#[benchmark]` attribute, i.e. `#[benchmark(extra, skip_meta)]`. Including either of these
+/// will enable the `extra` or `skip_meta` option, respectively. These options enable the same
+/// behavior they did in the old benchmarking syntax in `frame_benchmarking`, namely:
+///
+/// #### `extra`
+///
+/// Specifies that this benchmark should not normally run. To run benchmarks marked with
+/// `extra`, you will need to invoke the `frame-benchmarking-cli` with `--extra`.
+///
+/// #### `skip_meta`
+///
+/// Specifies that the benchmarking framework should not analyze the storage keys that
+/// benchmarked code read or wrote. This useful to suppress the prints in the form of unknown
+/// 0x… in case a storage key that does not have metadata. Note that this skips the analysis
+/// of all accesses, not just ones without metadata.
+///
+/// ## Where Clause
+///
+/// Some pallets require a where clause specifying constraints on their generics to make
+/// writing benchmarks feasible. To accomodate this situation, you can provide such a where
+/// clause as the (only) argument to the `#[benchmarks]` or `#[instance_benchmarks]` attribute
+/// macros. Below is an example of this taken from the `message-queue` pallet.
+///
+/// ```ignore
+/// #[benchmarks(
+/// 	where
+/// 		<<T as Config>::MessageProcessor as ProcessMessage>::Origin: From<u32> + PartialEq,
+/// 		<T as Config>::Size: From<u32>,
+/// )]
+/// mod benchmarks {
+/// 	use super::*;
+/// 	// ...
+/// }
+/// ```
+///
+/// ## Benchmark Tests
+///
+/// Benchmark tests can be generated using the old syntax in `frame_benchmarking`,
+/// including the `frame_benchmarking::impl_benchmark_test_suite` macro.
+///
+/// An example is shown below (taken from the `message-queue` pallet's `benchmarking` module):
+/// ```ignore
+/// #[benchmarks]
+/// mod benchmarks {
+/// 	use super::*;
+/// 	// ...
+/// 	impl_benchmark_test_suite!(
+/// 		MessageQueue,
+/// 		crate::mock::new_test_ext::<crate::integration_test::Test>(),
+/// 		crate::integration_test::Test
+/// 	);
+/// }
+/// ```
+pub mod benchmarking {
+	pub use frame_support_procedural::{
+		benchmark, benchmarks, block, extrinsic_call, instance_benchmarks,
+	};
+
+	// Used in #[benchmark] implementation to ensure that benchmark function arguments
+	// implement [`ParamRange`].
+	#[doc(hidden)]
+	pub use static_assertions::assert_impl_all;
+
+	/// Used by the new benchmarking code to specify that a benchmarking variable is linear
+	/// over some specified range, i.e. `Linear<0, 1_000>` means that the corresponding variable
+	/// is allowed to range from `0` to `1000`, inclusive.
+	///
+	/// See [`frame_support::benchmarking`] for more info.
+	pub struct Linear<const A: u32, const B: u32>;
+
+	/// Trait that must be implemented by all structs that can be used as parameter range types
+	/// in the new benchmarking code (i.e. `Linear<0, 1_000>`). Right now there is just
+	/// [`Linear`] but this could later be extended to support additional non-linear parameter
+	/// ranges.
+	///
+	/// See [`frame_support::benchmarking`] for more info.
+	pub trait ParamRange {
+		/// Represents the (inclusive) starting number of this [`ParamRange`].
+		fn start(&self) -> u32;
+
+		/// Represents the (inclusive) ending number of this [`ParamRange`]
+		fn end(&self) -> u32;
+	}
+
+	impl<const A: u32, const B: u32> ParamRange for Linear<A, B> {
+		fn start(&self) -> u32 {
+			return A
+		}
+
+		fn end(&self) -> u32 {
+			return B
+		}
+	}
+}
+
 // Generate a macro that will enable/disable code based on `std` feature being active.
 sp_core::generate_feature_enabled_macro!(std_enabled, feature = "std", $);