pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-04-26 21:37:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

[FRAME] Runtime Omni Bencher (#3512)

Browse Source 
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes https://github.com/paritytech/polkadot-sdk/issues/2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
This commit is contained in:
Oliver Tale-Yazdi
2024-04-08 18:03:56 +02:00
committed by GitHub
parent 216509dbaa
commit 9543d31474
33 changed files with 1064 additions and 307 deletions
Download Patch File Download Diff File Expand all files Collapse all files
substrate/utils/frame/benchmarking-cli/src/pallet/command.rs
+504 -256
View File
@@ -15,7 +15,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::{writer, ListOutput, PalletCmd};
use super::{
types::{ComponentRange, ComponentRangeMap},
writer, ListOutput, PalletCmd,
};
use crate::pallet::{types::FetchedCode, GenesisBuilder};
use codec::{Decode, Encode};
use frame_benchmarking::{
Analysis, BenchmarkBatch, BenchmarkBatchSplitResults, BenchmarkList, BenchmarkParameter,
@@ -23,23 +27,25 @@ use frame_benchmarking::{
};
use frame_support::traits::StorageInfo;
use linked_hash_map::LinkedHashMap;
use sc_cli::{execution_method_from_cli, CliConfiguration, Result, SharedParams};
use sc_chain_spec::json_patch::merge as json_merge;
use sc_cli::{execution_method_from_cli, ChainSpec, CliConfiguration, Result, SharedParams};
use sc_client_db::BenchmarkingState;
use sc_executor::{HeapAllocStrategy, WasmExecutor, DEFAULT_HEAP_ALLOC_STRATEGY};
use sc_service::Configuration;
use serde::Serialize;
use sp_core::{
offchain::{
testing::{TestOffchainExt, TestTransactionPoolExt},
OffchainDbExt, OffchainWorkerExt, TransactionPoolExt,
},
traits::{CallContext, ReadRuntimeVersionExt},
traits::{CallContext, CodeExecutor, ReadRuntimeVersionExt, WrappedRuntimeCode},
};
use sp_externalities::Extensions;
use sp_genesis_builder::{PresetId, Result as GenesisBuildResult};
use sp_keystore::{testing::MemoryKeystore, KeystoreExt};
use sp_runtime::traits::Hash;
use sp_state_machine::StateMachine;
use sp_state_machine::{OverlayedChanges, StateMachine};
use sp_wasm_interface::HostFunctions;
use std::{
borrow::Cow,
collections::{BTreeMap, BTreeSet, HashMap},
fmt::Debug,
fs,
@@ -50,17 +56,6 @@ use std::{
/// Logging target
const LOG_TARGET: &'static str = "polkadot_sdk_frame::benchmark::pallet";
/// The inclusive range of a component.
#[derive(Serialize, Debug, Clone, Eq, PartialEq)]
pub(crate) struct ComponentRange {
/// Name of the component.
name: String,
/// Minimal valid value of the component.
min: u32,
/// Maximal valid value of the component.
max: u32,
}
/// How the PoV size of a storage item should be estimated.
#[derive(clap::ValueEnum, Debug, Eq, PartialEq, Clone, Copy)]
pub enum PovEstimationMode {
@@ -87,7 +82,15 @@ impl FromStr for PovEstimationMode {
/// Maps (pallet, benchmark) -> ((pallet, storage) -> PovEstimationMode)
pub(crate) type PovModesMap =
HashMap<(Vec<u8>, Vec<u8>), HashMap<(String, String), PovEstimationMode>>;
HashMap<(String, String), HashMap<(String, String), PovEstimationMode>>;
#[derive(Debug, Clone)]
struct SelectedBenchmark {
pallet: String,
extrinsic: String,
components: Vec<(BenchmarkParameter, u32, u32)>,
pov_modes: Vec<(String, String)>,
}
// This takes multiple benchmark batches and combines all the results where the pallet, instance,
// and benchmark are the same.
@@ -145,41 +148,56 @@ This could mean that you either did not build the node correctly with the \
`--features runtime-benchmarks` flag, or the chain spec that you are using was \
not created by a node that was compiled with the flag";
/// When the runtime could not build the genesis storage.
const ERROR_CANNOT_BUILD_GENESIS: &str = "The runtime returned \
an error when trying to build the genesis storage. Please ensure that all pallets \
define a genesis config that can be built. This can be tested with: \
https://github.com/paritytech/polkadot-sdk/pull/3412";
/// Warn when using the chain spec to generate the genesis state.
const WARN_SPEC_GENESIS_CTOR: &'static str = "Using the chain spec instead of the runtime to \
generate the genesis state is deprecated. Please remove the `--chain`/`--dev`/`--local` argument, \
point `--runtime` to your runtime blob and set `--genesis-builder=runtime`. This warning may \
become a hard error any time after December 2024.";
/// The preset that we expect to find in the GenesisBuilder runtime API.
const GENESIS_PRESET: &str = "development";
impl PalletCmd {
/// Runs the command and benchmarks a pallet.
pub fn run<Hasher, ExtraHostFunctions>(&self, config: Configuration) -> Result<()>
#[deprecated(
note = "`run` will be removed after December 2024. Use `run_with_spec` instead or \
completely remove the code and use the `frame-benchmarking-cli` instead (see \
https://github.com/paritytech/polkadot-sdk/pull/3512)."
)]
pub fn run<Hasher, ExtraHostFunctions>(&self, config: sc_service::Configuration) -> Result<()>
where
Hasher: Hash,
ExtraHostFunctions: sp_wasm_interface::HostFunctions,
ExtraHostFunctions: HostFunctions,
{
self.run_with_spec::<Hasher, ExtraHostFunctions>(Some(config.chain_spec))
}
/// Runs the pallet benchmarking command.
pub fn run_with_spec<Hasher, ExtraHostFunctions>(
&self,
chain_spec: Option<Box<dyn ChainSpec>>,
) -> Result<()>
where
Hasher: Hash,
ExtraHostFunctions: HostFunctions,
{
self.check_args()?;
let _d = self.execution.as_ref().map(|exec| {
// We print the warning at the end, since there is often A LOT of output.
// We print the error at the end, since there is often A LOT of output.
sp_core::defer::DeferGuard::new(move || {
log::warn!(
log::error!(
target: LOG_TARGET,
"⚠️ Argument `--execution` is deprecated. Its value of `{exec}` has on effect.",
)
})
});
if let Some(output_path) = &self.output {
if !output_path.is_dir() && output_path.file_name().is_none() {
return Err("Output file or path is invalid!".into())
}
}
if let Some(header_file) = &self.header {
if !header_file.is_file() {
return Err("Header file is invalid!".into())
};
}
if let Some(handlebars_template_file) = &self.template {
if !handlebars_template_file.is_file() {
return Err("Handlebars template file is invalid!".into())
};
}
if let Some(json_input) = &self.json_input {
let raw_data = match std::fs::read(json_input) {
Ok(raw_data) => raw_data,
@@ -194,16 +212,10 @@ impl PalletCmd {
return self.output_from_results(&batches)
}
let spec = config.chain_spec;
let pallet = self.pallet.clone().unwrap_or_default();
let pallet = pallet.as_bytes();
let (genesis_storage, genesis_changes) =
self.genesis_storage::<Hasher, ExtraHostFunctions>(&chain_spec)?;
let mut changes = genesis_changes.clone();
let extrinsic = self.extrinsic.clone().unwrap_or_default();
let extrinsic_split: Vec<&str> = extrinsic.split(',').collect();
let extrinsics: Vec<_> = extrinsic_split.iter().map(|x| x.trim().as_bytes()).collect();
let genesis_storage = spec.build_storage()?;
let mut changes = Default::default();
let cache_size = Some(self.database_cache_size as usize);
let state_with_tracking = BenchmarkingState::<Hasher>::new(
genesis_storage.clone(),
@@ -225,11 +237,10 @@ impl PalletCmd {
let method =
execution_method_from_cli(self.wasm_method, self.wasmtime_instantiation_strategy);
let heap_pages =
self.heap_pages
.map_or(DEFAULT_HEAP_ALLOC_STRATEGY, |p| HeapAllocStrategy::Static {
extra_pages: p as _,
});
let state = &state_without_tracking;
let runtime = self.runtime_blob(&state_without_tracking)?;
let runtime_code = runtime.code()?;
let alloc_strategy = Self::alloc_strategy(runtime_code.heap_pages);
let executor = WasmExecutor::<(
sp_io::SubstrateHostFunctions,
@@ -237,91 +248,30 @@ impl PalletCmd {
ExtraHostFunctions,
)>::builder()
.with_execution_method(method)
.with_onchain_heap_alloc_strategy(heap_pages)
.with_offchain_heap_alloc_strategy(heap_pages)
.with_allow_missing_host_functions(self.allow_missing_host_functions)
.with_onchain_heap_alloc_strategy(alloc_strategy)
.with_offchain_heap_alloc_strategy(alloc_strategy)
.with_max_runtime_instances(2)
.with_runtime_cache_size(2)
.build();
let extensions = || -> Extensions {
let mut extensions = Extensions::default();
let (offchain, _) = TestOffchainExt::new();
let (pool, _) = TestTransactionPoolExt::new();
let keystore = MemoryKeystore::new();
extensions.register(KeystoreExt::new(keystore));
extensions.register(OffchainWorkerExt::new(offchain.clone()));
extensions.register(OffchainDbExt::new(offchain));
extensions.register(TransactionPoolExt::new(pool));
extensions.register(ReadRuntimeVersionExt::new(executor.clone()));
extensions
};
// Get Benchmark List
let state = &state_without_tracking;
let result = StateMachine::new(
state,
&mut changes,
&executor,
"Benchmark_benchmark_metadata",
&(self.extra).encode(),
&mut extensions(),
&sp_state_machine::backend::BackendRuntimeCode::new(state).runtime_code()?,
CallContext::Offchain,
)
.execute()
.map_err(|e| format!("{}: {}", ERROR_METADATA_NOT_FOUND, e))?;
let (list, storage_info) =
<(Vec<BenchmarkList>, Vec<StorageInfo>) as Decode>::decode(&mut &result[..])
.map_err(|e| format!("Failed to decode benchmark metadata: {:?}", e))?;
let (list, storage_info): (Vec<BenchmarkList>, Vec<StorageInfo>) =
Self::exec_state_machine(
StateMachine::new(
state,
&mut changes,
&executor,
"Benchmark_benchmark_metadata",
&(self.extra).encode(),
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
ERROR_METADATA_NOT_FOUND,
)?;
// Use the benchmark list and the user input to determine the set of benchmarks to run.
let mut benchmarks_to_run = Vec::new();
list.iter()
.filter(|item| pallet.is_empty() || pallet == &b"*"[..] || pallet == &item.pallet[..])
.for_each(|item| {
for benchmark in &item.benchmarks {
let benchmark_name = &benchmark.name;
if extrinsic.is_empty() ||
extrinsic.as_bytes() == &b"*"[..] ||
extrinsics.contains(&&benchmark_name[..])
{
benchmarks_to_run.push((
item.pallet.clone(),
benchmark.name.clone(),
benchmark.components.clone(),
benchmark.pov_modes.clone(),
))
}
}
});
// Convert `Vec<u8>` to `String` for better readability.
let benchmarks_to_run: Vec<_> = benchmarks_to_run
.into_iter()
.map(|(pallet, extrinsic, components, pov_modes)| {
let pallet_name =
String::from_utf8(pallet.clone()).expect("Encoded from String; qed");
let extrinsic_name =
String::from_utf8(extrinsic.clone()).expect("Encoded from String; qed");
(
pallet,
extrinsic,
components,
pov_modes
.into_iter()
.map(|(p, s)| {
(String::from_utf8(p).unwrap(), String::from_utf8(s).unwrap())
})
.collect(),
pallet_name,
extrinsic_name,
)
})
.collect();
if benchmarks_to_run.is_empty() {
return Err("No benchmarks found which match your input.".into())
}
let benchmarks_to_run = self.select_benchmarks_to_run(list)?;
if let Some(list_output) = self.list {
list_benchmark(benchmarks_to_run, list_output, self.no_csv_header);
@@ -333,15 +283,17 @@ impl PalletCmd {
let mut batches_db = Vec::new();
let mut timer = time::SystemTime::now();
// Maps (pallet, extrinsic) to its component ranges.
let mut component_ranges = HashMap::<(Vec<u8>, Vec<u8>), Vec<ComponentRange>>::new();
let mut component_ranges = HashMap::<(String, String), Vec<ComponentRange>>::new();
let pov_modes = Self::parse_pov_modes(&benchmarks_to_run)?;
let mut failed = Vec::<(String, String)>::new();
for (pallet, extrinsic, components, _, pallet_name, extrinsic_name) in
benchmarks_to_run.clone()
'outer: for (i, SelectedBenchmark { pallet, extrinsic, components, .. }) in
benchmarks_to_run.clone().into_iter().enumerate()
{
log::info!(
target: LOG_TARGET,
"Starting benchmark: {pallet_name}::{extrinsic_name}"
"[{: >3} % ] Starting benchmark: {pallet}::{extrinsic}",
(i * 100) / benchmarks_to_run.len(),
);
let all_components = if components.is_empty() {
vec![Default::default()]
@@ -392,100 +344,127 @@ impl PalletCmd {
for (s, selected_components) in all_components.iter().enumerate() {
// First we run a verification
if !self.no_verify {
let mut changes = genesis_changes.clone();
let state = &state_without_tracking;
let result = StateMachine::new(
state,
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
&pallet,
&extrinsic,
&selected_components.clone(),
true, // run verification code
1, // no need to do internal repeats
)
.encode(),
&mut extensions(),
&sp_state_machine::backend::BackendRuntimeCode::new(state)
.runtime_code()?,
CallContext::Offchain,
)
.execute()
.map_err(|e| {
format!("Error executing and verifying runtime benchmark: {}", e)
})?;
// Don't use these results since verification code will add overhead.
let _batch =
<std::result::Result<Vec<BenchmarkBatch>, String> as Decode>::decode(
&mut &result[..],
)
.map_err(|e| format!("Failed to decode benchmark results: {:?}", e))?
.map_err(|e| {
format!("Benchmark {pallet_name}::{extrinsic_name} failed: {e}",)
})?;
let _batch: Vec<BenchmarkBatch> = match Self::exec_state_machine::<
std::result::Result<Vec<BenchmarkBatch>, String>,
_,
_,
>(
StateMachine::new(
state,
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
pallet.as_bytes(),
extrinsic.as_bytes(),
&selected_components.clone(),
true, // run verification code
1, // no need to do internal repeats
)
.encode(),
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
"dispatch a benchmark",
) {
Err(e) => {
log::error!("Error executing and verifying runtime benchmark: {}", e);
failed.push((pallet.clone(), extrinsic.clone()));
continue 'outer
},
Ok(Err(e)) => {
log::error!("Error executing and verifying runtime benchmark: {}", e);
failed.push((pallet.clone(), extrinsic.clone()));
continue 'outer
},
Ok(Ok(b)) => b,
};
}
// Do one loop of DB tracking.
{
let mut changes = genesis_changes.clone();
let state = &state_with_tracking;
let result = StateMachine::new(
state, // todo remove tracking
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
&pallet.clone(),
&extrinsic.clone(),
&selected_components.clone(),
false, // don't run verification code for final values
self.repeat,
)
.encode(),
&mut extensions(),
&sp_state_machine::backend::BackendRuntimeCode::new(state)
.runtime_code()?,
CallContext::Offchain,
)
.execute()
.map_err(|e| format!("Error executing runtime benchmark: {}", e))?;
let batch =
<std::result::Result<Vec<BenchmarkBatch>, String> as Decode>::decode(
&mut &result[..],
)
.map_err(|e| format!("Failed to decode benchmark results: {:?}", e))??;
let batch: Vec<BenchmarkBatch> = match Self::exec_state_machine::<
std::result::Result<Vec<BenchmarkBatch>, String>,
_,
_,
>(
StateMachine::new(
state, // todo remove tracking
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
pallet.as_bytes(),
extrinsic.as_bytes(),
&selected_components.clone(),
false, // don't run verification code for final values
self.repeat,
)
.encode(),
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
"dispatch a benchmark",
) {
Err(e) => {
log::error!("Error executing runtime benchmark: {}", e);
failed.push((pallet.clone(), extrinsic.clone()));
continue 'outer
},
Ok(Err(e)) => {
log::error!("Benchmark {pallet}::{extrinsic} failed: {e}",);
failed.push((pallet.clone(), extrinsic.clone()));
continue 'outer
},
Ok(Ok(b)) => b,
};
batches_db.extend(batch);
}
// Finally run a bunch of loops to get extrinsic timing information.
for r in 0..self.external_repeat {
let mut changes = genesis_changes.clone();
let state = &state_without_tracking;
let result = StateMachine::new(
state, // todo remove tracking
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
&pallet.clone(),
&extrinsic.clone(),
&selected_components.clone(),
false, // don't run verification code for final values
self.repeat,
)
.encode(),
&mut extensions(),
&sp_state_machine::backend::BackendRuntimeCode::new(state)
.runtime_code()?,
CallContext::Offchain,
)
.execute()
.map_err(|e| format!("Error executing runtime benchmark: {}", e))?;
let batch =
<std::result::Result<Vec<BenchmarkBatch>, String> as Decode>::decode(
&mut &result[..],
)
.map_err(|e| format!("Failed to decode benchmark results: {:?}", e))??;
let batch = match Self::exec_state_machine::<
std::result::Result<Vec<BenchmarkBatch>, String>,
_,
_,
>(
StateMachine::new(
state, // todo remove tracking
&mut changes,
&executor,
"Benchmark_dispatch_benchmark",
&(
pallet.as_bytes(),
extrinsic.as_bytes(),
&selected_components.clone(),
false, // don't run verification code for final values
self.repeat,
)
.encode(),
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
"dispatch a benchmark",
) {
Err(e) => {
return Err(format!("Error executing runtime benchmark: {e}",).into());
},
Ok(Err(e)) => {
return Err(
format!("Benchmark {pallet}::{extrinsic} failed: {e}",).into()
);
},
Ok(Ok(b)) => b,
};
batches.extend(batch);
@@ -496,7 +475,8 @@ impl PalletCmd {
log::info!(
target: LOG_TARGET,
"Running benchmark: {pallet_name}::{extrinsic_name}({} args) {}/{} {}/{}",
"[{: >3} % ] Running benchmark: {pallet}::{extrinsic}({} args) {}/{} {}/{}",
(i * 100) / benchmarks_to_run.len(),
components.len(),
s + 1, // s starts at 0.
all_components.len(),
@@ -509,21 +489,276 @@ impl PalletCmd {
}
}
assert!(batches_db.len() == batches.len() / self.external_repeat as usize);
if !failed.is_empty() {
failed.sort();
eprintln!(
"The following {} benchmarks failed:\n{}",
failed.len(),
failed.iter().map(|(p, e)| format!("- {p}::{e}")).collect::<Vec<_>>().join("\n")
);
return Err(format!("{} benchmarks failed", failed.len()).into())
}
// Combine all of the benchmark results, so that benchmarks of the same pallet/function
// are together.
let batches = combine_batches(batches, batches_db);
self.output(&batches, &storage_info, &component_ranges, pov_modes)
}
fn select_benchmarks_to_run(&self, list: Vec<BenchmarkList>) -> Result<Vec<SelectedBenchmark>> {
let pallet = self.pallet.clone().unwrap_or_default();
let pallet = pallet.as_bytes();
let extrinsic = self.extrinsic.clone().unwrap_or_default();
let extrinsic_split: Vec<&str> = extrinsic.split(',').collect();
let extrinsics: Vec<_> = extrinsic_split.iter().map(|x| x.trim().as_bytes()).collect();
// Use the benchmark list and the user input to determine the set of benchmarks to run.
let mut benchmarks_to_run = Vec::new();
list.iter()
.filter(|item| pallet.is_empty() || pallet == &b"*"[..] || pallet == &item.pallet[..])
.for_each(|item| {
for benchmark in &item.benchmarks {
let benchmark_name = &benchmark.name;
if extrinsic.is_empty() ||
extrinsic.as_bytes() == &b"*"[..] ||
extrinsics.contains(&&benchmark_name[..])
{
benchmarks_to_run.push((
item.pallet.clone(),
benchmark.name.clone(),
benchmark.components.clone(),
benchmark.pov_modes.clone(),
))
}
}
});
// Convert `Vec<u8>` to `String` for better readability.
let benchmarks_to_run: Vec<_> = benchmarks_to_run
.into_iter()
.map(|(pallet, extrinsic, components, pov_modes)| {
let pallet = String::from_utf8(pallet.clone()).expect("Encoded from String; qed");
let extrinsic =
String::from_utf8(extrinsic.clone()).expect("Encoded from String; qed");
SelectedBenchmark {
pallet,
extrinsic,
components,
pov_modes: pov_modes
.into_iter()
.map(|(p, s)| {
(String::from_utf8(p).unwrap(), String::from_utf8(s).unwrap())
})
.collect(),
}
})
.collect();
if benchmarks_to_run.is_empty() {
return Err("No benchmarks found which match your input.".into())
}
Ok(benchmarks_to_run)
}
/// Produce a genesis storage and genesis changes.
///
/// It would be easier to only return one type, but there is no easy way to convert them.
// TODO: Re-write `BenchmarkingState` to not be such a clusterfuck and only accept
// `OverlayedChanges` instead of a mix between `OverlayedChanges` and `State`. But this can only
// be done once we deprecated and removed the legacy interface :(
fn genesis_storage<H: Hash, F: HostFunctions>(
&self,
chain_spec: &Option<Box<dyn ChainSpec>>,
) -> Result<(sp_storage::Storage, OverlayedChanges<H>)> {
Ok(match (self.genesis_builder, self.runtime.is_some()) {
(Some(GenesisBuilder::None), _) => Default::default(),
(Some(GenesisBuilder::Spec), _) | (None, false) => {
log::warn!("{WARN_SPEC_GENESIS_CTOR}");
let Some(chain_spec) = chain_spec else {
return Err("No chain spec specified to generate the genesis state".into());
};
let storage = chain_spec
.build_storage()
.map_err(|e| format!("{ERROR_CANNOT_BUILD_GENESIS}\nError: {e}"))?;
(storage, Default::default())
},
(Some(GenesisBuilder::Runtime), _) | (None, true) =>
(Default::default(), self.genesis_from_runtime::<H, F>()?),
})
}
/// Generate the genesis changeset by the runtime API.
fn genesis_from_runtime<H: Hash, F: HostFunctions>(&self) -> Result<OverlayedChanges<H>> {
let state = BenchmarkingState::<H>::new(
Default::default(),
Some(self.database_cache_size as usize),
false,
false,
)?;
// Create a dummy WasmExecutor just to build the genesis storage.
let method =
execution_method_from_cli(self.wasm_method, self.wasmtime_instantiation_strategy);
let executor = WasmExecutor::<(
sp_io::SubstrateHostFunctions,
frame_benchmarking::benchmarking::HostFunctions,
F,
)>::builder()
.with_execution_method(method)
.with_allow_missing_host_functions(self.allow_missing_host_functions)
.build();
let runtime = self.runtime_blob(&state)?;
let runtime_code = runtime.code()?;
// We cannot use the `GenesisConfigBuilderRuntimeCaller` here since it returns the changes
// as `Storage` item, but we need it as `OverlayedChanges`.
let genesis_json: Option<Vec<u8>> = Self::exec_state_machine(
StateMachine::new(
&state,
&mut Default::default(),
&executor,
"GenesisBuilder_get_preset",
&None::<PresetId>.encode(), // Use the default preset
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
"build the genesis spec",
)?;
let Some(base_genesis_json) = genesis_json else {
return Err("GenesisBuilder::get_preset returned no data".into())
};
let base_genesis_json = serde_json::from_slice::<serde_json::Value>(&base_genesis_json)
.map_err(|e| format!("GenesisBuilder::get_preset returned invalid JSON: {:?}", e))?;
let dev_genesis_json: Option<Vec<u8>> = Self::exec_state_machine(
StateMachine::new(
&state,
&mut Default::default(),
&executor,
"GenesisBuilder_get_preset",
&Some::<PresetId>(GENESIS_PRESET.into()).encode(), // Use the default preset
&mut Self::build_extensions(executor.clone()),
&runtime_code,
CallContext::Offchain,
),
"build the genesis spec",
)?;
let mut genesis_json = serde_json::Value::default();
json_merge(&mut genesis_json, base_genesis_json);
if let Some(dev) = dev_genesis_json {
let dev: serde_json::Value = serde_json::from_slice(&dev).map_err(|e| {
format!("GenesisBuilder::get_preset returned invalid JSON: {:?}", e)
})?;
json_merge(&mut genesis_json, dev);
} else {
log::warn!(
"Could not find genesis preset '{GENESIS_PRESET}'. Falling back to default."
);
}
let json_pretty_str = serde_json::to_string_pretty(&genesis_json)
.map_err(|e| format!("json to string failed: {e}"))?;
let mut changes = Default::default();
let build_res: GenesisBuildResult = Self::exec_state_machine(
StateMachine::new(
&state,
&mut changes,
&executor,
"GenesisBuilder_build_state",
&json_pretty_str.encode(),
&mut Extensions::default(),
&runtime_code,
CallContext::Offchain,
),
"populate the genesis state",
)?;
if let Err(e) = build_res {
return Err(format!("GenesisBuilder::build_state failed: {}", e).into())
}
Ok(changes)
}
/// Execute a state machine and decode its return value as `R`.
fn exec_state_machine<R: Decode, H: Hash, Exec: CodeExecutor>(
mut machine: StateMachine<BenchmarkingState<H>, H, Exec>,
hint: &str,
) -> Result<R> {
let res = machine
.execute()
.map_err(|e| format!("Could not call runtime API to {hint}: {}", e))?;
let res = R::decode(&mut &res[..])
.map_err(|e| format!("Failed to decode runtime API result to {hint}: {:?}", e))?;
Ok(res)
}
/// Build the extension that are available for pallet benchmarks.
fn build_extensions<E: CodeExecutor>(exe: E) -> Extensions {
let mut extensions = Extensions::default();
let (offchain, _) = TestOffchainExt::new();
let (pool, _) = TestTransactionPoolExt::new();
let keystore = MemoryKeystore::new();
extensions.register(KeystoreExt::new(keystore));
extensions.register(OffchainWorkerExt::new(offchain.clone()));
extensions.register(OffchainDbExt::new(offchain));
extensions.register(TransactionPoolExt::new(pool));
extensions.register(ReadRuntimeVersionExt::new(exe));
extensions
}
/// Load the runtime blob for this benchmark.
///
/// The blob will either be loaded from the `:code` key out of the chain spec, or from a file
/// when specified with `--runtime`.
fn runtime_blob<'a, H: Hash>(
&self,
state: &'a BenchmarkingState<H>,
) -> Result<FetchedCode<'a, BenchmarkingState<H>, H>> {
if let Some(runtime) = &self.runtime {
log::info!("Loading WASM from {}", runtime.display());
let code = fs::read(runtime)?;
let hash = sp_core::blake2_256(&code).to_vec();
let wrapped_code = WrappedRuntimeCode(Cow::Owned(code));
Ok(FetchedCode::FromFile { wrapped_code, heap_pages: self.heap_pages, hash })
} else {
log::info!("Loading WASM from genesis state");
let state = sp_state_machine::backend::BackendRuntimeCode::new(state);
Ok(FetchedCode::FromGenesis { state })
}
}
/// Allocation strategy for pallet benchmarking.
fn alloc_strategy(heap_pages: Option<u64>) -> HeapAllocStrategy {
heap_pages.map_or(DEFAULT_HEAP_ALLOC_STRATEGY, |p| HeapAllocStrategy::Static {
extra_pages: p as _,
})
}
fn output(
&self,
batches: &[BenchmarkBatchSplitResults],
storage_info: &[StorageInfo],
component_ranges: &HashMap<(Vec<u8>, Vec<u8>), Vec<ComponentRange>>,
component_ranges: &ComponentRangeMap,
pov_modes: PovModesMap,
) -> Result<()> {
// Jsonify the result and write it to a file or stdout if desired.
if !self.jsonify(&batches)? {
if !self.jsonify(&batches)? && !self.quiet {
// Print the summary only if `jsonify` did not write to stdout.
self.print_summary(&batches, &storage_info, pov_modes.clone())
}
@@ -546,11 +781,13 @@ impl PalletCmd {
/// Re-analyze a batch historic benchmark timing data. Will not take the PoV into account.
fn output_from_results(&self, batches: &[BenchmarkBatchSplitResults]) -> Result<()> {
let mut component_ranges =
HashMap::<(Vec<u8>, Vec<u8>), HashMap<String, (u32, u32)>>::new();
let mut component_ranges = HashMap::<(String, String), HashMap<String, (u32, u32)>>::new();
for batch in batches {
let range = component_ranges
.entry((batch.pallet.clone(), batch.benchmark.clone()))
.entry((
String::from_utf8(batch.pallet.clone()).unwrap(),
String::from_utf8(batch.benchmark.clone()).unwrap(),
))
.or_default();
for result in &batch.time_results {
for (param, value) in &result.components {
@@ -608,10 +845,13 @@ impl PalletCmd {
) {
for batch in batches.iter() {
// Print benchmark metadata
let pallet = String::from_utf8(batch.pallet.clone()).expect("Encoded from String; qed");
let benchmark =
String::from_utf8(batch.benchmark.clone()).expect("Encoded from String; qed");
println!(
"Pallet: {:?}, Extrinsic: {:?}, Lowest values: {:?}, Highest values: {:?}, Steps: {:?}, Repeat: {:?}",
String::from_utf8(batch.pallet.clone()).expect("Encoded from String; qed"),
String::from_utf8(batch.benchmark.clone()).expect("Encoded from String; qed"),
pallet,
benchmark,
self.lowest_range_values,
self.highest_range_values,
self.steps,
@@ -625,10 +865,7 @@ impl PalletCmd {
if !self.no_storage_info {
let mut storage_per_prefix = HashMap::<Vec<u8>, Vec<BenchmarkResult>>::new();
let pov_mode = pov_modes
.get(&(batch.pallet.clone(), batch.benchmark.clone()))
.cloned()
.unwrap_or_default();
let pov_mode = pov_modes.get(&(pallet, benchmark)).cloned().unwrap_or_default();
let comments = writer::process_storage_results(
&mut storage_per_prefix,
@@ -699,20 +936,11 @@ impl PalletCmd {
}
/// Parses the PoV modes per benchmark that were specified by the `#[pov_mode]` attribute.
fn parse_pov_modes(
benchmarks: &Vec<(
Vec<u8>,
Vec<u8>,
Vec<(BenchmarkParameter, u32, u32)>,
Vec<(String, String)>,
String,
String,
)>,
) -> Result<PovModesMap> {
fn parse_pov_modes(benchmarks: &Vec<SelectedBenchmark>) -> Result<PovModesMap> {
use std::collections::hash_map::Entry;
let mut parsed = PovModesMap::new();
for (pallet, call, _components, pov_modes, _, _) in benchmarks {
for SelectedBenchmark { pallet, extrinsic, pov_modes, .. } in benchmarks {
for (pallet_storage, mode) in pov_modes {
let mode = PovEstimationMode::from_str(&mode)?;
let splits = pallet_storage.split("::").collect::<Vec<_>>();
@@ -726,7 +954,7 @@ impl PalletCmd {
let (pov_pallet, pov_storage) = (splits[0], splits.get(1).unwrap_or(&"ALL"));
match parsed
.entry((pallet.clone(), call.clone()))
.entry((pallet.clone(), extrinsic.clone()))
.or_default()
.entry((pov_pallet.to_string(), pov_storage.to_string()))
{
@@ -744,6 +972,33 @@ impl PalletCmd {
}
Ok(parsed)
}
/// Sanity check the CLI arguments.
fn check_args(&self) -> Result<()> {
if self.runtime.is_some() && self.shared_params.chain.is_some() {
unreachable!("Clap should not allow both `--runtime` and `--chain` to be provided.")
}
if let Some(output_path) = &self.output {
if !output_path.is_dir() && output_path.file_name().is_none() {
return Err("Output file or path is invalid!".into())
}
}
if let Some(header_file) = &self.header {
if !header_file.is_file() {
return Err("Header file is invalid!".into())
};
}
if let Some(handlebars_template_file) = &self.template {
if !handlebars_template_file.is_file() {
return Err("Handlebars template file is invalid!".into())
};
}
Ok(())
}
}
impl CliConfiguration for PalletCmd {
@@ -761,35 +1016,28 @@ impl CliConfiguration for PalletCmd {
/// List the benchmarks available in the runtime, in a CSV friendly format.
fn list_benchmark(
benchmarks_to_run: Vec<(
Vec<u8>,
Vec<u8>,
Vec<(BenchmarkParameter, u32, u32)>,
Vec<(String, String)>,
String,
String,
)>,
benchmarks_to_run: Vec<SelectedBenchmark>,
list_output: ListOutput,
no_csv_header: bool,
) {
let mut benchmarks = BTreeMap::new();
// Sort and de-dub by pallet and function name.
benchmarks_to_run.iter().for_each(|(_, _, _, _, pallet_name, extrinsic_name)| {
benchmarks_to_run.iter().for_each(|bench| {
benchmarks
.entry(pallet_name)
.entry(&bench.pallet)
.or_insert_with(BTreeSet::new)
.insert(extrinsic_name);
.insert(&bench.extrinsic);
});
match list_output {
ListOutput::All => {
if !no_csv_header {
println!("pallet,extrinsic");
println!("pallet, extrinsic");
}
for (pallet, extrinsics) in benchmarks {
for extrinsic in extrinsics {
println!("{pallet},{extrinsic}");
println!("{pallet}, {extrinsic}");
}
}
},
substrate/utils/frame/benchmarking-cli/src/pallet/mod.rs
+23 -1
View File
@@ -16,9 +16,10 @@
// limitations under the License.
mod command;
mod types;
mod writer;
use crate::shared::HostInfoParams;
use crate::{pallet::types::GenesisBuilder, shared::HostInfoParams};
use clap::ValueEnum;
use sc_cli::{
WasmExecutionMethod, WasmtimeInstantiationStrategy, DEFAULT_WASMTIME_INSTANTIATION_STRATEGY,
@@ -166,6 +167,20 @@ pub struct PalletCmd {
)]
pub wasmtime_instantiation_strategy: WasmtimeInstantiationStrategy,
/// Optional runtime blob to use instead of the one from the genesis config.
#[arg(long, conflicts_with = "chain")]
pub runtime: Option<PathBuf>,
/// Do not fail if there are unknown but also unused host functions in the runtime.
#[arg(long)]
pub allow_missing_host_functions: bool,
/// How to construct the genesis state.
///
/// Uses `GenesisBuilder::Spec` by default and `GenesisBuilder::Runtime` if `runtime` is set.
#[arg(long, value_enum)]
pub genesis_builder: Option<GenesisBuilder>,
/// DEPRECATED: This argument has no effect.
#[arg(long = "execution")]
pub execution: Option<String>,
@@ -221,4 +236,11 @@ pub struct PalletCmd {
/// This exists only to restore legacy behaviour. It should never actually be needed.
#[arg(long)]
pub unsafe_overwrite_results: bool,
/// Do not print a summary at the end of the run.
///
/// These summaries can be very long when benchmarking multiple pallets at once. For CI
/// use-cases, this option reduces the noise.
#[arg(long)]
quiet: bool,
}
substrate/utils/frame/benchmarking-cli/src/pallet/types.rs
+78
View File
@@ -0,0 +1,78 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Various types used by this crate.
use sc_cli::Result;
use sp_core::traits::{RuntimeCode, WrappedRuntimeCode};
use sp_runtime::traits::Hash;
/// How the genesis state for benchmarking should be build.
#[derive(clap::ValueEnum, Debug, Eq, PartialEq, Clone, Copy)]
#[clap(rename_all = "kebab-case")]
pub enum GenesisBuilder {
/// Do not provide any genesis state.
///
/// Benchmarks are advised to function with this, since they should setup their own required
/// state. However, to keep backwards compatibility, this is not the default.
None,
/// Let the runtime build the genesis state through its `BuildGenesisConfig` runtime API.
Runtime,
/// Use the spec file to build the genesis state. This fails when there is no spec.
Spec,
}
/// A runtime blob that was either fetched from genesis storage or loaded from a file.
// NOTE: This enum is only needed for the annoying lifetime bounds on `RuntimeCode`. Otherwise we
// could just directly return the blob.
pub enum FetchedCode<'a, B, H> {
FromGenesis { state: sp_state_machine::backend::BackendRuntimeCode<'a, B, H> },
FromFile { wrapped_code: WrappedRuntimeCode<'a>, heap_pages: Option<u64>, hash: Vec<u8> },
}
impl<'a, B, H> FetchedCode<'a, B, H>
where
H: Hash,
B: sc_client_api::StateBackend<H>,
{
/// The runtime blob.
pub fn code(&'a self) -> Result<RuntimeCode<'a>> {
match self {
Self::FromGenesis { state } => state.runtime_code().map_err(Into::into),
Self::FromFile { wrapped_code, heap_pages, hash } => Ok(RuntimeCode {
code_fetcher: wrapped_code,
heap_pages: *heap_pages,
hash: hash.clone(),
}),
}
}
}
/// Maps a (pallet, benchmark) to its component ranges.
pub(crate) type ComponentRangeMap =
std::collections::HashMap<(String, String), Vec<ComponentRange>>;
/// The inclusive range of a component.
#[derive(serde::Serialize, Debug, Clone, Eq, PartialEq)]
pub(crate) struct ComponentRange {
/// Name of the component.
pub(crate) name: String,
/// Minimal valid value of the component.
pub(crate) min: u32,
/// Maximal valid value of the component.
pub(crate) max: u32,
}
substrate/utils/frame/benchmarking-cli/src/pallet/writer.rs
+14 -12
View File
@@ -28,7 +28,10 @@ use itertools::Itertools;
use serde::Serialize;
use crate::{
pallet::command::{ComponentRange, PovEstimationMode, PovModesMap},
pallet::{
command::{PovEstimationMode, PovModesMap},
types::{ComponentRange, ComponentRangeMap},
},
shared::UnderscoreHelper,
PalletCmd,
};
@@ -132,7 +135,7 @@ fn io_error(s: &str) -> std::io::Error {
fn map_results(
batches: &[BenchmarkBatchSplitResults],
storage_info: &[StorageInfo],
component_ranges: &HashMap<(Vec<u8>, Vec<u8>), Vec<ComponentRange>>,
component_ranges: &ComponentRangeMap,
pov_modes: PovModesMap,
default_pov_mode: PovEstimationMode,
analysis_choice: &AnalysisChoice,
@@ -188,7 +191,7 @@ fn get_benchmark_data(
batch: &BenchmarkBatchSplitResults,
storage_info: &[StorageInfo],
// Per extrinsic component ranges.
component_ranges: &HashMap<(Vec<u8>, Vec<u8>), Vec<ComponentRange>>,
component_ranges: &ComponentRangeMap,
pov_modes: PovModesMap,
default_pov_mode: PovEstimationMode,
analysis_choice: &AnalysisChoice,
@@ -207,6 +210,8 @@ fn get_benchmark_data(
AnalysisChoice::MedianSlopes => Analysis::median_slopes,
AnalysisChoice::Max => Analysis::max,
};
let pallet = String::from_utf8(batch.pallet.clone()).unwrap();
let benchmark = String::from_utf8(batch.benchmark.clone()).unwrap();
let extrinsic_time = analysis_function(&batch.time_results, BenchmarkSelector::ExtrinsicTime)
.expect("analysis function should return an extrinsic time for valid inputs");
@@ -282,10 +287,7 @@ fn get_benchmark_data(
// We add additional comments showing which storage items were touched.
// We find the worst case proof size, and use that as the final proof size result.
let mut storage_per_prefix = HashMap::<Vec<u8>, Vec<BenchmarkResult>>::new();
let pov_mode = pov_modes
.get(&(batch.pallet.clone(), batch.benchmark.clone()))
.cloned()
.unwrap_or_default();
let pov_mode = pov_modes.get(&(pallet.clone(), benchmark.clone())).cloned().unwrap_or_default();
let comments = process_storage_results(
&mut storage_per_prefix,
&batch.db_results,
@@ -351,12 +353,12 @@ fn get_benchmark_data(
.collect::<Vec<_>>();
let component_ranges = component_ranges
.get(&(batch.pallet.clone(), batch.benchmark.clone()))
.get(&(pallet.clone(), benchmark.clone()))
.map(|c| c.clone())
.unwrap_or_default();
BenchmarkData {
name: String::from_utf8(batch.benchmark.clone()).unwrap(),
name: benchmark,
components,
base_weight: extrinsic_time.base,
base_reads: reads.base,
@@ -378,7 +380,7 @@ fn get_benchmark_data(
pub(crate) fn write_results(
batches: &[BenchmarkBatchSplitResults],
storage_info: &[StorageInfo],
component_ranges: &HashMap<(Vec<u8>, Vec<u8>), Vec<ComponentRange>>,
component_ranges: &HashMap<(String, String), Vec<ComponentRange>>,
pov_modes: PovModesMap,
default_pov_mode: PovEstimationMode,
path: &PathBuf,
@@ -871,10 +873,10 @@ mod test {
fn test_pov_mode() -> PovModesMap {
let mut map = PovModesMap::new();
map.entry((b"scheduler".to_vec(), b"first_benchmark".to_vec()))
map.entry(("scheduler".into(), "first_benchmark".into()))
.or_default()
.insert(("scheduler".into(), "mel".into()), PovEstimationMode::MaxEncodedLen);
map.entry((b"scheduler".to_vec(), b"first_benchmark".to_vec()))
map.entry(("scheduler".into(), "first_benchmark".into()))
.or_default()
.insert(("scheduler".into(), "measured".into()), PovEstimationMode::Measured);
map