Add Proof Size to Weight Output (#11637)

* initial impl

* add template test

* linear fit proof size

* always record proof when tracking storage

* calculate worst case pov

* remove duplicate worst case

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* more comment output

* add cli for worst case map size

* update name

* clap does not support underscores

* rename

* expose worst case map values

* improve some comments

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* update template

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix fmt

* more fmt

* more fmt

* Dont panic when there is no proof

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Whitelist :extrinsic_index

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use whitelist when recording proof

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add logs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add PoV testing pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Deploy PoV testing pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Storage benches reside in the PoV pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Linear regress PoV per component

Splits the PoV calculation into "measured" and "estimated".
The measured part is reported by the Proof recorder and linear
regressed over all components at once.
The estimated part is calculated as worst-case by using the max
PoV size per storage access and calculating one linear regress per
component. This gives each component a (possibly) independent PoV.
For now the measured size will always be lower than the PoV on
Polkadot since it is measured on an empty snapshot. The measured
part is therefor only used as diagnostic for debugging.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Put PoV into the weight templates

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Extra alanysis choise for PoV

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add+Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make benches faster

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use same template comments

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_balances

* ".git/.scripts/bench-bot.sh" pallet dev pallet_democracy

* Update referenda mock BlockWeights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Take measured value size into account

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_scheduler

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* proof_size: None

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ugly, but works

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* wup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add pov_mode attribute to the benchmarks! macro

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use pov_mode attribute in PoV benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Scheduler, Whitelist: Add pov_mode attr

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update PoV weights

* Add CLI arg: default-pov-mode

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Update PoV weights"

This reverts commit 2f3ac2387396470b118122a6ff8fa4ee12216f4b.

* Revert "WIP"

This reverts commit c34b538cd2bc45da4544e887180184e30957904a.

* Revert first approach

This reverts commit range 8ddaa2fffe5930f225a30bee314d0b7c94c344dd^..4c84f8748e5395852a9e0e25b0404953fee1a59e

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add extra benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_alliance

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_whitelist

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_scheduler

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy 🤦

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add reference benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix doc comments

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Undo logging

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add 'Ignored' pov_mode

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Allow multiple attributes per benchmark

Turns out that the current benchmarking syntax does not support
multiple attributes per bench 🤦. Changing it to support that
since otherwise the `pov_mode` would conflict with the others.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Validate pov_mode syntax

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Ignore PoV for all contract benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bump macro recursion limit

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update contract weights

They dont have a PoV component anymore.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix test ffs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* pov_mode is unsupported in V2 syntax

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix pallet ui tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* update pallet ui

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix pallet ui tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Your Name <you@example.com>
This commit is contained in:
Shawn Tabrizi
2023-01-26 19:35:39 -03:00
committed by GitHub
parent 41f819eb48
commit 1b27ae9549
74 changed files with 18668 additions and 9111 deletions
@@ -0,0 +1,371 @@
// This file is part of Substrate.
// Copyright (C) 2022 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! All benchmarks in this file are just for debugging the PoV calculation logic, they are unused.
#![cfg(feature = "runtime-benchmarks")]
use super::*;
use frame_support::traits::UnfilteredDispatchable;
use frame_system::{Pallet as System, RawOrigin};
use sp_runtime::traits::Hash;
frame_benchmarking::benchmarks! {
storage_single_value_read {
Value::<T>::put(123);
}: {
assert_eq!(Value::<T>::get(), Some(123));
}
#[pov_mode = Ignored]
storage_single_value_ignored_read {
Value::<T>::put(123);
}: {
assert_eq!(Value::<T>::get(), Some(123));
}
storage_single_value_read_twice {
Value::<T>::put(123);
}: {
assert_eq!(Value::<T>::get(), Some(123));
assert_eq!(Value::<T>::get(), Some(123));
}
storage_single_value_write {
}: {
Value::<T>::put(123);
} verify {
assert_eq!(Value::<T>::get(), Some(123));
}
storage_single_value_kill {
Value::<T>::put(123);
}: {
Value::<T>::kill();
} verify {
assert!(!Value::<T>::exists());
}
// This benchmark and the following are testing a storage map with adjacent storage items.
//
// First a storage map is filled and a specific number of other storage items is
// created. Then the one value is read from the map. This demonstrates that the number of other
// nodes in the Trie influences the proof size. The number of inserted nodes can be interpreted
// as the number of `StorageMap`/`StorageValue` in the whole runtime.
#[pov_mode = Measured]
storage_1m_map_read_one_value_two_additional_layers {
(0..(1<<10)).for_each(|i| Map1M::<T>::insert(i, i));
// Assume there are 16-256 other storage items.
(0..(1u32<<4)).for_each(|i| {
let k = T::Hashing::hash(&i.to_be_bytes());
frame_support::storage::unhashed::put(k.as_ref(), &i);
});
}: {
assert_eq!(Map1M::<T>::get(1<<9), Some(1<<9));
}
#[pov_mode = Measured]
storage_1m_map_read_one_value_three_additional_layers {
(0..(1<<10)).for_each(|i| Map1M::<T>::insert(i, i));
// Assume there are 256-4096 other storage items.
(0..(1u32<<8)).for_each(|i| {
let k = T::Hashing::hash(&i.to_be_bytes());
frame_support::storage::unhashed::put(k.as_ref(), &i);
});
}: {
assert_eq!(Map1M::<T>::get(1<<9), Some(1<<9));
}
#[pov_mode = Measured]
storage_1m_map_read_one_value_four_additional_layers {
(0..(1<<10)).for_each(|i| Map1M::<T>::insert(i, i));
// Assume there are 4096-65536 other storage items.
(0..(1u32<<12)).for_each(|i| {
let k = T::Hashing::hash(&i.to_be_bytes());
frame_support::storage::unhashed::put(k.as_ref(), &i);
});
}: {
assert_eq!(Map1M::<T>::get(1<<9), Some(1<<9));
}
// Reads from both storage maps each `n` and `m` times. Should result in two linear components.
storage_map_read_per_component {
let n in 0 .. 100;
let m in 0 .. 100;
(0..m*10).for_each(|i| Map1M::<T>::insert(i, i));
(0..n*10).for_each(|i| Map16M::<T>::insert(i, i));
}: {
(0..m).for_each(|i|
assert_eq!(Map1M::<T>::get(i*10), Some(i*10)));
(0..n).for_each(|i|
assert_eq!(Map16M::<T>::get(i*10), Some(i*10)));
}
#[pov_mode = MaxEncodedLen {
Pov::Map1M: Ignored
}]
storage_map_read_per_component_one_ignored {
let n in 0 .. 100;
let m in 0 .. 100;
(0..m*10).for_each(|i| Map1M::<T>::insert(i, i));
(0..n*10).for_each(|i| Map16M::<T>::insert(i, i));
}: {
(0..m).for_each(|i|
assert_eq!(Map1M::<T>::get(i*10), Some(i*10)));
(0..n).for_each(|i|
assert_eq!(Map16M::<T>::get(i*10), Some(i*10)));
}
// Reads the same value from a storage map. Should not result in a component.
storage_1m_map_one_entry_repeated_read {
let n in 0 .. 100;
Map1M::<T>::insert(0, 0);
}: {
(0..n).for_each(|i|
assert_eq!(Map1M::<T>::get(0), Some(0)));
}
// Reads the same values from a storage map. Should result in a `1x` linear component.
storage_1m_map_multiple_entry_repeated_read {
let n in 0 .. 100;
(0..n).for_each(|i| Map1M::<T>::insert(i, i));
}: {
(0..n).for_each(|i| {
// Reading the same value 10 times does nothing.
(0..10).for_each(|j|
assert_eq!(Map1M::<T>::get(i), Some(i)));
});
}
storage_1m_double_map_read_per_component {
let n in 0 .. 1024;
(0..(1<<10)).for_each(|i| DoubleMap1M::<T>::insert(i, i, i));
}: {
(0..n).for_each(|i|
assert_eq!(DoubleMap1M::<T>::get(i, i), Some(i)));
}
storage_value_bounded_read {
}: {
assert!(BoundedValue::<T>::get().is_none());
}
// Reading unbounded values will produce no mathematical worst case PoV size for this component.
storage_value_unbounded_read {
}: {
assert!(UnboundedValue::<T>::get().is_none());
}
#[pov_mode = Ignored]
storage_value_unbounded_ignored_read {
}: {
assert!(UnboundedValue::<T>::get().is_none());
}
// Same as above, but we still expect a mathematical worst case PoV size for the bounded one.
storage_value_bounded_and_unbounded_read {
}: {
assert!(UnboundedValue::<T>::get().is_none());
assert!(BoundedValue::<T>::get().is_none());
}
#[pov_mode = Measured]
measured_storage_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
}
#[pov_mode = MaxEncodedLen]
mel_storage_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
}
#[pov_mode = Measured]
measured_storage_double_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
LargeValue2::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
assert!(LargeValue2::<T>::get().is_some());
}
#[pov_mode = MaxEncodedLen]
mel_storage_double_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
LargeValue2::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
assert!(LargeValue2::<T>::get().is_some());
}
#[pov_mode = MaxEncodedLen {
Pov::LargeValue2: Measured
}]
mel_mixed_storage_double_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
LargeValue2::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
assert!(LargeValue2::<T>::get().is_some());
}
#[pov_mode = Measured {
Pov::LargeValue2: MaxEncodedLen
}]
measured_mixed_storage_double_value_read_linear_size {
let l in 0 .. 1<<22;
let v: sp_runtime::BoundedVec<u8, _> = sp_std::vec![0u8; l as usize].try_into().unwrap();
LargeValue::<T>::put(&v);
LargeValue2::<T>::put(&v);
}: {
assert!(LargeValue::<T>::get().is_some());
assert!(LargeValue2::<T>::get().is_some());
}
#[pov_mode = Measured]
storage_map_unbounded_both_measured_read {
let i in 0 .. 1000;
UnboundedMap::<T>::insert(i, sp_std::vec![0; i as usize]);
UnboundedMap2::<T>::insert(i, sp_std::vec![0; i as usize]);
}: {
assert!(UnboundedMap::<T>::get(i).is_some());
assert!(UnboundedMap2::<T>::get(i).is_some());
}
#[pov_mode = MaxEncodedLen {
Pov::UnboundedMap: Measured
}]
storage_map_partial_unbounded_read {
let i in 0 .. 1000;
Map1M::<T>::insert(i, 0);
UnboundedMap::<T>::insert(i, sp_std::vec![0; i as usize]);
}: {
assert!(Map1M::<T>::get(i).is_some());
assert!(UnboundedMap::<T>::get(i).is_some());
}
#[pov_mode = MaxEncodedLen {
Pov::UnboundedMap: Ignored
}]
storage_map_partial_unbounded_ignored_read {
let i in 0 .. 1000;
Map1M::<T>::insert(i, 0);
UnboundedMap::<T>::insert(i, sp_std::vec![0; i as usize]);
}: {
assert!(Map1M::<T>::get(i).is_some());
assert!(UnboundedMap::<T>::get(i).is_some());
}
// Emitting an event will not incur any PoV.
emit_event {
// Emit a single event.
let call = Call::<T>::emit_event { };
}: { call.dispatch_bypass_filter(RawOrigin::Root.into()).unwrap(); }
verify {
assert_eq!(System::<T>::events().len(), 1);
}
// A No-OP will not incur any PoV.
noop {
let call = Call::<T>::noop { };
}: {
call.dispatch_bypass_filter(RawOrigin::Root.into()).unwrap();
}
impl_benchmark_test_suite!(
Pallet,
mock::new_test_ext(),
mock::Test,
);
}
#[cfg(test)]
mod mock {
use sp_runtime::testing::H256;
type AccountId = u64;
type AccountIndex = u32;
type BlockNumber = u64;
type UncheckedExtrinsic = frame_system::mocking::MockUncheckedExtrinsic<Test>;
type Block = frame_system::mocking::MockBlock<Test>;
frame_support::construct_runtime!(
pub enum Test where
Block = Block,
NodeBlock = Block,
UncheckedExtrinsic = UncheckedExtrinsic,
{
System: frame_system::{Pallet, Call, Config, Storage, Event<T>},
Baseline: crate::{Pallet, Call, Storage, Event<T>},
}
);
impl frame_system::Config for Test {
type BaseCallFilter = frame_support::traits::Everything;
type BlockWeights = ();
type BlockLength = ();
type DbWeight = ();
type RuntimeOrigin = RuntimeOrigin;
type Index = AccountIndex;
type BlockNumber = BlockNumber;
type RuntimeCall = RuntimeCall;
type Hash = H256;
type Hashing = ::sp_runtime::traits::BlakeTwo256;
type AccountId = AccountId;
type Lookup = sp_runtime::traits::IdentityLookup<Self::AccountId>;
type Header = sp_runtime::testing::Header;
type RuntimeEvent = RuntimeEvent;
type BlockHashCount = ();
type Version = ();
type PalletInfo = PalletInfo;
type AccountData = ();
type OnNewAccount = ();
type OnKilledAccount = ();
type SystemWeightInfo = ();
type SS58Prefix = ();
type OnSetCode = ();
type MaxConsumers = frame_support::traits::ConstU32<16>;
}
impl crate::Config for Test {
type RuntimeEvent = RuntimeEvent;
}
pub fn new_test_ext() -> sp_io::TestExternalities {
frame_system::GenesisConfig::default().build_storage::<Test>().unwrap().into()
}
}