Bound uses of Call (#11649)

* Introduce preimages module in traits

* Multisize Preimages

* Len not actually necessary

* Tweaks to the preimage API

* Fixes

* Get Scheduler building with new API

* Scheduler tests pass

* Bounded Scheduler 🎉

* Use Agenda holes and introduce IncompleteSince to avoid need to reschedule

* Tests pass with new weight system

* New benchmarks

* Add missing file

* Drop preimage when permenantly overeight

* Drop preimage when permenantly overeight

* Referenda uses latest preimage API

* Testing ok

* Adding tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add preimage migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Docs

* Remove dbg

* Refactor Democracy

* Refactor Democracy

* Add final MEL

* Remove silly maps

* Fixes

* Minor refactor

* Formatting

* Fixes

* Fixes

* Fixes

* Update frame/preimage/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Add migrations to Democracy

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Resolve conflicts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Resolve conflicts"

This reverts commit 734d66d69e54553471ffa54fa52e3e304dc8f106.

* Undo wrong resolves...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* massage clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy annoyance

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy annoyance

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add missing file

* Test <Preimage as QueryPreimage>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy harassment

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove old stuff

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test <Scheduler as Anon> trait functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update pallet-ui tests

Why is this needed? Should not be the case unless master is broken...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More scheduler trait test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Apply review suggestion

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add Scheduler test migration_v3_to_v4_works

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Merge fixup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Keep referenda benchmarks instantiatable

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use new scheduler weight functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use new democracy weight functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use weight compare functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update pallet-ui tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More renaming…

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More renaming…

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add comment

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Implement OnRuntimeUpgrade for scheduler::v3_to_v4 migration

Put the migration into a proper `MigrateToV4` struct and implement
the OnRuntimeUpgrade hooks for it. Also move the test to use that
instead.

This should make it easier for adding it to Polkadot.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Handle undecodable Agendas

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove trash

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use new OnRuntimeUpgrade functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix BoundedSlice::truncate_from

Co-authored-by: jakoblell

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix pre_upgrade hook return values

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add more error logging

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Find too large preimages in the pre_upgrade hook

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test that too large Calls in agendas are ignored

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use new OnRuntimeUpgrade hooks

Why did the CI not catch this?!

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* works fine - just more logs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix staking migration

Causing issues on Kusama...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix UI tests

No idea why this is needed. This is actually undoing an earlier change.
Maybe the CI has different rustc versions!?

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove multisig's Calls (#12072)

* Remove multisig's Calls

* Multisig: Fix tests and re-introduce reserve logic (#12241)

* Fix tests and re-introduce reserve logic

* fix benches

* add todo

* remove irrelevant bench

* [Feature] Add a migration that drains and refunds stored calls (#12313)

* [Feature] Add a migration that drains and refunds stored calls

* migration fixes

* fixes

* address review comments

* consume the whole block weight

* fix assertions

* license header

* fix interface

Co-authored-by: parity-processbot <>

Co-authored-by: parity-processbot <>
Co-authored-by: Roman Useinov <roman.useinov@gmail.com>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix multisig benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_democracy

* ".git/.scripts/bench-bot.sh" pallet dev pallet_scheduler

* ".git/.scripts/bench-bot.sh" pallet dev pallet_preimage

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: parity-processbot <>
Co-authored-by: Roman Useinov <roman.useinov@gmail.com>
This commit is contained in:
Gavin Wood
2022-10-05 19:21:37 +01:00
committed by GitHub
parent 24f0c3601c
commit 93e8ffed55
67 changed files with 5089 additions and 3488 deletions
+402
View File
@@ -0,0 +1,402 @@
// This file is part of Substrate.
// Copyright (C) 2017-2022 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Migrations for the scheduler pallet.
use super::*;
use frame_support::traits::OnRuntimeUpgrade;
/// The log target.
const TARGET: &'static str = "runtime::scheduler::migration";
pub mod v1 {
use super::*;
use frame_support::pallet_prelude::*;
#[frame_support::storage_alias]
pub(crate) type Agenda<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
<T as frame_system::Config>::BlockNumber,
Vec<
Option<
ScheduledV1<<T as Config>::RuntimeCall, <T as frame_system::Config>::BlockNumber>,
>,
>,
ValueQuery,
>;
#[frame_support::storage_alias]
pub(crate) type Lookup<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
Vec<u8>,
TaskAddress<<T as frame_system::Config>::BlockNumber>,
>;
}
pub mod v2 {
use super::*;
use frame_support::pallet_prelude::*;
#[frame_support::storage_alias]
pub(crate) type Agenda<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
<T as frame_system::Config>::BlockNumber,
Vec<Option<ScheduledV2Of<T>>>,
ValueQuery,
>;
#[frame_support::storage_alias]
pub(crate) type Lookup<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
Vec<u8>,
TaskAddress<<T as frame_system::Config>::BlockNumber>,
>;
}
pub mod v3 {
use super::*;
use frame_support::pallet_prelude::*;
#[frame_support::storage_alias]
pub(crate) type Agenda<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
<T as frame_system::Config>::BlockNumber,
Vec<Option<ScheduledV3Of<T>>>,
ValueQuery,
>;
#[frame_support::storage_alias]
pub(crate) type Lookup<T: Config> = StorageMap<
Pallet<T>,
Twox64Concat,
Vec<u8>,
TaskAddress<<T as frame_system::Config>::BlockNumber>,
>;
/// Migrate the scheduler pallet from V3 to V4.
pub struct MigrateToV4<T>(sp_std::marker::PhantomData<T>);
impl<T: Config<Hash = PreimageHash>> OnRuntimeUpgrade for MigrateToV4<T> {
#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, &'static str> {
assert_eq!(StorageVersion::get::<Pallet<T>>(), 3, "Can only upgrade from version 3");
let agendas = Agenda::<T>::iter_keys().count() as u32;
let decodable_agendas = Agenda::<T>::iter_values().count() as u32;
if agendas != decodable_agendas {
// This is not necessarily an error, but can happen when there are Calls
// in an Agenda that are not valid anymore with the new runtime.
log::error!(
target: TARGET,
"Can only decode {} of {} agendas - others will be dropped",
decodable_agendas,
agendas
);
}
log::info!(target: TARGET, "Trying to migrate {} agendas...", decodable_agendas);
// Check that no agenda overflows `MaxScheduledPerBlock`.
let max_scheduled_per_block = T::MaxScheduledPerBlock::get() as usize;
for (block_number, agenda) in Agenda::<T>::iter() {
if agenda.iter().cloned().filter_map(|s| s).count() > max_scheduled_per_block {
log::error!(
target: TARGET,
"Would truncate agenda of block {:?} from {} items to {} items.",
block_number,
agenda.len(),
max_scheduled_per_block,
);
return Err("Agenda would overflow `MaxScheduledPerBlock`.")
}
}
// Check that bounding the calls will not overflow `MAX_LENGTH`.
let max_length = T::Preimages::MAX_LENGTH as usize;
for (block_number, agenda) in Agenda::<T>::iter() {
for schedule in agenda.iter().cloned().filter_map(|s| s) {
match schedule.call {
frame_support::traits::schedule::MaybeHashed::Value(call) => {
let l = call.using_encoded(|c| c.len());
if l > max_length {
log::error!(
target: TARGET,
"Call in agenda of block {:?} is too large: {} byte",
block_number,
l,
);
return Err("Call is too large.")
}
},
_ => (),
}
}
}
Ok((decodable_agendas as u32).encode())
}
fn on_runtime_upgrade() -> Weight {
let version = StorageVersion::get::<Pallet<T>>();
if version != 3 {
log::warn!(
target: TARGET,
"skipping v3 to v4 migration: executed on wrong storage version.\
Expected version 3, found {:?}",
version,
);
return T::DbWeight::get().reads(1)
}
crate::Pallet::<T>::migrate_v3_to_v4()
}
#[cfg(feature = "try-runtime")]
fn post_upgrade(state: Vec<u8>) -> Result<(), &'static str> {
assert_eq!(StorageVersion::get::<Pallet<T>>(), 4, "Must upgrade");
// Check that everything decoded fine.
for k in crate::Agenda::<T>::iter_keys() {
assert!(crate::Agenda::<T>::try_get(k).is_ok(), "Cannot decode V4 Agenda");
}
let old_agendas: u32 =
Decode::decode(&mut &state[..]).expect("pre_upgrade provides a valid state; qed");
let new_agendas = crate::Agenda::<T>::iter_keys().count() as u32;
if old_agendas != new_agendas {
// This is not necessarily an error, but can happen when there are Calls
// in an Agenda that are not valid anymore in the new runtime.
log::error!(
target: TARGET,
"Did not migrate all Agendas. Previous {}, Now {}",
old_agendas,
new_agendas,
);
} else {
log::info!(target: TARGET, "Migrated {} agendas.", new_agendas);
}
Ok(())
}
}
}
#[cfg(test)]
#[cfg(feature = "try-runtime")]
mod test {
use super::*;
use crate::mock::*;
use frame_support::Hashable;
use sp_std::borrow::Cow;
use substrate_test_utils::assert_eq_uvec;
#[test]
#[allow(deprecated)]
fn migration_v3_to_v4_works() {
new_test_ext().execute_with(|| {
// Assume that we are at V3.
StorageVersion::new(3).put::<Scheduler>();
// Call that will be bounded to a `Lookup`.
let large_call =
RuntimeCall::System(frame_system::Call::remark { remark: vec![0; 1024] });
// Call that can be inlined.
let small_call =
RuntimeCall::System(frame_system::Call::remark { remark: vec![0; 10] });
// Call that is already hashed and can will be converted to `Legacy`.
let hashed_call =
RuntimeCall::System(frame_system::Call::remark { remark: vec![0; 2048] });
let bound_hashed_call = Preimage::bound(hashed_call.clone()).unwrap();
assert!(bound_hashed_call.lookup_needed());
// A Call by hash that will fail to decode becomes `None`.
let trash_data = vec![255u8; 1024];
let undecodable_hash = Preimage::note(Cow::Borrowed(&trash_data)).unwrap();
for i in 0..2u64 {
let k = i.twox_64_concat();
let old = vec![
Some(ScheduledV3Of::<Test> {
maybe_id: None,
priority: i as u8 + 10,
call: small_call.clone().into(),
maybe_periodic: None, // 1
origin: root(),
_phantom: PhantomData::<u64>::default(),
}),
None,
Some(ScheduledV3Of::<Test> {
maybe_id: Some(vec![i as u8; 32]),
priority: 123,
call: large_call.clone().into(),
maybe_periodic: Some((4u64, 20)),
origin: signed(i),
_phantom: PhantomData::<u64>::default(),
}),
Some(ScheduledV3Of::<Test> {
maybe_id: Some(vec![255 - i as u8; 320]),
priority: 123,
call: MaybeHashed::Hash(bound_hashed_call.hash()),
maybe_periodic: Some((8u64, 10)),
origin: signed(i),
_phantom: PhantomData::<u64>::default(),
}),
Some(ScheduledV3Of::<Test> {
maybe_id: Some(vec![i as u8; 320]),
priority: 123,
call: MaybeHashed::Hash(undecodable_hash.clone()),
maybe_periodic: Some((4u64, 20)),
origin: root(),
_phantom: PhantomData::<u64>::default(),
}),
];
frame_support::migration::put_storage_value(b"Scheduler", b"Agenda", &k, old);
}
let state = v3::MigrateToV4::<Test>::pre_upgrade().unwrap();
let _w = v3::MigrateToV4::<Test>::on_runtime_upgrade();
v3::MigrateToV4::<Test>::post_upgrade(state).unwrap();
let mut x = Agenda::<Test>::iter().map(|x| (x.0, x.1.into_inner())).collect::<Vec<_>>();
x.sort_by_key(|x| x.0);
let bound_large_call = Preimage::bound(large_call).unwrap();
assert!(bound_large_call.lookup_needed());
let bound_small_call = Preimage::bound(small_call).unwrap();
assert!(!bound_small_call.lookup_needed());
let expected = vec![
(
0,
vec![
Some(ScheduledOf::<Test> {
maybe_id: None,
priority: 10,
call: bound_small_call.clone(),
maybe_periodic: None,
origin: root(),
_phantom: PhantomData::<u64>::default(),
}),
None,
Some(ScheduledOf::<Test> {
maybe_id: Some(blake2_256(&[0u8; 32])),
priority: 123,
call: bound_large_call.clone(),
maybe_periodic: Some((4u64, 20)),
origin: signed(0),
_phantom: PhantomData::<u64>::default(),
}),
Some(ScheduledOf::<Test> {
maybe_id: Some(blake2_256(&[255u8; 320])),
priority: 123,
call: Bounded::from_legacy_hash(bound_hashed_call.hash()),
maybe_periodic: Some((8u64, 10)),
origin: signed(0),
_phantom: PhantomData::<u64>::default(),
}),
None,
],
),
(
1,
vec![
Some(ScheduledOf::<Test> {
maybe_id: None,
priority: 11,
call: bound_small_call.clone(),
maybe_periodic: None,
origin: root(),
_phantom: PhantomData::<u64>::default(),
}),
None,
Some(ScheduledOf::<Test> {
maybe_id: Some(blake2_256(&[1u8; 32])),
priority: 123,
call: bound_large_call.clone(),
maybe_periodic: Some((4u64, 20)),
origin: signed(1),
_phantom: PhantomData::<u64>::default(),
}),
Some(ScheduledOf::<Test> {
maybe_id: Some(blake2_256(&[254u8; 320])),
priority: 123,
call: Bounded::from_legacy_hash(bound_hashed_call.hash()),
maybe_periodic: Some((8u64, 10)),
origin: signed(1),
_phantom: PhantomData::<u64>::default(),
}),
None,
],
),
];
for (outer, (i, j)) in x.iter().zip(expected.iter()).enumerate() {
assert_eq!(i.0, j.0);
for (inner, (x, y)) in i.1.iter().zip(j.1.iter()).enumerate() {
assert_eq!(x, y, "at index: outer {} inner {}", outer, inner);
}
}
assert_eq_uvec!(x, expected);
assert_eq!(StorageVersion::get::<Scheduler>(), 4);
});
}
#[test]
#[allow(deprecated)]
fn migration_v3_to_v4_too_large_calls_are_ignored() {
new_test_ext().execute_with(|| {
// Assume that we are at V3.
StorageVersion::new(3).put::<Scheduler>();
let too_large_call = RuntimeCall::System(frame_system::Call::remark {
remark: vec![0; <Test as Config>::Preimages::MAX_LENGTH + 1],
});
let i = 0u64;
let k = i.twox_64_concat();
let old = vec![Some(ScheduledV3Of::<Test> {
maybe_id: None,
priority: 1,
call: too_large_call.clone().into(),
maybe_periodic: None,
origin: root(),
_phantom: PhantomData::<u64>::default(),
})];
frame_support::migration::put_storage_value(b"Scheduler", b"Agenda", &k, old);
// The pre_upgrade hook fails:
let err = v3::MigrateToV4::<Test>::pre_upgrade().unwrap_err();
assert!(err.contains("Call is too large"));
// But the migration itself works:
let _w = v3::MigrateToV4::<Test>::on_runtime_upgrade();
let mut x = Agenda::<Test>::iter().map(|x| (x.0, x.1.into_inner())).collect::<Vec<_>>();
x.sort_by_key(|x| x.0);
// The call becomes `None`.
let expected = vec![(0, vec![None])];
assert_eq_uvec!(x, expected);
assert_eq!(StorageVersion::get::<Scheduler>(), 4);
});
}
fn signed(i: u64) -> OriginCaller {
system::RawOrigin::Signed(i).into()
}
}