[doc] Example MBM pallet (#2119)

## Basic example showcasing a migration using the MBM framework

This PR has been built on top of
https://github.com/paritytech/polkadot-sdk/pull/1781 and adds two new
example crates to the `examples` pallet

### Changes Made:

Added the `pallet-example-mbm` crate: This crate provides a minimal
example of a pallet that uses MBM. It showcases a storage migration
where values are migrated from a `u32` to a `u64`.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
This commit is contained in:
Juan Girini
2024-04-04 13:47:24 +02:00
committed by GitHub
parent 0ef37c7540
commit bcb4d137c9
17 changed files with 686 additions and 11 deletions
@@ -0,0 +1,87 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![cfg_attr(not(feature = "std"), no_std)]
//! # Multi-Block Migrations Example Pallet
//!
//! This pallet serves as a minimal example of a pallet that uses the [Multi-Block Migrations
//! Framework](frame_support::migrations). You can observe how to configure it in a runtime in the
//! `kitchensink-runtime` crate.
//!
//! ## Introduction and Purpose
//!
//! The primary purpose of this pallet is to demonstrate the concept of Multi-Block Migrations in
//! Substrate. It showcases the migration of values from in the
//! [`MyMap`](`pallet::MyMap`) storage map a `u32` to a `u64` data type using the
//! [`SteppedMigration`](`frame_support::migrations::SteppedMigration`) implementation from the
//! [`migrations::v1`] module.
//!
//! The [`MyMap`](`pallet::MyMap`) storage item is defined in this `pallet`, and is
//! aliased to [`v0::MyMap`](`migrations::v1::v0::MyMap`) in the [`migrations::v1`]
//! module.
//!
//! ## How to Read the Documentation
//!
//! To access and navigate this documentation in your browser, use the following command:
//!
//! - `cargo doc --package pallet-example-mbm --open`
//!
//! This documentation is organized to help you understand the pallet's components, features, and
//! migration process.
//!
//! ## Example Usage
//!
//! To use this pallet and understand multi-block migrations, you can refer to the
//! [`migrations::v1`] module, which contains a step-by-step migration example.
//!
//! ## Pallet Structure
//!
//! The pallet is structured as follows:
//!
//! - [`migrations`]: Contains migration-related modules and migration logic.
//! - [`v1`](`migrations::v1`): Demonstrates the migration process for changing the data type in
//! the storage map.
//! - [`pallet`]: Defines the pallet configuration and storage items.
//!
//! ## Migration Safety
//!
//! When working with migrations, it's crucial to ensure the safety of your migrations. The
//! preferred tool to test migrations is
//! [`try-runtime-cli`](https://github.com/paritytech/try-runtime-cli). Support will be added to
//! dry-run MBMs once they are stable
//! (tracked: <https://github.com/paritytech/try-runtime-cli/issues/17>).
pub mod migrations;
mod mock;
pub use pallet::*;
#[frame_support::pallet]
pub mod pallet {
use frame_support::{pallet_prelude::StorageMap, Blake2_128Concat};
#[pallet::pallet]
pub struct Pallet<T>(_);
#[pallet::config]
pub trait Config: frame_system::Config {}
/// Define a storage item to illustrate multi-block migrations.
#[pallet::storage]
pub type MyMap<T: Config> = StorageMap<_, Blake2_128Concat, u32, u64>;
}
@@ -0,0 +1,29 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
/// # Multi-Block Migrations Module
///
/// This module showcases a simple use of the multi-block migrations framework.
pub mod v1;
/// A unique identifier across all pallets.
///
/// This constant represents a unique identifier for the migrations of this pallet.
/// It helps differentiate migrations for this pallet from those of others. Note that we don't
/// directly pull the crate name from the environment, since that would change if the crate were
/// ever to be renamed and could cause historic migrations to run again.
pub const PALLET_MIGRATIONS_ID: &[u8; 18] = b"pallet-example-mbm";
@@ -0,0 +1,54 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Benchmark the multi-block-migration.
#![cfg(feature = "runtime-benchmarks")]
use crate::{
migrations::{
v1,
v1::{weights, weights::WeightInfo},
},
Config, Pallet,
};
use frame_benchmarking::v2::*;
use frame_support::{migrations::SteppedMigration, weights::WeightMeter};
#[benchmarks]
mod benches {
use super::*;
/// Benchmark a single step of the `v1::LazyMigrationV1` migration.
#[benchmark]
fn step() {
v1::v0::MyMap::<T>::insert(0, 0);
let mut meter = WeightMeter::new();
#[block]
{
v1::LazyMigrationV1::<T, weights::SubstrateWeight<T>>::step(None, &mut meter).unwrap();
}
// Check that the new storage is decodable:
assert_eq!(crate::MyMap::<T>::get(0), Some(0));
// uses twice the weight once for migration and then for checking if there is another key.
assert_eq!(meter.consumed(), weights::SubstrateWeight::<T>::step() * 2);
}
impl_benchmark_test_suite!(Pallet, crate::mock::new_test_ext(), crate::mock::Runtime);
}
@@ -0,0 +1,118 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! # Multi-Block Migration v1
//!
//! This module showcases a simple migration that iterates over the values in the
//! [`v0::MyMap`](`crate::migrations::v1::v0::MyMap`) storage map, transforms them,
//! and inserts them into the [`MyMap`](`crate::pallet::MyMap`) storage map.
use super::PALLET_MIGRATIONS_ID;
use crate::pallet::{Config, MyMap};
use frame_support::{
migrations::{MigrationId, SteppedMigration, SteppedMigrationError},
pallet_prelude::PhantomData,
weights::WeightMeter,
};
mod benchmarks;
mod tests;
pub mod weights;
/// Module containing the OLD (v0) storage items.
///
/// Before running this migration, the storage alias defined here represents the
/// `on_chain` storage.
// This module is public only for the purposes of linking it in the documentation. It is not
// intended to be used by any other code.
pub mod v0 {
use super::Config;
use crate::pallet::Pallet;
use frame_support::{storage_alias, Blake2_128Concat};
#[storage_alias]
/// The storage item that is being migrated from.
pub type MyMap<T: Config> = StorageMap<Pallet<T>, Blake2_128Concat, u32, u32>;
}
/// Migrates the items of the [`crate::MyMap`] map from `u32` to `u64`.
///
/// The `step` function will be called once per block. It is very important that this function
/// *never* panics and never uses more weight than it got in its meter. The migrations should also
/// try to make maximal progress per step, so that the total time it takes to migrate stays low.
pub struct LazyMigrationV1<T: Config, W: weights::WeightInfo>(PhantomData<(T, W)>);
impl<T: Config, W: weights::WeightInfo> SteppedMigration for LazyMigrationV1<T, W> {
type Cursor = u32;
// Without the explicit length here the construction of the ID would not be infallible.
type Identifier = MigrationId<18>;
/// The identifier of this migration. Which should be globally unique.
fn id() -> Self::Identifier {
MigrationId { pallet_id: *PALLET_MIGRATIONS_ID, version_from: 0, version_to: 1 }
}
/// The actual logic of the migration.
///
/// This function is called repeatedly until it returns `Ok(None)`, indicating that the
/// migration is complete. Ideally, the migration should be designed in such a way that each
/// step consumes as much weight as possible. However, this is simplified to perform one stored
/// value mutation per block.
fn step(
mut cursor: Option<Self::Cursor>,
meter: &mut WeightMeter,
) -> Result<Option<Self::Cursor>, SteppedMigrationError> {
let required = W::step();
// If there is not enough weight for a single step, return an error. This case can be
// problematic if it is the first migration that ran in this block. But there is nothing
// that we can do about it here.
if meter.remaining().any_lt(required) {
return Err(SteppedMigrationError::InsufficientWeight { required });
}
// We loop here to do as much progress as possible per step.
loop {
if meter.try_consume(required).is_err() {
break;
}
let mut iter = if let Some(last_key) = cursor {
// If a cursor is provided, start iterating from the stored value
// corresponding to the last key processed in the previous step.
// Note that this only works if the old and the new map use the same way to hash
// storage keys.
v0::MyMap::<T>::iter_from(v0::MyMap::<T>::hashed_key_for(last_key))
} else {
// If no cursor is provided, start iterating from the beginning.
v0::MyMap::<T>::iter()
};
// If there's a next item in the iterator, perform the migration.
if let Some((last_key, value)) = iter.next() {
// Migrate the inner value: u32 -> u64.
let value = value as u64;
// We can just insert here since the old and the new map share the same key-space.
// Otherwise it would have to invert the concat hash function and re-hash it.
MyMap::<T>::insert(last_key, value);
cursor = Some(last_key) // Return the processed key as the new cursor.
} else {
cursor = None; // Signal that the migration is complete (no more items to process).
break
}
}
Ok(cursor)
}
}
@@ -0,0 +1,70 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![cfg(all(test, not(feature = "runtime-benchmarks")))]
use crate::{
migrations::{
v1,
v1::{weights, weights::WeightInfo as _},
},
mock::{
new_test_ext, run_to_block, AllPalletsWithSystem, MigratorServiceWeight, Runtime as T,
System,
},
};
use frame_support::traits::OnRuntimeUpgrade;
use pallet_migrations::WeightInfo as _;
#[test]
fn lazy_migration_works() {
new_test_ext().execute_with(|| {
frame_support::__private::sp_tracing::try_init_simple();
// Insert some values into the old storage map.
for i in 0..1024 {
v1::v0::MyMap::<T>::insert(i, i);
}
// Give it enough weight do do exactly 16 iterations:
let limit = <T as pallet_migrations::Config>::WeightInfo::progress_mbms_none() +
pallet_migrations::Pallet::<T>::exec_migration_max_weight() +
weights::SubstrateWeight::<T>::step() * 16;
MigratorServiceWeight::set(&limit);
System::set_block_number(1);
AllPalletsWithSystem::on_runtime_upgrade(); // onboard MBMs
let mut last_decodable = 0;
for block in 2..=65 {
run_to_block(block);
let mut decodable = 0;
for i in 0..1024 {
if crate::MyMap::<T>::get(i).is_some() {
decodable += 1;
}
}
assert_eq!(decodable, last_decodable + 16);
last_decodable = decodable;
}
// Check that everything is decodable now:
for i in 0..1024 {
assert_eq!(crate::MyMap::<T>::get(i), Some(i as u64));
}
});
}
@@ -0,0 +1,84 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Autogenerated weights for `pallet_example_mbm`
//!
//! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 32.0.0
//! DATE: 2024-03-26, STEPS: `50`, REPEAT: `20`, LOW RANGE: `[]`, HIGH RANGE: `[]`
//! WORST CASE MAP SIZE: `1000000`
//! HOSTNAME: `Olivers-MBP`, CPU: `<UNKNOWN>`
//! WASM-EXECUTION: `Compiled`, CHAIN: `None`, DB CACHE: `1024`
// Executed Command:
// polkadot-omni-bencher
// v1
// benchmark
// pallet
// --runtime
// target/release/wbuild/kitchensink-runtime/kitchensink_runtime.compact.compressed.wasm
// --pallet
// pallet_example_mbm
// --extrinsic
//
// --template
// substrate/.maintain/frame-weight-template.hbs
// --output
// substrate/frame/examples/multi-block-migrations/src/migrations/weights.rs
#![cfg_attr(rustfmt, rustfmt_skip)]
#![allow(unused_parens)]
#![allow(unused_imports)]
#![allow(missing_docs)]
use frame_support::{traits::Get, weights::{Weight, constants::RocksDbWeight}};
use core::marker::PhantomData;
/// Weight functions needed for `pallet_example_mbm`.
pub trait WeightInfo {
fn step() -> Weight;
}
/// Weights for `pallet_example_mbm` using the Substrate node and recommended hardware.
pub struct SubstrateWeight<T>(PhantomData<T>);
impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
/// Storage: `PalletExampleMbms::MyMap` (r:2 w:1)
/// Proof: `PalletExampleMbms::MyMap` (`max_values`: None, `max_size`: Some(28), added: 2503, mode: `MaxEncodedLen`)
fn step() -> Weight {
// Proof Size summary in bytes:
// Measured: `28`
// Estimated: `5996`
// Minimum execution time: 6_000_000 picoseconds.
Weight::from_parts(8_000_000, 5996)
.saturating_add(T::DbWeight::get().reads(2_u64))
.saturating_add(T::DbWeight::get().writes(1_u64))
}
}
// For backwards compatibility and tests.
impl WeightInfo for () {
/// Storage: `PalletExampleMbms::MyMap` (r:2 w:1)
/// Proof: `PalletExampleMbms::MyMap` (`max_values`: None, `max_size`: Some(28), added: 2503, mode: `MaxEncodedLen`)
fn step() -> Weight {
// Proof Size summary in bytes:
// Measured: `28`
// Estimated: `5996`
// Minimum execution time: 6_000_000 picoseconds.
Weight::from_parts(8_000_000, 5996)
.saturating_add(RocksDbWeight::get().reads(2_u64))
.saturating_add(RocksDbWeight::get().writes(1_u64))
}
}
@@ -0,0 +1,93 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![cfg(test)]
//! # Mock runtime for testing Multi-Block-Migrations
//!
//! This runtime is for testing only and should *never* be used in production. Please see the
//! comments on the specific config items. The core part of this runtime is the
//! [`pallet_migrations::Config`] implementation, where you define the migrations you want to run
//! using the [`Migrations`] type.
use frame_support::{
construct_runtime, derive_impl,
migrations::MultiStepMigrator,
pallet_prelude::Weight,
traits::{OnFinalize, OnInitialize},
};
type Block = frame_system::mocking::MockBlock<Runtime>;
impl crate::Config for Runtime {}
frame_support::parameter_types! {
pub storage MigratorServiceWeight: Weight = Weight::from_parts(100, 100); // do not use in prod
}
#[derive_impl(pallet_migrations::config_preludes::TestDefaultConfig)]
impl pallet_migrations::Config for Runtime {
// Here we inject the actual MBMs. Currently there is just one, but it accepts a tuple.
//
// # Example
// ```ignore
// type Migrations = (v1::Migration<Runtime>, v2::Migration<Runtime>, v3::Migration<Runtime>);
// ```
#[cfg(not(feature = "runtime-benchmarks"))]
type Migrations = (
crate::migrations::v1::LazyMigrationV1<
Runtime,
crate::migrations::v1::weights::SubstrateWeight<Runtime>,
>,
);
#[cfg(feature = "runtime-benchmarks")]
type Migrations = pallet_migrations::mock_helpers::MockedMigrations;
type MaxServiceWeight = MigratorServiceWeight;
}
#[derive_impl(frame_system::config_preludes::TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::Config for Runtime {
type Block = Block;
type MultiBlockMigrator = Migrator;
}
// Construct the runtime using the `construct_runtime` macro, specifying the pallet_migrations.
construct_runtime! {
pub struct Runtime
{
System: frame_system,
Pallet: crate,
Migrator: pallet_migrations,
}
}
pub fn new_test_ext() -> sp_io::TestExternalities {
sp_io::TestExternalities::new(Default::default())
}
#[allow(dead_code)]
pub fn run_to_block(n: u64) {
assert!(System::block_number() < n);
while System::block_number() < n {
let b = System::block_number();
AllPalletsWithSystem::on_finalize(b);
// Done by Executive:
<Runtime as frame_system::Config>::MultiBlockMigrator::step();
System::set_block_number(b + 1);
AllPalletsWithSystem::on_initialize(b + 1);
}
}