Runtime Upgrade ref docs and Single Block Migration example pallet (#1554)

Closes https://github.com/paritytech/polkadot-sdk-docs/issues/55

- Changes 'current storage version' terminology to less ambiguous
'in-code storage version' (suggestion by @ggwpez)
- Adds a new example pallet `pallet-example-single-block-migrations`
- Adds a new reference doc to replace
https://docs.substrate.io/maintain/runtime-upgrades/ (temporarily living
in the pallet while we wait for developer hub PR to merge)
- Adds documentation for the `storage_alias` macro
- Improves `trait Hooks` docs 
- Improves `trait GetStorageVersion` docs
- Update the suggested patterns for using `VersionedMigration`, so that
version unchecked migrations are never exported
- Prevents accidental usage of version unchecked migrations in runtimes

https://github.com/paritytech/substrate/pull/14421#discussion_r1255467895
- Unversioned migration code is kept inside `mod version_unchecked`,
versioned code is kept in `pub mod versioned`
- It is necessary to use modules to limit visibility because the inner
migration must be `pub`. See
https://github.com/rust-lang/rust/issues/30905 and

https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504/40
for more.

### todo

- [x] move to reference docs to proper place within sdk-docs (now that
https://github.com/paritytech/polkadot-sdk/pull/2102 is merged)
- [x] prdoc

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Juan <juangirini@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
This commit is contained in:
Liam Aharon
2024-02-28 18:32:02 +11:00
committed by GitHub
parent 7ec0b8741b
commit 12ce4f7d04
87 changed files with 1222 additions and 369 deletions
@@ -0,0 +1,213 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! # Single Block Migration Example Pallet
//!
//! An example pallet demonstrating best-practices for writing single-block migrations in the
//! context of upgrading pallet storage.
//!
//! ## Forwarning
//!
//! Single block migrations **MUST** execute in a single block, therefore when executed on a
//! parachain are only appropriate when guaranteed to not exceed block weight limits. If a
//! parachain submits a block that exceeds the block weight limit it will **brick the chain**!
//!
//! If weight is a concern or you are not sure which type of migration to use, you should probably
//! use a multi-block migration.
//!
//! TODO: Link above to multi-block migration example.
//!
//! ## Pallet Overview
//!
//! This example pallet contains a single storage item [`Value`](pallet::Value), which may be set by
//! any signed origin by calling the [`set_value`](crate::Call::set_value) extrinsic.
//!
//! For the purposes of this exercise, we imagine that in [`StorageVersion`] V0 of this pallet
//! [`Value`](pallet::Value) is a `u32`, and this what is currently stored on-chain.
//!
//! ```ignore
//! // (Old) Storage Version V0 representation of `Value`
//! #[pallet::storage]
//! pub type Value<T: Config> = StorageValue<_, u32>;
//! ```
//!
//! In [`StorageVersion`] V1 of the pallet a new struct [`CurrentAndPreviousValue`] is introduced:
#![doc = docify::embed!("src/lib.rs", CurrentAndPreviousValue)]
//! and [`Value`](pallet::Value) is updated to store this new struct instead of a `u32`:
#![doc = docify::embed!("src/lib.rs", Value)]
//!
//! In StorageVersion V1 of the pallet when [`set_value`](crate::Call::set_value) is called, the
//! new value is stored in the `current` field of [`CurrentAndPreviousValue`], and the previous
//! value (if it exists) is stored in the `previous` field.
#![doc = docify::embed!("src/lib.rs", pallet_calls)]
//!
//! ## Why a migration is necessary
//!
//! Without a migration, there will be a discrepancy between the on-chain storage for [`Value`] (in
//! V0 it is a `u32`) and the current storage for [`Value`] (in V1 it was changed to a
//! [`CurrentAndPreviousValue`] struct).
//!
//! The on-chain storage for [`Value`] would be a `u32` but the runtime would try to read it as a
//! [`CurrentAndPreviousValue`]. This would result in unacceptable undefined behavior.
//!
//! ## Adding a migration module
//!
//! Writing a pallets migrations in a seperate module is strongly recommended.
//!
//! Here's how the migration module is defined for this pallet:
//!
//! ```text
//! substrate/frame/examples/single-block-migrations/src/
//! ├── lib.rs <-- pallet definition
//! ├── Cargo.toml <-- pallet manifest
//! └── migrations/
//! ├── mod.rs <-- migrations module definition
//! └── v1.rs <-- migration logic for the V0 to V1 transition
//! ```
//!
//! This structure allows keeping migration logic separate from the pallet logic and
//! easily adding new migrations in the future.
//!
//! ## Writing the Migration
//!
//! All code related to the migration can be found under
//! [`v1.rs`](migrations::v1).
//!
//! See the migration source code for detailed comments.
//!
//! To keep the migration logic organised, it is split across additional modules:
//!
//! ### `mod v0`
//!
//! Here we define a [`storage_alias`](frame_support::storage_alias) for the old v0 [`Value`]
//! format.
//!
//! This allows reading the old v0 value from storage during the migration.
//!
//! ### `mod version_unchecked`
//!
//! Here we define our raw migration logic,
//! `version_unchecked::MigrateV0ToV1` which implements the [`OnRuntimeUpgrade`] trait.
//!
//! Importantly, it is kept in a private module so that it cannot be accidentally used in a runtime.
//!
//! Private modules cannot be referenced in docs, so please read the code directly.
//!
//! #### Standalone Struct or Pallet Hook?
//!
//! Note that the storage migration logic is attached to a standalone struct implementing
//! [`OnRuntimeUpgrade`], rather than implementing the
//! [`Hooks::on_runtime_upgrade`](frame_support::traits::Hooks::on_runtime_upgrade) hook directly on
//! the pallet. The pallet hook is better suited for special types of logic that need to execute on
//! every runtime upgrade, but not so much for one-off storage migrations.
//!
//! ### `pub mod versioned`
//!
//! Here, `version_unchecked::MigrateV0ToV1` is wrapped in a
//! [`VersionedMigration`] to define
//! [`versioned::MigrateV0ToV1`](crate::migrations::v1::versioned::MigrateV0ToV1), which may be used
//! in runtimes.
//!
//! Using [`VersionedMigration`] ensures that
//! - The migration only runs once when the on-chain storage version is `0`
//! - The on-chain storage version is updated to `1` after the migration executes
//! - Reads and writes from checking and setting the on-chain storage version are accounted for in
//! the final [`Weight`](frame_support::weights::Weight)
//!
//! This is the only public module exported from `v1`.
//!
//! ### `mod test`
//!
//! Here basic unit tests are defined for the migration.
//!
//! When writing migration tests, don't forget to check:
//! - `on_runtime_upgrade` returns the expected weight
//! - `post_upgrade` succeeds when given the bytes returned by `pre_upgrade`
//! - Pallet storage is in the expected state after the migration
//!
//! [`VersionedMigration`]: frame_support::migrations::VersionedMigration
//! [`GetStorageVersion`]: frame_support::traits::GetStorageVersion
//! [`OnRuntimeUpgrade`]: frame_support::traits::OnRuntimeUpgrade
//! [`MigrateV0ToV1`]: crate::migrations::v1::versioned::MigrationV0ToV1
// We make sure this pallet uses `no_std` for compiling to Wasm.
#![cfg_attr(not(feature = "std"), no_std)]
// allow non-camel-case names for storage version V0 value
#![allow(non_camel_case_types)]
// Re-export pallet items so that they can be accessed from the crate namespace.
pub use pallet::*;
// Export migrations so they may be used in the runtime.
pub mod migrations;
#[doc(hidden)]
mod mock;
use codec::{Decode, Encode, MaxEncodedLen};
use frame_support::traits::StorageVersion;
use sp_runtime::RuntimeDebug;
/// Example struct holding the most recently set [`u32`] and the
/// second most recently set [`u32`] (if one existed).
#[docify::export]
#[derive(
Clone, Eq, PartialEq, Encode, Decode, RuntimeDebug, scale_info::TypeInfo, MaxEncodedLen,
)]
pub struct CurrentAndPreviousValue {
/// The most recently set value.
pub current: u32,
/// The previous value, if one existed.
pub previous: Option<u32>,
}
// Pallet for demonstrating storage migrations.
#[frame_support::pallet(dev_mode)]
pub mod pallet {
use super::*;
use frame_support::pallet_prelude::*;
use frame_system::pallet_prelude::*;
/// Define the current [`StorageVersion`] of the pallet.
const STORAGE_VERSION: StorageVersion = StorageVersion::new(1);
#[pallet::pallet]
#[pallet::storage_version(STORAGE_VERSION)]
pub struct Pallet<T>(_);
#[pallet::config]
pub trait Config: frame_system::Config {}
/// [`StorageVersion`] V1 of [`Value`].
///
/// Currently used.
#[docify::export]
#[pallet::storage]
pub type Value<T: Config> = StorageValue<_, CurrentAndPreviousValue>;
#[docify::export(pallet_calls)]
#[pallet::call]
impl<T: Config> Pallet<T> {
pub fn set_value(origin: OriginFor<T>, value: u32) -> DispatchResult {
ensure_signed(origin)?;
let previous = Value::<T>::get().map(|v| v.current);
let new_struct = CurrentAndPreviousValue { current: value, previous };
<Value<T>>::put(new_struct);
Ok(())
}
}
}
@@ -0,0 +1,20 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
/// Module containing all logic associated with the example migration from
/// [`StorageVersion`](frame_support::traits::StorageVersion) V0 to V1.
pub mod v1;
@@ -0,0 +1,222 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use frame_support::{
storage_alias,
traits::{Get, OnRuntimeUpgrade},
};
#[cfg(feature = "try-runtime")]
use sp_std::vec::Vec;
/// Collection of storage item formats from the previous storage version.
///
/// Required so we can read values in the v0 storage format during the migration.
mod v0 {
use super::*;
/// V0 type for [`crate::Value`].
#[storage_alias]
pub type Value<T: crate::Config> = StorageValue<crate::Pallet<T>, u32>;
}
/// Private module containing *version unchecked* migration logic.
///
/// Should only be used by the [`VersionedMigration`](frame_support::migrations::VersionedMigration)
/// type in this module to create something to export.
///
/// The unversioned migration should be kept private so the unversioned migration cannot
/// accidentally be used in any runtimes.
///
/// For more about this pattern of keeping items private, see
/// - <https://github.com/rust-lang/rust/issues/30905>
/// - <https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504/40>
mod version_unchecked {
use super::*;
/// Implements [`OnRuntimeUpgrade`], migrating the state of this pallet from V0 to V1.
///
/// In V0 of the template [`crate::Value`] is just a `u32`. In V1, it has been upgraded to
/// contain the struct [`crate::CurrentAndPreviousValue`].
///
/// In this migration, update the on-chain storage for the pallet to reflect the new storage
/// layout.
pub struct MigrateV0ToV1<T: crate::Config>(sp_std::marker::PhantomData<T>);
impl<T: crate::Config> OnRuntimeUpgrade for MigrateV0ToV1<T> {
/// Return the existing [`crate::Value`] so we can check that it was correctly set in
/// `version_unchecked::MigrateV0ToV1::post_upgrade`.
#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, sp_runtime::TryRuntimeError> {
use codec::Encode;
// Access the old value using the `storage_alias` type
let old_value = v0::Value::<T>::get();
// Return it as an encoded `Vec<u8>`
Ok(old_value.encode())
}
/// Migrate the storage from V0 to V1.
///
/// - If the value doesn't exist, there is nothing to do.
/// - If the value exists, it is read and then written back to storage inside a
/// [`crate::CurrentAndPreviousValue`].
fn on_runtime_upgrade() -> frame_support::weights::Weight {
// Read the old value from storage
if let Some(old_value) = v0::Value::<T>::take() {
// Write the new value to storage
let new = crate::CurrentAndPreviousValue { current: old_value, previous: None };
crate::Value::<T>::put(new);
// One read for the old value, one write for the new value
T::DbWeight::get().reads_writes(1, 1)
} else {
// One read for trying to access the old value
T::DbWeight::get().reads(1)
}
}
/// Verifies the storage was migrated correctly.
///
/// - If there was no old value, the new value should not be set.
/// - If there was an old value, the new value should be a
/// [`crate::CurrentAndPreviousValue`].
#[cfg(feature = "try-runtime")]
fn post_upgrade(state: Vec<u8>) -> Result<(), sp_runtime::TryRuntimeError> {
use codec::Decode;
use frame_support::ensure;
let maybe_old_value = Option::<u32>::decode(&mut &state[..]).map_err(|_| {
sp_runtime::TryRuntimeError::Other("Failed to decode old value from storage")
})?;
match maybe_old_value {
Some(old_value) => {
let expected_new_value =
crate::CurrentAndPreviousValue { current: old_value, previous: None };
let actual_new_value = crate::Value::<T>::get();
ensure!(actual_new_value.is_some(), "New value not set");
ensure!(
actual_new_value == Some(expected_new_value),
"New value not set correctly"
);
},
None => {
ensure!(crate::Value::<T>::get().is_none(), "New value unexpectedly set");
},
};
Ok(())
}
}
}
/// Public module containing *version checked* migration logic.
///
/// This is the only module that should be exported from this module.
///
/// See [`VersionedMigration`](frame_support::migrations::VersionedMigration) docs for more about
/// how it works.
pub mod versioned {
use super::*;
/// `version_unchecked::MigrateV0ToV1` wrapped in a
/// [`VersionedMigration`](frame_support::migrations::VersionedMigration), which ensures that:
/// - The migration only runs once when the on-chain storage version is 0
/// - The on-chain storage version is updated to `1` after the migration executes
/// - Reads/Writes from checking/settings the on-chain storage version are accounted for
pub type MigrateV0ToV1<T> = frame_support::migrations::VersionedMigration<
0, // The migration will only execute when the on-chain storage version is 0
1, // The on-chain storage version will be set to 1 after the migration is complete
version_unchecked::MigrateV0ToV1<T>,
crate::pallet::Pallet<T>,
<T as frame_system::Config>::DbWeight,
>;
}
/// Tests for our migration.
///
/// When writing migration tests, it is important to check:
/// 1. `on_runtime_upgrade` returns the expected weight
/// 2. `post_upgrade` succeeds when given the bytes returned by `pre_upgrade`
/// 3. The storage is in the expected state after the migration
#[cfg(any(all(feature = "try-runtime", test), doc))]
mod test {
use super::*;
use crate::mock::{new_test_ext, MockRuntime};
use frame_support::assert_ok;
use version_unchecked::MigrateV0ToV1;
#[test]
fn handles_no_existing_value() {
new_test_ext().execute_with(|| {
// By default, no value should be set. Verify this assumption.
assert!(crate::Value::<MockRuntime>::get().is_none());
assert!(v0::Value::<MockRuntime>::get().is_none());
// Get the pre_upgrade bytes
let bytes = match MigrateV0ToV1::<MockRuntime>::pre_upgrade() {
Ok(bytes) => bytes,
Err(e) => panic!("pre_upgrade failed: {:?}", e),
};
// Execute the migration
let weight = MigrateV0ToV1::<MockRuntime>::on_runtime_upgrade();
// Verify post_upgrade succeeds
assert_ok!(MigrateV0ToV1::<MockRuntime>::post_upgrade(bytes));
// The weight should be just 1 read for trying to access the old value.
assert_eq!(weight, <MockRuntime as frame_system::Config>::DbWeight::get().reads(1));
// After the migration, no value should have been set.
assert!(crate::Value::<MockRuntime>::get().is_none());
})
}
#[test]
fn handles_existing_value() {
new_test_ext().execute_with(|| {
// Set up an initial value
let initial_value = 42;
v0::Value::<MockRuntime>::put(initial_value);
// Get the pre_upgrade bytes
let bytes = match MigrateV0ToV1::<MockRuntime>::pre_upgrade() {
Ok(bytes) => bytes,
Err(e) => panic!("pre_upgrade failed: {:?}", e),
};
// Execute the migration
let weight = MigrateV0ToV1::<MockRuntime>::on_runtime_upgrade();
// Verify post_upgrade succeeds
assert_ok!(MigrateV0ToV1::<MockRuntime>::post_upgrade(bytes));
// The weight used should be 1 read for the old value, and 1 write for the new
// value.
assert_eq!(
weight,
<MockRuntime as frame_system::Config>::DbWeight::get().reads_writes(1, 1)
);
// After the migration, the new value should be set as the `current` value.
let expected_new_value =
crate::CurrentAndPreviousValue { current: initial_value, previous: None };
assert_eq!(crate::Value::<MockRuntime>::get(), Some(expected_new_value));
})
}
}
@@ -0,0 +1,69 @@
// This file is part of Substrate.
// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![cfg(any(all(feature = "try-runtime", test), doc))]
use crate::*;
use frame_support::{derive_impl, traits::ConstU64, weights::constants::ParityDbWeight};
// Re-export crate as its pallet name for construct_runtime.
use crate as pallet_example_storage_migration;
type Block = frame_system::mocking::MockBlock<MockRuntime>;
// For testing the pallet, we construct a mock runtime.
frame_support::construct_runtime!(
pub struct MockRuntime {
System: frame_system::{Pallet, Call, Config<T>, Storage, Event<T>},
Balances: pallet_balances::{Pallet, Call, Storage, Config<T>, Event<T>},
Example: pallet_example_storage_migration::{Pallet, Call, Storage},
}
);
#[derive_impl(frame_system::config_preludes::TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::Config for MockRuntime {
type Block = Block;
type AccountData = pallet_balances::AccountData<u64>;
type DbWeight = ParityDbWeight;
}
impl pallet_balances::Config for MockRuntime {
type RuntimeHoldReason = RuntimeHoldReason;
type RuntimeFreezeReason = RuntimeFreezeReason;
type MaxLocks = ();
type MaxReserves = ();
type ReserveIdentifier = [u8; 8];
type Balance = u64;
type DustRemoval = ();
type RuntimeEvent = RuntimeEvent;
type ExistentialDeposit = ConstU64<1>;
type AccountStore = System;
type WeightInfo = ();
type FreezeIdentifier = ();
type MaxFreezes = ();
}
impl Config for MockRuntime {}
pub fn new_test_ext() -> sp_io::TestExternalities {
use sp_runtime::BuildStorage;
let t = RuntimeGenesisConfig { system: Default::default(), balances: Default::default() }
.build_storage()
.unwrap();
t.into()
}