4 Commits

Author	SHA1	Message	Date
Oliver Tale-Yazdi	eefd5fe449	Multi-Block-Migrations, poll hook and new System callbacks (#1781) ... This MR is the merge of https://github.com/paritytech/substrate/pull/14414 and https://github.com/paritytech/substrate/pull/14275. It implements [RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes https://github.com/paritytech/polkadot-sdk/issues/198. ----- This Merge request introduces three major topicals: 1. Multi-Block-Migrations 1. New pallet `poll` hook for periodic service work 1. Replacement hooks for `on_initialize` and `on_finalize` in cases where `poll` cannot be used and some more general changes to FRAME. The changes for each topical span over multiple crates. They are listed in topical order below. # 1.) Multi-Block-Migrations Multi-Block-Migrations are facilitated by creating `pallet_migrations` and configuring `System::Config::MultiBlockMigrator` to point to it. Executive picks this up and triggers one step of the migrations pallet per block. The chain is in lockdown mode for as long as an MBM is ongoing. Executive does this by polling `MultiBlockMigrator::ongoing` and not allowing any transaction in a block, if true. A MBM is defined through trait `SteppedMigration`. A condensed version looks like this: ```rust /// A migration that can proceed in multiple steps. pub trait SteppedMigration { type Cursor: FullCodec + MaxEncodedLen; type Identifier: FullCodec + MaxEncodedLen; fn id() -> Self::Identifier; fn max_steps() -> Option<u32>; fn step( cursor: Option<Self::Cursor>, meter: &mut WeightMeter, ) -> Result<Option<Self::Cursor>, SteppedMigrationError>; } ``` `pallet_migrations` can be configured with an aggregated tuple of these migrations. It then starts to migrate them one-by-one on the next runtime upgrade. Two things are important here: - 1. Doing another runtime upgrade while MBMs are ongoing is not a good idea and can lead to messed up state. - 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`, otherwise it is not used.** The pallet supports an `UpgradeStatusHandler` that can be used to notify external logic of upgrade start/finish (for example to pause XCM dispatch). Error recovery is very limited in the case that a migration errors or times out (exceeds its `max_steps`). Currently the runtime dev can decide in `FailedMigrationHandler::failed` how to handle this. One follow-up would be to pair this with the `SafeMode` pallet and enact safe mode when an upgrade fails, to allow governance to rescue the chain. This is currently not possible, since governance is not `Mandatory`. ## Runtime API - `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to inform the Block Author whether they can push transactions. ### Integration Add it to your runtime implementation of `Core` and `BlockBuilder`: ```patch diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs @@ impl_runtime_apis! { impl sp_block_builder::Core<Block> for Runtime { - fn initialize_block(header: &<Block as BlockT>::Header) { + fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode { Executive::initialize_block(header) } ... } ``` # 2.) `poll` hook A new pallet hook is introduced: `poll`. `Poll` is intended to replace mostly all usage of `on_initialize`. The reason for this is that any code that can be called from `on_initialize` cannot be migrated through an MBM. Currently there is no way to statically check this; the implication is to use `on_initialize` as rarely as possible. Failing to do so can result in broken storage invariants. The implementation of the poll hook depends on the `Runtime API` changes that are explained above. # 3.) Hard-Deadline callbacks Three new callbacks are introduced and configured on `System::Config`: `PreInherents`, `PostInherents` and `PostTransactions`. These hooks are meant as replacement for `on_initialize` and `on_finalize` in cases where the code that runs cannot be moved to `poll`. The reason for this is to make the usage of HD-code (hard deadline) more explicit - again to prevent broken invariants by MBMs. # 4.) FRAME (general changes) ## `frame_system` pallet A new memorize storage item `InherentsApplied` is added. It is used by executive to track whether inherents have already been applied. Executive and can then execute the MBMs directly between inherents and transactions. The `Config` gets five new items: - `SingleBlockMigrations` this is the new way of configuring migrations that run in a single block. Previously they were defined as last generic argument of `Executive`. This shift is brings all central configuration about migrations closer into view of the developer (migrations that are configured in `Executive` will still work for now but is deprecated). - `MultiBlockMigrator` this can be configured to an engine that drives MBMs. One example would be the `pallet_migrations`. Note that this is only the engine; the exact MBMs are injected into the engine. - `PreInherents` a callback that executes after `on_initialize` but before inherents. - `PostInherents` a callback that executes after all inherents ran (including MBMs and `poll`). - `PostTransactions` in symmetry to `PreInherents`, this one is called before `on_finalize` but after all transactions. A sane default is to set all of these to `()`. Example diff suitable for any chain: ```patch @@ impl frame_system::Config for Test { type MaxConsumers = ConstU32<16>; + type SingleBlockMigrations = (); + type MultiBlockMigrator = (); + type PreInherents = (); + type PostInherents = (); + type PostTransactions = (); } ``` An overview of how the block execution now looks like is here. The same graph is also in the rust doc. <details><summary>Block Execution Flow</summary> <p>  </p> </details> ## Inherent Order Moved to https://github.com/paritytech/polkadot-sdk/pull/2154 --------------- ## TODO - [ ] Check that `try-runtime` still works - [ ] Ensure backwards compatibility with old Runtime APIs - [x] Consume weight correctly - [x] Cleanup --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: command-bot <> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>	2024-02-28 19:49:00 +00:00
joe petrowski	757ae372f7	Switch All construct_runtimes to New Syntax (#2979) ... Clean up all the old syntax. --------- Co-authored-by: command-bot <> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com> Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>	2024-01-22 07:15:53 +00:00
Michal Kucharczyk	8ba7a6aba8	chain-spec: getting ready for native-runtime-free world (#1256) ... This PR prepares chains specs for _native-runtime-free_ world. This PR has following changes: - `substrate`: - adds support for: - JSON based `GenesisConfig` to `ChainSpec` allowing interaction with runtime `GenesisBuilder` API. - interacting with arbitrary runtime wasm blob to[ `chain-spec-builder`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/bin/utils/chain-spec-builder/src/lib.rs#L46) command line util, - removes [`code`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/frame/system/src/lib.rs#L660) from `system_pallet` - adds `code` to the `ChainSpec` - deprecates [`ChainSpec::from_genesis`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/client/chain-spec/src/chain_spec.rs#L263), but also changes the signature of this method extending it with `code` argument. [`ChainSpec::builder()`](https://github.com/paritytech/substrate/blob/20bee680ed098be7239cf7a6b804cd4de267983e/client/chain-spec/src/chain_spec.rs#L507) should be used instead. - `polkadot`: - all references to `RuntimeGenesisConfig` in `node/service` are removed, - all `(kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config` functions now return the JSON patch for default runtime `GenesisConfig`, - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed, - `cumulus`: - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed, - _JSON_ patch configuration used instead of `RuntimeGenesisConfig struct` in all chain specs. --------- Co-authored-by: command-bot <> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Kevin Krone <kevin@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>	2023-11-05 15:19:23 +01:00
Juan Girini	43415ef58c	feat: FRAME umbrella crate. (#1337) ... ### Original PR https://github.com/paritytech/substrate/pull/14137 This PR brings in the first version of the "_`frame` umbrella crate_". This crate is intended to serve two purposes: 1. documentation 2. easier development with frame. Ideally, we want most users to be able to build a frame-based pallet and runtime using just `frame` (plus `scale-codec` and `scale-info`). The crate is not finalized and is not yet intended for external use. Therefore, the version is set to `0.0.1-dev`, this PR is `silent`, and the entire crate is hidden behind the `experimental` flag. The main intention in merging it early on is to be able to iterate on it in the rest of [`developer-hub`](https://github.com/paritytech/polkadot-sdk-docs/) efforts. The public API of the `frame` crate is at the moment as follows: ``` pub mod frame pub use frame::log pub use frame::pallet pub mod frame::arithmetic pub use frame::arithmetic::<<sp_arithmetic::*>> pub use frame::arithmetic::<<sp_arithmetic::traits::*>> pub mod frame::deps pub use frame::deps::codec pub use frame::deps::frame_executive pub use frame::deps::frame_support pub use frame::deps::frame_system pub use frame::deps::scale_info pub use frame::deps::sp_api pub use frame::deps::sp_arithmetic pub use frame::deps::sp_block_builder pub use frame::deps::sp_consensus_aura pub use frame::deps::sp_consensus_grandpa pub use frame::deps::sp_core pub use frame::deps::sp_inherents pub use frame::deps::sp_io pub use frame::deps::sp_offchain pub use frame::deps::sp_runtime pub use frame::deps::sp_std pub use frame::deps::sp_version pub mod frame::derive pub use frame::derive::CloneNoBound pub use frame::derive::Debug pub use frame::derive::Debug pub use frame::derive::DebugNoBound pub use frame::derive::Decode pub use frame::derive::Decode pub use frame::derive::DefaultNoBound pub use frame::derive::Encode pub use frame::derive::Encode pub use frame::derive::EqNoBound pub use frame::derive::PartialEqNoBound pub use frame::derive::RuntimeDebug pub use frame::derive::RuntimeDebugNoBound pub use frame::derive::TypeInfo pub use frame::derive::TypeInfo pub mod frame::prelude pub use frame::prelude::<<frame_support::pallet_prelude::*>> pub use frame::prelude::<<frame_system::pallet_prelude::*>> pub use frame::prelude::<<sp_std::prelude::*>> pub use frame::prelude::CloneNoBound pub use frame::prelude::Debug pub use frame::prelude::Debug pub use frame::prelude::DebugNoBound pub use frame::prelude::Decode pub use frame::prelude::Decode pub use frame::prelude::DefaultNoBound pub use frame::prelude::Encode pub use frame::prelude::Encode pub use frame::prelude::EqNoBound pub use frame::prelude::PartialEqNoBound pub use frame::prelude::RuntimeDebug pub use frame::prelude::RuntimeDebugNoBound pub use frame::prelude::TypeInfo pub use frame::prelude::TypeInfo pub use frame::prelude::frame_system pub mod frame::primitives pub use frame::primitives::BlakeTwo256 pub use frame::primitives::H160 pub use frame::primitives::H256 pub use frame::primitives::H512 pub use frame::primitives::Hash pub use frame::primitives::Keccak256 pub use frame::primitives::U256 pub use frame::primitives::U512 pub mod frame::runtime pub mod frame::runtime::apis pub use frame::runtime::apis::<<frame_system_rpc_runtime_api::*>> pub use frame::runtime::apis::<<sp_api::*>> pub use frame::runtime::apis::<<sp_block_builder::*>> pub use frame::runtime::apis::<<sp_consensus_aura::*>> pub use frame::runtime::apis::<<sp_consensus_grandpa::*>> pub use frame::runtime::apis::<<sp_offchain::*>> pub use frame::runtime::apis::<<sp_session::runtime_api::*>> pub use frame::runtime::apis::<<sp_transaction_pool::runtime_api::*>> pub use frame::runtime::apis::ApplyExtrinsicResult pub use frame::runtime::apis::CheckInherentsResult pub use frame::runtime::apis::InherentData pub use frame::runtime::apis::OpaqueMetadata pub use frame::runtime::apis::impl_runtime_apis pub use frame::runtime::apis::sp_api pub mod frame::runtime::prelude pub use frame::runtime::prelude::<<frame_executive::*>> pub use frame::runtime::prelude::ConstBool pub use frame::runtime::prelude::ConstI128 pub use frame::runtime::prelude::ConstI16 pub use frame::runtime::prelude::ConstI32 pub use frame::runtime::prelude::ConstI64 pub use frame::runtime::prelude::ConstI8 pub use frame::runtime::prelude::ConstU128 pub use frame::runtime::prelude::ConstU16 pub use frame::runtime::prelude::ConstU32 pub use frame::runtime::prelude::ConstU64 pub use frame::runtime::prelude::ConstU8 pub use frame::runtime::prelude::NativeVersion pub use frame::runtime::prelude::RuntimeVersion pub use frame::runtime::prelude::construct_runtime pub use frame::runtime::prelude::create_runtime_str pub use frame::runtime::prelude::derive_impl pub use frame::runtime::prelude::frame_support pub use frame::runtime::prelude::ord_parameter_types pub use frame::runtime::prelude::parameter_types pub use frame::runtime::prelude::runtime_version pub mod frame::runtime::testing_prelude pub use frame::runtime::testing_prelude::BuildStorage pub use frame::runtime::testing_prelude::Storage pub mod frame::runtime::types_common pub type frame::runtime::types_common::AccountId = <<frame::runtime::types_common::Signature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId pub type frame::runtime::types_common::BlockNumber = u32 pub type frame::runtime::types_common::BlockOf<T, Extra> = sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<frame::runtime::types_common::BlockNumber, sp_runtime::traits::BlakeTwo256>, sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<frame::runtime::types_common::AccountId, ()>, <T as frame_system::pallet::Config>::RuntimeCall, frame::runtime::types_common::Signature, Extra>> pub type frame::runtime::types_common::OpaqueBlock = sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<frame::runtime::types_common::BlockNumber, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic> pub type frame::runtime::types_common::Signature = sp_runtime::MultiSignature pub type frame::runtime::types_common::SystemSignedExtensionsOf<T> = (frame_system::extensions::check_non_zero_sender::CheckNonZeroSender<T>, frame_system::extensions::check_spec_version::CheckSpecVersion<T>, frame_system::extensions::check_tx_version::CheckTxVersion<T>, frame_system::extensions::check_genesis::CheckGenesis<T>, frame_system::extensions::check_mortality::CheckMortality<T>, frame_system::extensions::check_nonce::CheckNonce<T>, frame_system::extensions::check_weight::CheckWeight<T>) pub mod frame::testing_prelude pub use frame::testing_prelude::<<frame_executive::*>> pub use frame::testing_prelude::<<frame_system::mocking::*>> pub use frame::testing_prelude::BuildStorage pub use frame::testing_prelude::ConstBool pub use frame::testing_prelude::ConstI128 pub use frame::testing_prelude::ConstI16 pub use frame::testing_prelude::ConstI32 pub use frame::testing_prelude::ConstI64 pub use frame::testing_prelude::ConstI8 pub use frame::testing_prelude::ConstU128 pub use frame::testing_prelude::ConstU16 pub use frame::testing_prelude::ConstU32 pub use frame::testing_prelude::ConstU64 pub use frame::testing_prelude::ConstU8 pub use frame::testing_prelude::NativeVersion pub use frame::testing_prelude::RuntimeVersion pub use frame::testing_prelude::Storage pub use frame::testing_prelude::TestState pub use frame::testing_prelude::assert_err pub use frame::testing_prelude::assert_err_ignore_postinfo pub use frame::testing_prelude::assert_error_encoded_size pub use frame::testing_prelude::assert_noop pub use frame::testing_prelude::assert_ok pub use frame::testing_prelude::assert_storage_noop pub use frame::testing_prelude::construct_runtime pub use frame::testing_prelude::create_runtime_str pub use frame::testing_prelude::derive_impl pub use frame::testing_prelude::frame_support pub use frame::testing_prelude::frame_system pub use frame::testing_prelude::if_std pub use frame::testing_prelude::ord_parameter_types pub use frame::testing_prelude::parameter_types pub use frame::testing_prelude::runtime_version pub use frame::testing_prelude::storage_alias pub mod frame::traits pub use frame::traits::<<frame_support::traits::*>> pub use frame::traits::<<sp_runtime::traits::*>> ``` --- The road to full stabilization is - [ ] https://github.com/paritytech/polkadot-sdk/issues/127 - [ ] have a more intentional version bump, as opposed to the current bi weekly force-major-bump - [ ] revise the internal API of `frame`, especially what goes into the `prelude`s. - [ ] migrate all internal pallets and runtime to use `frame` --------- Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>	2023-10-27 11:38:16 +02:00