Oliver Tale-Yazdi
eefd5fe449
This MR is the merge of https://github.com/paritytech/substrate/pull/14414 and https://github.com/paritytech/substrate/pull/14275. It implements [RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes https://github.com/paritytech/polkadot-sdk/issues/198. ----- This Merge request introduces three major topicals: 1. Multi-Block-Migrations 1. New pallet `poll` hook for periodic service work 1. Replacement hooks for `on_initialize` and `on_finalize` in cases where `poll` cannot be used and some more general changes to FRAME. The changes for each topical span over multiple crates. They are listed in topical order below. # 1.) Multi-Block-Migrations Multi-Block-Migrations are facilitated by creating `pallet_migrations` and configuring `System::Config::MultiBlockMigrator` to point to it. Executive picks this up and triggers one step of the migrations pallet per block. The chain is in lockdown mode for as long as an MBM is ongoing. Executive does this by polling `MultiBlockMigrator::ongoing` and not allowing any transaction in a block, if true. A MBM is defined through trait `SteppedMigration`. A condensed version looks like this: ```rust /// A migration that can proceed in multiple steps. pub trait SteppedMigration { type Cursor: FullCodec + MaxEncodedLen; type Identifier: FullCodec + MaxEncodedLen; fn id() -> Self::Identifier; fn max_steps() -> Option<u32>; fn step( cursor: Option<Self::Cursor>, meter: &mut WeightMeter, ) -> Result<Option<Self::Cursor>, SteppedMigrationError>; } ``` `pallet_migrations` can be configured with an aggregated tuple of these migrations. It then starts to migrate them one-by-one on the next runtime upgrade. Two things are important here: - 1. Doing another runtime upgrade while MBMs are ongoing is not a good idea and can lead to messed up state. - 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`, otherwise it is not used.** The pallet supports an `UpgradeStatusHandler` that can be used to notify external logic of upgrade start/finish (for example to pause XCM dispatch). Error recovery is very limited in the case that a migration errors or times out (exceeds its `max_steps`). Currently the runtime dev can decide in `FailedMigrationHandler::failed` how to handle this. One follow-up would be to pair this with the `SafeMode` pallet and enact safe mode when an upgrade fails, to allow governance to rescue the chain. This is currently not possible, since governance is not `Mandatory`. ## Runtime API - `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to inform the Block Author whether they can push transactions. ### Integration Add it to your runtime implementation of `Core` and `BlockBuilder`: ```patch diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs @@ impl_runtime_apis! { impl sp_block_builder::Core<Block> for Runtime { - fn initialize_block(header: &<Block as BlockT>::Header) { + fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode { Executive::initialize_block(header) } ... } ``` # 2.) `poll` hook A new pallet hook is introduced: `poll`. `Poll` is intended to replace mostly all usage of `on_initialize`. The reason for this is that any code that can be called from `on_initialize` cannot be migrated through an MBM. Currently there is no way to statically check this; the implication is to use `on_initialize` as rarely as possible. Failing to do so can result in broken storage invariants. The implementation of the poll hook depends on the `Runtime API` changes that are explained above. # 3.) Hard-Deadline callbacks Three new callbacks are introduced and configured on `System::Config`: `PreInherents`, `PostInherents` and `PostTransactions`. These hooks are meant as replacement for `on_initialize` and `on_finalize` in cases where the code that runs cannot be moved to `poll`. The reason for this is to make the usage of HD-code (hard deadline) more explicit - again to prevent broken invariants by MBMs. # 4.) FRAME (general changes) ## `frame_system` pallet A new memorize storage item `InherentsApplied` is added. It is used by executive to track whether inherents have already been applied. Executive and can then execute the MBMs directly between inherents and transactions. The `Config` gets five new items: - `SingleBlockMigrations` this is the new way of configuring migrations that run in a single block. Previously they were defined as last generic argument of `Executive`. This shift is brings all central configuration about migrations closer into view of the developer (migrations that are configured in `Executive` will still work for now but is deprecated). - `MultiBlockMigrator` this can be configured to an engine that drives MBMs. One example would be the `pallet_migrations`. Note that this is only the engine; the exact MBMs are injected into the engine. - `PreInherents` a callback that executes after `on_initialize` but before inherents. - `PostInherents` a callback that executes after all inherents ran (including MBMs and `poll`). - `PostTransactions` in symmetry to `PreInherents`, this one is called before `on_finalize` but after all transactions. A sane default is to set all of these to `()`. Example diff suitable for any chain: ```patch @@ impl frame_system::Config for Test { type MaxConsumers = ConstU32<16>; + type SingleBlockMigrations = (); + type MultiBlockMigrator = (); + type PreInherents = (); + type PostInherents = (); + type PostTransactions = (); } ``` An overview of how the block execution now looks like is here. The same graph is also in the rust doc. <details><summary>Block Execution Flow</summary> <p>  </p> </details> ## Inherent Order Moved to https://github.com/paritytech/polkadot-sdk/pull/2154 --------------- ## TODO - [ ] Check that `try-runtime` still works - [ ] Ensure backwards compatibility with old Runtime APIs - [x] Consume weight correctly - [x] Cleanup --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: command-bot <> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>
Parity Bridges Common
This is a collection of components for building bridges.
These components include Substrate pallets for syncing headers, passing arbitrary messages, as well as libraries for building relayers to provide cross-chain communication capabilities.
Three bridge nodes are also available. The nodes can be used to run test networks which bridge other Substrate chains.
🚧 The bridges are currently under construction - a hardhat is recommended beyond this point 🚧
Contents
- Installation
- High-Level Architecture
- Project Layout
- Running the Bridge
- How to send a message
- Community
Installation
To get up and running you need both stable and nightly Rust. Rust nightly is used to build the Web Assembly (WASM) runtime for the node. You can configure the WASM support as so:
rustup install nightly
rustup target add wasm32-unknown-unknown --toolchain nightly
Once this is configured you can build and test the repo as follows:
git clone https://github.com/paritytech/parity-bridges-common.git
cd parity-bridges-common
cargo build --all
cargo test --all
Also you can build the repo with Parity CI Docker image:
docker pull paritytech/bridges-ci:production
mkdir ~/cache
chown 1000:1000 ~/cache #processes in the container runs as "nonroot" user with UID 1000
docker run --rm -it -w /shellhere/parity-bridges-common \
-v /home/$(whoami)/cache/:/cache/ \
-v "$(pwd)":/shellhere/parity-bridges-common \
-e CARGO_HOME=/cache/cargo/ \
-e SCCACHE_DIR=/cache/sccache/ \
-e CARGO_TARGET_DIR=/cache/target/ paritytech/bridges-ci:production cargo build --all
#artifacts can be found in ~/cache/target
If you want to reproduce other steps of CI process you can use the following guide.
If you need more information about setting up your development environment Substrate's Installation page is a good resource.
High-Level Architecture
This repo has support for bridging foreign chains together using a combination of Substrate pallets and external processes called relayers. A bridge chain is one that is able to follow the consensus of a foreign chain independently. For example, consider the case below where we want to bridge two Substrate based chains.
+---------------+ +---------------+
| | | |
| Rococo | | Westend |
| | | |
+-------+-------+ +-------+-------+
^ ^
| +---------------+ |
| | | |
+-----> | Bridge Relay | <-------+
| |
+---------------+
The Rococo chain must be able to accept Westend headers and verify their integrity. It does this by using a runtime module designed to track GRANDPA finality. Since two blockchains can't interact directly they need an external service, called a relayer, to communicate. The relayer will subscribe to new Rococo headers via RPC and submit them to the Westend chain for verification.
Take a look at Bridge High Level Documentation for more in-depth description of the bridge interaction.
Project Layout
Here's an overview of how the project is laid out. The main bits are the bin, which is the actual "blockchain", the
modules which are used to build the blockchain's logic (a.k.a the runtime) and the relays which are used to pass
messages between chains.
├── modules // Substrate Runtime Modules (a.k.a Pallets)
│ ├── beefy // On-Chain BEEFY Light Client (in progress)
│ ├── grandpa // On-Chain GRANDPA Light Client
│ ├── messages // Cross Chain Message Passing
│ ├── parachains // On-Chain Parachains Light Client
│ ├── relayers // Relayer Rewards Registry
│ ├── xcm-bridge-hub // Multiple Dynamic Bridges Support
│ ├── xcm-bridge-hub-router // XCM Router that may be used to Connect to XCM Bridge Hub
├── primitives // Code shared between modules, runtimes, and relays
│ └── ...
├── relays // Application for sending finality proofs and messages between chains
│ └── ...
└── scripts // Useful development and maintenance scripts
Running the Bridge
Apart from live Rococo <> Westend bridge, you may spin up local networks and test see how it works locally. More details may be found in this document.