Oliver Tale-Yazdi
e1c033ebe1
(imported from https://github.com/paritytech/cumulus/pull/2157) ## Changes This MR refactores the XCMP, Parachains System and DMP pallets to use the [MessageQueue](https://github.com/paritytech/substrate/pull/12485) for delayed execution of incoming messages. The DMP pallet is entirely replaced by the MQ and thereby removed. This allows for PoV-bounded execution and resolves a number of issues that stem from the current work-around. All System Parachains adopt this change. The most important changes are in `primitives/core/src/lib.rs`, `parachains/common/src/process_xcm_message.rs`, `pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs` and the runtime configs. ### DMP Queue Pallet The pallet got removed and its logic refactored into parachain-system. Overweight message management can be done directly through the MQ pallet. Final undeployment migrations are provided by `cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that can be configured with an aux config trait like: ```rust parameter_types! { pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME; pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent; } impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime { type PalletName = DmpQueuePalletName; type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>; type DbWeight = <Runtime as frame_system::Config>::DbWeight; } // And adding them to your Migrations tuple: pub type Migrations = ( ... cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>, cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>, ); ``` ### XCMP Queue pallet Removed all dispatch queue functionality. Incoming XCMP messages are now either: Immediately handled if they are Signals, enqueued into the MQ pallet otherwise. New config items for the XCMP queue pallet: ```rust /// The actual queue implementation that retains the messages for later processing. type XcmpQueue: EnqueueMessage<ParaId>; /// How a XCM over HRMP from a sibling parachain should be processed. type XcmpProcessor: ProcessMessage<Origin = ParaId>; /// The maximal number of suspended XCMP channels at the same time. #[pallet::constant] type MaxInboundSuspended: Get<u32>; ``` How to configure those: ```rust // Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since // the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s. type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>; // Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched // with origin `Junction::Sibling(…)`. type XcmpProcessor = ProcessFromSibling< ProcessXcmMessage< AggregateMessageOrigin, xcm_executor::XcmExecutor<xcm_config::XcmConfig>, RuntimeCall, >, >; // Not really important what to choose here. Just something larger than the maximal number of channels. type MaxInboundSuspended = sp_core::ConstU32<1_000>; ``` The `InboundXcmpStatus` storage item was replaced by `InboundXcmpSuspended` since it now only tracks inbound queue suspension and no message indices anymore. Now only sends the most recent channel `Signals`, as all prio ones are out-dated anyway. ### Parachain System pallet For `DMP` messages instead of forwarding them to the `DMP` pallet, it now pushes them to the configured `DmpQueue`. The message processing which was triggered in `set_validation_data` is now being done by the MQ pallet `on_initialize`. XCMP messages are still handed off to the `XcmpMessageHandler` (XCMP-Queue pallet) - no change here. New config items for the parachain system pallet: ```rust /// Queues inbound downward messages for delayed processing. /// /// Analogous to the `XcmpQueue` of the XCMP queue pallet. type DmpQueue: EnqueueMessage<AggregateMessageOrigin>; ``` How to configure: ```rust /// Use the MQ pallet to store DMP messages for delayed processing. type DmpQueue = MessageQueue; ``` ## Message Flow The flow of messages on the parachain side. Messages come in from the left via the `Validation Data` and finally end up at the `Xcm Executor` on the right.  ## Further changes - Bumped the default suspension, drop and resume thresholds in `QueueConfigData::default()`. - `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when they would be a noop. - Properly validate the `QueueConfigData` before setting it. - Marked weight files as auto-generated so they wont auto-expand in the MR files view. - Move the `hypothetical` asserts to `frame_support` under the name `experimental_hypothetically` Questions: - [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in the runtimes? Not sure how to best fix. Just having them like this makes tests fail that rely on the real message processor when the feature is enabled. - [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler already takes 80% so I put it to 10% but that is quite low. TODO: - [x] Remove c&p code after https://github.com/paritytech/polkadot/pull/6271 - [x] Use `HandleMessage` once it is public in Substrate - [x] fix `runtime-benchmarks` feature https://github.com/paritytech/polkadot/pull/6966 - [x] Benchmarks - [x] Tests - [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended` - [x] Possibly cleanup Migrations (DMP+XCMP) - [x] optional: create `TransformProcessMessageOrigin` in Substrate and replace `ProcessFromSibling` - [ ] Rerun weights on ref HW --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: command-bot <>
Polkadot
Implementation of a https://polkadot.network node in Rust based on the Substrate framework.
The README provides information about installing the polkadot binary and developing on the codebase. For more specific
guides, like how to run a validator node, see the Polkadot Wiki.
Installation
Using a pre-compiled binary
If you just wish to run a Polkadot node without compiling it yourself, you may either run the latest binary from our releases page, or install Polkadot from one of our package repositories.
Debian-based (Debian, Ubuntu)
Currently supports Debian 10 (Buster) and Ubuntu 20.04 (Focal), and derivatives. Run the following
commands as the root user.
# Import the security@parity.io GPG key
gpg --recv-keys --keyserver hkps://keys.mailvelope.com 9D4B2B6EB8F97156D19669A9FF0812D491B96798
gpg --export 9D4B2B6EB8F97156D19669A9FF0812D491B96798 > /usr/share/keyrings/parity.gpg
# Add the Parity repository and update the package index
echo 'deb [signed-by=/usr/share/keyrings/parity.gpg] https://releases.parity.io/deb release main' > /etc/apt/sources.list.d/parity.list
apt update
# Install the `parity-keyring` package - This will ensure the GPG key
# used by APT remains up-to-date
apt install parity-keyring
# Install polkadot
apt install polkadot
Installation from the Debian repository will create a systemd service that can be used to run a
Polkadot node. This is disabled by default, and can be started by running systemctl start polkadot
on demand (use systemctl enable polkadot to make it auto-start after reboot). By default, it will
run as the polkadot user. Command-line flags passed to the binary can be customized by editing
/etc/default/polkadot. This file will not be overwritten on updating Polkadot. You may also just
run the node directly from the command-line.
Building
Since the Polkadot node is based on Substrate, first set up your build environment according to the Substrate installation instructions.
Install via Cargo
Make sure you have the support software installed from the Build from Source section below this section.
If you want to install Polkadot in your PATH, you can do so with:
cargo install --git https://github.com/paritytech/polkadot-sdk --tag <version> polkadot --locked
Build from Source
Build the client by cloning this repository and running the following commands from the root directory of the repo:
git checkout <latest tagged release>
./scripts/init.sh
cargo build --release
Note: if you want to move the built polkadot binary somewhere (e.g. into $PATH) you will also
need to move polkadot-execute-worker and polkadot-prepare-worker. You can let cargo do all this
for you by running:
cargo install --path . --locked
Build from Source with Docker
You can also build from source using Parity CI docker image:
git checkout <latest tagged release>
docker run --rm -it -w /shellhere/polkadot \
-v $(pwd):/shellhere/polkadot \
paritytech/ci-linux:production cargo build --release
sudo chown -R $(id -u):$(id -g) target/
If you want to reproduce other steps of CI process you can use the following guide.
Networks
This repo supports runtimes for Polkadot, Kusama, and Westend.
Connect to Polkadot Mainnet
Connect to the global Polkadot Mainnet network by running:
../target/release/polkadot --chain=polkadot
You can see your node on [telemetry] (set a custom name with --name "my custom name").
telemetry: https://telemetry.polkadot.io/#list/Polkadot
Connect to the "Kusama" Canary Network
Connect to the global Kusama canary network by running:
../target/release/polkadot --chain=kusama
You can see your node on [telemetry] (set a custom name with --name "my custom name").
telemetry: https://telemetry.polkadot.io/#list/Kusama
Connect to the Westend Testnet
Connect to the global Westend testnet by running:
../target/release/polkadot --chain=westend
You can see your node on [telemetry] (set a custom name with --name "my custom name").
telemetry: https://telemetry.polkadot.io/#list/Westend
Obtaining DOTs
If you want to do anything on Polkadot, Kusama, or Westend, then you'll need to get an account and some DOT, KSM, or WND tokens, respectively. Follow the instructions on the Wiki to obtain tokens for your testnet of choice.
Hacking on Polkadot
If you'd actually like to hack on Polkadot, you can grab the source code and build it. Ensure you have Rust and the support software installed.
Then, grab the Polkadot source code:
git clone https://github.com/paritytech/polkadot-sdk.git
cd polkadot
Then build the code. You will need to build in release mode (--release) to start a network. Only
use debug mode for development (faster compile times for development and testing).
cargo build
You can run the tests if you like:
cargo test --workspace --profile testnet
# Or run only the tests for specified crated
cargo test -p <crate-name> --profile testnet
You can start a development chain with:
cargo run --bin polkadot -- --dev
Detailed logs may be shown by running the node with the following environment variables set:
RUST_LOG=debug RUST_BACKTRACE=1 cargo run --bin polkadot -- --dev
Development
You can run a simple single-node development "network" on your machine by running:
cargo run --bin polkadot --release -- --dev
You can muck around by heading to https://polkadot.js.org/apps and choose "Local Node" from the Settings menu.
Local Two-node Testnet
If you want to see the multi-node consensus algorithm in action locally, then you can create a local testnet. You'll need two terminals open. In one, run:
polkadot --dev --alice -d /tmp/alice
And in the other, run:
polkadot --dev --bob -d /tmp/bob --bootnodes '/ip4/127.0.0.1/tcp/30333/p2p/ALICE_BOOTNODE_ID_HERE'
Ensure you replace ALICE_BOOTNODE_ID_HERE with the node ID from the output of the first terminal.
Monitoring
Once you set this up you can take a look at the Polkadot Grafana dashboards that we currently maintain.
Using Docker
Shell Completion
Contributing
Contributing Guidelines
Contributor Code of Conduct
License
Polkadot is GPL 3.0 licensed.