Use Message Queue pallet for UMP dispatch (#6271)

* Add ProcessXcmMessage struct

* Migrate away from weights in host config

* New well-known key to report UMPQ capacity

* Add missing file

* Fixes

* Remove original UMP files

* Docs

* Update runtime/parachains/src/inclusion/mod.rs

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* Add benchmarking

* Benchmarks

* Mock example of using the QueueChangeHandler to update the WKK

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Merge remote-tracking branch 'origin/master' into gav-message-queue

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update remove-weight migration

The migration got touched on master; just resolving conflicts here.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add message- to dispatch-origin conversion for XCM processing

Just using the `impl Into<MultiLocation>` was a bit inflexible.
Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* DNM: Temporarily comment code since XCMv3 is not merged yet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use u64 for queue-wide limits on UmpAcceptanceCheckErr

Using u32 here was one audit finding for the queue pallet.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Define one sub-queue per *MP queue

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Harden check_upward_messages

Using safe math and casts.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add type-safe well_known_keys

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add message-queue weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Deploy MessageQueue to Polkadot

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Migrate to parachain config V5

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update UMP tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert messed up merge 🤦

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update remove-weight migration

The migration got touched on master; just resolving conflicts here.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add message- to dispatch-origin conversion for XCM processing

Just using the `impl Into<MultiLocation>` was a bit inflexible.
Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* DNM: Temporarily comment code since XCMv3 is not merged yet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use u64 for queue-wide limits on UmpAcceptanceCheckErr

Using u32 here was one audit finding for the queue pallet.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Define one sub-queue per *MP queue

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Harden check_upward_messages

Using safe math and casts.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add type-safe well_known_keys

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add message-queue weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Deploy MessageQueue to Polkadot

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Migrate to parachain config V5

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update UMP tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make stuff compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Move DMP and HRMP messages to the MessageQueue

It currently does not compile in the CIbecause of some local
tweaks to Substrate.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Diener for CI

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* diener update cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert wrong changes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "DNM: Temporarily comment code since XCMv3 is not merged yet"

This reverts commit 820aa235cb21dd1d2621843607f7682bf035434e.

* Make compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup runtimes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Define benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Lockfile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add AggregateMessageOrigin

This enum currently only holds one value, but having it will make
it easier in the future to extend.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Forbid UMP for off-boarding paras

- Reject candidates with UMP messages for off-boarding paras
- Forbid scheduling off-boarding when a para has unprocess UMPs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Delete stupid test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use BoundedVec for upward messages

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add weights and fix MessageProcessor

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bound receive_upward_messages and check bound in configuration pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bound Debug impl

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test runtime

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix xcm-simulator

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Properly fix xcm-simulator and fuzzer

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* cargo update -p sp-io

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Adapt to upstream Substrate changes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix ProcesseMessage impls

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Some tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* cargo update -p sp-io

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use new MQ API

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add UMP while Para offboarding tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use Mocked message processor for benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use variables for constants

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add MQ pallet weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use MQ pallet weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Configure QueueChangeHandler

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add config test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix MQ serive weight

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup outgoing UMP dispatch queues

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use Master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Weight mul is not const

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove merge marker

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update runtime/parachains/src/inclusion/mod.rs

Co-authored-by: Gavin Wood <gavin@parity.io>

* Update runtime/kusama/src/lib.rs

Co-authored-by: Gavin Wood <gavin@parity.io>

* Use lowercase UMP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clarify comment

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use Weight::from_parts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix doc

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Emit event after the fact

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add defensive_proof to receive_upward_messages

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Reapply "Remove original UMP files"

Looks like they came back from the dead. Re-apply commit cf6d316f0

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove old files

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rename MaxUmpMessageLen -> MaxUmpMessageLenOf

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test defensive message dropping of receive_upward_messages

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup imports

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update implementors guide

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove FAIL-CI mark

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Delete unused code

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add another test for MQ change hook

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Imports

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Keep Kusama runtime formatting

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Delete unused code"

This reverts commit dd76bca5025b7e1ef846a9539c3607eed185f16a.

* Feature gate mock functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Review: Use saturating_add

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test RelayDispatchQueueSize storage key

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Move migration to own file to avoid merge conflicts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Migration in own file

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rococo: configure MQ pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Rococo: configure MQ pallet"

Going to do this as follow up, since it needs Substrate changes
and i dont want to stall this MR any longer.

This reverts commit b9c15e8a8339c4e877d654ee3f09903af4210736.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Fixup tests"

This reverts commit 88f1cbe20774d20e5e1e554e798960ae39437af1.

* Fixup migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix CI

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix other migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bump MAX_CODE_SIZE to 10MiB

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add ForceUpdateUmpLimits migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* imports

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use defensive instead of defensive_proof

'defensive_proof' also prints the 'self', which spams the console
too much when running the tests. Just the length is enough.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rename to ScheduleConfigUpdate

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup migration checks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add MAX_CODE_SIZE to ScheduleConfigUpdate

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Set MAX_CODE_SIZE to 4MiB

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix benchmark

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix formatting

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Add MAX_CODE_SIZE to ScheduleConfigUpdate"

This reverts commit 7caffb09e83083b57affd548215e45b25c3d64dc.

* Revert "Set MAX_CODE_SIZE to 4MiB"

This reverts commit 103ffbaf686487d2fbe0082a16826af17cacc1a1.

* Revert "Bump MAX_CODE_SIZE to 10MiB"

This reverts commit 530734b7b0da5b7680054e0242348fcc79a666fe.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove consistency check from migration

Re-addig these checks is blocked on https://github.com/paritytech/polkadot/issues/7108

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix constants

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bump MAX_UPWARD_MESSAGE_SIZE_BOUND for Westend

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix migrations

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use old nightly for fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixes

* cargo fmt

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixes

* Add MQ pallet to fuzzer

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix XMC simulator example

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove runtime-benchmarks from fuzzers

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Remove runtime-benchmarks from fuzzers"

This reverts commit e1f2bb01b6dea2dd465539d3658719895b58b557.

* Fix example simulator

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add V6 migration and remove old ones

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Actually make old migrations reusable

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Dont delete old migrations

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Future proof AggregateMessageOrigin and review fixes

There are indications that Loopback and Bridged will be needed soon.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix fuzzer build

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Review

Co-authored-by: muharem <ismailov.m.h@gmail.com>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove old migration

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Set MQ service weight to 20%

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tabs in Markdown

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
This commit is contained in:
Gavin Wood
2023-05-19 18:14:13 +02:00
committed by GitHub
parent 9e4bca6895
commit 400864c352
74 changed files with 4034 additions and 2536 deletions
+341 -53
View File
@@ -21,20 +21,32 @@
//! to included.
use crate::{
configuration, disputes, dmp, hrmp, paras, paras_inherent::DisputedBitfield,
scheduler::CoreAssignment, shared, ump,
configuration::{self, HostConfiguration},
disputes, dmp, hrmp, paras,
paras_inherent::DisputedBitfield,
scheduler::CoreAssignment,
shared,
};
use bitvec::{order::Lsb0 as BitOrderLsb0, vec::BitVec};
use frame_support::pallet_prelude::*;
use frame_support::{
defensive,
pallet_prelude::*,
traits::{Defensive, EnqueueMessage},
BoundedSlice,
};
use pallet_message_queue::OnQueueChanged;
use parity_scale_codec::{Decode, Encode};
use primitives::{
supermajority_threshold, AvailabilityBitfield, BackedCandidate, CandidateCommitments,
CandidateDescriptor, CandidateHash, CandidateReceipt, CommittedCandidateReceipt, CoreIndex,
GroupIndex, Hash, HeadData, Id as ParaId, SigningContext, UncheckedSignedAvailabilityBitfields,
ValidatorId, ValidatorIndex, ValidityAttestation,
supermajority_threshold, well_known_keys, AvailabilityBitfield, BackedCandidate,
CandidateCommitments, CandidateDescriptor, CandidateHash, CandidateReceipt,
CommittedCandidateReceipt, CoreIndex, GroupIndex, Hash, HeadData, Id as ParaId, SigningContext,
UncheckedSignedAvailabilityBitfields, UpwardMessage, ValidatorId, ValidatorIndex,
ValidityAttestation,
};
use scale_info::TypeInfo;
use sp_runtime::{traits::One, DispatchError};
use sp_runtime::{traits::One, DispatchError, SaturatedConversion, Saturating};
#[cfg(feature = "std")]
use sp_std::fmt;
use sp_std::{collections::btree_set::BTreeSet, prelude::*};
pub use pallet::*;
@@ -42,6 +54,32 @@ pub use pallet::*;
#[cfg(test)]
pub(crate) mod tests;
#[cfg(feature = "runtime-benchmarks")]
mod benchmarking;
pub trait WeightInfo {
fn receive_upward_messages(i: u32) -> Weight;
}
pub struct TestWeightInfo;
impl WeightInfo for TestWeightInfo {
fn receive_upward_messages(_: u32) -> Weight {
Weight::MAX
}
}
impl WeightInfo for () {
fn receive_upward_messages(_: u32) -> Weight {
Weight::zero()
}
}
/// Maximum value that `config.max_upward_message_size` can be set to.
///
/// This is used for benchmarking sanely bounding relevant storage items. It is expected from the `configuration`
/// pallet to check these values before setting.
pub const MAX_UPWARD_MESSAGE_SIZE_BOUND: u32 = 128 * 1024;
/// A bitfield signed by a validator indicating that it is keeping its piece of the erasure-coding
/// for any backed candidates referred to by a `1` bit available.
///
@@ -179,6 +217,55 @@ pub fn minimum_backing_votes(n_validators: usize) -> usize {
sp_std::cmp::min(n_validators, 2)
}
/// Reads the footprint of queues for a specific origin type.
pub trait QueueFootprinter {
type Origin;
fn message_count(origin: Self::Origin) -> u64;
}
impl QueueFootprinter for () {
type Origin = UmpQueueId;
fn message_count(_: Self::Origin) -> u64 {
0
}
}
/// Aggregate message origin for the `MessageQueue` pallet.
///
/// Can be extended to serve further use-cases besides just UMP. Is stored in storage, so any change
/// to existing values will require a migration.
#[derive(Encode, Decode, Clone, MaxEncodedLen, Eq, PartialEq, RuntimeDebug, TypeInfo)]
pub enum AggregateMessageOrigin {
/// Inbound upward message.
#[codec(index = 0)]
Ump(UmpQueueId),
}
/// Identifies a UMP queue inside the `MessageQueue` pallet.
///
/// It is written in verbose form since future variants like `Loopback` and `Bridged` are already
/// forseeable.
#[derive(Encode, Decode, Clone, MaxEncodedLen, Eq, PartialEq, RuntimeDebug, TypeInfo)]
pub enum UmpQueueId {
/// The message originated from this parachain.
#[codec(index = 0)]
Para(ParaId),
}
#[cfg(feature = "runtime-benchmarks")]
impl From<u32> for AggregateMessageOrigin {
fn from(n: u32) -> Self {
// Some dummy for the benchmarks.
Self::Ump(UmpQueueId::Para(n.into()))
}
}
/// The maximal length of a UMP message.
pub type MaxUmpMessageLenOf<T> =
<<T as Config>::MessageQueue as EnqueueMessage<AggregateMessageOrigin>>::MaxMessageLen;
#[frame_support::pallet]
pub mod pallet {
use super::*;
@@ -193,13 +280,22 @@ pub mod pallet {
+ shared::Config
+ paras::Config
+ dmp::Config
+ ump::Config
+ hrmp::Config
+ configuration::Config
{
type RuntimeEvent: From<Event<Self>> + IsType<<Self as frame_system::Config>::RuntimeEvent>;
type DisputesHandler: disputes::DisputesHandler<Self::BlockNumber>;
type RewardValidators: RewardValidators;
/// The system message queue.
///
/// The message queue provides general queueing and processing functionality. Currently it
/// replaces the old `UMP` dispatch queue. Other use-cases can be implemented as well by
/// adding new variants to `AggregateMessageOrigin`.
type MessageQueue: EnqueueMessage<AggregateMessageOrigin>;
/// Weight info for the calls of this pallet.
type WeightInfo: WeightInfo;
}
#[pallet::event]
@@ -211,6 +307,8 @@ pub mod pallet {
CandidateIncluded(CandidateReceipt<T::Hash>, HeadData, CoreIndex, GroupIndex),
/// A candidate timed out. `[candidate, head_data]`
CandidateTimedOut(CandidateReceipt<T::Hash>, HeadData, CoreIndex),
/// Some upward messages have been received and will be processed.
UpwardMessagesReceived { from: ParaId, count: u32 },
}
#[pallet::error]
@@ -299,6 +397,71 @@ pub mod pallet {
const LOG_TARGET: &str = "runtime::inclusion";
/// The reason that a candidate's outputs were rejected for.
#[derive(derive_more::From)]
#[cfg_attr(feature = "std", derive(Debug))]
enum AcceptanceCheckErr<BlockNumber> {
HeadDataTooLarge,
/// Code upgrades are not permitted at the current time.
PrematureCodeUpgrade,
/// The new runtime blob is too large.
NewCodeTooLarge,
/// The candidate violated this DMP acceptance criteria.
ProcessedDownwardMessages(dmp::ProcessedDownwardMessagesAcceptanceErr),
/// The candidate violated this UMP acceptance criteria.
UpwardMessages(UmpAcceptanceCheckErr),
/// The candidate violated this HRMP watermark acceptance criteria.
HrmpWatermark(hrmp::HrmpWatermarkAcceptanceErr<BlockNumber>),
/// The candidate violated this outbound HRMP acceptance criteria.
OutboundHrmp(hrmp::OutboundHrmpAcceptanceErr),
}
/// An error returned by [`check_upward_messages`] that indicates a violation of one of acceptance
/// criteria rules.
#[cfg_attr(test, derive(PartialEq))]
pub enum UmpAcceptanceCheckErr {
/// The maximal number of messages that can be submitted in one batch was exceeded.
MoreMessagesThanPermitted { sent: u32, permitted: u32 },
/// The maximal size of a single message was exceeded.
MessageSize { idx: u32, msg_size: u32, max_size: u32 },
/// The allowed number of messages in the queue was exceeded.
CapacityExceeded { count: u64, limit: u64 },
/// The allowed combined message size in the queue was exceeded.
TotalSizeExceeded { total_size: u64, limit: u64 },
/// A para-chain cannot send UMP messages while it is offboarding.
IsOffboarding,
}
#[cfg(feature = "std")]
impl fmt::Debug for UmpAcceptanceCheckErr {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
match *self {
UmpAcceptanceCheckErr::MoreMessagesThanPermitted { sent, permitted } => write!(
fmt,
"more upward messages than permitted by config ({} > {})",
sent, permitted,
),
UmpAcceptanceCheckErr::MessageSize { idx, msg_size, max_size } => write!(
fmt,
"upward message idx {} larger than permitted by config ({} > {})",
idx, msg_size, max_size,
),
UmpAcceptanceCheckErr::CapacityExceeded { count, limit } => write!(
fmt,
"the ump queue would have more items than permitted by config ({} > {})",
count, limit,
),
UmpAcceptanceCheckErr::TotalSizeExceeded { total_size, limit } => write!(
fmt,
"the ump queue would have grown past the max size permitted by config ({} > {})",
total_size, limit,
),
UmpAcceptanceCheckErr::IsOffboarding =>
write!(fmt, "upward message rejected because the para is off-boarding",),
}
}
}
impl<T: Config> Pallet<T> {
/// Block initialization logic, called by initializer.
pub(crate) fn initializer_initialize(_now: T::BlockNumber) -> Weight {
@@ -311,12 +474,25 @@ impl<T: Config> Pallet<T> {
/// Handle an incoming session change.
pub(crate) fn initializer_on_new_session(
_notification: &crate::initializer::SessionChangeNotification<T::BlockNumber>,
outgoing_paras: &[ParaId],
) {
// unlike most drain methods, drained elements are not cleared on `Drop` of the iterator
// and require consumption.
for _ in <PendingAvailabilityCommitments<T>>::drain() {}
for _ in <PendingAvailability<T>>::drain() {}
for _ in <AvailabilityBitfields<T>>::drain() {}
Self::cleanup_outgoing_ump_dispatch_queues(outgoing_paras);
}
pub(crate) fn cleanup_outgoing_ump_dispatch_queues(outgoing: &[ParaId]) {
for outgoing_para in outgoing {
Self::cleanup_outgoing_ump_dispatch_queue(*outgoing_para);
}
}
pub(crate) fn cleanup_outgoing_ump_dispatch_queue(para: ParaId) {
T::MessageQueue::sweep_queue(AggregateMessageOrigin::Ump(UmpQueueId::Para(para)));
}
/// Extract the freed cores based on cores that became available.
@@ -698,20 +874,22 @@ impl<T: Config> Pallet<T> {
let relay_parent_number = now;
let check_ctx = CandidateCheckContext::<T>::new(now, relay_parent_number);
if let Err(err) = check_ctx.check_validation_outputs(
para_id,
&validation_outputs.head_data,
&validation_outputs.new_validation_code,
validation_outputs.processed_downward_messages,
&validation_outputs.upward_messages,
T::BlockNumber::from(validation_outputs.hrmp_watermark),
&validation_outputs.horizontal_messages,
) {
if check_ctx
.check_validation_outputs(
para_id,
&validation_outputs.head_data,
&validation_outputs.new_validation_code,
validation_outputs.processed_downward_messages,
&validation_outputs.upward_messages,
T::BlockNumber::from(validation_outputs.hrmp_watermark),
&validation_outputs.horizontal_messages,
)
.is_err()
{
log::debug!(
target: LOG_TARGET,
"Validation outputs checking for parachain `{}` failed: {:?}",
"Validation outputs checking for parachain `{}` failed",
u32::from(para_id),
err,
);
false
} else {
@@ -750,31 +928,31 @@ impl<T: Config> Pallet<T> {
// initial weight is config read.
let mut weight = T::DbWeight::get().reads_writes(1, 0);
if let Some(new_code) = commitments.new_validation_code {
weight += <paras::Pallet<T>>::schedule_code_upgrade(
weight.saturating_add(<paras::Pallet<T>>::schedule_code_upgrade(
receipt.descriptor.para_id,
new_code,
relay_parent_number,
&config,
);
));
}
// enact the messaging facet of the candidate.
weight += <dmp::Pallet<T>>::prune_dmq(
weight.saturating_accrue(<dmp::Pallet<T>>::prune_dmq(
receipt.descriptor.para_id,
commitments.processed_downward_messages,
);
weight += <ump::Pallet<T>>::receive_upward_messages(
));
weight.saturating_accrue(Self::receive_upward_messages(
receipt.descriptor.para_id,
commitments.upward_messages,
);
weight += <hrmp::Pallet<T>>::prune_hrmp(
commitments.upward_messages.as_slice(),
));
weight.saturating_accrue(<hrmp::Pallet<T>>::prune_hrmp(
receipt.descriptor.para_id,
T::BlockNumber::from(commitments.hrmp_watermark),
);
weight += <hrmp::Pallet<T>>::queue_outbound_hrmp(
));
weight.saturating_accrue(<hrmp::Pallet<T>>::queue_outbound_hrmp(
receipt.descriptor.para_id,
commitments.horizontal_messages,
);
));
Self::deposit_event(Event::<T>::CandidateIncluded(
plain,
@@ -783,12 +961,105 @@ impl<T: Config> Pallet<T> {
backing_group,
));
weight +
<paras::Pallet<T>>::note_new_head(
receipt.descriptor.para_id,
commitments.head_data,
relay_parent_number,
)
weight.saturating_add(<paras::Pallet<T>>::note_new_head(
receipt.descriptor.para_id,
commitments.head_data,
relay_parent_number,
))
}
/// Check that all the upward messages sent by a candidate pass the acceptance criteria.
pub(crate) fn check_upward_messages(
config: &HostConfiguration<T::BlockNumber>,
para: ParaId,
upward_messages: &[UpwardMessage],
) -> Result<(), UmpAcceptanceCheckErr> {
// Cannot send UMP messages while off-boarding.
if <paras::Pallet<T>>::is_offboarding(para) {
ensure!(upward_messages.is_empty(), UmpAcceptanceCheckErr::IsOffboarding);
}
let additional_msgs = upward_messages.len();
if additional_msgs > config.max_upward_message_num_per_candidate as usize {
return Err(UmpAcceptanceCheckErr::MoreMessagesThanPermitted {
sent: additional_msgs as u32,
permitted: config.max_upward_message_num_per_candidate,
})
}
let fp = T::MessageQueue::footprint(AggregateMessageOrigin::Ump(UmpQueueId::Para(para)));
let (para_queue_count, mut para_queue_size) = (fp.count, fp.size);
if para_queue_count.saturating_add(additional_msgs as u64) >
config.max_upward_queue_count as u64
{
return Err(UmpAcceptanceCheckErr::CapacityExceeded {
count: para_queue_count.saturating_add(additional_msgs as u64),
limit: config.max_upward_queue_count as u64,
})
}
for (idx, msg) in upward_messages.into_iter().enumerate() {
let msg_size = msg.len();
if msg_size > config.max_upward_message_size as usize {
return Err(UmpAcceptanceCheckErr::MessageSize {
idx: idx as u32,
msg_size: msg_size as u32,
max_size: config.max_upward_message_size,
})
}
// make sure that the queue is not overfilled.
// we do it here only once since returning false invalidates the whole relay-chain block.
if para_queue_size.saturating_add(msg_size as u64) > config.max_upward_queue_size as u64
{
return Err(UmpAcceptanceCheckErr::TotalSizeExceeded {
total_size: para_queue_size.saturating_add(msg_size as u64),
limit: config.max_upward_queue_size as u64,
})
}
para_queue_size.saturating_accrue(msg_size as u64);
}
Ok(())
}
/// Enqueues `upward_messages` from a `para`'s accepted candidate block.
///
/// This function is infallible since the candidate was already accepted and we therefore need
/// to deal with the messages as given. Messages that are too long will be ignored since such
/// candidates should have already been rejected in [`Self::check_upward_messages`].
pub(crate) fn receive_upward_messages(para: ParaId, upward_messages: &[Vec<u8>]) -> Weight {
let bounded = upward_messages
.iter()
.filter_map(|d| {
BoundedSlice::try_from(&d[..])
.map_err(|e| {
defensive!("Accepted candidate contains too long msg, len=", d.len());
e
})
.ok()
})
.collect();
Self::receive_bounded_upward_messages(para, bounded)
}
/// Enqueues storage-bounded `upward_messages` from a `para`'s accepted candidate block.
pub(crate) fn receive_bounded_upward_messages(
para: ParaId,
messages: Vec<BoundedSlice<'_, u8, MaxUmpMessageLenOf<T>>>,
) -> Weight {
let count = messages.len() as u32;
if count == 0 {
return Weight::zero()
}
T::MessageQueue::enqueue_messages(
messages.into_iter(),
AggregateMessageOrigin::Ump(UmpQueueId::Para(para)),
);
let weight = <T as Config>::WeightInfo::receive_upward_messages(count);
Self::deposit_event(Event::UpwardMessagesReceived { from: para, count });
weight
}
/// Cleans up all paras pending availability that the predicate returns true for.
@@ -902,17 +1173,6 @@ const fn availability_threshold(n_validators: usize) -> usize {
supermajority_threshold(n_validators)
}
#[derive(derive_more::From, Debug)]
enum AcceptanceCheckErr<BlockNumber> {
HeadDataTooLarge,
PrematureCodeUpgrade,
NewCodeTooLarge,
ProcessedDownwardMessages(dmp::ProcessedDownwardMessagesAcceptanceErr),
UpwardMessages(ump::AcceptanceCheckErr),
HrmpWatermark(hrmp::HrmpWatermarkAcceptanceErr<BlockNumber>),
OutboundHrmp(hrmp::OutboundHrmpAcceptanceErr),
}
impl<BlockNumber> AcceptanceCheckErr<BlockNumber> {
/// Returns the same error so that it can be threaded through a needle of `DispatchError` and
/// ultimately returned from a `Dispatchable`.
@@ -930,6 +1190,26 @@ impl<BlockNumber> AcceptanceCheckErr<BlockNumber> {
}
}
impl<T: Config> OnQueueChanged<AggregateMessageOrigin> for Pallet<T> {
// Write back the remaining queue capacity into `relay_dispatch_queue_remaining_capacity`.
fn on_queue_changed(origin: AggregateMessageOrigin, count: u64, size: u64) {
let para = match origin {
AggregateMessageOrigin::Ump(UmpQueueId::Para(p)) => p,
};
// TODO maybe migrate this to u64
let (count, size) = (count.saturated_into(), size.saturated_into());
// TODO paritytech/polkadot#6283: Remove all usages of `relay_dispatch_queue_size`
#[allow(deprecated)]
well_known_keys::relay_dispatch_queue_size_typed(para).set((count, size));
let config = <configuration::Pallet<T>>::config();
let remaining_count = config.max_upward_queue_count.saturating_sub(count);
let remaining_size = config.max_upward_queue_size.saturating_sub(size);
well_known_keys::relay_dispatch_queue_remaining_capacity(para)
.set((remaining_count, remaining_size));
}
}
/// A collection of data required for checking a candidate.
pub(crate) struct CandidateCheckContext<T: Config> {
config: configuration::HostConfiguration<T::BlockNumber>,
@@ -965,12 +1245,13 @@ impl<T: Config> CandidateCheckContext<T> {
let relay_parent_number = now - One::one();
{
// this should never fail because the para is registered
let persisted_validation_data = match crate::util::make_persisted_validation_data::<T>(
para_id,
relay_parent_number,
parent_storage_root,
) {
)
.defensive_proof("the para is registered")
{
Some(l) => l,
None => return Ok(Err(FailedToCreatePVD)),
};
@@ -1018,10 +1299,9 @@ impl<T: Config> CandidateCheckContext<T> {
) {
log::debug!(
target: LOG_TARGET,
"Validation outputs checking during inclusion of a candidate {} for parachain `{}` failed: {:?}",
"Validation outputs checking during inclusion of a candidate {} for parachain `{}` failed",
candidate_idx,
u32::from(para_id),
err,
);
Err(err.strip_into_dispatch_err::<T>())?;
};
@@ -1059,10 +1339,18 @@ impl<T: Config> CandidateCheckContext<T> {
// check if the candidate passes the messaging acceptance criteria
<dmp::Pallet<T>>::check_processed_downward_messages(para_id, processed_downward_messages)?;
<ump::Pallet<T>>::check_upward_messages(&self.config, para_id, upward_messages)?;
Pallet::<T>::check_upward_messages(&self.config, para_id, upward_messages)?;
<hrmp::Pallet<T>>::check_hrmp_watermark(para_id, self.relay_parent_number, hrmp_watermark)?;
<hrmp::Pallet<T>>::check_outbound_hrmp(&self.config, para_id, horizontal_messages)?;
Ok(())
}
}
impl<T: Config> QueueFootprinter for Pallet<T> {
type Origin = UmpQueueId;
fn message_count(origin: Self::Origin) -> u64 {
T::MessageQueue::footprint(AggregateMessageOrigin::Ump(origin)).count
}
}