mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-05-31 05:11:02 +00:00
Use Message Queue as DMP and XCMP dispatch queue (#1246)
(imported from https://github.com/paritytech/cumulus/pull/2157) ## Changes This MR refactores the XCMP, Parachains System and DMP pallets to use the [MessageQueue](https://github.com/paritytech/substrate/pull/12485) for delayed execution of incoming messages. The DMP pallet is entirely replaced by the MQ and thereby removed. This allows for PoV-bounded execution and resolves a number of issues that stem from the current work-around. All System Parachains adopt this change. The most important changes are in `primitives/core/src/lib.rs`, `parachains/common/src/process_xcm_message.rs`, `pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs` and the runtime configs. ### DMP Queue Pallet The pallet got removed and its logic refactored into parachain-system. Overweight message management can be done directly through the MQ pallet. Final undeployment migrations are provided by `cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that can be configured with an aux config trait like: ```rust parameter_types! { pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME; pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent; } impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime { type PalletName = DmpQueuePalletName; type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>; type DbWeight = <Runtime as frame_system::Config>::DbWeight; } // And adding them to your Migrations tuple: pub type Migrations = ( ... cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>, cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>, ); ``` ### XCMP Queue pallet Removed all dispatch queue functionality. Incoming XCMP messages are now either: Immediately handled if they are Signals, enqueued into the MQ pallet otherwise. New config items for the XCMP queue pallet: ```rust /// The actual queue implementation that retains the messages for later processing. type XcmpQueue: EnqueueMessage<ParaId>; /// How a XCM over HRMP from a sibling parachain should be processed. type XcmpProcessor: ProcessMessage<Origin = ParaId>; /// The maximal number of suspended XCMP channels at the same time. #[pallet::constant] type MaxInboundSuspended: Get<u32>; ``` How to configure those: ```rust // Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since // the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s. type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>; // Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched // with origin `Junction::Sibling(…)`. type XcmpProcessor = ProcessFromSibling< ProcessXcmMessage< AggregateMessageOrigin, xcm_executor::XcmExecutor<xcm_config::XcmConfig>, RuntimeCall, >, >; // Not really important what to choose here. Just something larger than the maximal number of channels. type MaxInboundSuspended = sp_core::ConstU32<1_000>; ``` The `InboundXcmpStatus` storage item was replaced by `InboundXcmpSuspended` since it now only tracks inbound queue suspension and no message indices anymore. Now only sends the most recent channel `Signals`, as all prio ones are out-dated anyway. ### Parachain System pallet For `DMP` messages instead of forwarding them to the `DMP` pallet, it now pushes them to the configured `DmpQueue`. The message processing which was triggered in `set_validation_data` is now being done by the MQ pallet `on_initialize`. XCMP messages are still handed off to the `XcmpMessageHandler` (XCMP-Queue pallet) - no change here. New config items for the parachain system pallet: ```rust /// Queues inbound downward messages for delayed processing. /// /// Analogous to the `XcmpQueue` of the XCMP queue pallet. type DmpQueue: EnqueueMessage<AggregateMessageOrigin>; ``` How to configure: ```rust /// Use the MQ pallet to store DMP messages for delayed processing. type DmpQueue = MessageQueue; ``` ## Message Flow The flow of messages on the parachain side. Messages come in from the left via the `Validation Data` and finally end up at the `Xcm Executor` on the right.  ## Further changes - Bumped the default suspension, drop and resume thresholds in `QueueConfigData::default()`. - `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when they would be a noop. - Properly validate the `QueueConfigData` before setting it. - Marked weight files as auto-generated so they wont auto-expand in the MR files view. - Move the `hypothetical` asserts to `frame_support` under the name `experimental_hypothetically` Questions: - [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in the runtimes? Not sure how to best fix. Just having them like this makes tests fail that rely on the real message processor when the feature is enabled. - [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler already takes 80% so I put it to 10% but that is quite low. TODO: - [x] Remove c&p code after https://github.com/paritytech/polkadot/pull/6271 - [x] Use `HandleMessage` once it is public in Substrate - [x] fix `runtime-benchmarks` feature https://github.com/paritytech/polkadot/pull/6966 - [x] Benchmarks - [x] Tests - [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended` - [x] Possibly cleanup Migrations (DMP+XCMP) - [x] optional: create `TransformProcessMessageOrigin` in Substrate and replace `ProcessFromSibling` - [ ] Rerun weights on ref HW --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: command-bot <>
This commit is contained in:
Oliver Tale-Yazdi
committed by
GitHub
GitHub
parent
7df0417bcd
commit
e1c033ebe1
@@ -30,16 +30,17 @@
|
||||
use codec::{Decode, Encode, MaxEncodedLen};
|
||||
use cumulus_primitives_core::{
|
||||
relay_chain, AbridgedHostConfiguration, ChannelInfo, ChannelStatus, CollationInfo,
|
||||
DmpMessageHandler, GetChannelInfo, InboundDownwardMessage, InboundHrmpMessage,
|
||||
MessageSendError, OutboundHrmpMessage, ParaId, PersistedValidationData, UpwardMessage,
|
||||
UpwardMessageSender, XcmpMessageHandler, XcmpMessageSource,
|
||||
GetChannelInfo, InboundDownwardMessage, InboundHrmpMessage, MessageSendError,
|
||||
OutboundHrmpMessage, ParaId, PersistedValidationData, UpwardMessage, UpwardMessageSender,
|
||||
XcmpMessageHandler, XcmpMessageSource,
|
||||
};
|
||||
use cumulus_primitives_parachain_inherent::{MessageQueueChain, ParachainInherentData};
|
||||
use frame_support::{
|
||||
defensive,
|
||||
dispatch::{DispatchResult, Pays, PostDispatchInfo},
|
||||
ensure,
|
||||
inherent::{InherentData, InherentIdentifier, ProvideInherent},
|
||||
traits::Get,
|
||||
traits::{Get, HandleMessage},
|
||||
weights::Weight,
|
||||
};
|
||||
use frame_system::{ensure_none, ensure_root, pallet_prelude::HeaderFor};
|
||||
@@ -52,15 +53,20 @@ use sp_runtime::{
|
||||
InvalidTransaction, TransactionLongevity, TransactionSource, TransactionValidity,
|
||||
ValidTransaction,
|
||||
},
|
||||
DispatchError, FixedU128, RuntimeDebug, Saturating,
|
||||
BoundedSlice, DispatchError, FixedU128, RuntimeDebug, Saturating,
|
||||
};
|
||||
use sp_std::{cmp, collections::btree_map::BTreeMap, prelude::*};
|
||||
use xcm::latest::XcmHash;
|
||||
|
||||
mod benchmarking;
|
||||
pub mod migration;
|
||||
|
||||
mod mock;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
pub mod weights;
|
||||
|
||||
pub use weights::WeightInfo;
|
||||
|
||||
mod unincluded_segment;
|
||||
|
||||
pub mod consensus_hook;
|
||||
@@ -177,6 +183,9 @@ where
|
||||
check_version: bool,
|
||||
}
|
||||
|
||||
/// The max length of a DMP message.
|
||||
pub type MaxDmpMessageLenOf<T> = <<T as Config>::DmpQueue as HandleMessage>::MaxMessageLen;
|
||||
|
||||
pub mod ump_constants {
|
||||
use super::FixedU128;
|
||||
|
||||
@@ -216,17 +225,18 @@ pub mod pallet {
|
||||
/// The place where outbound XCMP messages come from. This is queried in `finalize_block`.
|
||||
type OutboundXcmpMessageSource: XcmpMessageSource;
|
||||
|
||||
/// The message handler that will be invoked when messages are received via DMP.
|
||||
type DmpMessageHandler: DmpMessageHandler;
|
||||
/// Queues inbound downward messages for delayed processing.
|
||||
///
|
||||
/// All inbound DMP messages from the relay are pushed into this. The handler is expected to
|
||||
/// eventually process all the messages that are pushed to it.
|
||||
type DmpQueue: HandleMessage;
|
||||
|
||||
/// The weight we reserve at the beginning of the block for processing DMP messages.
|
||||
type ReservedDmpWeight: Get<Weight>;
|
||||
|
||||
/// The message handler that will be invoked when messages are received via XCMP.
|
||||
///
|
||||
/// The messages are dispatched in the order they were relayed by the relay chain. If
|
||||
/// multiple messages were relayed at one block, these will be dispatched in ascending
|
||||
/// order of the sender's para ID.
|
||||
/// This should normally link to the XCMP Queue pallet.
|
||||
type XcmpMessageHandler: XcmpMessageHandler;
|
||||
|
||||
/// The weight we reserve at the beginning of the block for processing XCMP messages.
|
||||
@@ -235,6 +245,9 @@ pub mod pallet {
|
||||
/// Something that can check the associated relay parent block number.
|
||||
type CheckAssociatedRelayNumber: CheckAssociatedRelayNumber;
|
||||
|
||||
/// Weight info for functions and calls.
|
||||
type WeightInfo: WeightInfo;
|
||||
|
||||
/// An entry-point for higher-level logic to manage the backlog of unincluded parachain
|
||||
/// blocks and authorship rights for those blocks.
|
||||
///
|
||||
@@ -631,15 +644,15 @@ pub mod pallet {
|
||||
|
||||
<T::OnSystemEvent as OnSystemEvent>::on_validation_data(&vfp);
|
||||
|
||||
total_weight += Self::process_inbound_downward_messages(
|
||||
total_weight.saturating_accrue(Self::enqueue_inbound_downward_messages(
|
||||
relevant_messaging_state.dmq_mqc_head,
|
||||
downward_messages,
|
||||
);
|
||||
total_weight += Self::process_inbound_horizontal_messages(
|
||||
));
|
||||
total_weight.saturating_accrue(Self::enqueue_inbound_horizontal_messages(
|
||||
&relevant_messaging_state.ingress_channels,
|
||||
horizontal_messages,
|
||||
vfp.relay_parent_number,
|
||||
);
|
||||
));
|
||||
|
||||
Ok(PostDispatchInfo { actual_weight: Some(total_weight), pays_fee: Pays::No })
|
||||
}
|
||||
@@ -1130,7 +1143,7 @@ impl<T: Config> Pallet<T> {
|
||||
// inherent.
|
||||
}
|
||||
|
||||
/// Process all inbound downward messages relayed by the collator.
|
||||
/// Enqueue all inbound downward messages relayed by the collator into the MQ pallet.
|
||||
///
|
||||
/// Checks if the sequence of the messages is valid, dispatches them and communicates the
|
||||
/// number of processed messages to the collator via a storage update.
|
||||
@@ -1139,26 +1152,33 @@ impl<T: Config> Pallet<T> {
|
||||
///
|
||||
/// If it turns out that after processing all messages the Message Queue Chain
|
||||
/// hash doesn't match the expected.
|
||||
fn process_inbound_downward_messages(
|
||||
fn enqueue_inbound_downward_messages(
|
||||
expected_dmq_mqc_head: relay_chain::Hash,
|
||||
downward_messages: Vec<InboundDownwardMessage>,
|
||||
) -> Weight {
|
||||
let dm_count = downward_messages.len() as u32;
|
||||
let mut dmq_head = <LastDmqMqcHead<T>>::get();
|
||||
|
||||
let mut weight_used = Weight::zero();
|
||||
let weight_used = T::WeightInfo::enqueue_inbound_downward_messages(dm_count);
|
||||
if dm_count != 0 {
|
||||
Self::deposit_event(Event::DownwardMessagesReceived { count: dm_count });
|
||||
let max_weight =
|
||||
<ReservedDmpWeightOverride<T>>::get().unwrap_or_else(T::ReservedDmpWeight::get);
|
||||
|
||||
let message_iter = downward_messages
|
||||
.into_iter()
|
||||
.inspect(|m| {
|
||||
dmq_head.extend_downward(m);
|
||||
})
|
||||
.map(|m| (m.sent_at, m.msg));
|
||||
weight_used += T::DmpMessageHandler::handle_dmp_messages(message_iter, max_weight);
|
||||
// Eagerly update the MQC head hash:
|
||||
for m in &downward_messages {
|
||||
dmq_head.extend_downward(m);
|
||||
}
|
||||
let bounded = downward_messages
|
||||
.iter()
|
||||
// Note: we are not using `.defensive()` here since that prints the whole value to
|
||||
// console. In case that the message is too long, this clogs up the log quite badly.
|
||||
.filter_map(|m| match BoundedSlice::try_from(&m.msg[..]) {
|
||||
Ok(bounded) => Some(bounded),
|
||||
Err(_) => {
|
||||
defensive!("Inbound Downward message was too long; dropping");
|
||||
None
|
||||
},
|
||||
});
|
||||
T::DmpQueue::handle_messages(bounded);
|
||||
<LastDmqMqcHead<T>>::put(&dmq_head);
|
||||
|
||||
Self::deposit_event(Event::DownwardMessagesProcessed {
|
||||
@@ -1181,14 +1201,15 @@ impl<T: Config> Pallet<T> {
|
||||
|
||||
/// Process all inbound horizontal messages relayed by the collator.
|
||||
///
|
||||
/// This is similar to `Pallet::process_inbound_downward_messages`, but works on multiple
|
||||
/// inbound channels.
|
||||
/// This is similar to [`enqueue_inbound_downward_messages`], but works with multiple inbound
|
||||
/// channels. It immediately dispatches signals and queues all other XCMs. Blob messages are
|
||||
/// ignored.
|
||||
///
|
||||
/// **Panics** if either any of horizontal messages submitted by the collator was sent from
|
||||
/// a para which has no open channel to this parachain or if after processing
|
||||
/// messages across all inbound channels MQCs were obtained which do not
|
||||
/// correspond to the ones found on the relay-chain.
|
||||
fn process_inbound_horizontal_messages(
|
||||
fn enqueue_inbound_horizontal_messages(
|
||||
ingress_channels: &[(ParaId, cumulus_primitives_core::AbridgedHrmpChannel)],
|
||||
horizontal_messages: BTreeMap<ParaId, Vec<InboundHrmpMessage>>,
|
||||
relay_parent_number: relay_chain::BlockNumber,
|
||||
|
||||
Reference in New Issue
Block a user