Use Message Queue as DMP and XCMP dispatch queue (#1246)

(imported from https://github.com/paritytech/cumulus/pull/2157)

## Changes

This MR refactores the XCMP, Parachains System and DMP pallets to use
the [MessageQueue](https://github.com/paritytech/substrate/pull/12485)
for delayed execution of incoming messages. The DMP pallet is entirely
replaced by the MQ and thereby removed. This allows for PoV-bounded
execution and resolves a number of issues that stem from the current
work-around.

All System Parachains adopt this change.  
The most important changes are in `primitives/core/src/lib.rs`,
`parachains/common/src/process_xcm_message.rs`,
`pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs`
and the runtime configs.

### DMP Queue Pallet

The pallet got removed and its logic refactored into parachain-system.
Overweight message management can be done directly through the MQ
pallet.

Final undeployment migrations are provided by
`cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that
can be configured with an aux config trait like:

```rust
parameter_types! {
	pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME;
	pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent;
}

impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime {
	type PalletName = DmpQueuePalletName;
	type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>;
	type DbWeight = <Runtime as frame_system::Config>::DbWeight;
}

// And adding them to your Migrations tuple:
pub type Migrations = (
	...
	cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>,
	cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>,
);
```

### XCMP Queue pallet

Removed all dispatch queue functionality. Incoming XCMP messages are now
either: Immediately handled if they are Signals, enqueued into the MQ
pallet otherwise.

New config items for the XCMP queue pallet:
```rust
/// The actual queue implementation that retains the messages for later processing.
type XcmpQueue: EnqueueMessage<ParaId>;

/// How a XCM over HRMP from a sibling parachain should be processed.
type XcmpProcessor: ProcessMessage<Origin = ParaId>;

/// The maximal number of suspended XCMP channels at the same time.
#[pallet::constant]
type MaxInboundSuspended: Get<u32>;
```

How to configure those:

```rust
// Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since
// the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s.
type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>;

// Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched
// with origin `Junction::Sibling(…)`.
type XcmpProcessor = ProcessFromSibling<
	ProcessXcmMessage<
		AggregateMessageOrigin,
		xcm_executor::XcmExecutor<xcm_config::XcmConfig>,
		RuntimeCall,
	>,
>;

// Not really important what to choose here. Just something larger than the maximal number of channels.
type MaxInboundSuspended = sp_core::ConstU32<1_000>;
```

The `InboundXcmpStatus` storage item was replaced by
`InboundXcmpSuspended` since it now only tracks inbound queue suspension
and no message indices anymore.

Now only sends the most recent channel `Signals`, as all prio ones are
out-dated anyway.

### Parachain System pallet

For `DMP` messages instead of forwarding them to the `DMP` pallet, it
now pushes them to the configured `DmpQueue`. The message processing
which was triggered in `set_validation_data` is now being done by the MQ
pallet `on_initialize`.

XCMP messages are still handed off to the `XcmpMessageHandler`
(XCMP-Queue pallet) - no change here.

New config items for the parachain system pallet:
```rust
/// Queues inbound downward messages for delayed processing. 
///
/// Analogous to the `XcmpQueue` of the XCMP queue pallet.
type DmpQueue: EnqueueMessage<AggregateMessageOrigin>;
``` 

How to configure:
```rust
/// Use the MQ pallet to store DMP messages for delayed processing.
type DmpQueue = MessageQueue;
``` 

## Message Flow

The flow of messages on the parachain side. Messages come in from the
left via the `Validation Data` and finally end up at the `Xcm Executor`
on the right.

![Untitled
(1)](https://github.com/paritytech/cumulus/assets/10380170/6cf8b377-88c9-4aed-96df-baace266e04d)

## Further changes

- Bumped the default suspension, drop and resume thresholds in
`QueueConfigData::default()`.
- `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when
they would be a noop.
- Properly validate the `QueueConfigData` before setting it.
- Marked weight files as auto-generated so they wont auto-expand in the
MR files view.
- Move the `hypothetical` asserts to `frame_support` under the name
`experimental_hypothetically`

Questions:
- [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in
the runtimes? Not sure how to best fix. Just having them like this makes
tests fail that rely on the real message processor when the feature is
enabled.
- [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler
already takes 80% so I put it to 10% but that is quite low.

TODO:
- [x] Remove c&p code after
https://github.com/paritytech/polkadot/pull/6271
- [x] Use `HandleMessage` once it is public in Substrate
- [x] fix `runtime-benchmarks` feature
https://github.com/paritytech/polkadot/pull/6966
- [x] Benchmarks
- [x] Tests
- [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended`
- [x] Possibly cleanup Migrations (DMP+XCMP)
- [x] optional: create `TransformProcessMessageOrigin` in Substrate and
replace `ProcessFromSibling`
- [ ] Rerun weights on ref HW

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: command-bot <>

cumulus/pallets/xcmp-queue/src/migration.rs

@@ -16,20 +16,23 @@
 
 //! A module that is responsible for migration of storage.
 
-use crate::{Config, Overweight, Pallet, QueueConfig, DEFAULT_POV_SIZE};
+use crate::{Config, OverweightIndex, Pallet, ParaId, QueueConfig, DEFAULT_POV_SIZE};
+use cumulus_primitives_core::XcmpMessageFormat;
 use frame_support::{
 	pallet_prelude::*,
-	traits::{OnRuntimeUpgrade, StorageVersion},
+	traits::{EnqueueMessage, OnRuntimeUpgrade, StorageVersion},
 	weights::{constants::WEIGHT_REF_TIME_PER_MILLIS, Weight},
 };
 
 /// The current storage version.
 pub const STORAGE_VERSION: StorageVersion = StorageVersion::new(3);
 
-/// Migrates the pallet storage to the most recent version.
-pub struct Migration<T: Config>(PhantomData<T>);
+pub const LOG: &str = "runtime::xcmp-queue-migration";
 
-impl<T: Config> OnRuntimeUpgrade for Migration<T> {
+/// Migrates the pallet storage to the most recent version.
+pub struct MigrationToV3<T: Config>(PhantomData<T>);
+
+impl<T: Config> OnRuntimeUpgrade for MigrationToV3<T> {
 	fn on_runtime_upgrade() -> Weight {
 		let mut weight = T::DbWeight::get().reads(1);
 
@@ -77,11 +80,55 @@ mod v1 {
 	}
 }
 
+pub mod v3 {
+	use super::*;
+	use crate::*;
+
+	/// Status of the inbound XCMP channels.
+	#[frame_support::storage_alias]
+	pub(crate) type InboundXcmpStatus<T: Config> =
+		StorageValue<Pallet<T>, Vec<InboundChannelDetails>, OptionQuery>;
+
+	/// Inbound aggregate XCMP messages. It can only be one per ParaId/block.
+	#[frame_support::storage_alias]
+	pub(crate) type InboundXcmpMessages<T: Config> = StorageDoubleMap<
+		Pallet<T>,
+		Blake2_128Concat,
+		ParaId,
+		Twox64Concat,
+		RelayBlockNumber,
+		Vec<u8>,
+		OptionQuery,
+	>;
+
+	#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, TypeInfo)]
+	pub struct InboundChannelDetails {
+		/// The `ParaId` of the parachain that this channel is connected with.
+		pub sender: ParaId,
+		/// The state of the channel.
+		pub state: InboundState,
+		/// The ordered metadata of each inbound message.
+		///
+		/// Contains info about the relay block number that the message was sent at, and the format
+		/// of the incoming message.
+		pub message_metadata: Vec<(RelayBlockNumber, XcmpMessageFormat)>,
+	}
+
+	#[derive(
+		Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Encode, Decode, RuntimeDebug, TypeInfo,
+	)]
+	pub enum InboundState {
+		Ok,
+		Suspended,
+	}
+}
+
 /// Migrates `QueueConfigData` from v1 (using only reference time weights) to v2 (with
 /// 2D weights).
 ///
 /// NOTE: Only use this function if you know what you're doing. Default to using
 /// `migrate_to_latest`.
+#[allow(deprecated)]
 pub fn migrate_to_v2<T: Config>() -> Weight {
 	let translate = |pre: v1::QueueConfigData| -> super::QueueConfigData {
 		super::QueueConfigData {
@@ -108,17 +155,70 @@ pub fn migrate_to_v2<T: Config>() -> Weight {
 }
 
 pub fn migrate_to_v3<T: Config>() -> Weight {
+	#[frame_support::storage_alias]
+	type Overweight<T: Config> =
+		CountedStorageMap<Pallet<T>, Twox64Concat, OverweightIndex, ParaId>;
 	let overweight_messages = Overweight::<T>::initialize_counter() as u64;
 
 	T::DbWeight::get().reads_writes(overweight_messages, 1)
 }
 
+pub fn lazy_migrate_inbound_queue<T: Config>() {
+	let Some(mut states) = v3::InboundXcmpStatus::<T>::get() else {
+		log::debug!(target: LOG, "Lazy migration finished: item gone");
+		return
+	};
+	let Some(ref mut next) = states.first_mut() else {
+		log::debug!(target: LOG, "Lazy migration finished: item empty");
+		v3::InboundXcmpStatus::<T>::kill();
+		return
+	};
+	log::debug!(
+		"Migrating inbound HRMP channel with sibling {:?}, msgs left {}.",
+		next.sender,
+		next.message_metadata.len()
+	);
+	// We take the last element since the MQ is a FIFO and we want to keep the order.
+	let Some((block_number, format)) = next.message_metadata.pop() else {
+		states.remove(0);
+		v3::InboundXcmpStatus::<T>::put(states);
+		return
+	};
+	if format != XcmpMessageFormat::ConcatenatedVersionedXcm {
+		log::warn!(target: LOG,
+			"Dropping message with format {:?} (not ConcatenatedVersionedXcm)",
+			format
+		);
+		v3::InboundXcmpMessages::<T>::remove(&next.sender, &block_number);
+		v3::InboundXcmpStatus::<T>::put(states);
+		return
+	}
+
+	let Some(msg) = v3::InboundXcmpMessages::<T>::take(&next.sender, &block_number) else {
+		defensive!("Storage corrupted: HRMP message missing:", (next.sender, block_number));
+		v3::InboundXcmpStatus::<T>::put(states);
+		return
+	};
+
+	let Ok(msg): Result<BoundedVec<_, _>, _> = msg.try_into() else {
+		log::error!(target: LOG, "Message dropped: too big");
+		v3::InboundXcmpStatus::<T>::put(states);
+		return
+	};
+
+	// Finally! We have a proper message.
+	T::XcmpQueue::enqueue_message(msg.as_bounded_slice(), next.sender);
+	log::debug!(target: LOG, "Migrated HRMP message to MQ: {:?}", (next.sender, block_number));
+	v3::InboundXcmpStatus::<T>::put(states);
+}
+
 #[cfg(test)]
 mod tests {
 	use super::*;
 	use crate::mock::{new_test_ext, Test};
 
 	#[test]
+	#[allow(deprecated)]
 	fn test_migration_to_v2() {
 		let v1 = v1::QueueConfigData {
 			suspend_threshold: 5,