mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-05-01 04:17:57 +00:00
General Message Queue Pallet (#12485)
* The message queue * Make fully generic * Refactor * Docs * Refactor * Use iter not slice * Per-origin queues * Multi-queue processing * Introduce MaxReady * Remove MaxReady in favour of ready ring * Cleanups * ReadyRing and tests * Stale page reaping * from_components -> from_parts Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move WeightCounter to sp_weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add MockedWeightInfo Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Deploy to kitchensink Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use WeightCounter Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Small fixes and logging Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add service_page Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Typo Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move service_page below service_queue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add service_message Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use correct weight function Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Overweight execution * Refactor * Missing file * Fix WeightCounter usage in scheduler Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix peek_index Take into account that decoding from a mutable slice modifies it. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add tests and bench service_page_item Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add debug_info Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add no-progress check to service_queues Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add more benches Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bound from_message and try_append_message Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add PageReaped event Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename BookStateOf and BookStateFor Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update tests and remove logging Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove redundant per-message origins; add footprint() and sweep_queue() * Move testing stuff to mock.rs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add integration test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix no-progress check Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix debug_info Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup merge and tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix footprint tracking * Introduce * Formatting * OverweightEnqueued event, auto-servicing config item * Update tests and benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Provide change handler * Add missing BookStateFor::insert and call QueueChangeHandler Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update benchmarks and weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use weight metering functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * weightInfo::process_message_payload is gone Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add defensive_saturating_accrue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename WeightCounter to WeightMeter Ctr+Shift+H should do the trick. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test on_initialize Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add module docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove origin from MaxMessageLen The message origin is not encoded into the heap and does therefore not influence the max message length anymore. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add BoundedVec::as_slice Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test Page::{from_message, try_append_message} Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs * Do nothing in sweep_queue if the queue does not exist ... otherwise it inserts default values into the storage. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test ring (un)knitting Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Upgrade stress-test Change the test to not assume that all queued messages will be processed in the next block but split it over multiple. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename BoundedVec::as_slice to as_bounded_slice Conflicts with deref().as_slice() otherwise. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix imports Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove ReadyRing struct Was used for testing only. Instead use 'fn assert_ring' which also check the service head and backlinks. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix stale page watermark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test feature and clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * QueueChanged handler is called correctly Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update benches Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Abstract testing functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Simplify tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make stuff compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Extend overweight execution benchmark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove TODOs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test service queue with faulty MessageProcessor Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update pallet ui tests to 1.65 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review doc fixes Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add weight_limit to extrinsic weight of execute_overweight * Correctly return unused weight * Return actual weight consumed in do_execute_overweight * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Set version 7.0.0-dev Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make it compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Switch message_size to u64 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Switch message_count to u64 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make CI green Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs * Update tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue * Dont mention README.md in the Cargo.toml Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove reference to readme Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: parity-processbot <> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
This commit is contained in:
Gavin Wood
@@ -0,0 +1,185 @@
|
||||
// Copyright 2022 Parity Technologies (UK) Ltd.
|
||||
// This file is part of Polkadot.
|
||||
|
||||
// Polkadot is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
|
||||
// Polkadot is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU General Public License for more details.
|
||||
|
||||
// You should have received a copy of the GNU General Public License
|
||||
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
//! Std setup helpers for testing and benchmarking.
|
||||
//!
|
||||
//! Cannot be put into mock.rs since benchmarks require no-std and mock.rs is std.
|
||||
|
||||
use crate::*;
|
||||
use frame_support::traits::Defensive;
|
||||
|
||||
/// Converts `Self` into a `Weight` by using `Self` for all components.
|
||||
pub trait IntoWeight {
|
||||
fn into_weight(self) -> Weight;
|
||||
}
|
||||
|
||||
impl IntoWeight for u64 {
|
||||
fn into_weight(self) -> Weight {
|
||||
Weight::from_parts(self, self)
|
||||
}
|
||||
}
|
||||
|
||||
/// Mocked message origin for testing.
|
||||
#[derive(Copy, Clone, Eq, PartialEq, Encode, Decode, MaxEncodedLen, TypeInfo, Debug)]
|
||||
pub enum MessageOrigin {
|
||||
Here,
|
||||
There,
|
||||
Everywhere(u32),
|
||||
}
|
||||
|
||||
impl From<u32> for MessageOrigin {
|
||||
fn from(i: u32) -> Self {
|
||||
Self::Everywhere(i)
|
||||
}
|
||||
}
|
||||
|
||||
/// Processes any message and consumes (1, 1) weight per message.
|
||||
pub struct NoopMessageProcessor;
|
||||
impl ProcessMessage for NoopMessageProcessor {
|
||||
type Origin = MessageOrigin;
|
||||
|
||||
fn process_message(
|
||||
_message: &[u8],
|
||||
_origin: Self::Origin,
|
||||
weight_limit: Weight,
|
||||
) -> Result<(bool, Weight), ProcessMessageError> {
|
||||
let weight = Weight::from_parts(1, 1);
|
||||
|
||||
if weight.all_lte(weight_limit) {
|
||||
Ok((true, weight))
|
||||
} else {
|
||||
Err(ProcessMessageError::Overweight(weight))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Create a message from the given data.
|
||||
pub fn msg<N: Get<u32>>(x: &'static str) -> BoundedSlice<u8, N> {
|
||||
BoundedSlice::defensive_truncate_from(x.as_bytes())
|
||||
}
|
||||
|
||||
pub fn vmsg(x: &'static str) -> Vec<u8> {
|
||||
x.as_bytes().to_vec()
|
||||
}
|
||||
|
||||
/// Create a page from a single message.
|
||||
pub fn page<T: Config>(msg: &[u8]) -> PageOf<T> {
|
||||
PageOf::<T>::from_message::<T>(msg.try_into().unwrap())
|
||||
}
|
||||
|
||||
pub fn single_page_book<T: Config>() -> BookStateOf<T> {
|
||||
BookState { begin: 0, end: 1, count: 1, ready_neighbours: None, message_count: 0, size: 0 }
|
||||
}
|
||||
|
||||
pub fn empty_book<T: Config>() -> BookStateOf<T> {
|
||||
BookState { begin: 0, end: 1, count: 1, ready_neighbours: None, message_count: 0, size: 0 }
|
||||
}
|
||||
|
||||
/// Returns a full page of messages with their index as payload and the number of messages.
|
||||
pub fn full_page<T: Config>() -> (PageOf<T>, usize) {
|
||||
let mut msgs = 0;
|
||||
let mut page = PageOf::<T>::default();
|
||||
for i in 0..u32::MAX {
|
||||
let r = i.using_encoded(|d| page.try_append_message::<T>(d.try_into().unwrap()));
|
||||
if r.is_err() {
|
||||
break
|
||||
} else {
|
||||
msgs += 1;
|
||||
}
|
||||
}
|
||||
assert!(msgs > 0, "page must hold at least one message");
|
||||
(page, msgs)
|
||||
}
|
||||
|
||||
/// Returns a page filled with empty messages and the number of messages.
|
||||
pub fn book_for<T: Config>(page: &PageOf<T>) -> BookStateOf<T> {
|
||||
BookState {
|
||||
count: 1,
|
||||
begin: 0,
|
||||
end: 1,
|
||||
ready_neighbours: None,
|
||||
message_count: page.remaining.into() as u64,
|
||||
size: page.remaining_size.into() as u64,
|
||||
}
|
||||
}
|
||||
|
||||
/// Assert the last event that was emitted.
|
||||
#[cfg(any(feature = "std", feature = "runtime-benchmarks", test))]
|
||||
pub fn assert_last_event<T: Config>(generic_event: <T as Config>::RuntimeEvent) {
|
||||
assert!(
|
||||
!frame_system::Pallet::<T>::block_number().is_zero(),
|
||||
"The genesis block has n o events"
|
||||
);
|
||||
frame_system::Pallet::<T>::assert_last_event(generic_event.into());
|
||||
}
|
||||
|
||||
/// Provide a setup for `bump_service_head`.
|
||||
pub fn setup_bump_service_head<T: Config>(
|
||||
current: <<T as Config>::MessageProcessor as ProcessMessage>::Origin,
|
||||
next: <<T as Config>::MessageProcessor as ProcessMessage>::Origin,
|
||||
) {
|
||||
let mut book = single_page_book::<T>();
|
||||
book.ready_neighbours = Some(Neighbours::<MessageOriginOf<T>> { prev: next.clone(), next });
|
||||
ServiceHead::<T>::put(¤t);
|
||||
BookStateFor::<T>::insert(¤t, &book);
|
||||
}
|
||||
|
||||
/// Knit a queue into the ready-ring and write it back to storage.
|
||||
pub fn knit<T: Config>(o: &<<T as Config>::MessageProcessor as ProcessMessage>::Origin) {
|
||||
let mut b = BookStateFor::<T>::get(o);
|
||||
b.ready_neighbours = crate::Pallet::<T>::ready_ring_knit(o).ok().defensive();
|
||||
BookStateFor::<T>::insert(o, b);
|
||||
}
|
||||
|
||||
/// Unknit a queue into the ready-ring and write it back to storage.
|
||||
pub fn unknit<T: Config>(o: &<<T as Config>::MessageProcessor as ProcessMessage>::Origin) {
|
||||
let mut b = BookStateFor::<T>::get(o);
|
||||
crate::Pallet::<T>::ready_ring_unknit(o, b.ready_neighbours.unwrap());
|
||||
b.ready_neighbours = None;
|
||||
BookStateFor::<T>::insert(o, b);
|
||||
}
|
||||
|
||||
/// Build a ring with three queues: `Here`, `There` and `Everywhere(0)`.
|
||||
pub fn build_ring<T: Config>(
|
||||
queues: &[<<T as Config>::MessageProcessor as ProcessMessage>::Origin],
|
||||
) {
|
||||
for queue in queues {
|
||||
BookStateFor::<T>::insert(queue, empty_book::<T>());
|
||||
}
|
||||
for queue in queues {
|
||||
knit::<T>(queue);
|
||||
}
|
||||
assert_ring::<T>(queues);
|
||||
}
|
||||
|
||||
/// Check that the Ready Ring consists of `queues` in that exact order.
|
||||
///
|
||||
/// Also check that all backlinks are valid and that the first element is the service head.
|
||||
pub fn assert_ring<T: Config>(
|
||||
queues: &[<<T as Config>::MessageProcessor as ProcessMessage>::Origin],
|
||||
) {
|
||||
for (i, origin) in queues.iter().enumerate() {
|
||||
let book = BookStateFor::<T>::get(origin);
|
||||
assert_eq!(
|
||||
book.ready_neighbours,
|
||||
Some(Neighbours {
|
||||
prev: queues[(i + queues.len() - 1) % queues.len()].clone(),
|
||||
next: queues[(i + 1) % queues.len()].clone(),
|
||||
})
|
||||
);
|
||||
}
|
||||
assert_eq!(ServiceHead::<T>::get(), queues.first().cloned());
|
||||
}
|
||||
Reference in New Issue
Block a user