Asynchronous backing PR (#2300)

* Update substrate & polkadot

* min changes to make async backing compile

* (async backing) parachain-system: track limitations for unincluded blocks (#2438)

* unincluded segment draft

* read para head from storage proof

* read_para_head -> read_included_para_head

* Provide pub interface

* add errors

* fix unincluded segment update

* BlockTracker -> Ancestor

* add a dmp limit

* Read para head depending on the storage switch

* doc comments

* storage items docs

* add a sanity check on block initialize

* Check watermark

* append to the segment on block finalize

* Move segment update into set_validation_data

* Resolve para head todo

* option watermark

* fix comment

* Drop dmq check

* fix weight

* doc-comments on inherent invariant

* Remove TODO

* add todo

* primitives tests

* pallet tests

* doc comments

* refactor unincluded segment length into a ConsensusHook (#2501)

* refactor unincluded segment length into a ConsensusHook

* add docs

* refactor bandwidth_out calculation

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* test for limits from impl

* fmt

* make tests compile

* update comment

* uncomment test

* fix collator test by adding parent to state proof

* patch HRMP watermark rules for unincluded segment

* get consensus-common tests to pass, using unincluded segment

* fix unincluded segment tests

* get all tests passing

* fmt

* rustdoc CI

* aura-ext: limit the number of authored blocks per slot (#2551)

* aura_ext consensus hook

* reverse dependency

* include weight into hook

* fix tests

* remove stray println

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* fix test warning

* fix doc link

---------

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>

* parachain-system: ignore go ahead signal once upgrade is processed (#2594)

* handle goahead signal for unincluded segment

* doc comment

* add test

* parachain-system: drop processed messages from inherent data (#2590)

* implement `drop_processed_messages`

* drop messages based on relay parent number

* adjust tests

* drop changes to mqc

* fix comment

* drop test

* drop more dead code

* clippy

* aura-ext: check slot in consensus hook and remove all `CheckInherents` logic (#2658)

* aura-ext: check slot in consensus hook

* convert relay chain slot

* Make relay chain slot duration generic

* use fixed velocity hook for pallets with aura

* purge timestamp inherent

* fix warning

* adjust runtime tests

* fix slots in tests

* Make `xcm-emulator` test pass for new consensus hook (#2722)

* add pallets on_initialize

* tests pass

* add AuraExt on_init

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>

* update polkadot git refs

* CollationGenerationConfig closure is now optional (#2772)

* CollationGenerationConfig closure is now optional

* fix test

* propagate network-protocol-staging feature (#2899)

* Feature Flagging Consensus Hook Type Parameter (#2911)

* First pass

* fmt

* Added as default feature in tomls

* Changed to direct dependency feature

* Dealing with clippy error

* Update pallets/parachain-system/src/lib.rs

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

---------

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* fmt

* bump deps and remove warning

* parachain-system: update RelevantMessagingState according to the unincluded segment (#2948)

* mostly address 2471 with a bug introduced

* adjust relevant messaging state after computing total

* fmt

* max -> min

* fix test implementation of xcmp source

* add test

* fix test message sending logic

* fix + test

* add more to unincluded segment test

* fmt

---------

Co-authored-by: Chris Sosnin <chris125_@live.com>

* Integrate new Aura / Parachain Consensus Logic in Parachain-Template / Polkadot-Parachain (#2864)

* add a comment

* refactor client/service utilities

* deprecate start_collator

* update parachain-template

* update test-service in the same way

* update polkadot-parachain crate

* fmt

* wire up new SubmitCollation message

* some runtime utilities for implementing unincluded segment runtime APIs

* allow parachains to configure their level of sybil-resistance when starting the network

* make aura-ext compile

* update to specify sybil resistance levels

* fmt

* specify relay chain slot duration in milliseconds

* update Aura to explicitly produce Send futures

also, make relay_chain_slot_duration a Duration

* add authoring duration to basic collator and document params

* integrate new basic collator into parachain-template

* remove assert_send used for testing

* basic-aura: only author when parent included

* update polkadot-parachain-bin

* fmt

* some fixes

* fixes

* add a RelayNumberMonotonicallyIncreases

* add a utility function for initializing subsystems

* some logging for timestamp adjustment

* fmt

* some fixes for lookahead collator

* add a log

* update `find_potential_parents` to account for sessions

* bound the loop

* restore & deprecate old start_collator and start_full_node functions.

* remove unnecessary await calls

* fix warning

* clippy

* more clippy

* remove unneeded logic

* ci

* update comment

Co-authored-by: Marcin S. <marcin@bytedude.com>

* (async backing) restore `CheckInherents` for backwards-compatibility (#2977)

* bring back timestamp

* Restore CheckInherents

* revert to empty CheckInherents

* make CheckInherents optional

* attempt

* properly end system blocks

* add some more comments

* ignore failing system parachain tests

* update refs after main feature branch merge

* comment out the offending tests because CI runs ignored tests

* fix warnings

* fmt

* revert to polkadot master

* cargo update -p polkadot-primitives -p sp-io

---------

Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
This commit is contained in:
Chris Sosnin
2023-08-18 19:57:34 +03:00
committed by GitHub
parent 4f699c70a4
commit 6ef1117685
105 changed files with 4707 additions and 2359 deletions
@@ -50,7 +50,7 @@ sp-weights = { git = "https://github.com/paritytech/substrate", default-features
# Polkadot
pallet-xcm = { git = "https://github.com/paritytech/polkadot", default-features = false, branch = "master" }
pallet-xcm-benchmarks = { git = "https://github.com/paritytech/polkadot", branch = "master", default-features = false, optional = true }
pallet-xcm-benchmarks = { git = "https://github.com/paritytech/polkadot", default-features = false, optional = true , branch = "master" }
polkadot-core-primitives = { git = "https://github.com/paritytech/polkadot", default-features = false, branch = "master" }
polkadot-parachain = { git = "https://github.com/paritytech/polkadot", default-features = false, branch = "master" }
polkadot-runtime-common = { git = "https://github.com/paritytech/polkadot", default-features = false, branch = "master" }
@@ -62,12 +62,11 @@ xcm-executor = { git = "https://github.com/paritytech/polkadot", default-feature
# Cumulus
cumulus-pallet-aura-ext = { path = "../../../../pallets/aura-ext", default-features = false }
cumulus-pallet-dmp-queue = { path = "../../../../pallets/dmp-queue", default-features = false }
cumulus-pallet-parachain-system = { path = "../../../../pallets/parachain-system", default-features = false }
cumulus-pallet-parachain-system = { path = "../../../../pallets/parachain-system", default-features = false, features = ["parameterized-consensus-hook",] }
cumulus-pallet-session-benchmarking = { path = "../../../../pallets/session-benchmarking", default-features = false, version = "3.0.0" }
cumulus-pallet-xcm = { path = "../../../../pallets/xcm", default-features = false }
cumulus-pallet-xcmp-queue = { path = "../../../../pallets/xcmp-queue", default-features = false }
cumulus-primitives-core = { path = "../../../../primitives/core", default-features = false }
cumulus-primitives-timestamp = { path = "../../../../primitives/timestamp", default-features = false }
cumulus-primitives-utility = { path = "../../../../primitives/utility", default-features = false }
pallet-collator-selection = { path = "../../../../pallets/collator-selection", default-features = false }
parachain-info = { path = "../../../pallets/parachain-info", default-features = false }
@@ -82,7 +81,9 @@ asset-test-utils = { path = "../test-utils"}
substrate-wasm-builder = { git = "https://github.com/paritytech/substrate", branch = "master", optional = true }
[features]
default = [ "std" ]
default = [
"std",
]
runtime-benchmarks = [
"hex-literal",
"frame-benchmarking/runtime-benchmarks",
@@ -181,7 +182,6 @@ std = [
"cumulus-pallet-xcm/std",
"cumulus-pallet-xcmp-queue/std",
"cumulus-primitives-core/std",
"cumulus-primitives-timestamp/std",
"cumulus-primitives-utility/std",
"pallet-collator-selection/std",
"parachain-info/std",
@@ -105,3 +105,15 @@ pub mod fee {
}
}
}
/// Consensus-related.
pub mod consensus {
/// Maximum number of blocks simultaneously accepted by the Runtime, not yet included
/// into the relay chain.
pub const UNINCLUDED_SEGMENT_CAPACITY: u32 = 1;
/// How many parachain blocks are processed by the relay chain per parent. Limits the
/// number of blocks authored per slot.
pub const BLOCK_PROCESSING_VELOCITY: u32 = 1;
/// Relay chain slot duration, in milliseconds.
pub const RELAY_CHAIN_SLOT_DURATION_MILLIS: u32 = 6000;
}
@@ -82,7 +82,7 @@ use sp_version::NativeVersion;
use sp_version::RuntimeVersion;
use codec::{Decode, Encode, MaxEncodedLen};
use constants::{currency::*, fee::WeightToFee};
use constants::{consensus::*, currency::*, fee::WeightToFee};
use frame_support::{
construct_runtime,
dispatch::DispatchClass,
@@ -551,6 +551,12 @@ impl cumulus_pallet_parachain_system::Config for Runtime {
type XcmpMessageHandler = XcmpQueue;
type ReservedXcmpWeight = ReservedXcmpWeight;
type CheckAssociatedRelayNumber = RelayNumberStrictlyIncreases;
type ConsensusHook = cumulus_pallet_aura_ext::FixedVelocityConsensusHook<
Runtime,
RELAY_CHAIN_SLOT_DURATION_MILLIS,
BLOCK_PROCESSING_VELOCITY,
UNINCLUDED_SEGMENT_CAPACITY,
>;
}
impl parachain_info::Config for Runtime {}
@@ -1211,33 +1217,9 @@ impl_runtime_apis! {
}
}
struct CheckInherents;
impl cumulus_pallet_parachain_system::CheckInherents<Block> for CheckInherents {
fn check_inherents(
block: &Block,
relay_state_proof: &cumulus_pallet_parachain_system::RelayChainStateProof,
) -> sp_inherents::CheckInherentsResult {
let relay_chain_slot = relay_state_proof
.read_slot()
.expect("Could not read the relay chain slot from the proof");
let inherent_data =
cumulus_primitives_timestamp::InherentDataProvider::from_relay_chain_slot_and_duration(
relay_chain_slot,
sp_std::time::Duration::from_secs(6),
)
.create_inherent_data()
.expect("Could not create the timestamp inherent data");
inherent_data.check_extrinsics(block)
}
}
cumulus_pallet_parachain_system::register_validate_block! {
Runtime = Runtime,
BlockExecutor = cumulus_pallet_aura_ext::BlockExecutor::<Runtime, Executive>,
CheckInherents = CheckInherents,
}
#[cfg(test)]
@@ -22,11 +22,12 @@ use asset_hub_polkadot_runtime::xcm_config::{
ForeignCreatorsSovereignAccountOf, TrustBackedAssetsPalletLocation, XcmConfig,
};
pub use asset_hub_polkadot_runtime::{
constants::fee::WeightToFee, AssetDeposit, Assets, Balances, ExistentialDeposit, ForeignAssets,
ForeignAssetsInstance, MetadataDepositBase, MetadataDepositPerByte, ParachainSystem, Runtime,
RuntimeCall, RuntimeEvent, SessionKeys, System, TrustBackedAssetsInstance,
constants::fee::WeightToFee, AllPalletsWithoutSystem, AssetDeposit, Assets, Balances,
ExistentialDeposit, ForeignAssets, ForeignAssetsInstance, MetadataDepositBase,
MetadataDepositPerByte, ParachainSystem, Runtime, RuntimeCall, RuntimeEvent, SessionKeys,
System, TrustBackedAssetsInstance,
};
use asset_test_utils::{CollatorSessionKeys, ExtBuilder, RuntimeHelper};
use asset_test_utils::{CollatorSessionKeys, ExtBuilder};
use codec::{Decode, Encode};
use cumulus_primitives_utility::ChargeWeightInFungibles;
use frame_support::{
@@ -47,6 +48,8 @@ const SOME_ASSET_ADMIN: [u8; 32] = [5u8; 32];
type AssetIdForTrustBackedAssetsConvert =
assets_common::AssetIdForTrustBackedAssetsConvert<TrustBackedAssetsPalletLocation>;
type RuntimeHelper = asset_test_utils::RuntimeHelper<Runtime, AllPalletsWithoutSystem>;
fn collator_session_keys() -> CollatorSessionKeys<Runtime> {
CollatorSessionKeys::new(
AccountId::from(ALICE),
@@ -70,7 +73,7 @@ fn test_asset_xcm_trader() {
let minimum_asset_balance = 333333333_u128;
let local_asset_id = 1;
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
local_asset_id.into(),
AccountId::from(ALICE).into(),
true,
@@ -79,7 +82,7 @@ fn test_asset_xcm_trader() {
// We first mint enough asset for the account to exist for assets
assert_ok!(Assets::mint(
RuntimeHelper::<Runtime>::origin_of(AccountId::from(ALICE)),
RuntimeHelper::origin_of(AccountId::from(ALICE)),
local_asset_id.into(),
AccountId::from(ALICE).into(),
minimum_asset_balance
@@ -90,7 +93,7 @@ fn test_asset_xcm_trader() {
AssetIdForTrustBackedAssetsConvert::convert_back(&local_asset_id).unwrap();
// Set Alice as block author, who will receive fees
RuntimeHelper::<Runtime>::run_to_block(2, Some(AccountId::from(ALICE)));
RuntimeHelper::run_to_block(2, AccountId::from(ALICE));
// We are going to buy 400e9 weight
// Because of the ED being higher in kusama's asset hub
@@ -152,7 +155,7 @@ fn test_asset_xcm_trader_with_refund() {
// We need root origin to create a sufficient asset
// We set existential deposit to be identical to the one for Balances first
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
1.into(),
AccountId::from(ALICE).into(),
true,
@@ -161,7 +164,7 @@ fn test_asset_xcm_trader_with_refund() {
// We first mint enough asset for the account to exist for assets
assert_ok!(Assets::mint(
RuntimeHelper::<Runtime>::origin_of(AccountId::from(ALICE)),
RuntimeHelper::origin_of(AccountId::from(ALICE)),
1.into(),
AccountId::from(ALICE).into(),
ExistentialDeposit::get()
@@ -171,7 +174,7 @@ fn test_asset_xcm_trader_with_refund() {
let ctx = XcmContext { origin: None, message_id: XcmHash::default(), topic: None };
// Set Alice as block author, who will receive fees
RuntimeHelper::<Runtime>::run_to_block(2, Some(AccountId::from(ALICE)));
RuntimeHelper::run_to_block(2, AccountId::from(ALICE));
// We are going to buy 400e9 weight
// Because of the ED being higher in kusama's asset hub
@@ -235,7 +238,7 @@ fn test_asset_xcm_trader_refund_not_possible_since_amount_less_than_ed() {
// We need root origin to create a sufficient asset
// We set existential deposit to be identical to the one for Balances first
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
1.into(),
AccountId::from(ALICE).into(),
true,
@@ -246,7 +249,7 @@ fn test_asset_xcm_trader_refund_not_possible_since_amount_less_than_ed() {
let ctx = XcmContext { origin: None, message_id: XcmHash::default(), topic: None };
// Set Alice as block author, who will receive fees
RuntimeHelper::<Runtime>::run_to_block(2, Some(AccountId::from(ALICE)));
RuntimeHelper::run_to_block(2, AccountId::from(ALICE));
// We are going to buy 50e9 weight
// Because of the ED being higher in kusama's asset hub
@@ -290,7 +293,7 @@ fn test_that_buying_ed_refund_does_not_refund() {
// We need root origin to create a sufficient asset
// We set existential deposit to be identical to the one for Balances first
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
1.into(),
AccountId::from(ALICE).into(),
true,
@@ -301,7 +304,7 @@ fn test_that_buying_ed_refund_does_not_refund() {
let ctx = XcmContext { origin: None, message_id: XcmHash::default(), topic: None };
// Set Alice as block author, who will receive fees
RuntimeHelper::<Runtime>::run_to_block(2, Some(AccountId::from(ALICE)));
RuntimeHelper::run_to_block(2, AccountId::from(ALICE));
// We are gonna buy ED
let bought = Weight::from_parts(ExistentialDeposit::get().try_into().unwrap(), 0);
@@ -355,7 +358,7 @@ fn test_asset_xcm_trader_not_possible_for_non_sufficient_assets() {
// Create a non-sufficient asset
let minimum_asset_balance = 1_000_000_u128;
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
1.into(),
AccountId::from(ALICE).into(),
false,
@@ -364,7 +367,7 @@ fn test_asset_xcm_trader_not_possible_for_non_sufficient_assets() {
// We first mint enough asset for the account to exist for assets
assert_ok!(Assets::mint(
RuntimeHelper::<Runtime>::origin_of(AccountId::from(ALICE)),
RuntimeHelper::origin_of(AccountId::from(ALICE)),
1.into(),
AccountId::from(ALICE).into(),
minimum_asset_balance
@@ -374,7 +377,7 @@ fn test_asset_xcm_trader_not_possible_for_non_sufficient_assets() {
let ctx = XcmContext { origin: None, message_id: XcmHash::default(), topic: None };
// Set Alice as block author, who will receive fees
RuntimeHelper::<Runtime>::run_to_block(2, Some(AccountId::from(ALICE)));
RuntimeHelper::run_to_block(2, AccountId::from(ALICE));
// We are going to buy 400e9 weight
// Because of the ED being higher in kusama's asset hub
@@ -441,7 +444,7 @@ fn test_assets_balances_api_works() {
// We need root origin to create a sufficient asset
let minimum_asset_balance = 3333333_u128;
assert_ok!(Assets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
local_asset_id.into(),
AccountId::from(ALICE).into(),
true,
@@ -450,7 +453,7 @@ fn test_assets_balances_api_works() {
// We first mint enough asset for the account to exist for assets
assert_ok!(Assets::mint(
RuntimeHelper::<Runtime>::origin_of(AccountId::from(ALICE)),
RuntimeHelper::origin_of(AccountId::from(ALICE)),
local_asset_id.into(),
AccountId::from(ALICE).into(),
minimum_asset_balance
@@ -459,7 +462,7 @@ fn test_assets_balances_api_works() {
// create foreign asset
let foreign_asset_minimum_asset_balance = 3333333_u128;
assert_ok!(ForeignAssets::force_create(
RuntimeHelper::<Runtime>::root_origin(),
RuntimeHelper::root_origin(),
foreign_asset_id_multilocation,
AccountId::from(SOME_ASSET_ADMIN).into(),
false,
@@ -468,7 +471,7 @@ fn test_assets_balances_api_works() {
// We first mint enough asset for the account to exist for assets
assert_ok!(ForeignAssets::mint(
RuntimeHelper::<Runtime>::origin_of(AccountId::from(SOME_ASSET_ADMIN)),
RuntimeHelper::origin_of(AccountId::from(SOME_ASSET_ADMIN)),
foreign_asset_id_multilocation,
AccountId::from(ALICE).into(),
6 * foreign_asset_minimum_asset_balance
@@ -515,6 +518,7 @@ fn test_assets_balances_api_works() {
asset_test_utils::include_teleports_for_native_asset_works!(
Runtime,
AllPalletsWithoutSystem,
XcmConfig,
CheckingAccount,
WeightToFee,
@@ -538,6 +542,7 @@ asset_test_utils::include_teleports_for_native_asset_works!(
asset_test_utils::include_teleports_for_foreign_assets_works!(
Runtime,
AllPalletsWithoutSystem,
XcmConfig,
CheckingAccount,
WeightToFee,