Asynchronous backing PR (#2300)

* Update substrate & polkadot

* min changes to make async backing compile

* (async backing) parachain-system: track limitations for unincluded blocks (#2438)

* unincluded segment draft

* read para head from storage proof

* read_para_head -> read_included_para_head

* Provide pub interface

* add errors

* fix unincluded segment update

* BlockTracker -> Ancestor

* add a dmp limit

* Read para head depending on the storage switch

* doc comments

* storage items docs

* add a sanity check on block initialize

* Check watermark

* append to the segment on block finalize

* Move segment update into set_validation_data

* Resolve para head todo

* option watermark

* fix comment

* Drop dmq check

* fix weight

* doc-comments on inherent invariant

* Remove TODO

* add todo

* primitives tests

* pallet tests

* doc comments

* refactor unincluded segment length into a ConsensusHook (#2501)

* refactor unincluded segment length into a ConsensusHook

* add docs

* refactor bandwidth_out calculation

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* test for limits from impl

* fmt

* make tests compile

* update comment

* uncomment test

* fix collator test by adding parent to state proof

* patch HRMP watermark rules for unincluded segment

* get consensus-common tests to pass, using unincluded segment

* fix unincluded segment tests

* get all tests passing

* fmt

* rustdoc CI

* aura-ext: limit the number of authored blocks per slot (#2551)

* aura_ext consensus hook

* reverse dependency

* include weight into hook

* fix tests

* remove stray println

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* fix test warning

* fix doc link

---------

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>

* parachain-system: ignore go ahead signal once upgrade is processed (#2594)

* handle goahead signal for unincluded segment

* doc comment

* add test

* parachain-system: drop processed messages from inherent data (#2590)

* implement `drop_processed_messages`

* drop messages based on relay parent number

* adjust tests

* drop changes to mqc

* fix comment

* drop test

* drop more dead code

* clippy

* aura-ext: check slot in consensus hook and remove all `CheckInherents` logic (#2658)

* aura-ext: check slot in consensus hook

* convert relay chain slot

* Make relay chain slot duration generic

* use fixed velocity hook for pallets with aura

* purge timestamp inherent

* fix warning

* adjust runtime tests

* fix slots in tests

* Make `xcm-emulator` test pass for new consensus hook (#2722)

* add pallets on_initialize

* tests pass

* add AuraExt on_init

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>

* update polkadot git refs

* CollationGenerationConfig closure is now optional (#2772)

* CollationGenerationConfig closure is now optional

* fix test

* propagate network-protocol-staging feature (#2899)

* Feature Flagging Consensus Hook Type Parameter (#2911)

* First pass

* fmt

* Added as default feature in tomls

* Changed to direct dependency feature

* Dealing with clippy error

* Update pallets/parachain-system/src/lib.rs

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

---------

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* fmt

* bump deps and remove warning

* parachain-system: update RelevantMessagingState according to the unincluded segment (#2948)

* mostly address 2471 with a bug introduced

* adjust relevant messaging state after computing total

* fmt

* max -> min

* fix test implementation of xcmp source

* add test

* fix test message sending logic

* fix + test

* add more to unincluded segment test

* fmt

---------

Co-authored-by: Chris Sosnin <chris125_@live.com>

* Integrate new Aura / Parachain Consensus Logic in Parachain-Template / Polkadot-Parachain (#2864)

* add a comment

* refactor client/service utilities

* deprecate start_collator

* update parachain-template

* update test-service in the same way

* update polkadot-parachain crate

* fmt

* wire up new SubmitCollation message

* some runtime utilities for implementing unincluded segment runtime APIs

* allow parachains to configure their level of sybil-resistance when starting the network

* make aura-ext compile

* update to specify sybil resistance levels

* fmt

* specify relay chain slot duration in milliseconds

* update Aura to explicitly produce Send futures

also, make relay_chain_slot_duration a Duration

* add authoring duration to basic collator and document params

* integrate new basic collator into parachain-template

* remove assert_send used for testing

* basic-aura: only author when parent included

* update polkadot-parachain-bin

* fmt

* some fixes

* fixes

* add a RelayNumberMonotonicallyIncreases

* add a utility function for initializing subsystems

* some logging for timestamp adjustment

* fmt

* some fixes for lookahead collator

* add a log

* update `find_potential_parents` to account for sessions

* bound the loop

* restore & deprecate old start_collator and start_full_node functions.

* remove unnecessary await calls

* fix warning

* clippy

* more clippy

* remove unneeded logic

* ci

* update comment

Co-authored-by: Marcin S. <marcin@bytedude.com>

* (async backing) restore `CheckInherents` for backwards-compatibility (#2977)

* bring back timestamp

* Restore CheckInherents

* revert to empty CheckInherents

* make CheckInherents optional

* attempt

* properly end system blocks

* add some more comments

* ignore failing system parachain tests

* update refs after main feature branch merge

* comment out the offending tests because CI runs ignored tests

* fix warnings

* fmt

* revert to polkadot master

* cargo update -p polkadot-primitives -p sp-io

---------

Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>

cumulus/client/collator/src/lib.rs

@@ -26,7 +26,8 @@ use sp_core::traits::SpawnNamed;
 use sp_runtime::traits::{Block as BlockT, Header as HeaderT};
 
 use cumulus_client_consensus_common::ParachainConsensus;
-use polkadot_node_primitives::{CollationResult, MaybeCompressedPoV};
+use polkadot_node_primitives::{CollationGenerationConfig, CollationResult, MaybeCompressedPoV};
+use polkadot_node_subsystem::messages::{CollationGenerationMessage, CollatorProtocolMessage};
 use polkadot_overseer::Handle as OverseerHandle;
 use polkadot_primitives::{CollatorPair, Id as ParaId};
 
@@ -200,7 +201,7 @@ pub mod relay_chain_driven {
 		let config = CollationGenerationConfig {
 			key,
 			para_id,
-			collator: Box::new(move |relay_parent, validation_data| {
+			collator: Some(Box::new(move |relay_parent, validation_data| {
 				// Cloning the channel on each usage effectively makes the channel
 				// unbounded. The channel is actually bounded by the block production
 				// and consensus systems of Polkadot, which limits the amount of possible
@@ -218,7 +219,7 @@ pub mod relay_chain_driven {
 
 					this_rx.await.ok().flatten()
 				})
-			}),
+			})),
 		};
 
 		overseer_handle
@@ -233,6 +234,31 @@ pub mod relay_chain_driven {
 	}
 }
 
+/// Initialize the collation-related subsystems on the relay-chain side.
+///
+/// This must be done prior to collation, and does not set up any callback for collation.
+/// For callback-driven collators, use the [`relay_chain_driven`] module.
+pub async fn initialize_collator_subsystems(
+	overseer_handle: &mut OverseerHandle,
+	key: CollatorPair,
+	para_id: ParaId,
+) {
+	overseer_handle
+		.send_msg(
+			CollationGenerationMessage::Initialize(CollationGenerationConfig {
+				key,
+				para_id,
+				collator: None,
+			}),
+			"StartCollator",
+		)
+		.await;
+
+	overseer_handle
+		.send_msg(CollatorProtocolMessage::CollateOn(para_id), "StartCollator")
+		.await;
+}
+
 /// Parameters for [`start_collator`].
 pub struct StartCollatorParams<Block: BlockT, RA, BS, Spawner> {
 	pub para_id: ParaId,
@@ -246,7 +272,24 @@ pub struct StartCollatorParams<Block: BlockT, RA, BS, Spawner> {
 }
 
 /// Start the collator.
+#[deprecated = "Collators should run consensus futures which handle this logic internally"]
 pub async fn start_collator<Block, RA, BS, Spawner>(
+	params: StartCollatorParams<Block, RA, BS, Spawner>,
+) where
+	Block: BlockT,
+	BS: BlockBackend<Block> + Send + Sync + 'static,
+	Spawner: SpawnNamed + Clone + Send + Sync + 'static,
+	RA: ProvideRuntimeApi<Block> + Send + Sync + 'static,
+	RA::Api: CollectCollationInfo<Block>,
+{
+	// This never needed to be asynchronous, but shouldn't be changed due to backcompat.
+	#[allow(deprecated)]
+	start_collator_sync(params);
+}
+
+/// Start the collator in a synchronous function.
+#[deprecated = "Collators should run consensus futures which handle this logic internally"]
+pub fn start_collator_sync<Block, RA, BS, Spawner>(
 	StartCollatorParams {
 		para_id,
 		block_status,
@@ -269,9 +312,8 @@ pub async fn start_collator<Block, RA, BS, Spawner>(
 
 	let collator = Collator::new(collator_service, parachain_consensus);
 
-	let mut request_stream = relay_chain_driven::init(key, para_id, overseer_handle).await;
-
 	let collation_future = Box::pin(async move {
+		let mut request_stream = relay_chain_driven::init(key, para_id, overseer_handle).await;
 		while let Some(request) = request_stream.next().await {
 			let collation = collator
 				.clone()
@@ -298,11 +340,14 @@ mod tests {
 		Client, ClientBlockImportExt, DefaultTestClientBuilderExt, InitBlockBuilder,
 		TestClientBuilder, TestClientBuilderExt,
 	};
+	use cumulus_test_relay_sproof_builder::RelayStateSproofBuilder;
 	use cumulus_test_runtime::{Block, Header};
 	use futures::{channel::mpsc, executor::block_on, StreamExt};
+	use polkadot_node_primitives::CollationGenerationConfig;
 	use polkadot_node_subsystem::messages::CollationGenerationMessage;
 	use polkadot_node_subsystem_test_helpers::ForwardSubsystem;
 	use polkadot_overseer::{dummy::dummy_overseer_builder, HeadSupportsParachains};
+	use polkadot_primitives::HeadData;
 	use sp_consensus::BlockOrigin;
 	use sp_core::{testing::TaskExecutor, Pair};
 	use sp_runtime::traits::BlakeTwo256;
@@ -330,10 +375,14 @@ mod tests {
 			_: PHash,
 			validation_data: &PersistedValidationData,
 		) -> Option<ParachainCandidate<Block>> {
+			let mut sproof = RelayStateSproofBuilder::default();
+			sproof.included_para_head = Some(HeadData(parent.encode()));
+			sproof.para_id = cumulus_test_runtime::PARACHAIN_ID.into();
+
 			let builder = self.client.init_block_builder_at(
 				parent.hash(),
 				Some(validation_data.clone()),
-				Default::default(),
+				sproof,
 			);
 
 			let (block, _, proof) = builder.build().expect("Creates block").into_inner();
@@ -368,6 +417,7 @@ mod tests {
 
 		spawner.spawn("overseer", None, overseer.run().then(|_| async {}).boxed());
 
+		#[allow(deprecated)]
 		let collator_start = start_collator(StartCollatorParams {
 			runtime_api: client.clone(),
 			block_status: client.clone(),
@@ -384,13 +434,19 @@ mod tests {
 			.0
 			.expect("message should be send by `start_collator` above.");
 
-		let CollationGenerationMessage::Initialize(config) = msg;
+		let collator_fn = match msg {
+			CollationGenerationMessage::Initialize(CollationGenerationConfig {
+				collator: Some(c),
+				..
+			}) => c,
+			_ => panic!("unexpected message or no collator fn"),
+		};
 
 		let validation_data =
 			PersistedValidationData { parent_head: header.encode().into(), ..Default::default() };
 		let relay_parent = Default::default();
 
-		let collation = block_on((config.collator)(relay_parent, &validation_data))
+		let collation = block_on(collator_fn(relay_parent, &validation_data))
 			.expect("Collation is build")
 			.collation;