Asynchronous Backing MegaPR (#5022)

* inclusion emulator logic for asynchronous backing (#4790) * initial stab at candidate_context * fmt * docs & more TODOs * some cleanups * reframe as inclusion_emulator * documentations yes * update types * add constraint modifications * watermark * produce modifications * v2 primitives: re-export all v1 for consistency * vstaging primitives * emulator constraints: handle code upgrades * produce outbound HRMP modifications * stack. * method for applying modifications * method just for sanity-checking modifications * fragments produce modifications, not prospectives * make linear * add some TODOs * remove stacking; handle code upgrades * take `fragment` private * reintroduce stacking. * fragment constructor * add TODO * allow validating fragments against future constraints * docs * relay-parent number and min code size checks * check code upgrade restriction * check max hrmp per candidate * fmt * remove GoAhead logic because it wasn't helpful * docs on code upgrade failure * test stacking * test modifications against constraints * fmt * test fragments * descending or duplicate test * fmt * remove unused imports in vstaging * wrong primitives * spellcheck * Runtime changes for Asynchronous Backing (#4786) * inclusion: utility for allowed relay-parents * inclusion: use prev number instead of prev hash * track most recent context of paras * inclusion: accept previous relay-parents * update dmp advancement rule for async backing * fmt * add a comment about validation outputs * clean up a couple of TODOs * weights * fix weights * fmt * Resolve dmp todo * Restore inclusion tests * Restore paras_inherent tests * MostRecentContext test * Benchmark for new paras dispatchable * Prepare check_validation_outputs for upgrade * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs * Implementers guide changes * More tests for allowed relay parents * Add a github issue link * Compute group index based on relay parent * Storage migration * Move allowed parents tracker to shared * Compile error * Get group assigned to core at the next block * Test group assignment * fmt * Error instead of panic * Update guide * Extend doc-comment * Update runtime/parachains/src/shared.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Prospective Parachains Subsystem (#4913) * docs and skeleton * subsystem skeleton * main loop * fragment tree basics & fmt * begin fragment trees & view * flesh out more of view update logic * further flesh out update logic * some refcount functions for fragment trees * add fatal/non-fatal errors * use non-fatal results * clear up some TODOs * ideal format for scheduling info * add a bunch of TODOs * some more fluff * extract fragment graph to submodule * begin fragment graph API * trees, not graphs * improve docs * scope and constructor for trees * add some test TODOs * limit max ancestors and store constraints * constructor * constraints: fix bug in HRMP watermarks * fragment tree population logic * set::retain * extract population logic * implement add_and_populate * fmt * add some TODOs in tests * implement child-selection * strip out old stuff based on wrong assumptions * use fatality * implement pruning * remove unused ancestor constraints * fragment tree instantiation * remove outdated comment * add message/request types and skeleton for handling * fmt * implement handle_candidate_seconded * candidate storage: handle backed * implement handle_candidate_backed * implement answer_get_backable_candidate * remove async where not needed * implement fetch_ancestry * add logic for run_iteration * add some docs * remove global allow(unused), fix warnings * make spellcheck happy (despite English) * fmt * bump Cargo.lock * replace tracing with gum * introduce PopulateFrom trait * implement GetHypotheticalDepths * revise docs slightly * first fragment tree scope test * more scope tests * test add_candidate * fmt * test retain * refactor test code * test populate is recursive * test contiguity of depth 0 is maintained * add_and_populate tests * cycle tests * remove PopulateFrom trait * fmt * test hypothetical depths (non-recursive) * have CandidateSeconded return membership * tree membership requests * Add a ProspectiveParachainsSubsystem struct * add a staging API for base constraints * add a `From` impl * add runtime API for staging_validity_constraints * implement fetch_base_constraints * implement `fetch_upcoming_paras` * remove reconstruction of candidate receipt; no obvious usecase * fmt * export message to broader module * remove last TODO * correctly export * fix compilation and add GetMinimumRelayParent request * make provisioner into a real subsystem with proper mesage bounds * fmt * fix ChannelsOut in overseer test * fix overseer tests * fix again * fmt * Integrate prospective parachains subsystem into backing: Part 1 (#5557) * BEGIN ASYNC candidate-backing CHANGES * rename & document modes * answer prospective validation data requests * GetMinimumRelayParents request is now plural * implement an implicit view utility for backing subsystems * implicit-view: get allowed relay parents * refactorings and improvements to implicit view * add some TODOs for tests * split implicit view updates into 2 functions * backing: define State to prepare for functional refactor * add some docs * backing: implement bones of new leaf activation logic * backing: create per-relay-parent-states * use new handle_active_leaves_update * begin extracting logic from CandidateBackingJob * mostly extract statement import from job logic * handle statement imports outside of job logic * do some TODO planning for prospective parachains integration * finish rewriting backing subsystem in functional style * add prospective parachains mode to relay parent entries * fmt * add a RejectedByProspectiveParachains error * notify prospective parachains of seconded and backed candidates * always validate candidates exhaustively in backing. * return persisted_validation_data from validation * handle rejections by prospective parachains * implement seconding sanity check * invoke validate_and_second * Alter statement table to allow multiple seconded messages per validator * refactor backing to have statements carry PVD * clean up all warnings * Add tests for implicit view * Improve doc comments * Prospective parachains mode based on Runtime API version * Add a TODO * Rework seconding_sanity_check * Iterate over responses * Update backing tests * collator-protocol: load PVD from runtime * Fix validator side tests * Update statement-distribution to fetch PVD * Fix statement-distribution tests * Backing tests with prospective paras #1 * fix per_relay_parent pruning in backing * Test multiple leaves * Test seconding sanity check * Import statement order Before creating an entry in `PerCandidateState` map wait for the approval from the prospective parachains * Add a test for correct state updates * Second multiple candidates per relay parent test * Add backing tests with prospective paras * Second more than one test without prospective paras * Add a test for prospective para blocks * Update malus * typos Co-authored-by: Chris Sosnin <chris125_@live.com> * Track occupied depth in backing per parachain (#5778) * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * fmt * Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991) * BEGIN STATEMENT DISTRIBUTION WORK create a vstaging network protocol which is the same as v1 * mostly make network bridge amenable to vstaging * network-bridge: fully adapt to vstaging * add some TODOs for tests * fix fallout in bitfield-distribution * bitfield distribution tests + TODOs * fix fallout in gossip-support * collator-protocol: fix message fallout * collator-protocol: load PVD from runtime * add TODO for vstaging tests * make things compile * set used network protocol version using a feature * fmt * get approval-distribution building * fix approval-distribution tests * spellcheck * nits * approval distribution net protocol test * bitfield distribution net protocol test * Revert "collator-protocol: fix message fallout" This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1. * Network bridge tests Co-authored-by: Chris Sosnin <chris125_@live.com> * remove max_pov_size requirement from prospective pvd request (#6014) * remove max_pov_size requirement from prospective pvd request * fmt * Extract legacy statement distribution to its own module (#6026) * add compatibility type to v2 statement distribution message * warning cleanup * handle compatibility layer for v2 * clean up an unimplemented!() block * circulate statements based on version * extract legacy v1 code into separate module * remove unimplemented * clean up naming of from_requester/responder * remove TODOs * have backing share seconded statements with PVD * fmt * fix warning * Quick fix unused warning for not yet implemented/used staging messages. * Fix network bridge test * Fix wrong merge. We now have 23 subsystems (network bridge split + prospective parachains) Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> * Version 3 is already live. * Fix tests (#6055) * Fix backing tests * Fix warnings. * fmt * collator-protocol: asynchronous backing changes (#5740) * Draft collator side changes * Start working on collations management * Handle peer's view change * Versioning on advertising * Versioned collation fetching request * Handle versioned messages * Improve docs for collation requests * Add spans * Add request receiver to overseer * Fix collator side tests * Extract relay parent mode to lib * Validator side draft * Add more checks for advertisement * Request pvd based on async backing mode * review * Validator side improvements * Make old tests green * More fixes * Collator side tests draft * Send collation test * fmt * Collator side network protocol versioning * cleanup * merge artifacts * Validator side net protocol versioning * Remove fragment tree membership request * Resolve todo * Collator side core state test * Improve net protocol compatibility * Validator side tests * more improvements * style fixes * downgrade log * Track implicit assignments * Limit the number of seconded candidates per para * Add a sanity check * Handle fetched candidate * fix tests * Retry fetch * Guard against dequeueing while already fetching * Reintegrate connection management * Timeout on advertisements * fmt * spellcheck * update tests after merge * validator assignment fixes for backing and collator protocol (#6158) * Rename depth->ancestry len in tests * Refactor group assignments * Remove implicit assignments * backing: consider occupied core assignments * Track a single para on validator side * Refactor prospective parachains mode request (#6179) * Extract prospective parachains mode into util * Skip activations depending on the mode * backing: don't send backed candidate to provisioner (#6185) * backing: introduce `CanSecond` request for advertisements filtering (#6225) * Drop BoundToRelayParent * draft changes * fix backing tests * Fix genesis ancestry * Fix validator side tests * more tests * cargo generate-lockfile * Implement `StagingValidityConstraints` Runtime API method (#6258) * Implement StagingValidityConstraints * spellcheck * fix ump params * Update hrmp comment * Introduce ump per candidate limit * hypothetical earliest block * refactor primitives usage * hypothetical earliest block number test * fix build * Prepare the Runtime for asynchronous backing upgrade (#6287) * Introduce async backing params to runtime config * fix cumulus config * use config * finish runtimes * Introduce new staging API * Update collator protocol * Update provisioner * Update prospective parachains * Update backing * Move async backing params lower in the config * make naming consistent * misc * Use real prospective parachains subsystem (#6407) * Backport `HypotheticalFrontier` into the feature branch (#6605) * implement more general HypotheticalFrontier * fmt * drop unneeded request Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Resolve todo about legacy leaf activation (#6447) * fix bug/warning in handling membership answers * Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521) * Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest` * Update tests * Fix (removed wrong docstring) * Fix can_second request * Patch some dead_code errors --------- Co-authored-by: Chris Sosnin <chris125_@live.com> * Async Backing: Send Statement Distribution "Backed" messages (#6634) * Backing: Send Statement Distribution "Backed" messages Closes #6590. **TODO:** - [ ] Adjust tests * Fix compile errors * (Mostly) fix tests * Fix comment * Fix test and compile error * Test that `StatementDistributionMessage::Backed` is sent * Fix compile error * Fix some clippy errors * Add prospective parachains subsystem tests (#6454) * Add prospective parachains subsystem test * Add `should_do_no_work_if_async_backing_disabled_for_leaf` test * Implement `activate_leaf` helper, up to getting ancestry * Finish implementing `activate_leaf` * Small refactor in `activate_leaf` * Get `CandidateSeconded` working * Finish `send_candidate_and_check_if_found` test * Refactor; send more leaves & candidates * Refactor test * Implement `check_candidate_parent_leaving_view` test * Start work on `check_candidate_on_multiple_forks` test * Don’t associate specific parachains with leaf * Finish `correctly_updates_leaves` test * Fix cycle due to reused head data * Fix `check_backable_query` test * Fix `check_candidate_on_multiple_forks` test * Add `check_depth_and_pvd_queries` test * Address review comments * Remove TODO * add a new index for output head data to candidate storage * Resolve test TODOs * Fix compile errors * test candidate storage pruning, make sure new index is cleaned up --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Node-side metrics for asynchronous backing (#6549) * Add metrics for `prune_view_candidate_storage` * Add metrics for `request_unblocked_collations` * Fix docstring * Couple fixes from review comments * Fix `check_depth_query` test * inclusion-emulator: mirror advancement rule check (#6361) * inclusion-emulator: mirror advancement rule check * fix build * prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649) * Introduce `backed_in_path_only` flag for depth request * fmt * update doc comment * fmt * Add async-backing zombienet tests (#6314) * Async backing: impl guide for statement distribution (#6738) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> * Asynchronous backing statement distribution: Take III (#5999) * add notification types for v2 statement-distribution * improve protocol docs * add empty vstaging module * fmt * add backed candidate packet request types * start putting down structure of new logic * handle activated leaf * some sanity-checking on outbound statements * fmt * update vstaging share to use statements with PVD * tiny refactor, candidate_hash location * import local statements * refactor statement import * first stab at broadcast logic * fmt * fill out some TODOs * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * fmt, fix grid test after topology change * typo Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * address review * adjust comment, make easier to understand * Fix typo --------- Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * miscellaneous fixes to make asynchronous backing work (#6791) * propagate network-protocol-staging feature * add feature to adder-collator as well * allow collation-generation of occupied cores * prospective parachains: special treatment for pending availability candidates * runtime: fetch candidates pending availability * lazily construct PVD for pending candidates * fix fallout in prospective parachains hypothetical/select_child * runtime: enact candidates when creating paras-inherent * make tests compile * test pending availability in the scope * add prospective parachains test * fix validity constraints leftovers * drop prints * Fix typos --------- Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Marcin S <marcin@realemail.net> * Remove restart from test (#6840) * Async Backing: Statement Distribution Tests (#6755) * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * Hook up request sender * Add `valid_statement_without_prior_seconded_is_ignored` test * Fix `valid_statement_without_prior_seconded_is_ignored` test * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Remove obsolete test * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * First draft of `ensure_seconding_limit_is_respected` test * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * Fix `ensure_seconding_limit_is_respected` test * Start `backed_candidate_leads_to_advertisement` test * fmt, fix grid test after topology change * Send Backed notification * Finish `backed_candidate_leads_to_advertisement` test * Finish `peer_reported_for_duplicate_statements` test * Finish `received_advertisement_before_confirmation_leads_to_request` * Add `advertisements_rejected_from_incorrect_peers` test * Add `manifest_rejected_*` tests * Add `manifest_rejected_when_group_does_not_match_para` test * Add `local_node_sanity_checks_incoming_requests` test * Add `local_node_respects_statement_mask` test * Add tests where peer is reported for providing invalid signatures * Add `cluster_peer_allowed_to_send_incomplete_statements` test * Add `received_advertisement_after_backing_leads_to_acknowledgement` * Add `received_advertisement_after_confirmation_before_backing` test * peer_reported_for_advertisement_conflicting_with_confirmed_candidate * Add `peer_reported_for_not_enough_statements` test * Add `peer_reported_for_providing_statements_meant_to_be_masked_out` * Add `additional_statements_are_shared_after_manifest_exchange` * Add `grid_statements_imported_to_backing` test * Add `relay_parent_entering_peer_view_leads_to_advertisement` test * Add `advertisement_not_re_sent_when_peer_re_enters_view` test * Update node/network/statement-distribution/src/vstaging/tests/grid.rs Co-authored-by: asynchronous rob <rphmeier@gmail.com> * Resolve TODOs, update test * Address unused code * Add check after every test for unhandled requests * Refactor (`make_dummy_leaf` and `handle_sent_request`) * Refactor (`make_dummy_topology`) * Minor refactor --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * Fix some clippy lints in tests * Async backing: minor fixes (#6920) * bitfield-distribution test * implicit view tests * Refactor parameters -> params * scheduler: update storage migration (#6963) * update scheduler migration * Adjust weight to account for storage read * Statement Distribution Guide Edits (#7025) * Statement distribution guide edits * Addressed Marcin's comments * Add attested candidate request retry timeouts (#6833) Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Fix async backing statement distribution tests (#6621) Resolve some todos in async backing statement-distribution branch (#6482) Fix clippy errors in statement distribution branch (#6720) * Async backing: add Prospective Parachains impl guide (#6933) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> * Updates to Provisioner Guide for Async Backing (#7106) * Initial corrections and clarifications * Partial first draft * Finished first draft * Adding back wrongly removed test bit * fmt * Update roadmap/implementers-guide/src/node/utility/provisioner.md Co-authored-by: Marcin S. <marcin@realemail.net> * Addressing comments * Reorganization * fmt --------- Co-authored-by: Marcin S. <marcin@realemail.net> * fmt * Renaming Parathread Mentions (#7287) * Renaming parathreads * Renaming module to pallet * More updates * PVF: Refactor workers into separate crates, remove host dependency (#7253) * PVF: Refactor workers into separate crates, remove host dependency * Fix compile error * Remove some leftover code * Fix compile errors * Update Cargo.lock * Remove worker main.rs files I accidentally copied these from the other PR. This PR isn't intended to introduce standalone workers yet. * Address review comments * cargo fmt * Update a couple of comments * Update log targets * Update quote to 1.0.27 (#7280) Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: parity-processbot <> * pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271) * pallets: implement Default for GenesisConfig in no_std This change is follow-up of: https://github.com/paritytech/substrate/pull/14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334 * Cargo.lock updated * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * cli: enable BEEFY by default on test networks (#7293) We consider BEEFY mature enough to run by default on all nodes for test networks (Rococo/Wococo/Versi). Right now, most nodes are not running it since it's opt-in using --beefy flag. Switch to an opt-out model for test networks. Replace --beefy flag from CLI with --no-beefy and have BEEFY client start by default on test networks. Signed-off-by: acatangiu <adrian@parity.io> * runtime: past session slashing runtime API (#6667) * runtime/vstaging: unapplied_slashes runtime API * runtime/vstaging: key_ownership_proof runtime API * runtime/ParachainHost: submit_report_dispute_lost * fix key_ownership_proof API * runtime: submit_report_dispute_lost runtime API * nits * Update node/subsystem-types/src/messages.rs Co-authored-by: Marcin S. <marcin@bytedude.com> * revert unrelated fmt changes * post merge fixes * fix compilation --------- Co-authored-by: Marcin S. <marcin@bytedude.com> * Correcting git mishap * Document usage of `gum` crate (#7294) * Document usage of gum crate * Small fix * Add some more basic info * Update node/gum/src/lib.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Update target docs --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * XCM: Fix issue with RequestUnlock (#7278) * XCM: Fix issue with RequestUnlock * Leave API changes for v4 * Fix clippy errors * Fix tests --------- Co-authored-by: parity-processbot <> * Companion for Substrate#14228 (#7295) * Companion for Substrate#14228 https://github.com/paritytech/substrate/pull/14228 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * Companion for #14237: Use latest sp-crates (#7300) * To revert: Update substrate branch to "lexnv/bump_sp_crates" Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates"" This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008. * Update cargo lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * bounded-collections bump to 0.1.7 (#7305) * bounded-collections bump to 0.1.7 Companion for: paritytech/substrate#14225 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * bump to quote 1.0.28 (#7306) * `RollingSessionWindow` cleanup (#7204) * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit * Fix tests in import * Fix the rest of the tests * Remove dead code * Fix todos * Simplify session caching * Comments for `SessionInfoProvider` * Separate `SessionInfoProvider` from `State` * `cache_session_info_for_head` becomes freestanding function * Remove unneeded `mut` usage * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider` * Fix SessionInfo retrieval * Code cleanup * Don't wrap `SessionInfoProvider` in an `Option` * Remove `earliest_session()` * Remove pre-caching -> wip * Fix some tests and code cleanup * Fix all tests * Fixes in tests * Fix comments, variable names and small style changes * Fix a warning * impl From<SessionWindowSize> for NonZeroUsize * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG * Code review feedback * Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db * Remove `col_session_data` usages * Storage migration clearing columns w/o removing them * Remove session data column usages from `approval-voting` and `dispute-coordinator` tests * Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests * Fix formatting in initialized.rs * Fix a corner case in `SessionInfo` caching for `dispute-coordinator` * Remove `RollingSessionWindow` ;( * Revert "Fix formatting in initialized.rs" This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc. * v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it * Fix `NUM_COLUMNS` in `approval-voting` * Use `columns::v3::NUM_COLUMNS` when opening db * Update node/service/src/parachains_db/upgrade.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3` * Fix `NUM+COLUMNS` in approval_voting * Fix formatting * Fix columns usage * Clarification comments about the different db versions --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * pallet-para-config: Remove remnant WeightInfo functions (#7308) * pallet-para-config: Remove remnant WeightInfo functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * set_config_with_weight begone Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> * XCM: PayOverXcm config (#6900) * Move XCM query functionality to trait * Fix tests * Add PayOverXcm implementation * fix the PayOverXcm trait to compile * moved doc comment out of trait implmeentation and to the trait * PayOverXCM documentation * Change documentation a bit * Added empty benchmark methods implementation and changed docs * update PayOverXCM to convert AccountIds to MultiLocations * Implement benchmarking method * Change v3 to latest * Descend origin to an asset sender (#6970) * descend origin to an asset sender * sender as tuple of dest and sender * Add more variants to the QueryResponseStatus enum * Change Beneficiary to Into<[u8; 32]> * update PayOverXcm to return concrete errors and use AccountId as sender * use polkadot-primitives for AccountId * fix dependency to use polkadot-core-primitives * force Unpaid instruction to the top of the instructions list * modify report_outcome to accept interior argument * use new_query directly for building final xcm query, instead of report_outcome * fix usage of new_query to use the XcmQueryHandler * fix usage of new_query to use the XcmQueryHandler * tiny method calling fix * xcm query handler (#7198) * drop redundant query status * rename ReportQueryStatus to OuterQueryStatus * revert rename of QueryResponseStatus * update mapping * Update xcm/xcm-builder/src/pay.rs Co-authored-by: Gavin Wood <gavin@parity.io> * Updates * Docs * Fix benchmarking stuff * Destination can be determined based on asset_kind * Tweaking API to minimise clones * Some repotting and docs --------- Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * Companion for #14265 (#7307) * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: parity-processbot <> * bump serde to 1.0.163 (#7315) * bump serde to 1.0.163 * bump ci * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * fmt * Updated fmt * Removing changes accidentally pulled from master * fix another master pull issue * Another master pull fix * fmt * Fixing implementers guide build * Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread" This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Marcin S. <marcin@bytedude.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * fix bitfield distribution test * approval distribution tests * fix bridge tests * update Cargo.lock * [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457) * Optimize collator-protocol validator-side request fetching * address feedback: replace tuples with structs * feedback: add doc comments * move collation types to subfolder --------- Signed-off-by: alindima <alin@parity.io> * Update collation generation for asynchronous backing (#7405) * break candidate receipt construction and distribution into own function * update implementers' guide to include SubmitCollation * implement SubmitCollation for collation-generation * fmt * fix test compilation & remove unnecessary submodule * add some TODOs for a test suite. * Update roadmap/implementers-guide/src/types/overseer-protocol.md Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * add new test harness and first test * refactor to avoid requiring background sender * ensure collation gets packaged and distributed * tests for the fallback case with no hint * add parent rp-number hint tests * fmt * update uses of CollationGenerationConfig * fix remaining test * address review comments * use subsystemsender for background tasks * fmt * remove ValidationCodeHashHint and related tests --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * fix some more fallout from merge * fmt * remove staging APIs from Rococo & Westend (#7513) * send network messages on main protocol name (#7515) * misc async backing improvements for allowed ancestry blocks (#7532) * shared: fix acquire_info * backwards-compat test for prospective parachains * same relay parent is allowed * provisioner: request candidate receipt by relay parent (#7527) * return candidates hash from prospective parachains * update provisioner * update tests * guide changes * send a single message to backing * fix test * revert to old `handle_new_activations` logic in some cases (#7514) * revert to old `handle_new_activations` logic * gate sending messages on scheduled cores to max_depth >= 2 * fmt * 2->1 * Omnibus asynchronous backing bugfix PR (#7529) * fix a bug in backing * add some more logs * prospective parachains: take ancestry only up to session bounds * add test * fix zombienet tests (#7614) Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix runtime compilation * make bitfield distribution tests compile * attempt to fix zombienet disputes (#7618) * update metric name * update some metric names * avoid cycles when creating fake candidates * make undying collator more friendly to malformed parents * fix a bug in malus * fmt * clippy * add RUN_IN_CONTAINER to new ZombieNet tests (#7631) * remove duplicated migration happened because of master-merge --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: alindima <alin@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Mattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com> Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Alin Dima <alin@parity.io>
2026-06-14 04:01:10 +00:00 · 2023-08-18 11:11:56 -05:00
parent ad539f0e41
commit 5174b9d2d7
175 changed files with 40882 additions and 6462 deletions
@@ -12,18 +12,17 @@
  - [Messaging Overview](messaging.md)
  - [PVF Pre-checking](pvf-prechecking.md)
 - [Runtime Architecture](runtime/README.md)
-  - [`Initializer` Module](runtime/initializer.md)
-  - [`Configuration` Module](runtime/configuration.md)
-  - [`Shared`](runtime/shared.md)
-  - [`Disputes` Module](runtime/disputes.md)
-  - [`Paras` Module](runtime/paras.md)
-  - [`Scheduler` Module](runtime/scheduler.md)
-  - [`Inclusion` Module](runtime/inclusion.md)
-  - [`ParaInherent` Module](runtime/parainherent.md)
-  - [`DMP` Module](runtime/dmp.md)
-  - [`UMP` Module](runtime/ump.md)
-  - [`HRMP` Module](runtime/hrmp.md)
-  - [`Session Info` Module](runtime/session_info.md)
+  - [`Initializer` Pallet](runtime/initializer.md)
+  - [`Configuration` Pallet](runtime/configuration.md)
+  - [`Shared` Pallet](runtime/shared.md)
+  - [`Disputes` Pallet](runtime/disputes.md)
+  - [`Paras` Pallet](runtime/paras.md)
+  - [`Scheduler` Pallet](runtime/scheduler.md)
+  - [`Inclusion` Pallet](runtime/inclusion.md)
+  - [`ParaInherent` Pallet](runtime/parainherent.md)
+  - [`DMP` Pallet](runtime/dmp.md)
+  - [`HRMP` Pallet](runtime/hrmp.md)
+  - [`Session Info` Pallet](runtime/session_info.md)
 - [Runtime APIs](runtime-api/README.md)
  - [Validators](runtime-api/validators.md)
  - [Validator Groups](runtime-api/validator-groups.md)
@@ -45,7 +44,9 @@
    - [Collator Protocol](node/collators/collator-protocol.md)
  - [Backing Subsystems](node/backing/README.md)
    - [Candidate Backing](node/backing/candidate-backing.md)
+    - [Prospective Parachains](node/backing/prospective-parachains.md)
    - [Statement Distribution](node/backing/statement-distribution.md)
+    - [Statement Distribution (Legacy)](node/backing/statement-distribution-legacy.md)
  - [Availability Subsystems](node/availability/README.md)
    - [Availability Distribution](node/availability/availability-distribution.md)
    - [Availability Recovery](node/availability/availability-recovery.md)
@@ -24,11 +24,12 @@ exactly one downward message queue.
 - **Parablock:** A block in a parachain.
 - **Parachain:** A constituent chain secured by the Relay Chain's validators.
 - **Parachain Validators:** A subset of validators assigned during a period of time to back candidates for a specific parachain
- **Parathread:** A parachain which is scheduled on a pay-as-you-go basis.
+- **On-demand parachain:** A parachain which is scheduled on a pay-as-you-go basis.
+- **Lease holding parachain:** A parachain possessing an active slot lease. The lease holder is assigned a single availability core for the duration of the lease, granting consistent blockspace scheduling at the rate 1 parablock per relay block.
 - **PDK (Parachain Development Kit):** A toolset that allows one to develop a parachain. Cumulus is a PDK.
 - **Preimage:** In our context, if `H(X) = Y` where `H` is a hash function and `Y` is the hash, then `X` is the hash preimage.
 - **Proof-of-Validity (PoV):** A stateless-client proof that a parachain candidate is valid, with respect to some validation function.
- **PVF:** Parachain Validation Function. The validation code that is run by validators on parachains or parathreads.
+- **PVF:** Parachain Validation Function. The validation code that is run by validators on parachains.
 - **PVF Prechecking:** This is the process of initially checking the PVF when it is first added. We attempt preparation of the PVF and make sure it succeeds within a given timeout, plus some additional checks.
 - **PVF Preparation:** This is the process of preparing the WASM blob and includes both prevalidation and compilation. As there is no prevalidation right now, preparation just consists of compilation.
 - **Relay Parent:** A block in the relay chain, referred to in a context where work is being done in the context of the state at this block.
@@ -130,7 +130,7 @@ Dispatch a `CandidateValidationMessage::Validate(validation function, candidate,

 ### Distribute Signed Statement

-Dispatch a [`StatementDistributionMessage`][SDM]`::Share(relay_parent, SignedFullStatement)`.
+Dispatch a [`StatementDistributionMessage`][SDM]`::Share(relay_parent, SignedFullStatementWithPVD)`.

 [OverseerSignal]: ../../types/overseer-protocol.md#overseer-signal
 [Statement]: ../../types/backing.md#statement-type
@@ -0,0 +1,158 @@
+# Prospective Parachains
+
+## Overview
+
+**Purpose:** Tracks and handles prospective parachain fragments and informs
+other backing-stage subsystems of work to be done.
+
+"prospective":
+- [*prə'spɛktɪv*] adj.
+- future, likely, potential
+
+Asynchronous backing changes the runtime to accept parachain candidates from a
+certain allowed range of historic relay-parents. This means we can now build
+*prospective parachains* – that is, trees of potential (but likely) future
+parachain blocks. This is the subsystem responsible for doing so.
+
+Other subsystems such as Backing rely on Prospective Parachains, e.g. for
+determining if a candidate can be seconded. This subsystem is the main
+coordinator of work within the node for the collation and backing phases of
+parachain consensus.
+
+Prospective Parachains is primarily an implementation of fragment trees. It also
+handles concerns such as:
+
+- the relay-chain being forkful
+- session changes
+
+See the following sections for more details.
+
+### Fragment Trees
+
+This subsystem builds up fragment trees, which are trees of prospective para
+candidates. Each path through the tree represents a possible state transition
+path for the para. Each potential candidate is a fragment, or a node, in the
+tree. Candidates are validated against constraints as they are added.
+
+This subsystem builds up trees for each relay-chain block in the view, for each
+para. These fragment trees are used for:
+
+- providing backable candidates to other subsystems
+- sanity-checking that candidates can be seconded
+- getting seconded candidates under active leaves
+- etc.
+
+For example, here is a tree with several possible paths:
+
+```
+Para Head registered by the relay chain:     included_head
+                                                  ↲  ↳
+depth 0:                                  head_0_a    head_0_b
+                                             ↲            ↳
+depth 1:                             head_1_a              head_1_b
+                                  ↲      |     ↳
+depth 2:                 head_2_a1   head_2_a2  head_2_a3
+```
+
+### The Relay-Chain Being Forkful
+
+We account for the same candidate possibly appearing in different forks. While
+we still build fragment trees for each head in each fork, we are efficient with
+how we reference candidates to save space.
+
+### Session Changes
+
+Allowed ancestry doesn't cross session boundary. That is, you can only build on
+top of the freshest relay parent when the session starts. This is a current
+limitation that may be lifted in the future.
+
+Also, runtime configuration values needed for constraints (such as
+`max_pov_size`) are constant within a session. This is important when building
+prospective validation data. This is unlikely to change.
+
+## Messages
+
+### Incoming
+
+- `ActiveLeaves`
+  - Notification of a change in the set of active leaves.
+  - Constructs fragment trees for each para for each new leaf.
+- `ProspectiveParachainsMessage::IntroduceCandidate`
+  - Informs the subsystem of a new candidate.
+  - Sent by the Backing Subsystem when it is importing a statement for a
+    new candidate.
+- `ProspectiveParachainsMessage::CandidateSeconded`
+  - Informs the subsystem that a previously introduced candidate has
+    been seconded.
+  - Sent by the Backing Subsystem when it is importing a statement for a
+    new candidate after it sends `IntroduceCandidate`, if that wasn't
+    rejected by Prospective Parachains.
+- `ProspectiveParachainsMessage::CandidateBacked`
+  - Informs the subsystem that a previously introduced candidate has
+    been backed.
+  - Sent by the Backing Subsystem after it successfully imports a
+    statement giving a candidate the necessary quorum of backing votes.
+- `ProspectiveParachainsMessage::GetBackableCandidate`
+  - Get a backable candidate hash along with its relay parent for a given parachain,
+    under a given relay-parent (leaf) hash, which is a descendant of given candidate hashes.
+  - Sent by the Provisioner when requesting backable candidates, when
+    selecting candidates for a given relay-parent.
+- `ProspectiveParachainsMessage::GetHypotheticalFrontier`
+  - Gets the hypothetical frontier membership of candidates with the
+    given properties under the specified active leaves' fragment trees.
+  - Sent by the Backing Subsystem when sanity-checking whether a candidate can
+    be seconded based on its hypothetical frontiers.
+- `ProspectiveParachainsMessage::GetTreeMembership`
+  - Gets the membership of the candidate in all fragment trees.
+  - Sent by the Backing Subsystem when it needs to update the candidates
+    seconded at various depths under new active leaves.
+- `ProspectiveParachainsMessage::GetMinimumRelayParents`
+  - Gets the minimum accepted relay-parent number for each para in the
+    fragment tree for the given relay-chain block hash.
+  - That is, this returns the minimum relay-parent block number in the
+    same branch of the relay-chain which is accepted in the fragment
+    tree for each para-id.
+  - Sent by the Backing, Statement Distribution, and Collator Protocol
+    subsystems when activating leaves in the implicit view.
+- `ProspectiveParachainsMessage::GetProspectiveValidationData`
+  - Gets the validation data of some prospective candidate. The
+    candidate doesn't need to be part of any fragment tree.
+  - Sent by the Collator Protocol subsystem (validator side) when
+    handling a fetched collation result.
+
+### Outgoing
+
+- `RuntimeApiRequest::StagingParaBackingState`
+  - Gets the backing state of the given para (the constraints of the para and
+    candidates pending availability).
+- `RuntimeApiRequest::AvailabilityCores`
+  - Gets information on all availability cores.
+- `ChainApiMessage::Ancestors`
+  - Requests the `k` ancestor block hashes of a block with the given
+    hash.
+- `ChainApiMessage::BlockHeader`
+  - Requests the block header by hash.
+
+## Glossary
+
+- **Candidate storage:** Stores candidates and information about them
+  such as their relay-parents and their backing states. Is indexed in
+  various ways.
+- **Constraints:**
+  - Constraints on the actions that can be taken by a new parachain
+    block.
+  - Exhaustively define the set of valid inputs and outputs to parachain
+    execution.
+- **Fragment:** A prospective para block (that is, a block not yet referenced by
+  the relay-chain). Fragments are anchored to the relay-chain at a particular
+  relay-parent.
+- **Fragment tree:**
+  - A tree of fragments. Together, these fragments define one or more
+    prospective paths a parachain's state may transition through.
+  - See the "Fragment Tree" section.
+- **Inclusion emulation:** Emulation of the logic that the runtime uses
+  for checking parachain blocks.
+- **Relay-parent:** A particular relay-chain block that a fragment is
+  anchored to.
+- **Scope:** The scope of a fragment tree, defining limits on nodes
+  within the tree.
@@ -0,0 +1,119 @@
+# Statement Distribution (Legacy)
+
+This describes the legacy, backwards-compatible version of the Statement
+Distribution subsystem.
+
+**Note:** All the V1 (legacy) code was extracted out to a `legacy_v1` module of
+the `statement-distribution` crate, which doesn't alter any logic. V2 (new
+protocol) peers also run `legacy_v1` and communicate with V1 peers using V1
+messages and with V2 peers using V2 messages. Once the runtime upgrade goes
+through on all networks, this `legacy_v1` code will no longer be triggered and
+will be vestigial and can be removed.
+
+## Overview
+
+The Statement Distribution Subsystem is responsible for distributing statements about seconded candidates between validators.
+
+## Protocol
+
+`PeerSet`: `Validation`
+
+Input:
+
+- `NetworkBridgeUpdate(update)`
+- `StatementDistributionMessage`
+
+Output:
+
+- `NetworkBridge::SendMessage(PeerId, message)`
+- `NetworkBridge::SendRequests(StatementFetchingV1)`
+- `NetworkBridge::ReportPeer(PeerId, cost_or_benefit)`
+
+## Functionality
+
+Implemented as a gossip protocol. Handles updates to our view and peers' views. Neighbor packets are used to inform peers which chain heads we are interested in data for.
+
+The Statement Distribution Subsystem is responsible for distributing signed statements that we have generated and for forwarding statements generated by other validators. It also detects a variety of Validator misbehaviors for reporting to the [Provisioner Subsystem](../utility/provisioner.md). During the Backing stage of the inclusion pipeline, Statement Distribution is the main point of contact with peer nodes. On receiving a signed statement from a peer in the same backing group, assuming the peer receipt state machine is in an appropriate state, it sends the Candidate Receipt to the [Candidate Backing subsystem](candidate-backing.md) to handle the validator's statement. On receiving `StatementDistributionMessage::Share` we make sure to send messages to our backing group in addition to random other peers, to ensure a fast backing process and getting all statements quickly for distribution.
+
+This subsystem tracks equivocating validators and stops accepting information from them. It establishes a data-dependency order:
+
+- In order to receive a `Seconded` message we have the corresponding chain head in our view
+- In order to receive a `Valid` message we must have received the corresponding `Seconded` message.
+
+And respect this data-dependency order from our peers by respecting their views. This subsystem is responsible for checking message signatures.
+
+The Statement Distribution subsystem sends statements to peer nodes.
+
+## Peer Receipt State Machine
+
+There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any [`SignedFullStatement`](../../types/backing.md#signed-statement-type), validate that the provided signature does in fact sign the included data. Note that each individual parablock candidate gets its own instance of this state machine; it is perfectly legal to receive a `Valid(X)` before a `Seconded(Y)`, as long as a `Seconded(X)` has been received.
+
+A: Initial State. Receive `SignedFullStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedFullStatement` variant: drop it.
+
+B: Receive any `SignedFullStatement`: check signature and determine whether the statement is new to us. if new, forward to Candidate Backing and circulate to other peers. Receive `OverseerMessage::StopWork`: proceed to C.
+
+C: Receive any message for this block: drop it.
+
+For large statements (see below), we also keep track of the total received large
+statements per peer and have a hard limit on that number for flood protection.
+This is necessary as in the current code we only forward statements once we have
+all the data, therefore flood protection for large statement is a bit more
+subtle. This will become an obsolete problem once [off chain code
+upgrades](https://github.com/paritytech/polkadot/issues/2979) are implemented.
+
+## Peer Knowledge Tracking
+
+The peer receipt state machine implies that for parsimony of network resources, we should model the knowledge of our peers, and help them out. For example, let's consider a case with peers A, B, and C, validators X and Y, and candidate M. A sends us a `Statement::Second(M)` signed by X. We've double-checked it, and it's valid. While we're checking it, we receive a copy of X's `Statement::Second(M)` from `B`, along with a `Statement::Valid(M)` signed by Y.
+
+Our response to A is just the `Statement::Valid(M)` signed by Y. However, we haven't heard anything about this from C. Therefore, we send it everything we have: first a copy of X's `Statement::Second`, then Y's `Statement::Valid`.
+
+This system implies a certain level of duplication of messages--we received X's `Statement::Second` from both our peers, and C may experience the same--but it minimizes the degree to which messages are simply dropped.
+
+And respect this data-dependency order from our peers. This subsystem is responsible for checking message signatures.
+
+No jobs. We follow view changes from the [`NetworkBridge`](../utility/network-bridge.md), which in turn is updated by the overseer.
+
+## Equivocations and Flood Protection
+
+An equivocation is a double-vote by a validator. The [Candidate Backing](candidate-backing.md) Subsystem is better-suited than this one to detect equivocations as it adds votes to quorum trackers.
+
+At this level, we are primarily concerned about flood-protection, and to some extent, detecting equivocations is a part of that. In particular, we are interested in detecting equivocations of `Seconded` statements. Since every other statement is dependent on `Seconded` statements, ensuring that we only ever hold a bounded number of `Seconded` statements is sufficient for flood-protection.
+
+The simple approach is to say that we only receive up to two `Seconded` statements per validator per chain head. However, the marginal cost of equivocation, conditional on having already equivocated, is close to 0, since a single double-vote offence is counted as all double-vote offences for a particular chain-head. Even if it were not, there is some amount of equivocations that can be done such that the marginal cost of issuing further equivocations is close to 0, as there would be an amount of equivocations necessary to be completely and totally obliterated by the slashing algorithm. We fear the validator with nothing left to lose.
+
+With that in mind, this simple approach has a caveat worth digging deeper into.
+
+First: We may be aware of two equivocated `Seconded` statements issued by a validator. A totally honest peer of ours can also be aware of one or two different `Seconded` statements issued by the same validator. And yet another peer may be aware of one or two _more_ `Seconded` statements. And so on. This interacts badly with pre-emptive sending logic. Upon sending a `Seconded` statement to a peer, we will want to pre-emptively follow up with all statements relative to that candidate. Waiting for acknowledgment introduces latency at every hop, so that is best avoided. What can happen is that upon receipt of the `Seconded` statement, the peer will discard it as it falls beyond the bound of 2 that it is allowed to store. It cannot store anything in memory about discarded candidates as that would introduce a DoS vector. Then, the peer would receive from us all of the statements pertaining to that candidate, which, from its perspective, would be undesired - they are data-dependent on the `Seconded` statement we sent them, but they have erased all record of that from their memory. Upon receiving a potential flood of undesired statements, this 100% honest peer may choose to disconnect from us. In this way, an adversary may be able to partition the network with careful distribution of equivocated `Seconded` statements.
+
+The fix is to track, per-peer, the hashes of up to 4 candidates per validator (per relay-parent) that the peer is aware of. It is 4 because we may send them 2 and they may send us 2 different ones. We track the data that they are aware of as the union of things we have sent them and things they have sent us. If we receive a 1st or 2nd `Seconded` statement from a peer, we note it in the peer's known candidates even if we do disregard the data locally. And then, upon receipt of any data dependent on that statement, we do not reduce that peer's standing in our eyes, as the data was not undesired.
+
+There is another caveat to the fix: we don't want to allow the peer to flood us because it has set things up in a way that it knows we will drop all of its traffic.
+We also track how many statements we have received per peer, per candidate, and per chain-head. This is any statement concerning a particular candidate: `Seconded`, `Valid`, or `Invalid`. If we ever receive a statement from a peer which would push any of these counters beyond twice the amount of validators at the chain-head, we begin to lower the peer's standing and eventually disconnect. This bound is a massive overestimate and could be reduced to twice the number of validators in the corresponding validator group. It is worth noting that the goal at the time of writing is to ensure any finite bound on the amount of stored data, as any equivocation results in a large slash.
+
+## Large statements
+
+Seconded statements can become quite large on parachain runtime upgrades for
+example. For this reason, there exists a `LargeStatement` constructor for the
+`StatementDistributionMessage` wire message, which only contains light metadata
+of a statement. The actual candidate data is not included. This message type is
+used whenever a message is deemed large. The receiver of such a message needs to
+request the actual payload via request/response by means of a
+`StatementFetchingV1` request.
+
+This is necessary as distribution of a large payload (mega bytes) via gossip
+would make the network collapse and timely distribution of statements would no
+longer be possible. By using request/response it is ensured that each peer only
+transferes large data once. We only take good care to detect an overloaded
+peer early and immediately move on to a different peer for fetching the data.
+This mechanism should result in a good load distribution and therefore a rather
+optimal distribution path.
+
+With these optimizations, distribution of payloads in the size of up to 3 to 4
+MB should work with Kusama validator specifications. For scaling up even more,
+runtime upgrades and message passing should be done off chain at some point.
+
+Flood protection considerations: For making DoS attacks slightly harder on this
+subsystem, nodes will only respond to large statement requests, when they
+previously notified that peer via gossip about that statement. So, it is not
+possible to DoS nodes at scale, by requesting candidate data over and over
+again.
@@ -1,107 +1,479 @@
 # Statement Distribution

-The Statement Distribution Subsystem is responsible for distributing statements about seconded candidates between validators.
+This subsystem is responsible for distributing signed statements that we have generated and forwarding statements generated by our peers. Received candidate receipts and statements are passed to the [Candidate Backing subsystem](candidate-backing.md) to handle producing local statements. On receiving `StatementDistributionMessage::Share`, this subsystem distributes the message across the network with redundency to ensure a fast backing process.
+
+## Overview
+
+**Goal:** every well-connected node is aware of every next potential parachain
+block.
+
+Validators can either:
+
+- receive parachain block from collator, check block, and gossip statement.
+- receive statements from other validators, check the parachain block if it
+  originated within their own group, gossip forward statement if valid.
+
+Validators must have statements, candidates, and persisted validation from all
+other validators. This is because we need to store statements from validators
+who've checked the candidate on the relay chain, so we know who to hold
+accountable in case of disputes. Any validator can be selected as the next
+relay-chain block author, and this is not revealed in advance for security
+reasons. As a result, all validators must have a up to date view of all possible
+parachain candidates + backing statements that could be placed on-chain in the
+next block.
+
+[This blog post](https://polkadot.network/blog/polkadot-v1-0-sharding-and-economic-security)
+puts it another way: "Validators who aren't assigned to the parachain still
+listen for the attestations [statements] because whichever validator ends up
+being the author of the relay-chain block needs to bundle up attested parachain
+blocks for several parachains and place them into the relay-chain block."
+
+Backing-group quorum (that is, enough backing group votes) must be reached
+before the block author will consider the candidate. Therefore, validators need
+to consider _all_ seconded candidates within their own group, because that's
+what they're assigned to work on. Validators only need to consider _backable_
+candidates from other groups. This informs the design of the statement
+distribution protocol to have separate phases for in-group and out-group
+distribution, respectively called "cluster" and "grid" mode (see below).
+
+### With Async Backing
+
+Asynchronous backing changes the runtime to accept parachain candidates from a
+certain allowed range of historic relay-parents. These candidates must be backed
+by the group assigned to the parachain as-of their corresponding relay parents.

 ## Protocol

-`PeerSet`: `Validation`
+To address the concern of dealing with large numbers of spam candidates or
+statements, the overall design approach is to combine a focused "clustering"
+protocol for legitimate fresh candidates with a broad-distribution "grid"
+protocol to quickly get backed candidates into the hands of many validators.
+Validators do not eagerly send each other heavy `CommittedCandidateReceipt`,
+but instead request these lazily through request/response protocols.

-Input:
+A high-level description of the protocol follows:

- `NetworkBridgeUpdate(update)`
- `StatementDistributionMessage`
+### Messages

-Output:
+Nodes can send each other a few kinds of messages: `Statement`,
+`BackedCandidateManifest`, `BackedCandidateAcknowledgement`.

- `NetworkBridge::SendMessage(PeerId, message)`
- `NetworkBridge::SendRequests(StatementFetchingV1)`
- `NetworkBridge::ReportPeer(PeerId, cost_or_benefit)`
+- `Statement` messages contain only a signed compact statement, without full
+  candidate info.
+- `BackedCandidateManifest` messages advertise a description of a backed
+  candidate and stored statements.
+- `BackedCandidateAcknowledgement` messages acknowledge that a backed candidate
+  is fully known.

-## Functionality
+### Request/response protocol

-Implemented as a gossip protocol. Handle updates to our view and peers' views. Neighbor packets are used to inform peers which chain heads we are interested in data for.
+Nodes can request the full `CommittedCandidateReceipt` and
+`PersistedValidationData`, along with statements, over a request/response
+protocol. This is the `AttestedCandidateRequest`; the response is
+`AttestedCandidateResponse`.

-It is responsible for distributing signed statements that we have generated and forwarding them, and for detecting a variety of Validator misbehaviors for reporting to the [Provisioner Subsystem](../utility/provisioner.md). During the Backing stage of the inclusion pipeline, it's the main point of contact with peer nodes. On receiving a signed statement from a peer in the same backing group, assuming the peer receipt state machine is in an appropriate state, it sends the Candidate Receipt to the [Candidate Backing subsystem](candidate-backing.md) to handle the validator's statement. On receiving `StatementDistributionMessage::Share` we make sure to send messages to our backing group in addition to random other peers, to ensure a fast backing process and getting all statements quickly for distribution.
+### Importability and the Hypothetical Frontier

-Track equivocating validators and stop accepting information from them. Establish a data-dependency order:
+The **prospective parachains** subsystem maintains prospective "fragment trees"
+which can be used to determine whether a particular parachain candidate could
+possibly be included in the future. Candidates which either are within a
+fragment tree or _would be_ part of a fragment tree if accepted are said to be
+in the "hypothetical frontier".

- In order to receive a `Seconded` message we have the corresponding chain head in our view
- In order to receive an `Valid` message we must have received the corresponding `Seconded` message.
+The **statement-distribution** subsystem keeps track of all candidates, and
+updates its knowledge of the hypothetical frontier based on events such as new
+relay parents, new confirmed candidates, and newly backed candidates.

-And respect this data-dependency order from our peers by respecting their views. This subsystem is responsible for checking message signatures.
+We only consider statements as "importable" when the corresponding candidate is
+part of the hypothetical frontier, and only send "importable" statements to the
+backing subsystem itself.

-The Statement Distribution subsystem sends statements to peer nodes.
+### Cluster Mode

-## Peer Receipt State Machine
+- Validator nodes are partitioned into groups (with some exceptions), and
+  validators within a group at a relay-parent can send each other `Statement`
+  messages for any candidates within that group and based on that relay-parent.
+- This is referred to as the "cluster" mode.
+  - Right now these are the same as backing groups, though "cluster"
+    specifically refers to the set of nodes communicating with each other in the
+    first phase of distribution.
+- `Seconded` statements must be sent before `Valid` statements.
+- `Seconded` statements may only be sent to other members of the group when the
+  candidate is fully known by the local validator.
+  - "Fully known" means the validator has the full `CommittedCandidateReceipt`
+    and `PersistedValidationData`, which it receives on request from other
+    validators or from a collator.
+  - The reason for this is that sending a statement (which is always a
+    `CompactStatement` carrying nothing but a hash and signature) to the
+    cluster, is also a signal that the sending node is available to request the
+    candidate from.
+  - This makes the protocol easier to reason about, while also reducing network
+    messages about candidates that don't really exist.
+- Validators in a cluster receiving messages about unknown candidates request
+  the candidate (and statements) from other cluster members which have it.
+- Spam considerations
+  - The maximum depth of candidates allowed in asynchronous backing determines
+    the maximum amount of `Seconded` statements originating from a validator V
+    which each validator in a cluster may send to others. This bounds the number
+    of candidates.
+  - There is a small number of validators in each group, which further limits
+    the amount of candidates.
+- We accept candidates which don't fit in the fragment trees of any relay
+  parents.
+  - "Accept" means "attempt to request and store in memory until useful or
+    expired".
+  - We listen to prospective parachains subsystem to learn of new additions to
+    the fragment trees.
+  - Use this to attempt to import the candidate later.

-There is a very simple state machine which governs which messages we are willing to receive from peers. Not depicted in the state machine: on initial receipt of any [`SignedFullStatement`](../../types/backing.md#signed-statement-type), validate that the provided signature does in fact sign the included data. Note that each individual parablock candidate gets its own instance of this state machine; it is perfectly legal to receive a `Valid(X)` before a `Seconded(Y)`, as long as a `Seconded(X)` has been received.
+### Grid Mode

-A: Initial State. Receive `SignedFullStatement(Statement::Second)`: extract `Statement`, forward to Candidate Backing, proceed to B. Receive any other `SignedFullStatement` variant: drop it.
+- Every consensus session provides randomness and a fixed validator set, which
+  is used to build a redundant grid topology.
+  - It's redundant in the sense that there are 2 paths from every node to every
+    other node. See "Grid Topology" section for more details.
+- This grid topology is used to create a sending path from each validator group
+  to every validator.
+- When a node observes a candidate as backed, it sends a
+  `BackedCandidateManifest` to their "receiving" nodes.
+- If receiving nodes don't yet know the candidate, they request it.
+- Once they know the candidate, they respond with a
+  `BackedCandidateAcknowledgement`.
+- Once two nodes perform a manifest/acknowledgement exchange, they can send
+  `Statement` messages directly to each other for any new statements they might
+  need.
+  - This limits the amount of statements we'd have to deal with w.r.t.
+    candidates that don't really exist. See "Manifest Exchange" section.
+- There are limitations on the number of candidates that can be advertised by
+  each peer, similar to those in the cluster. Validators do not request
+  candidates which exceed these limitations.
+- Validators request candidates as soon as they are advertised, but do not
+  import the statements until the candidate is part of the hypothetical
+  frontier, and do not re-advertise or acknowledge until the candidate is
+  considered both backable and part of the hypothetical frontier.
+- Note that requesting is not an implicit acknowledgement, and an explicit
+  acknowledgement must be sent upon receipt.

-B: Receive any `SignedFullStatement`: check signature and determine whether the statement is new to us. if new, forward to Candidate Backing and circulate to other peers. Receive `OverseerMessage::StopWork`: proceed to C.
+## Messages

-C: Receive any message for this block: drop it.
+### Incoming

-For large statements (see below), we also keep track of the total received large
-statements per peer and have a hard limit on that number for flood protection.
-This is necessary as in the current code we only forward statements once we have
-all the data, therefore flood protection for large statement is a bit more
-subtle. This will become an obsolete problem once [off chain code
-upgrades](https://github.com/paritytech/polkadot/issues/2979) are implemented.
+- `ActiveLeaves`
+  - Notification of a change in the set of active leaves.
+- `StatementDistributionMessage::Share`
+  - Notification of a locally-originating statement. That is, this statement
+    comes from our node and should be distributed to other nodes.
+  - Sent by the Backing Subsystem after it successfully imports a
+    locally-originating statement.
+- `StatementDistributionMessage::Backed`
+  - Notification of a candidate being backed (received enough validity votes
+    from the backing group).
+  - Sent by the Backing Subsystem after it successfully imports a statement for
+    the first time and after sending ~Share~.
+- `StatementDistributionMessage::NetworkBridgeUpdate`
+  - See next section.

-## Peer Knowledge Tracking
+#### Network bridge events

-The peer receipt state machine implies that for parsimony of network resources, we should model the knowledge of our peers, and help them out. For example, let's consider a case with peers A, B, and C, validators X and Y, and candidate M. A sends us a `Statement::Second(M)` signed by X. We've double-checked it, and it's valid. While we're checking it, we receive a copy of X's `Statement::Second(M)` from `B`, along with a `Statement::Valid(M)` signed by Y.
+- v1 compatibility
+  - Messages for the v1 protocol are routed to the legacy statement
+    distribution.
+- `Statement`
+  - Notification of a signed statement.
+  - Sent by a peer's Statement Distribution subsystem when circulating
+    statements.
+- `BackedCandidateManifest`
+  - Notification of a backed candidate being known by the sending node.
+  - For the candidate being requested by the receiving node if needed.
+  - Announcement.
+  - Sent by a peer's Statement Distribution subsystem.
+- `BackedCandidateKnown`
+  - Notification of a backed candidate being known by the sending node.
+  - For informing a receiving node which already has the candidate.
+  - Acknowledgement.
+  - Sent by a peer's Statement Distribution subsystem.

-Our response to A is just the `Statement::Valid(M)` signed by Y. However, we haven't heard anything about this from C. Therefore, we send it everything we have: first a copy of X's `Statement::Second`, then Y's `Statement::Valid`.
+### Outgoing

-This system implies a certain level of duplication of messages--we received X's `Statement::Second` from both our peers, and C may experience the same--but it minimizes the degree to which messages are simply dropped.
+- `NetworkBridgeTxMessage::SendValidationMessages`
+  - Sends a peer all pending messages / acknowledgements / statements for a
+    relay parent, either through the cluster or the grid.
+- `NetworkBridgeTxMessage::SendValidationMessage`
+  - Circulates a compact statement to all peers who need it, either through the
+    cluster or the grid.
+- `NetworkBridgeTxMessage::ReportPeer`
+  - Reports a peer (either good or bad).
+- `CandidateBackingMessage::Statement`
+  - Note a validator's statement about a particular candidate.
+- `ProspectiveParachainsMessage::GetHypotheticalFrontier`
+  - Gets the hypothetical frontier membership of candidates under active leaves'
+    fragment trees.
+- `NetworkBridgeTxMessage::SendRequests`
+  - Sends requests, initiating the request/response protocol.

-And respect this data-dependency order from our peers. This subsystem is responsible for checking message signatures.
+## Request/Response

-No jobs. We follow view changes from the [`NetworkBridge`](../utility/network-bridge.md), which in turn is updated by the overseer.
+We also have a request/response protocol because validators do not eagerly send
+each other heavy `CommittedCandidateReceipt`, but instead need to request these
+lazily.

-## Equivocations and Flood Protection
+### Protocol

-An equivocation is a double-vote by a validator. The [Candidate Backing](candidate-backing.md) Subsystem is better-suited than this one to detect equivocations as it adds votes to quorum trackers.
+1. Requesting Validator

-At this level, we are primarily concerned about flood-protection, and to some extent, detecting equivocations is a part of that. In particular, we are interested in detecting equivocations of `Seconded` statements. Since every other statement is dependent on `Seconded` statements, ensuring that we only ever hold a bounded number of `Seconded` statements is sufficient for flood-protection.
+   - Requests are queued up with `RequestManager::get_or_insert`.
+     - Done as needed, when handling incoming manifests/statements.
+   - `RequestManager::dispatch_requests` sends any queued-up requests.
+      - Calls `RequestManager::next_request` to completion.
+        - Creates the `OutgoingRequest`, saves the receiver in
+          `RequestManager::pending_responses`.
+      - Does nothing if we have more responses pending than the limit of parallel
+        requests.

-The simple approach is to say that we only receive up to two `Seconded` statements per validator per chain head. However, the marginal cost of equivocation, conditional on having already equivocated, is close to 0, since a single double-vote offence is counted as all double-vote offences for a particular chain-head. Even if it were not, there is some amount of equivocations that can be done such that the marginal cost of issuing further equivocations is close to 0, as there would be an amount of equivocations necessary to be completely and totally obliterated by the slashing algorithm. We fear the validator with nothing left to lose.
+2. Peer

-With that in mind, this simple approach has a caveat worth digging deeper into.
+   - Requests come in on a peer on the `IncomingRequestReceiver`.
+     - Runs in a background responder task which feeds requests to `answer_request`
+       through `MuxedMessage`.
+     - This responder task has a limit on the number of parallel requests.
+   - `answer_request` on the peer takes the request and sends a response.
+     - Does this using the response sender on the request.

-First: We may be aware of two equivocated `Seconded` statements issued by a validator. A totally honest peer of ours can also be aware of one or two different `Seconded` statements issued by the same validator. And yet another peer may be aware of one or two _more_ `Seconded` statements. And so on. This interacts badly with pre-emptive sending logic. Upon sending a `Seconded` statement to a peer, we will want to pre-emptively follow up with all statements relative to that candidate. Waiting for acknowledgment introduces latency at every hop, so that is best avoided. What can happen is that upon receipt of the `Seconded` statement, the peer will discard it as it falls beyond the bound of 2 that it is allowed to store. It cannot store anything in memory about discarded candidates as that would introduce a DoS vector. Then, the peer would receive from us all of the statements pertaining to that candidate, which, from its perspective, would be undesired - they are data-dependent on the `Seconded` statement we sent them, but they have erased all record of that from their memory. Upon receiving a potential flood of undesired statements, this 100% honest peer may choose to disconnect from us. In this way, an adversary may be able to partition the network with careful distribution of equivocated `Seconded` statements.
+3. Requesting Validator

-The fix is to track, per-peer, the hashes of up to 4 candidates per validator (per relay-parent) that the peer is aware of. It is 4 because we may send them 2 and they may send us 2 different ones. We track the data that they are aware of as the union of things we have sent them and things they have sent us. If we receive a 1st or 2nd `Seconded` statement from a peer, we note it in the peer's known candidates even if we do disregard the data locally. And then, upon receipt of any data dependent on that statement, we do not reduce that peer's standing in our eyes, as the data was not undesired.
+   - `receive_response` on the original validator yields a response.
+     - Response was sent on the request's response sender.
+     - Uses `RequestManager::await_incoming` to await on pending responses in an
+       unordered fashion.
+     - Runs on the `MuxedMessage` receiver.
+   - `handle_response` handles the response.

-There is another caveat to the fix: we don't want to allow the peer to flood us because it has set things up in a way that it knows we will drop all of its traffic.
-We also track how many statements we have received per peer, per candidate, and per chain-head. This is any statement concerning a particular candidate: `Seconded`, `Valid`, or `Invalid`. If we ever receive a statement from a peer which would push any of these counters beyond twice the amount of validators at the chain-head, we begin to lower the peer's standing and eventually disconnect. This bound is a massive overestimate and could be reduced to twice the number of validators in the corresponding validator group. It is worth noting that the goal at the time of writing is to ensure any finite bound on the amount of stored data, as any equivocation results in a large slash.
+### API

-## Large statements
+- `dispatch_requests`
+  - Dispatches pending requests for candidate data & statements.
+- `answer_request`
+  - Answers an incoming request for a candidate.
+  - Takes an incoming `AttestedCandidateRequest`.
+- `receive_response`
+  - Wait on the next incoming response.
+  - If there are no requests pending, this future never resolves.
+  - Returns `UnhandledResponse`
+- `handle_response`
+  - Handles an incoming response.
+  - Takes `UnhandledResponse`

-Seconded statements can become quite large on parachain runtime upgrades for
-example. For this reason, there exists a `LargeStatement` constructor for the
-`StatementDistributionMessage` wire message, which only contains light metadata
-of a statement. The actual candidate data is not included. This message type is
-used whenever a message is deemed large. The receiver of such a message needs to
-request the actual payload via request/response by means of a
-`StatementFetchingV1` request.
+## Manifests

-This is necessary as distribution of a large payload (mega bytes) via gossip
-would make the network collapse and timely distribution of statements would no
-longer be possible. By using request/response it is ensured that each peer only
-transferes large data once. We only take good care to detect an overloaded
-peer early and immediately move on to a different peer for fetching the data.
-This mechanism should result in a good load distribution and therefore a rather
-optimal distribution path.
+A manifest is a message about a known backed candidate, along with a description
+of the statements backing it. It can be one of two kinds:

-With these optimizations, distribution of payloads in the size of up to 3 to 4
-MB should work with Kusama validator specifications. For scaling up even more,
-runtime upgrades and message passing should be done off chain at some point.
+- `Full`: Contains information about the candidate and should be sent to peers
+  who may not have the candidate yet. This is also called an `Announcement`.
+- `Acknowledgement`: Omits information implicit in the candidate, and should be
+  sent to peers which are guaranteed to have the candidate already.

-Flood protection considerations: For making DoS attacks slightly harder on this
-subsystem, nodes will only respond to large statement requests, when they
-previously notified that peer via gossip about that statement. So, it is not
-possible to DoS nodes at scale, by requesting candidate data over and over
-again.
+### Manifest Exchange
+
+Manifest exchange is when a receiving node received a `Full` manifest and
+replied with an `Acknowledgement`. It indicates that both nodes know the
+candidate as valid and backed. This allows the nodes to send `Statement`
+messages directly to each other for any new statements.
+
+Why? This limits the amount of statements we'd have to deal with w.r.t.
+candidates that don't really exist. Limiting out-of-group statement distribution
+between peers to only candidates that both peers agree are backed and exist
+ensures we only have to store statements about real candidates.
+
+In practice, manifest exchange means that one of three things have happened:
+
+- They announced, we acknowledged.
+- We announced, they acknowledged.
+- We announced, they announced.
+
+Concerning the last case, note that it is possible for two nodes to have each
+other in their sending set. Consider:
+
+```
+1 2
+3 4
+```
+
+If validators 2 and 4 are in group B, then there is a path `2->1->3` and
+`4->3->1`. Therefore, 1 and 3 might send each other manifests for the same
+candidate at the same time, without having seen the other's yet. This also
+counts as a manifest exchange, but is only allowed to occur in this way.
+
+After the exchange is complete, we update pending statements. Pending statements
+are those we know locally that the remote node does not.
+
+#### Alternative Paths Through The Topology
+
+Nodes should send a `BackedCandidateAcknowledgement(CandidateHash,
+StatementFilter)` notification to any peer which has sent a manifest, and the
+candidate has been acquired by other means. This keeps alternative paths through
+the topology open, which allows nodes to receive additional statements that come
+later, but not after the candidate has been posted on-chain.
+
+This is mostly about the limitation that the runtime has no way for block
+authors to post statements that come after the parablock is posted on-chain and
+ensure those validators still get rewarded. Technically, we only need enough
+statements to back the candidate and the manifest + request will provide that.
+But more statements might come shortly afterwards, and we want those to end up
+on-chain as well to ensure all validators in the group are rewarded.
+
+For clarity, here is the full timeline:
+
+1. candidate seconded
+1. backable in cluster
+1. distributed along grid
+1. latecomers issue statements
+1. candidate posted on chain
+1. really latecomers issue statements
+
+## Cluster Module
+
+The cluster module provides direct distribution of unbacked candidates within a
+group. By utilizing this initial phase of propagating only within
+clusters/groups, we bound the number of `Seconded` messages per validator per
+relay-parent, helping us prevent spam. Validators can try to circumvent this,
+but they would only consume a few KB of memory and it is trivially slashable on
+chain.
+
+The cluster module determines whether to accept/reject messages from other
+validators in the same group. It keeps track of what we have sent to other
+validators in the group, and pending statements. For the full protocol, see
+"Protocol".
+
+## Grid Module
+
+The grid module provides distribution of backed candidates and late statements
+outside the backing group. For the full protocol, see the "Protocol" section.
+
+### Grid Topology
+
+For distributing outside our cluster (aka backing group) we use a 2D grid 
+topology. This limits the amount of peers we send messages to, and handles 
+view updates.
+
+The basic operation of the grid topology is that:
+
+- A validator producing a message sends it to its row-neighbors and its
+  column-neighbors.
+- A validator receiving a message originating from one of its row-neighbors
+  sends it to its column-neighbors.
+- A validator receiving a message originating from one of its column-neighbors
+  sends it to its row-neighbors.
+
+This grid approach defines 2 unique paths for every validator to reach every
+other validator in at most 2 hops, providing redundancy.
+
+Propagation follows these rules:
+
+- Each node has a receiving set and a sending set. These are different for each
+  group. That is, if a node receives a candidate from group A, it checks if it
+  is allowed to receive from that node for candidates from group A.
+- For groups that we are in, receive from nobody and send to our X/Y peers.
+- For groups that we are not part of:
+  - We receive from any validator in the group we share a slice with and send to
+    the corresponding X/Y slice in the other dimension.
+  - For any validators we don't share a slice with, we receive from the nodes
+    which share a slice with them.
+
+### Example
+
+For size 11, the matrix would be:
+
+```
+0  1  2
+3  4  5
+6  7  8
+9 10
+```
+
+e.g. for index 10, the neighbors would be 1, 4, 7, 9 -- these are the nodes we
+could directly communicate with (e.g. either send to or receive from).
+
+Now, which of these neighbors can 10 receive from? Recall that the
+sending/receiving sets for 10 would be different for different groups. Here are
+some hypothetical scenarios:
+
+- **Scenario 1:** 9 belongs to group A but not 10. Here, 10 can directly receive
+  candidates from group A from 9. 10 would propagate them to the nodes in {1, 4,
+  7} that are not in A.
+- **Scenario 2:** 6 is in group A instead of 9, and 7 is not in group A. 10 can
+  receive group A messages from 7 or 9. 10 will try to relay these messages, but
+  7 and 9 together should have already propagated the message to all x/y 
+  peers of 10. If so, then 10 will just receive acknowledgements in reply rather 
+  than requests.
+- **Scenario 3:** 10 itself is in group A. 10 would not receive candidates from
+  this group from any other nodes through the grid. It would itself send such
+  candidates to all its neighbors that are not in A.
+
+### Seconding Limit
+
+The seconding limit is a per-validator limit. Before asynchronous backing, we
+had a rule that every validator was only allowed to second one candidate per
+relay parent. With asynchronous backing, we have a 'maximum depth' which makes
+it possible to second multiple candidates per relay parent. The seconding limit
+is set to `max depth + 1` to set an upper bound on candidates entering the
+system.
+
+## Candidates Module
+
+The candidates module provides a tracker for all known candidates in the view,
+whether they are confirmed or not, and how peers have advertised the candidates.
+What is a confirmed candidate? It is a candidate for which we have the full
+receipt and the persisted validation data. This module gets confirmed candidates
+from two sources:
+
+- It can be that a validator fetched a collation directly from the collator and
+  validated it.
+- The first time a validator gets an announcement for an unknown candidate, it
+  will send a request for the candidate. Upon receiving a response and
+  validating it (see `UnhandledResponse::validate_response`), it will mark the
+  candidate as confirmed.
+
+## Requests Module
+
+The requests module provides a manager for pending requests for candidate data,
+as well as pending responses. See "Request/Response Protocol" for a high-level
+description of the flow. See module-docs for full details.
+
+## Glossary
+
+- **Acknowledgement:** A partial manifest sent to a validator that already has the 
+  candidate to inform them that the sending node also knows the candidate. 
+  Concludes a manifest exchange.
+- **Announcement:** A full manifest indicating that a backed candidate is known by 
+  the sending node. Initiates a manifest exchange.
+- **Attestation:** See "Statement".
+- **Backable vs. Backed:**
+  - Note that we sometimes use "backed" to refer to candidates that are
+    "backable", but not yet backed on chain.
+  - **Backed** should technically mean that the parablock candidate and its
+    backing statements have been added to a relay chain block.
+  - **Backable** is when the necessary backing statements have been acquired but
+    those statements and the parablock candidate haven't been backed in a relay
+    chain block yet.
+- **Fragment tree:** A parachain fragment not referenced by the relay-chain.
+  It is a tree of prospective parachain blocks.
+- **Manifest:** A message about a known backed candidate, along with a
+  description of the statements backing it. There are two kinds of manifest, 
+  `Acknowledgement` and `Announcement`. See "Manifests" section.
+- **Peer:** Another validator that a validator is connected to.
+- **Request/response:** A protocol used to lazily request and receive heavy
+  candidate data when needed.
+- **Reputation:** Tracks reputation of peers. Applies annoyance cost and good
+  behavior benefits.
+- **Statement:** Signed statements that can be made about parachain candidates.
+  - **Seconded:** Proposal of a parachain candidate. Implicit validity vote.
+  - **Valid:** States that a parachain candidate is valid.
+- **Target:** Target validator to send a statement to.
+- **View:** Current knowledge of the chain state.
+  - **Explicit view** / **immediate view**
+    - The view a peer has of the relay chain heads and highest finalized block.
+  - **Implicit view**
+    - Derived from the immediate view. Composed of active leaves and minimum
+      relay-parents allowed for candidates of various parachains at those
+      leaves.
@@ -9,7 +9,7 @@ Collation generation for Parachains currently works in the following way:
 1.  A new relay chain block is imported.
 2.  The collation generation subsystem checks if the core associated to
    the parachain is free and if yes, continues.
-3.  Collation generation calls our collator callback to generate a PoV.
+3.  Collation generation calls our collator callback, if present, to generate a PoV. If none exists, do nothing.
 4.  Authoring logic determines if the current node should build a PoV.
 5.  Build new PoV and give it back to collation generation.

@@ -25,6 +25,10 @@ Collation generation for Parachains currently works in the following way:
  - No more than one initialization message should ever be sent to the collation
    generation subsystem.
  - Sent by a collator to initialize this subsystem.
+- `CollationGenerationMessage::SubmitCollation`
+  - If the subsystem isn't initialized or the relay-parent is too old to be relevant, ignore the message.
+  - Otherwise, use the provided parameters to generate a [`CommittedCandidateReceipt`]
+  - Submit the collation to the collator-protocol with `CollatorProtocolMessage::DistributeCollation`.

 ### Outgoing

@@ -101,7 +105,7 @@ pub struct CollationGenerationConfig {
  /// Collator's authentication key, so it can sign things.
  pub key: CollatorPair,
  /// Collation function. See [`CollatorFn`] for more details.
-  pub collator: CollatorFn,
+  pub collator: Option<CollatorFn>,
  /// The parachain that this collator collates for
  pub para_id: ParaId,
 }
@@ -136,7 +140,7 @@ The configuration should be optional, to allow for the case where the node is no

 - **Collation generation config**

-  - Contains collator's authentication key, collator function, and
+  - Contains collator's authentication key, optional collator function, and
    parachain ID.

 [CP]: collator-protocol.md
@@ -1,8 +1,6 @@
 # Provisioner

-Relay chain block authorship authority is governed by BABE and is beyond the scope of the Overseer and the rest of the subsystems. That said, ultimately the block author needs to select a set of backable parachain candidates and other consensus data, and assemble a block from them. This subsystem is responsible for providing the necessary data to all potential block authors.
-
-A major feature of the provisioner: this subsystem is responsible for ensuring that parachain block candidates are sufficiently available before sending them to potential block authors.
+Relay chain block authorship authority is governed by BABE and is beyond the scope of the Overseer and the rest of the subsystems. That said, ultimately the block author needs to select a set of backable parachain candidates and other consensus data, and assemble a block from them. This subsystem is responsible for providing the necessary data to all potential block authors. 

 ## Provisionable Data

@@ -10,7 +8,7 @@ There are several distinct types of provisionable data, but they share this prop

 ### Backed Candidates

-The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backed candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's; the provisioner must ensure that block authors are aware of all available [`BackedCandidate`s](../../types/backing.md#backed-candidate).
+The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backable candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's. The provisioner subsystem is how those block authors make this choice in practice.

 ### Signed Bitfields

@@ -30,6 +28,23 @@ Dispute resolution is complex and is explained in substantially more detail [her

 ## Protocol

+The subsystem should maintain a set of handles to Block Authorship Provisioning iterations that are currently live.
+
+### On Overseer Signal
+
+- `ActiveLeavesUpdate`:
+  - For each `activated` head:
+    - spawn a Block Authorship Provisioning iteration with the given relay parent, storing a bidirectional channel with that iteration.
+  - For each `deactivated` head:
+    - terminate the Block Authorship Provisioning iteration for the given relay parent, if any.
+- `Conclude`: Forward `Conclude` to all iterations, waiting a small amount of time for them to join, and then hard-exiting.
+
+### On `ProvisionerMessage`
+
+Forward the message to the appropriate Block Authorship Provisioning iteration, or discard if no appropriate iteration is currently active.
+
+### Per Provisioning Iteration
+
 Input: [`ProvisionerMessage`](../../types/overseer-protocol.md#provisioner-message). Backed candidates come from the [Candidate Backing subsystem](../backing/candidate-backing.md), signed bitfields come from the [Bitfield Distribution subsystem](../availability/bitfield-distribution.md), and disputes come from the [Disputes Subsystem](../disputes/dispute-coordinator.md). Misbehavior reports are currently sent from the [Candidate Backing subsystem](../backing/candidate-backing.md) and contain the following misbehaviors:

 1. `Misbehavior::ValidityDoubleVote`
@@ -45,37 +60,17 @@ Block authors request the inherent data they should use for constructing the inh

 ## Block Production

-When a validator is selected by BABE to author a block, it becomes a block producer. The provisioner is the subsystem best suited to choosing which specific backed candidates and availability bitfields should be assembled into the block. To engage this functionality, a `ProvisionerMessage::RequestInherentData` is sent; the response is a [`ParaInherentData`](../../types/runtime.md#parainherentdata). There are never two distinct parachain candidates included for the same parachain and that new parachain candidates cannot be backed until the previous one either gets declared available or expired. Appropriate bitfields, as outlined in the section on [bitfield selection](#bitfield-selection), and any dispute statements should be attached as well.
+When a validator is selected by BABE to author a block, it becomes a block producer. The provisioner is the subsystem best suited to choosing which specific backed candidates and availability bitfields should be assembled into the block. To engage this functionality, a `ProvisionerMessage::RequestInherentData` is sent; the response is a [`ParaInherentData`](../../types/runtime.md#parainherentdata). Each relay chain block backs at most one backable parachain block candidate per parachain. Additionally no further block candidate can be backed until the previous one either gets declared available or expired. If bitfields indicate that candidate A, predecessor of B, should be declared available, then B can be backed in the same relay block. Appropriate bitfields, as outlined in the section on [bitfield selection](#bitfield-selection), and any dispute statements should be attached as well.

 ### Bitfield Selection

 Our goal with respect to bitfields is simple: maximize availability. However, it's not quite as simple as always including all bitfields; there are constraints which still need to be met:

- We cannot choose more than one bitfield per validator.
- Each bitfield must correspond to an occupied core.
+- not more than one bitfield per validator
+- each 1 bit must correspond to an occupied core

 Beyond that, a semi-arbitrary selection policy is fine. In order to meet the goal of maximizing availability, a heuristic of picking the bitfield with the greatest number of 1 bits set in the event of conflict is useful.

-### Candidate Selection
-
-The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate appropriate to each free core.
-
-To determine availability:
-
- Get the list of core states from the runtime API
- For each core state:
-  - On `CoreState::Scheduled`, then we can make an `OccupiedCoreAssumption::Free`.
-  - On `CoreState::Occupied`, then we may be able to make an assumption:
-    - If the bitfields indicate availability and there is a scheduled `next_up_on_available`, then we can make an `OccupiedCoreAssumption::Included`.
-    - If the bitfields do not indicate availability, and there is a scheduled `next_up_on_time_out`, and `occupied_core.time_out_at == block_number_under_production`, then we can make an `OccupiedCoreAssumption::TimedOut`.
-  - If we did not make an `OccupiedCoreAssumption`, then continue on to the next core.
-  - Now compute the core's `validation_data_hash`: get the `PersistedValidationData` from the runtime, given the known `ParaId` and `OccupiedCoreAssumption`;
-  - Find an appropriate candidate for the core.
-    - There are two constraints: `backed_candidate.candidate.descriptor.para_id == scheduled_core.para_id && candidate.candidate.descriptor.validation_data_hash == computed_validation_data_hash`.
-    - In the event that more than one candidate meets the constraints, selection between the candidates is arbitrary. However, not more than one candidate can be selected per core.
-
-The end result of this process is a vector of `BackedCandidate`s, sorted in order of their core index. Furthermore, this process should select at maximum one candidate which upgrades the runtime validation code.
-
 ### Dispute Statement Selection

 This is the point at which the block author provides further votes to active disputes or initiates new disputes in the runtime state.
@@ -100,27 +95,92 @@ To compute bitfield availability, then:
  - Update the availability. Conceptually, assuming bit vectors: `availability[validator_index] |= bitfield[core_idx]`
 - Availability has a 2/3 threshold. Therefore: `3 * availability.count_ones() >= 2 * availability.len()`

-### Notes
+### Candidate Selection: Prospective Parachains Mode

-See also: [Scheduler Module: Availability Cores](../../runtime/scheduler.md#availability-cores).
+The state of the provisioner `PerRelayParent` tracks an important setting, `ProspectiveParachainsMode`. This setting determines which backable candidate selection method the provisioner uses.

-## Functionality
+`ProspectiveParachainsMode::Disabled` - The provisioner uses its own internal legacy candidate selection. 
+`ProspectiveParachainsMode::Enabled` - The provisioner requests that [prospective parachains](../backing/prospective-parachains.md) provide selected candidates.

-The subsystem should maintain a set of handles to Block Authorship Provisioning Jobs that are currently live.
+Candidates selected with `ProspectiveParachainsMode::Enabled` are able to benefit from the increased block production time asynchronous backing allows. For this reason all Polkadot protocol networks will eventually use prospective parachains candidate selection. Then legacy candidate selection will be removed as obsolete.

-### On Overseer Signal
+### Prospective Parachains Candidate Selection

- `ActiveLeavesUpdate`:
-  - For each `activated` head:
-    - spawn a Block Authorship Provisioning Job with the given relay parent, storing a bidirectional channel with that job.
-  - For each `deactivated` head:
-    - terminate the Block Authorship Provisioning Job for the given relay parent, if any.
- `Conclude`: Forward `Conclude` to all jobs, waiting a small amount of time for them to join, and then hard-exiting.
+The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate appropriate to each free core. In prospective parachains candidate selection the provisioner handles the former process while [prospective parachains](../backing/prospective-parachains.md) handles the latter.

-### On `ProvisionerMessage`
+To select backable candidates:

-Forward the message to the appropriate Block Authorship Provisioning Job, or discard if no appropriate job is currently active.
+- Get the list of core states from the runtime API
+- For each core state:
+  - On `CoreState::Free`
+    - The core is unscheduled and doesn’t need to be provisioned with a candidate
+  - On `CoreState::Scheduled`
+    - The core is unoccupied and scheduled to accept a backed block for a particular `para_id`.
+    - The provisioner requests a backable candidate from [prospective parachains](../backing/prospective-parachains.md) with the desired relay parent, the core’s scheduled `para_id`, and an empty required path. 
+  - On `CoreState::Occupied`
+    - The availability core is occupied by a parachain block candidate pending availability. A further candidate need not be provided by the provisioner unless the core will be vacated this block. This is the case when either bitfields indicate the current core occupant has been made available or a timeout is reached.
+    - If `bitfields_indicate_availability`
+      - If `Some(scheduled_core) = occupied_core.next_up_on_available`, the core will be vacated and in need of a provisioned candidate. The provisioner requests a backable candidate from [prospective parachains](../backing/prospective-parachains.md) with the core’s scheduled `para_id` and a required path with one entry. This entry corresponds to the parablock candidate previously occupying this core, which was made available and can be built upon even though it hasn’t been seen as included in a relay chain block yet. See the Required Path section below for more detail.
+      - If `occupied_core.next_up_on_available` is `None`, then the core being vacated is unscheduled and doesn’t need to be provisioned with a candidate.
+    - Else-if `occupied_core.time_out_at == block_number`
+      - If `Some(scheduled_core) = occupied_core.next_up_on_timeout`, the core will be vacated and in need of a provisioned candidate. A candidate is requested in exactly the same way as with `CoreState::Scheduled`.
+      - Else the core being vacated is unscheduled and doesn’t need to be provisioned with a candidate
+The end result of this process is a vector of `CandidateHash`s, sorted in order of their core index.

-## Block Authorship Provisioning Job
+#### Required Path

-Maintain the set of channels to block authors. On receiving provisionable data, send a copy over each channel.
+Required path is a parameter for `ProspectiveParachainsMessage::GetBackableCandidate`, which the provisioner sends in candidate selection. 
+
+An empty required path indicates that the requested candidate should be a direct child of the most recently included parablock for the given `para_id` as of the given relay parent. 
+
+In contrast, a required path with one or more entries prompts [prospective parachains](../backing/prospective-parachains.md) to step forward through its fragment tree for the given `para_id` and relay parent until the desired parablock is reached. We then select a direct child of that parablock to pass to the provisioner. 
+
+The parablocks making up a required path do not need to have been previously seen as included in relay chain blocks. Thus the ability to provision backable candidates based on a required path effectively decouples backing from inclusion.
+
+### Legacy Candidate Selection 
+
+Legacy candidate selection takes place in the provisioner. Thus the provisioner needs to keep an up to date record of all [backed_candidates](../../types/backing.md#backed-candidate) `PerRelayParent` to pick from. 
+
+The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate appropriate to each free core.
+
+To determine availability:
+
+- Get the list of core states from the runtime API
+- For each core state:
+  - On `CoreState::Scheduled`, then we can make an `OccupiedCoreAssumption::Free`.
+  - On `CoreState::Occupied`, then we may be able to make an assumption:
+    - If the bitfields indicate availability and there is a scheduled `next_up_on_available`, then we can make an `OccupiedCoreAssumption::Included`.
+    - If the bitfields do not indicate availability, and there is a scheduled `next_up_on_time_out`, and `occupied_core.time_out_at == block_number_under_production`, then we can make an `OccupiedCoreAssumption::TimedOut`.
+  - If we did not make an `OccupiedCoreAssumption`, then continue on to the next core.
+  - Now compute the core's `validation_data_hash`: get the `PersistedValidationData` from the runtime, given the known `ParaId` and `OccupiedCoreAssumption`;
+  - Find an appropriate candidate for the core.
+    - There are two constraints: `backed_candidate.candidate.descriptor.para_id == scheduled_core.para_id && candidate.candidate.descriptor.validation_data_hash == computed_validation_data_hash`.
+    - In the event that more than one candidate meets the constraints, selection between the candidates is arbitrary. However, not more than one candidate can be selected per core.
+
+The end result of this process is a vector of `CandidateHash`s, sorted in order of their core index.
+
+### Retrieving Full `BackedCandidate`s for Selected Hashes
+
+Legacy candidate selection and prospective parachains candidate selection both leave us with a vector of `CandidateHash`s. These are passed to the backing subsystem with `CandidateBackingMessage::GetBackedCandidates`.
+
+The response is a vector of `BackedCandidate`s, sorted in order of their core index and ready to be provisioned to block authoring. The candidate selection and retrieval process should select at maximum one candidate which upgrades the runtime validation code.
+
+## Glossary
+
+- **Relay-parent:** 
+  - A particular relay-chain block which serves as an anchor and reference point for processes and data which depend on relay-chain state.
+- **Active Leaf:** 
+  - A relay chain block which is the head of an active fork of the relay chain. 
+  - Block authorship provisioning jobs are spawned per active leaf and concluded for any leaves which become inactive.
+- **Candidate Selection:** 
+  - The process by which the provisioner selects backable parachain block candidates to pass to block authoring.
+  - Two versions, prospective parachains candidate selection and legacy candidate selection. See their respective protocol sections for details.
+- **Availability Core:** 
+  - Often referred to simply as "cores", availability cores are an abstraction used for resource management. For the provisioner, availability cores are most relevant in that core states determine which `para_id`s to provision backable candidates for.
+  - For more on availability cores see [Scheduler Module: Availability Cores](../../runtime/scheduler.md#availability-cores)
+- **Availability Bitfield:**
+  - Often referred to simply as a "bitfield", an availability bitfield represents the view of parablock candidate availability from a particular validator's perspective. Each bit in the bitfield corresponds to a single [availability core](../../runtime-api/availability-cores.md).
+  - For more on availability bitfields see [availability](../../types/availability.md)
+- **Backable vs. Backed:**
+  - Note that we sometimes use "backed" to refer to candidates that are "backable", but not yet backed on chain.
+  - Backable means that a quorum of the candidate's assigned backing group have provided signed affirming statements.
@@ -2,7 +2,7 @@

 ## Motivation

-Parachains' and parathreads' validation function is described by a wasm module that we refer to as a PVF. Since a PVF is a wasm module the typical way of executing it is to compile it to machine code.
+Parachains' validation function is described by a wasm module that we refer to as a PVF. Since a PVF is a wasm module the typical way of executing it is to compile it to machine code.

 Typically an optimizing compiler consists of algorithms that are able to optimize the resulting machine code heavily. However, while those algorithms perform quite well for a typical wasm code produced by standard toolchains (e.g. rustc/LLVM), those algorithms can be abused to consume a lot of resources. Moreover, since those algorithms are rather complex there is a lot of room for a bug that can crash the compiler.

@@ -31,8 +31,8 @@ We also have an additional step where we attempt to instantiate the WASM runtime

 Pre-checking is run when a new validation code is included in the chain. A new PVF can be added in two cases:

- A new parachain or parathread is registered.
- An existing parachain or parathread signalled an upgrade of its validation code.
+- A new parachain is registered.
+- An existing parachain signalled an upgrade of its validation code.

 Before any of those operations finish, the PVF pre-checking vote is initiated. The PVF pre-checking vote is identified by the PVF code hash that is being voted on. If there is already PVF pre-checking process running, then no
 new PVF pre-checking vote will be started. Instead, the operation just subscribes to the existing vote.
@@ -58,7 +58,7 @@ On the node-side, there is a PVF pre-checking [subsystem][pvf-prechecker-subsyst

 ## Summary

-Parachains' and parathreads' validation function is described by a wasm module that we refer to as a PVF.
+Parachains' validation function is described by a wasm module that we refer to as a PVF.

 In order to make the PVF usable for candidate validation it has to be registered on-chain.

@@ -52,8 +52,8 @@ enum CoreState {
    /// If a particular Collator is required to author this block, that is also present in this
    /// variant.
    Scheduled(ScheduledCore),
-    /// The core is currently free and there is nothing scheduled. This can be the case for parathread
-    /// cores when there are no parathread blocks queued. Parachain cores will never be left idle.
+    /// The core is currently free and there is nothing scheduled. This can be the case for on-demand
+    /// cores when there are no on-demand parachain blocks queued. Leased cores will never be left idle.
    Free,
 }
 ```
@@ -6,7 +6,7 @@ Due to the (lack of) guarantees provided by a particular blockchain-runtime fram

 We also expect, although it's beyond the scope of this guide, that these runtime modules will exist alongside various other modules. This has two facets to consider. First, even if the modules that we describe here don't invoke each others' entry points or routines during initialization, we still have to protect against those other modules doing that. Second, some of those modules are expected to provide governance capabilities for the chain. Configuration exposed by parachain-host modules is mostly for the benefit of these governance modules, to allow the operators or community of the chain to tweak parameters.

-The runtime's primary roles to manage scheduling and updating of parachains and parathreads, as well as handling misbehavior reports and slashing. This guide doesn't focus on how parachains or parathreads are registered, only that they are. Also, this runtime description assumes that validator sets are selected somehow, but doesn't assume any other details than a periodic _session change_ event. Session changes give information about the incoming validator set and the validator set of the following session.
+The runtime's primary role is to manage scheduling and updating of parachains, as well as handling misbehavior reports and slashing. This guide doesn't focus on how parachains are registered, only that they are. Also, this runtime description assumes that validator sets are selected somehow, but doesn't assume any other details than a periodic _session change_ event. Session changes give information about the incoming validator set and the validator set of the following session.

 The runtime also serves another role, which is to make data available to the Node-side logic via Runtime APIs. These Runtime APIs should be sufficient for the Node-side code to author blocks correctly.

@@ -17,9 +17,9 @@ We will split the logic of the runtime up into these modules:
 * Initializer: manages initialization order of the other modules.
 * Shared: manages shared storage and configurations for other modules.
 * Configuration: manages configuration and configuration updates in a non-racy manner.
-* Paras: manages chain-head and validation code for parachains and parathreads.
-* Scheduler: manages parachain and parathread scheduling as well as validator assignments.
-* Inclusion: handles the inclusion and availability of scheduled parachains and parathreads.
+* Paras: manages chain-head and validation code for parachains.
+* Scheduler: manages parachain scheduling as well as validator assignments.
+* Inclusion: handles the inclusion and availability of scheduled parachains.
 * SessionInfo: manages various session keys of validators and other params stored per session.
 * Disputes: handles dispute resolution for included, available parablocks.
 * Slashing: handles slashing logic for concluded disputes.
@@ -1,4 +1,4 @@
-# Configuration Module
+# Configuration Pallet

 This module is responsible for managing all configuration of the parachain host in-flight. It provides a central point for configuration updates to prevent races between configuration changes and parachain-processing logic. Configuration can only change during the session change routine, and as this module handles the session change notification first it provides an invariant that the configuration does not change throughout the entire session. Both the [scheduler](scheduler.md) and [inclusion](inclusion.md) modules rely on this invariant to ensure proper behavior of the scheduler.

@@ -1,4 +1,4 @@
-# Disputes Module
+# Disputes Pallet

 After a backed candidate is made available, it is included and proceeds into an acceptance period during which validators are randomly selected to do (secondary) approval checks of the parablock. Any reports disputing the validity of the candidate will cause escalation, where even more validators are requested to check the block, and so on, until either the parablock is determined to be invalid or valid. Those on the wrong side of the dispute are slashed and, if the parablock is deemed invalid, the relay chain is rolled back to a point before that block was included.

@@ -1,4 +1,4 @@
-# DMP Module
+# DMP Pallet

 A module responsible for Downward Message Processing (DMP). See [Messaging Overview](../messaging.md) for more details.

@@ -27,9 +27,9 @@ No initialization routine runs for this module.

 Candidate Acceptance Function:

-* `check_processed_downward_messages(P: ParaId, processed_downward_messages: u32)`:
+* `check_processed_downward_messages(P: ParaId, relay_parent_number: BlockNumber, processed_downward_messages: u32)`:
+    1. Checks that `processed_downward_messages` is at least 1 if `DownwardMessageQueues` for `P` is not empty at the given `relay_parent_number`.
    1. Checks that `DownwardMessageQueues` for `P` is at least `processed_downward_messages` long.
-    1. Checks that `processed_downward_messages` is at least 1 if `DownwardMessageQueues` for `P` is not empty.

 Candidate Enactment:

@@ -1,4 +1,4 @@
-# HRMP Module
+# HRMP Pallet

 A module responsible for Horizontally Relay-routed Message Passing (HRMP). See [Messaging Overview](../messaging.md) for more details.

@@ -1,6 +1,6 @@
-# Inclusion Module
+# Inclusion Pallet

-The inclusion module is responsible for inclusion and availability of scheduled parachains and parathreads. It also manages the UMP dispatch queue of each parachain/thread.
+The inclusion module is responsible for inclusion and availability of scheduled parachains. It also manages the UMP dispatch queue of each parachain.

 ## Storage

@@ -61,9 +61,9 @@ No initialization routine runs for this module. However, the initialization of t
 All failed checks should lead to an unrecoverable error making the block invalid.

 * `process_bitfields(expected_bits, Bitfields, core_lookup: Fn(CoreIndex) -> Option<ParaId>)`:
-  1. call `sanitize_bitfields<true>` and use the sanitized `signed_bitfields` from now on.
-  1. call `sanitize_backed_candidates<true>` and use the sanitized `backed_candidates` from now on.
-  1. apply each bit of bitfield to the corresponding pending candidate. looking up parathread cores using the `core_lookup`. Disregard bitfields that have a `1` bit for any free cores.
+  1. Call `sanitize_bitfields<true>` and use the sanitized `signed_bitfields` from now on.
+  1. Call `sanitize_backed_candidates<true>` and use the sanitized `backed_candidates` from now on.
+  1. Apply each bit of bitfield to the corresponding pending candidate, looking up on-demand parachain cores using the `core_lookup`. Disregard bitfields that have a `1` bit for any free cores.
  1. For each applied bit of each availability-bitfield, set the bit for the validator in the `CandidatePendingAvailability`'s `availability_votes` bitfield. Track all candidates that now have >2/3 of bits set in their `availability_votes`. These candidates are now available and can be enacted.
  1. For all now-available candidates, invoke the `enact_candidate` routine with the candidate and relay-parent number.
  1. Return a list of `(CoreIndex, CandidateHash)` from freed cores consisting of the cores where candidates have become available.
@@ -84,26 +84,26 @@ All failed checks should lead to an unrecoverable error making the block invalid
  1. check that the validator bit index is not out of bounds.
  1. check the validators signature, iff `full_check=FullCheck::Yes`.

-* `sanitize_backed_candidates<T: crate::inclusion::Config, F: Fn(CandidateHash) -> bool>(
-    relay_parent: T::Hash,
+* `sanitize_backed_candidates<T: crate::inclusion::Config, F: FnMut(usize, &BackedCandidate<T::Hash>) -> bool>(
    mut backed_candidates: Vec<BackedCandidate<T::Hash>>,
    candidate_has_concluded_invalid_dispute: F,
    scheduled: &[CoreAssignment],
  ) `
  1. filter out any backed candidates that have concluded invalid.
-  1. filter out backed candidates that don't have a matching `relay_parent`.
  1. filters backed candidates whom's paraid was scheduled by means of the provided `scheduled` parameter.
+  1. sorts remaining candidates with respect to the core index assigned to them.

-* `process_candidates(parent_storage_root, BackedCandidates, scheduled: Vec<CoreAssignment>, group_validators: Fn(GroupIndex) -> Option<Vec<ValidatorIndex>>)`:
+* `process_candidates(allowed_relay_parents, BackedCandidates, scheduled: Vec<CoreAssignment>, group_validators: Fn(GroupIndex) -> Option<Vec<ValidatorIndex>>)`:
+    > For details on `AllowedRelayParentsTracker` see documentation for [Shared](./shared.md) module.
  1. check that each candidate corresponds to a scheduled core and that they are ordered in the same order the cores appear in assignments in `scheduled`.
  1. check that `scheduled` is sorted ascending by `CoreIndex`, without duplicates.
+  1. check that the relay-parent from each candidate receipt is one of the allowed relay-parents.
  1. check that there is no candidate pending availability for any scheduled `ParaId`.
-  1. check that each candidate's `validation_data_hash` corresponds to a `PersistedValidationData` computed from the current state.
-    > NOTE: With contextual execution in place, validation data will be obtained as of the state of the context block. However, only the state of the current block can be used for such a query.
+  1. check that each candidate's `validation_data_hash` corresponds to a `PersistedValidationData` computed from the state of the context block.
  1. If the core assignment includes a specific collator, ensure the backed candidate is issued by that collator.
  1. Ensure that any code upgrade scheduled by the candidate does not happen within `config.validation_upgrade_cooldown` of `Paras::last_code_upgrade(para_id, true)`, if any, comparing against the value of `Paras::FutureCodeUpgrades` for the given para ID.
  1. Check the collator's signature on the candidate data.
-  1. check the backing of the candidate using the signatures and the bitfields, comparing against the validators assigned to the groups, fetched with the `group_validators` lookup.
+  1. check the backing of the candidate using the signatures and the bitfields, comparing against the validators assigned to the groups, fetched with the `group_validators` lookup, while group indices are computed by `Scheduler` according to group rotation info. 
  1. call `check_upward_messages(config, para, commitments.upward_messages)` to check that the upward messages are valid.
  1. call `Dmp::check_processed_downward_messages(para, commitments.processed_downward_messages)` to check that the DMQ is properly drained.
  1. call `Hrmp::check_hrmp_watermark(para, commitments.hrmp_watermark)` for each candidate to check rules of processing the HRMP watermark.
@@ -1,4 +1,4 @@
-# Initializer Module
+# Initializer Pallet

 This module is responsible for initializing the other modules in a deterministic order. It also has one other purpose as described in the overview of the runtime: accepting and forwarding session change notifications.

@@ -1,7 +1,7 @@
-# Paras Module
+# Paras Pallet

-The Paras module is responsible for storing information on parachains and parathreads. Registered
-parachains and parathreads cannot change except at session boundaries and after at least a full
+The Paras module is responsible for storing information on parachains. Registered
+parachains cannot change except at session boundaries and after at least a full
 session has passed. This is primarily to ensure that the number and meaning of bits required for the
 availability bitfields does not change except at session boundaries.

@@ -54,15 +54,15 @@ struct ParaGenesisArgs {
 pub enum ParaLifecycle {
  /// A Para is new and is onboarding.
  Onboarding,
-  /// Para is a Parathread.
+  /// Para is a Parathread (on-demand parachain).
  Parathread,
-  /// Para is a Parachain.
+  /// Para is a lease holding Parachain.
  Parachain,
-  /// Para is a Parathread which is upgrading to a Parachain.
+  /// Para is a Parathread (on-demand Parachain) which is upgrading to a lease holding Parachain.
  UpgradingParathread,
-  /// Para is a Parachain which is downgrading to a Parathread.
+  /// Para is a lease holding Parachain which is downgrading to an on-demand parachain.
  DowngradingParachain,
-  /// Parathread is being offboarded.
+  /// Parathread (on-demand parachain) is being offboarded.
  OutgoingParathread,
  /// Parachain is being offboarded.
  OutgoingParachain,
@@ -102,11 +102,11 @@ struct PvfCheckActiveVoteState {

 #### Para Lifecycle

-Because the state changes of parachains and parathreads are delayed, we track the specific state of 
+Because the state changes of parachains are delayed, we track the specific state of 
 the para using the `ParaLifecycle` enum.

 ```
-None                 Parathread                  Parachain
+None         Parathread (on-demand parachain)    Parachain
 +                        +                          +
 |                        |                          |
 |   (≈2 Session Delay)   |                          |
@@ -148,12 +148,14 @@ use frame_system::pallet_prelude::BlockNumberFor;
 PvfActiveVoteMap: map ValidationCodeHash => PvfCheckActiveVoteState;
 /// The list of all currently active PVF votes. Auxiliary to `PvfActiveVoteMap`.
 PvfActiveVoteList: Vec<ValidationCodeHash>;
-/// All parachains. Ordered ascending by ParaId. Parathreads are not included.
+/// All parachains. Ordered ascending by ParaId. On-demand parachains are not included.
 Parachains: Vec<ParaId>,
 /// The current lifecycle state of all known Para Ids.
 ParaLifecycle: map ParaId => Option<ParaLifecycle>,
 /// The head-data of every registered para.
 Heads: map ParaId => Option<HeadData>;
+/// The context (relay-chain block number) of the most recent parachain head.
+MostRecentContext: map ParaId => BlockNumber;
 /// The validation code hash of every live para.
 CurrentCodeHash: map ParaId => Option<ValidationCodeHash>;
 /// Actual past code hash, indicated by the para id as well as the block number at which it became outdated.
@@ -221,19 +223,17 @@ CodeByHash: map ValidationCodeHash => Option<ValidationCode>

 1. Execute all queued actions for paralifecycle changes:
  1. Clean up outgoing paras.
-     1. This means removing the entries under `Heads`, `CurrentCode`, `FutureCodeUpgrades`, and
-        `FutureCode`. An according entry should be added to `PastCode`, `PastCodeMeta`, and
-        `PastCodePruning` using the outgoing `ParaId` and removed `CurrentCode` value. This is
-        because any outdated validation code must remain available on-chain for a determined amount
+     1. This means removing the entries under `Heads`, `CurrentCode`, `FutureCodeUpgrades`,
+        `FutureCode` and `MostRecentContext`. An according entry should be added to `PastCode`, `PastCodeMeta`, and `PastCodePruning` using the outgoing `ParaId` and removed `CurrentCode` value. This is because any outdated validation code must remain available on-chain for a determined amount
        of blocks, and validation code outdated by de-registering the para is still subject to that
        invariant.
  1. Apply all incoming paras by initializing the `Heads` and `CurrentCode` using the genesis
-     parameters.
+     parameters as well as `MostRecentContext` to `0`.
  1. Amend the `Parachains` list and `ParaLifecycle` to reflect changes in registered parachains.
-  1. Amend the `ParaLifecycle` set to reflect changes in registered parathreads.
-  1. Upgrade all parathreads that should become parachains, updating the `Parachains` list and
+  1. Amend the `ParaLifecycle` set to reflect changes in registered on-demand parachains.
+  1. Upgrade all on-demand parachains that should become lease holding parachains, updating the `Parachains` list and
     `ParaLifecycle`.
-  1. Downgrade all parachains that should become parathreads, updating the `Parachains` list and
+  1. Downgrade all lease holding parachains that should become on-demand parachains, updating the `Parachains` list and
     `ParaLifecycle`.
  1. (Deferred) Return list of outgoing paras to the initializer for use by other modules.
 1. Go over all active PVF pre-checking votes:
@@ -255,22 +255,21 @@ CodeByHash: map ValidationCodeHash => Option<ValidationCode>
 * `schedule_para_initialize(ParaId, ParaGenesisArgs)`: Schedule a para to be initialized at the next
  session. Noop if para is already registered in the system with some `ParaLifecycle`.
 * `schedule_para_cleanup(ParaId)`: Schedule a para to be cleaned up after the next full session.
-* `schedule_parathread_upgrade(ParaId)`: Schedule a parathread to be upgraded to a parachain.
-* `schedule_parachain_downgrade(ParaId)`: Schedule a parachain to be downgraded to a parathread.
+* `schedule_parathread_upgrade(ParaId)`: Schedule a parathread (on-demand parachain) to be upgraded to a parachain.
+* `schedule_parachain_downgrade(ParaId)`: Schedule a parachain to be downgraded from lease holding to on-demand.
 * `schedule_code_upgrade(ParaId, new_code, relay_parent: BlockNumber, HostConfiguration)`: Schedule a future code
  upgrade of the given parachain. In case the PVF pre-checking is disabled, or the new code is already present in the storage, the upgrade will be applied after inclusion of a block of the same parachain
  executed in the context of a relay-chain block with number >= `relay_parent + config.validation_upgrade_delay`. If the upgrade is scheduled `UpgradeRestrictionSignal` is set and it will remain set until `relay_parent + config.validation_upgrade_cooldown`.
 In case the PVF pre-checking is enabled, or the new code is not already present in the storage, then the PVF pre-checking run will be scheduled for that validation code. If the pre-checking concludes with rejection, then the upgrade is canceled. Otherwise, after pre-checking is concluded the upgrade will be scheduled and be enacted as described above.
 * `note_new_head(ParaId, HeadData, BlockNumber)`: note that a para has progressed to a new head,
-  where the new head was executed in the context of a relay-chain block with given number. This will
-  apply pending code upgrades based on the block number provided. If an upgrade took place it will clear the `UpgradeGoAheadSignal`.
+  where the new head was executed in the context of a relay-chain block with given number, the latter value is inserted into the `MostRecentContext` mapping. This will apply pending code upgrades based on the block number provided. If an upgrade took place it will clear the `UpgradeGoAheadSignal`.
 * `lifecycle(ParaId) -> Option<ParaLifecycle>`: Return the `ParaLifecycle` of a para.
-* `is_parachain(ParaId) -> bool`: Returns true if the para ID references any live parachain,
-  including those which may be transitioning to a parathread in the future.
-* `is_parathread(ParaId) -> bool`: Returns true if the para ID references any live parathread,
-  including those which may be transitioning to a parachain in the future.
-* `is_valid_para(ParaId) -> bool`: Returns true if the para ID references either a live parathread
-  or live parachain.
+* `is_parachain(ParaId) -> bool`: Returns true if the para ID references any live lease holding parachain,
+  including those which may be transitioning to an on-demand parachain in the future.
+* `is_parathread(ParaId) -> bool`: Returns true if the para ID references any live parathread (on-demand parachain),
+  including those which may be transitioning to a lease holding parachain in the future.
+* `is_valid_para(ParaId) -> bool`: Returns true if the para ID references either a live on-demand parachain
+  or live lease holding parachain.
 * `can_upgrade_validation_code(ParaId) -> bool`: Returns true if the given para can signal code upgrade right now.
 * `pvfs_require_prechecking() -> Vec<ValidationCodeHash>`: Returns the list of PVF validation code hashes that require PVF pre-checking votes.

@@ -1,24 +1,24 @@
-# Scheduler Module
+# Scheduler Pallet

 > TODO: this section is still heavily under construction. key questions about availability cores and validator assignment are still open and the flow of the the section may be contradictory or inconsistent

 The Scheduler module is responsible for two main tasks:

- Partitioning validators into groups and assigning groups to parachains and parathreads.
- Scheduling parachains and parathreads
+- Partitioning validators into groups and assigning groups to parachains.
+- Scheduling parachains for each block

 It aims to achieve these tasks with these goals in mind:

 - It should be possible to know at least a block ahead-of-time, ideally more, which validators are going to be assigned to which parachains.
 - Parachains that have a candidate pending availability in this fork of the chain should not be assigned.
 - Validator assignments should not be gameable. Malicious cartels should not be able to manipulate the scheduler to assign themselves as desired.
- High or close to optimal throughput of parachains and parathreads. Work among validator groups should be balanced.
+- High or close to optimal throughput of parachains. Work among validator groups should be balanced.

 ## Availability Cores

-The Scheduler manages resource allocation using the concept of "Availability Cores". There will be one availability core for each parachain, and a fixed number of cores used for multiplexing parathreads. Validators will be partitioned into groups, with the same number of groups as availability cores. Validator groups will be assigned to different availability cores over time.
+The Scheduler manages resource allocation using the concept of "Availability Cores". There will be one availability core for each lease holding parachain, and a fixed number of cores used for multiplexing on-demand parachains. Validators will be partitioned into groups, with the same number of groups as availability cores. Validator groups will be assigned to different availability cores over time.

-An availability core can exist in either one of two states at the beginning or end of a block: free or occupied. A free availability core can have a parachain or parathread assigned to it for the potential to have a backed candidate included. After backing, the core enters the occupied state as the backed candidate is pending availability. There is an important distinction: a core is not considered occupied until it is in charge of a block pending availability, although the implementation may treat scheduled cores the same as occupied ones for brevity. A core exits the occupied state when the candidate is no longer pending availability - either on timeout or on availability. A core starting in the occupied state can move to the free state and back to occupied all within a single block, as availability bitfields are processed before backed candidates. At the end of the block, there is a possible timeout on availability which can move the core back to the free state if occupied.
+An availability core can exist in either one of two states at the beginning or end of a block: free or occupied. A free availability core can have a lease holding or on-demand parachain assigned to it for the potential to have a backed candidate included. After backing, the core enters the occupied state as the backed candidate is pending availability. There is an important distinction: a core is not considered occupied until it is in charge of a block pending availability, although the implementation may treat scheduled cores the same as occupied ones for brevity. A core exits the occupied state when the candidate is no longer pending availability - either on timeout or on availability. A core starting in the occupied state can move to the free state and back to occupied all within a single block, as availability bitfields are processed before backed candidates. At the end of the block, there is a possible timeout on availability which can move the core back to the free state if occupied.

 Cores are treated as an ordered list and are typically referred to by their index in that list.

@@ -82,54 +82,57 @@ digraph {

 ## Validator Groups

-Validator group assignments do not need to change very quickly. The security benefits of fast rotation are redundant with the challenge mechanism in the [Approval process](../protocol-approval.md). Because of this, we only divide validators into groups at the beginning of the session and do not shuffle membership during the session. However, we do take steps to ensure that no particular validator group has dominance over a single parachain or parathread-multiplexer for an entire session to provide better guarantees of live-ness.
+Validator group assignments do not need to change very quickly. The security benefits of fast rotation are redundant with the challenge mechanism in the [Approval process](../protocol-approval.md). Because of this, we only divide validators into groups at the beginning of the session and do not shuffle membership during the session. However, we do take steps to ensure that no particular validator group has dominance over a single lease holding parachain or on-demand parachain-multiplexer for an entire session to provide better guarantees of live-ness.

 Validator groups rotate across availability cores in a round-robin fashion, with rotation occurring at fixed intervals. The i'th group will be assigned to the `(i+k)%n`'th core at any point in time, where `k` is the number of rotations that have occurred in the session, and `n` is the number of cores. This makes upcoming rotations within the same session predictable.

-When a rotation occurs, validator groups are still responsible for distributing availability chunks for any previous cores that are still occupied and pending availability. In practice, rotation and availability-timeout frequencies should be set so this will only be the core they have just been rotated from. It is possible that a validator group is rotated onto a core which is currently occupied. In this case, the validator group will have nothing to do until the previously-assigned group finishes their availability work and frees the core or the availability process times out. Depending on if the core is for a parachain or parathread, a different timeout `t` from the [`HostConfiguration`](../types/runtime.md#host-configuration) will apply. Availability timeouts should only be triggered in the first `t-1` blocks after the beginning of a rotation.
+When a rotation occurs, validator groups are still responsible for distributing availability chunks for any previous cores that are still occupied and pending availability. In practice, rotation and availability-timeout frequencies should be set so this will only be the core they have just been rotated from. It is possible that a validator group is rotated onto a core which is currently occupied. In this case, the validator group will have nothing to do until the previously-assigned group finishes their availability work and frees the core or the availability process times out. Depending on if the core is for a lease holding parachain or on-demand parachain, a different timeout `t` from the [`HostConfiguration`](../types/runtime.md#host-configuration) will apply. Availability timeouts should only be triggered in the first `t-1` blocks after the beginning of a rotation.

 ## Claims

-Parathreads operate on a system of claims. Collators participate in auctions to stake a claim on authoring the next block of a parathread, although the auction mechanism is beyond the scope of the scheduler. The scheduler guarantees that they'll be given at least a certain number of attempts to author a candidate that is backed. Attempts that fail during the availability phase are not counted, since ensuring availability at that stage is the responsibility of the backing validators, not of the collator. When a claim is accepted, it is placed into a queue of claims, and each claim is assigned to a particular parathread-multiplexing core in advance. Given that the current assignments of validator groups to cores are known, and the upcoming assignments are predictable, it is possible for parathread collators to know who they should be talking to now and how they should begin establishing connections with as a fallback.
+On-demand parachains operate on a system of claims. Collators purchase claims on authoring the next block of an on-demand parachain, although the purchase mechanism is beyond the scope of the scheduler. The scheduler guarantees that they'll be given at least a certain number of attempts to author a candidate that is backed. Attempts that fail during the availability phase are not counted, since ensuring availability at that stage is the responsibility of the backing validators, not of the collator. When a claim is accepted, it is placed into a queue of claims, and each claim is assigned to a particular on-demand parachain-multiplexing core in advance. Given that the current assignments of validator groups to cores are known, and the upcoming assignments are predictable, it is possible for on-demand parachain collators to know who they should be talking to now and how they should begin establishing connections with as a fallback.

-With this information, the Node-side can be aware of which parathreads have a good chance of being includable within the relay-chain block and can focus any additional resources on backing candidates from those parathreads. Furthermore, Node-side code is aware of which validator group will be responsible for that thread. If the necessary conditions are reached for core reassignment, those candidates can be backed within the same block as the core being freed.
+With this information, the Node-side can be aware of which on-demand parachains have a good chance of being includable within the relay-chain block and can focus any additional resources on backing candidates from those on-demand parachains. Furthermore, Node-side code is aware of which validator group will be responsible for that thread. If the necessary conditions are reached for core reassignment, those candidates can be backed within the same block as the core being freed.

-Parathread claims, when scheduled onto a free core, may not result in a block pending availability. This may be due to collator error, networking timeout, or censorship by the validator group. In this case, the claims should be retried a certain number of times to give the collator a fair shot.
+On-demand claims, when scheduled onto a free core, may not result in a block pending availability. This may be due to collator error, networking timeout, or censorship by the validator group. In this case, the claims should be retried a certain number of times to give the collator a fair shot.

 ## Storage

 Utility structs:

 ```rust
-// A claim on authoring the next block for a given parathread.
+// A claim on authoring the next block for a given parathread (on-demand parachain).
 struct ParathreadClaim(ParaId, CollatorId);

-// An entry tracking a claim to ensure it does not pass the maximum number of retries.
+// An entry tracking a parathread (on-demand parachain) claim to ensure it does not 
+// pass the maximum number of retries.
 struct ParathreadEntry {
  claim: ParathreadClaim,
  retries: u32,
 }

-// A queued parathread entry, pre-assigned to a core.
+// A queued parathread (on-demand parachain) entry, pre-assigned to a core.
 struct QueuedParathread {
  claim: ParathreadEntry,
-  /// offset within the set of para-threads ranged `0..config.parathread_cores`.
+  /// offset within the set of parathreads (on-demand parachains) ranged `0..config.parathread_cores`.
  core_offset: u32,
 }

 struct ParathreadQueue {
  queue: Vec<QueuedParathread>,
-  /// offset within the set of para-threads ranged `0..config.parathread_cores`.
+  /// offset within the set of parathreads (on-demand parachains) ranged `0..config.parathread_cores`.
  next_core_offset: u32,
 }

 enum CoreOccupied {
+  // On-demand parachain
  Parathread(ParathreadEntry), // claim & retries
  Parachain,
 }

 enum AssignmentKind {
  Parachain,
+  // On-demand parachain
  Parathread(CollatorId, u32),
 }

@@ -137,7 +140,6 @@ struct CoreAssignment {
  core: CoreIndex,
  para_id: ParaId,
  kind: AssignmentKind,
-  group_idx: GroupIndex,
 }
 // reasons a core might be freed.
 enum FreedReason {
@@ -151,13 +153,13 @@ Storage layout:
 ```rust
 /// All the validator groups. One for each core. Indices are into the `ActiveValidators` storage.
 ValidatorGroups: Vec<Vec<ValidatorIndex>>;
-/// A queue of upcoming claims and which core they should be mapped onto.
+/// A queue of upcoming parathread (on-demand parachain) claims and which core they should be mapped onto.
 ParathreadQueue: ParathreadQueue;
 /// One entry for each availability core. Entries are `None` if the core is not currently occupied.
-/// The i'th parachain belongs to the i'th core, with the remaining cores all being
-/// parathread-multiplexers.
+/// The i'th parachain lease belongs to the i'th core, with the remaining cores all being
+/// on-demand parachain-multiplexers.
 AvailabilityCores: Vec<Option<CoreOccupied>>;
-/// An index used to ensure that only one claim on a parathread exists in the queue or is
+/// An index used to ensure that only one claim on a parathread (on-demand parachain) exists in the queue or is
 /// currently being handled by an occupied core.
 ParathreadClaimIndex: Vec<ParaId>;
 /// The block number where the session start occurred. Used to track how many group rotations have occurred.
@@ -187,11 +189,11 @@ Actions:
   - Note that the total number of validators `V` in AV may not be evenly divided by `n_cores`.
   - The groups are selected by partitioning AV.  The first `V % N` groups will have `(V / n_cores) + 1` members, while the remaining groups will have `(V / N)` members each.
   - Instead of using the indices within AV, which point to the broader set, indices _into_ AV should be used. This implies that groups should have simply ascending validator indices.
-1. Prune the parathread queue to remove all retries beyond `configuration.parathread_retries`.
-   - Also prune all parathread claims corresponding to de-registered parathreads.
-   - all pruned claims should have their entry removed from the parathread index.
-   - assign all non-pruned claims to new cores if the number of parathread cores has changed between the `new_config` and `old_config` of the `SessionChangeNotification`.
-   - Assign claims in equal balance across all cores if rebalancing, and set the `next_core` of the `ParathreadQueue` by incrementing the relative index of the last assigned core and taking it modulo the number of parathread cores. 
+1. Prune the parathread (on-demand parachain) queue to remove all retries beyond `configuration.parathread_retries`.
+   - Also prune all on-demand claims corresponding to de-registered parachains.
+   - all pruned claims should have their entry removed from the parathread (on-demand parachain) index.
+   - assign all non-pruned claims to new cores if the number of on-demand parachain cores has changed between the `new_config` and `old_config` of the `SessionChangeNotification`.
+   - Assign claims in equal balance across all cores if rebalancing, and set the `next_core` of the `ParathreadQueue` (on-demand queue) by incrementing the relative index of the last assigned core and taking it modulo the number of on-demand cores. 

 ## Initialization

@@ -203,17 +205,17 @@ No finalization routine runs for this module.

 ## Routines

- `add_parathread_claim(ParathreadClaim)`: Add a parathread claim to the queue.
-  - Fails if any parathread claim on the same parathread is currently indexed.
+- `add_parathread_claim(ParathreadClaim)`: Add a parathread (on-demand parachain) claim to the queue.
+  - Fails if any on-demand claim on the same parachain is currently indexed.
  - Fails if the queue length is >= `config.scheduling_lookahead * config.parathread_cores`.
-  - The core used for the parathread claim is the `next_core` field of the `ParathreadQueue` and adding `Paras::parachains().len()` to it.
+  - The core used for the on-demand claim is the `next_core` field of the `ParathreadQueue` (on-demand queue) and adding `Paras::parachains().len()` to it.
  - `next_core` is then updated by adding 1 and taking it modulo `config.parathread_cores`.
  - The claim is then added to the claim index.
 - `free_cores(Vec<(CoreIndex, FreedReason)>)`: indicate previosuly-occupied cores which are to be considered returned and why they are being returned.
-  - All freed parachain cores should be assigned to their respective parachain
-  - All freed parathread cores whose reason for freeing was `FreedReason::Concluded` should have the claim removed from the claim index.
-  - All freed parathread cores whose reason for freeing was `FreedReason::TimedOut` should have the claim added to the parathread queue again without retries incremented
-  - All freed parathread cores should take the next parathread entry from the queue.
+  - All freed lease holding parachain cores should be assigned to their respective parachain
+  - All freed on-demand parachain cores whose reason for freeing was `FreedReason::Concluded` should have the claim removed from the claim index.
+  - All freed on-demand cores whose reason for freeing was `FreedReason::TimedOut` should have the claim added to the parathread queue (on-demand queue) again without retries incremented
+  - All freed on-demand cores should take the next on-demand parachain entry from the queue.
 - `schedule(Vec<(CoreIndex, FreedReason)>, now: BlockNumber)`: schedule new core assignments, with a parameter indicating previously-occupied cores which are to be considered returned and why they are being returned.
  - Invoke `free_cores(freed_cores)`
  - The i'th validator group will be assigned to the `(i+k)%n`'th core at any point in time, where `k` is the number of rotations that have occurred in the session, and `n` is the total number of cores. This makes upcoming rotations within the same session predictable. Rotations are based off of `now`.
@@ -225,9 +227,9 @@ No finalization routine runs for this module.
  - Since both the availability cores and the newly-occupied cores lists are sorted ascending, this method can be implemented efficiently.
 - `core_para(CoreIndex) -> ParaId`: return the currently-scheduled or occupied ParaId for the given core.
 - `group_validators(GroupIndex) -> Option<Vec<ValidatorIndex>>`: return all validators in a given group, if the group index is valid for this session.
- `availability_timeout_predicate() -> Option<impl Fn(CoreIndex, BlockNumber) -> bool>`: returns an optional predicate that should be used for timing out occupied cores. if `None`, no timing-out should be done. The predicate accepts the index of the core, and the block number since which it has been occupied. The predicate should be implemented based on the time since the last validator group rotation, and the respective parachain and parathread timeouts, i.e. only within `max(config.chain_availability_period, config.thread_availability_period)` of the last rotation would this return `Some`.
+- `availability_timeout_predicate() -> Option<impl Fn(CoreIndex, BlockNumber) -> bool>`: returns an optional predicate that should be used for timing out occupied cores. if `None`, no timing-out should be done. The predicate accepts the index of the core, and the block number since which it has been occupied. The predicate should be implemented based on the time since the last validator group rotation, and the respective parachain timeouts, i.e. only within `max(config.chain_availability_period, config.thread_availability_period)` of the last rotation would this return `Some`.
 - `group_rotation_info(now: BlockNumber) -> GroupRotationInfo`: Returns a helper for determining group rotation.
- `next_up_on_available(CoreIndex) -> Option<ScheduledCore>`: Return the next thing that will be scheduled on this core assuming it is currently occupied and the candidate occupying it became available. Returns in `ScheduledCore` format (todo: link to Runtime APIs page; linkcheck doesn't allow this right now). For parachains, this is always the ID of the parachain and no specified collator. For parathreads, this is based on the next item in the `ParathreadQueue` assigned to that core, and is `None` if there isn't one.
- `next_up_on_time_out(CoreIndex) -> Option<ScheduledCore>`: Return the next thing that will be scheduled on this core assuming it is currently occupied and the candidate occupying it timed out. Returns in `ScheduledCore` format (todo: link to Runtime APIs page; linkcheck doesn't allow this right now). For parachains, this is always the ID of the parachain and no specified collator. For parathreads, this is based on the next item in the `ParathreadQueue` assigned to that core, or if there isn't one, the claim that is currently occupying the core. Otherwise `None`.
+- `next_up_on_available(CoreIndex) -> Option<ScheduledCore>`: Return the next thing that will be scheduled on this core assuming it is currently occupied and the candidate occupying it became available. Returns in `ScheduledCore` format (todo: link to Runtime APIs page; linkcheck doesn't allow this right now). For lease holding parachains, this is always the ID of the parachain and no specified collator. For on-demand parachains, this is based on the next item in the `ParathreadQueue` (on-demand queue) assigned to that core, and is `None` if there isn't one.
+- `next_up_on_time_out(CoreIndex) -> Option<ScheduledCore>`: Return the next thing that will be scheduled on this core assuming it is currently occupied and the candidate occupying it timed out. Returns in `ScheduledCore` format (todo: link to Runtime APIs page; linkcheck doesn't allow this right now). For parachains, this is always the ID of the parachain and no specified collator. For on-demand parachains, this is based on the next item in the `ParathreadQueue` (on-demand queue) assigned to that core, or if there isn't one, the claim that is currently occupying the core. Otherwise `None`.
 - `clear()`:
-  - Free all scheduled cores and return parathread claims to queue, with retries incremented. Skip parathreads which no longer exist under paras.
+  - Free all scheduled cores and return on-demand claims to queue, with retries incremented. Skip on-demand parachains which no longer exist under paras.
@@ -1,4 +1,4 @@
-# Shared Module
+# Shared Pallet

 This module is responsible for managing shared storage and configuration for other modules.

@@ -19,6 +19,27 @@ pub(crate) const SESSION_DELAY: SessionIndex = 2;

 ## Storage

+Helper structs:
+
+```rust
+struct AllowedRelayParentsTracker<Hash, BlockNumber> {
+  // The past relay parents, paired with state roots, that are viable to build upon.
+  //
+  // They are in ascending chronologic order, so the newest relay parents are at
+  // the back of the deque.
+  //
+  // (relay_parent, state_root)
+  //
+  // NOTE: the size limit of look-back is currently defined as a constant in Runtime.
+  buffer: VecDeque<(Hash, Hash)>,
+
+  // The number of the most recent relay-parent, if any.
+  latest_number: BlockNumber,
+}
+```
+
+Storage Layout:
+
 ```rust
 /// The current session index within the Parachains Runtime system.
 CurrentSessionIndex: SessionIndex;
@@ -28,6 +49,8 @@ ActiveValidatorIndices: Vec<ValidatorIndex>,
 /// The parachain attestation keys of the validators actively participating in parachain consensus.
 /// This should be the same length as `ActiveValidatorIndices`.
 ActiveValidatorKeys: Vec<ValidatorId>
+/// Relay-parents allowed to build candidates upon.
+AllowedRelayParents: AllowedRelayParentsTracker<T::Hash, T::BlockNumber>,
 ```

 ## Initialization
@@ -51,6 +74,8 @@ This information is used in the:
  passed.
 * Paras Module: For delaying updates to paras until at least one full session has passed.

+Allowed relay parents buffer, which is maintained by [ParaInherent](./parainherent.md) module, is cleared on every session change.
+
 ## Finalization

 The Shared Module currently has no finalization routines.
@@ -1,7 +1,7 @@
 # Candidate Types

 Para candidates are some of the most common types, both within the runtime and on the Node-side.
-Candidates are the fundamental datatype for advancing parachains and parathreads, encapsulating the collator's signature, the context of the parablock, the commitments to the output, and a commitment to the data which proves it valid.
+Candidates are the fundamental datatype for advancing parachains, encapsulating the collator's signature, the context of the parablock, the commitments to the output, and a commitment to the data which proves it valid.

 In a way, this entire guide is about these candidates: how they are scheduled, constructed, backed, included, and challenged.

@@ -142,7 +142,7 @@ struct PersistedValidationData {

 ## `HeadData`

-Head data is a type-safe abstraction around bytes (`Vec<u8>`) for the purposes of representing heads of parachains or parathreads.
+Head data is a type-safe abstraction around bytes (`Vec<u8>`) for the purposes of representing heads of parachains.

 ```rust
 struct HeadData(Vec<u8>);
@@ -150,7 +150,7 @@ struct HeadData(Vec<u8>);

 ## Candidate Commitments

-The execution and validation of parachain or parathread candidates produces a number of values which either must be committed to on the relay chain or committed to the state of the relay chain.
+The execution and validation of parachain candidates produces a number of values which either must be committed to blocks on the relay chain or committed to the state of the relay chain.

 ```rust
 /// Commitments made in a `CandidateReceipt`. Many of these are outputs of validation.
@@ -345,9 +345,10 @@ enum BitfieldSigningMessage { }

 ```rust
 enum CandidateBackingMessage {
-  /// Requests a set of backable candidates that could be backed in a child of the given
-  /// relay-parent, referenced by its hash.
-  GetBackedCandidates(Hash, Vec<CandidateHash>, ResponseChannel<Vec<BackedCandidate>>),
+  /// Requests a set of backable candidates attested by the subsystem.
+  ///
+  /// Each pair is (candidate_hash, candidate_relay_parent).
+  GetBackedCandidates(Vec<(CandidateHash, Hash)>, oneshot::Sender<Vec<BackedCandidate>>),
  /// Note that the Candidate Backing subsystem should second the given candidate in the context of the
  /// given relay-parent (ref. by hash). This candidate must be validated using the provided PoV.
  /// The PoV is expected to match the `pov_hash` in the descriptor.
@@ -447,6 +448,57 @@ enum CollatorProtocolMessage {
 }
 ```

+## Collation Generation Message
+
+Messages received by the [Collation Generation subsystem](../node/collators/collation-generation.md)
+
+This is the core interface by which collators built on top of a Polkadot node submit collations to validators. As such, these messages are not sent by any subsystem but are instead sent from outside of the overseer.
+
+```rust
+/// A function provided to the subsystem which it uses to pull new collations.
+///
+/// This mode of querying collations is obsoleted by `CollationGenerationMessages::SubmitCollation`
+///
+/// The response channel, if present, is meant to receive a `Seconded` statement as a
+/// form of authentication, for collation mechanisms which rely on this for anti-spam.
+type CollatorFn = Fn(Hash, PersistedValidationData) -> Future<Output = (Collation, Option<ResponseChannel<SignedStatement>>)>;
+
+/// Configuration for the collation generator
+struct CollationGenerationConfig {
+    /// Collator's authentication key, so it can sign things.
+    key: CollatorPair,
+    /// Collation function. See [`CollatorFn`] for more details.
+    collator: CollatorFn,
+    /// The parachain that this collator collates for
+    para_id: ParaId,
+}
+
+/// Parameters for submitting a collation
+struct SubmitCollationParams {
+    /// The relay-parent the collation is built against.
+    relay_parent: Hash,
+    /// The collation itself (PoV and commitments)
+    collation: Collation,
+    /// The parent block's head-data.
+    parent_head: HeadData,
+    /// The hash of the validation code the collation was created against.
+    validation_code_hash: ValidationCodeHash,
+    /// A response channel for receiving a `Seconded` message about the candidate
+    /// once produced by a validator. This is not guaranteed to provide anything.
+    result_sender: Option<ResponseChannel<SignedStatement>>,
+}
+
+enum CollationGenerationMessage {
+    /// Initialize the collation generation subsystem
+	Initialize(CollationGenerationConfig),
+    /// Submit a collation to the subsystem. This will package it into a signed
+    /// [`CommittedCandidateReceipt`] and distribute along the network to validators.
+    ///
+    /// If sent before `Initialize`, this will be ignored.
+    SubmitCollation(SubmitCollationParams),
+}
+```
+
 ## Dispute Coordinator Message

 Messages received by the [Dispute Coordinator subsystem](../node/disputes/dispute-coordinator.md)
@@ -797,7 +849,7 @@ enum StatementDistributionMessage {
    ///
    /// The statement distribution subsystem assumes that the statement should be correctly
    /// signed.
-    Share(Hash, SignedFullStatement),
+    Share(Hash, SignedFullStatementWithPVD),
 }
 ```

@@ -19,9 +19,9 @@ struct HostConfiguration {
 	pub max_code_size: u32,
 	/// The maximum head-data size, in bytes.
 	pub max_head_data_size: u32,
-	/// The amount of availability cores to dedicate to parathreads.
+	/// The amount of availability cores to dedicate to parathreads (on-demand parachains).
 	pub parathread_cores: u32,
-	/// The number of retries that a parathread author has to submit their block.
+	/// The number of retries that a parathread (on-demand parachain) author has to submit their block.
 	pub parathread_retries: u32,
 	/// How often parachain groups should be rotated across parachains.
 	pub group_rotation_frequency: BlockNumber,
@@ -29,10 +29,10 @@ struct HostConfiguration {
 	/// after inclusion that validators have to make the block available and signal its availability to
 	/// the chain. Must be at least 1.
 	pub chain_availability_period: BlockNumber,
-	/// The availability period, in blocks, for parathreads. Same as the `chain_availability_period`,
+	/// The availability period, in blocks, for parathreads (on-demand parachains). Same as the `chain_availability_period`,
 	/// but a differing timeout due to differing requirements. Must be at least 1.
 	pub thread_availability_period: BlockNumber,
-	/// The amount of blocks ahead to schedule parathreads.
+	/// The amount of blocks ahead to schedule on-demand parachains.
 	pub scheduling_lookahead: u32,
 	/// The maximum number of validators to have per core. `None` means no maximum.
 	pub max_validators_per_core: Option<u32>,
@@ -88,7 +88,7 @@ struct HostConfiguration {
 	pub hrmp_channel_max_total_size: u32,
 	/// The maximum number of inbound HRMP channels a parachain is allowed to accept.
 	pub hrmp_max_parachain_inbound_channels: u32,
-	/// The maximum number of inbound HRMP channels a parathread is allowed to accept.
+	/// The maximum number of inbound HRMP channels a parathread (on-demand parachain) is allowed to accept.
 	pub hrmp_max_parathread_inbound_channels: u32,
 	/// The maximum size of a message that could ever be put into an HRMP channel.
 	///
@@ -96,7 +96,7 @@ struct HostConfiguration {
 	pub hrmp_channel_max_message_size: u32,
 	/// The maximum number of outbound HRMP channels a parachain is allowed to open.
 	pub hrmp_max_parachain_outbound_channels: u32,
-	/// The maximum number of outbound HRMP channels a parathread is allowed to open.
+	/// The maximum number of outbound HRMP channels a parathread (on-demand parachain) is allowed to open.
 	pub hrmp_max_parathread_outbound_channels: u32,
 	/// The maximum number of outbound HRMP messages can be sent by a candidate.
 	///