Asynchronous Backing MegaPR (#5022)

* inclusion emulator logic for asynchronous backing (#4790) * initial stab at candidate_context * fmt * docs & more TODOs * some cleanups * reframe as inclusion_emulator * documentations yes * update types * add constraint modifications * watermark * produce modifications * v2 primitives: re-export all v1 for consistency * vstaging primitives * emulator constraints: handle code upgrades * produce outbound HRMP modifications * stack. * method for applying modifications * method just for sanity-checking modifications * fragments produce modifications, not prospectives * make linear * add some TODOs * remove stacking; handle code upgrades * take `fragment` private * reintroduce stacking. * fragment constructor * add TODO * allow validating fragments against future constraints * docs * relay-parent number and min code size checks * check code upgrade restriction * check max hrmp per candidate * fmt * remove GoAhead logic because it wasn't helpful * docs on code upgrade failure * test stacking * test modifications against constraints * fmt * test fragments * descending or duplicate test * fmt * remove unused imports in vstaging * wrong primitives * spellcheck * Runtime changes for Asynchronous Backing (#4786) * inclusion: utility for allowed relay-parents * inclusion: use prev number instead of prev hash * track most recent context of paras * inclusion: accept previous relay-parents * update dmp advancement rule for async backing * fmt * add a comment about validation outputs * clean up a couple of TODOs * weights * fix weights * fmt * Resolve dmp todo * Restore inclusion tests * Restore paras_inherent tests * MostRecentContext test * Benchmark for new paras dispatchable * Prepare check_validation_outputs for upgrade * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs * Implementers guide changes * More tests for allowed relay parents * Add a github issue link * Compute group index based on relay parent * Storage migration * Move allowed parents tracker to shared * Compile error * Get group assigned to core at the next block * Test group assignment * fmt * Error instead of panic * Update guide * Extend doc-comment * Update runtime/parachains/src/shared.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Prospective Parachains Subsystem (#4913) * docs and skeleton * subsystem skeleton * main loop * fragment tree basics & fmt * begin fragment trees & view * flesh out more of view update logic * further flesh out update logic * some refcount functions for fragment trees * add fatal/non-fatal errors * use non-fatal results * clear up some TODOs * ideal format for scheduling info * add a bunch of TODOs * some more fluff * extract fragment graph to submodule * begin fragment graph API * trees, not graphs * improve docs * scope and constructor for trees * add some test TODOs * limit max ancestors and store constraints * constructor * constraints: fix bug in HRMP watermarks * fragment tree population logic * set::retain * extract population logic * implement add_and_populate * fmt * add some TODOs in tests * implement child-selection * strip out old stuff based on wrong assumptions * use fatality * implement pruning * remove unused ancestor constraints * fragment tree instantiation * remove outdated comment * add message/request types and skeleton for handling * fmt * implement handle_candidate_seconded * candidate storage: handle backed * implement handle_candidate_backed * implement answer_get_backable_candidate * remove async where not needed * implement fetch_ancestry * add logic for run_iteration * add some docs * remove global allow(unused), fix warnings * make spellcheck happy (despite English) * fmt * bump Cargo.lock * replace tracing with gum * introduce PopulateFrom trait * implement GetHypotheticalDepths * revise docs slightly * first fragment tree scope test * more scope tests * test add_candidate * fmt * test retain * refactor test code * test populate is recursive * test contiguity of depth 0 is maintained * add_and_populate tests * cycle tests * remove PopulateFrom trait * fmt * test hypothetical depths (non-recursive) * have CandidateSeconded return membership * tree membership requests * Add a ProspectiveParachainsSubsystem struct * add a staging API for base constraints * add a `From` impl * add runtime API for staging_validity_constraints * implement fetch_base_constraints * implement `fetch_upcoming_paras` * remove reconstruction of candidate receipt; no obvious usecase * fmt * export message to broader module * remove last TODO * correctly export * fix compilation and add GetMinimumRelayParent request * make provisioner into a real subsystem with proper mesage bounds * fmt * fix ChannelsOut in overseer test * fix overseer tests * fix again * fmt * Integrate prospective parachains subsystem into backing: Part 1 (#5557) * BEGIN ASYNC candidate-backing CHANGES * rename & document modes * answer prospective validation data requests * GetMinimumRelayParents request is now plural * implement an implicit view utility for backing subsystems * implicit-view: get allowed relay parents * refactorings and improvements to implicit view * add some TODOs for tests * split implicit view updates into 2 functions * backing: define State to prepare for functional refactor * add some docs * backing: implement bones of new leaf activation logic * backing: create per-relay-parent-states * use new handle_active_leaves_update * begin extracting logic from CandidateBackingJob * mostly extract statement import from job logic * handle statement imports outside of job logic * do some TODO planning for prospective parachains integration * finish rewriting backing subsystem in functional style * add prospective parachains mode to relay parent entries * fmt * add a RejectedByProspectiveParachains error * notify prospective parachains of seconded and backed candidates * always validate candidates exhaustively in backing. * return persisted_validation_data from validation * handle rejections by prospective parachains * implement seconding sanity check * invoke validate_and_second * Alter statement table to allow multiple seconded messages per validator * refactor backing to have statements carry PVD * clean up all warnings * Add tests for implicit view * Improve doc comments * Prospective parachains mode based on Runtime API version * Add a TODO * Rework seconding_sanity_check * Iterate over responses * Update backing tests * collator-protocol: load PVD from runtime * Fix validator side tests * Update statement-distribution to fetch PVD * Fix statement-distribution tests * Backing tests with prospective paras #1 * fix per_relay_parent pruning in backing * Test multiple leaves * Test seconding sanity check * Import statement order Before creating an entry in `PerCandidateState` map wait for the approval from the prospective parachains * Add a test for correct state updates * Second multiple candidates per relay parent test * Add backing tests with prospective paras * Second more than one test without prospective paras * Add a test for prospective para blocks * Update malus * typos Co-authored-by: Chris Sosnin <chris125_@live.com> * Track occupied depth in backing per parachain (#5778) * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * fmt * Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991) * BEGIN STATEMENT DISTRIBUTION WORK create a vstaging network protocol which is the same as v1 * mostly make network bridge amenable to vstaging * network-bridge: fully adapt to vstaging * add some TODOs for tests * fix fallout in bitfield-distribution * bitfield distribution tests + TODOs * fix fallout in gossip-support * collator-protocol: fix message fallout * collator-protocol: load PVD from runtime * add TODO for vstaging tests * make things compile * set used network protocol version using a feature * fmt * get approval-distribution building * fix approval-distribution tests * spellcheck * nits * approval distribution net protocol test * bitfield distribution net protocol test * Revert "collator-protocol: fix message fallout" This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1. * Network bridge tests Co-authored-by: Chris Sosnin <chris125_@live.com> * remove max_pov_size requirement from prospective pvd request (#6014) * remove max_pov_size requirement from prospective pvd request * fmt * Extract legacy statement distribution to its own module (#6026) * add compatibility type to v2 statement distribution message * warning cleanup * handle compatibility layer for v2 * clean up an unimplemented!() block * circulate statements based on version * extract legacy v1 code into separate module * remove unimplemented * clean up naming of from_requester/responder * remove TODOs * have backing share seconded statements with PVD * fmt * fix warning * Quick fix unused warning for not yet implemented/used staging messages. * Fix network bridge test * Fix wrong merge. We now have 23 subsystems (network bridge split + prospective parachains) Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> * Version 3 is already live. * Fix tests (#6055) * Fix backing tests * Fix warnings. * fmt * collator-protocol: asynchronous backing changes (#5740) * Draft collator side changes * Start working on collations management * Handle peer's view change * Versioning on advertising * Versioned collation fetching request * Handle versioned messages * Improve docs for collation requests * Add spans * Add request receiver to overseer * Fix collator side tests * Extract relay parent mode to lib * Validator side draft * Add more checks for advertisement * Request pvd based on async backing mode * review * Validator side improvements * Make old tests green * More fixes * Collator side tests draft * Send collation test * fmt * Collator side network protocol versioning * cleanup * merge artifacts * Validator side net protocol versioning * Remove fragment tree membership request * Resolve todo * Collator side core state test * Improve net protocol compatibility * Validator side tests * more improvements * style fixes * downgrade log * Track implicit assignments * Limit the number of seconded candidates per para * Add a sanity check * Handle fetched candidate * fix tests * Retry fetch * Guard against dequeueing while already fetching * Reintegrate connection management * Timeout on advertisements * fmt * spellcheck * update tests after merge * validator assignment fixes for backing and collator protocol (#6158) * Rename depth->ancestry len in tests * Refactor group assignments * Remove implicit assignments * backing: consider occupied core assignments * Track a single para on validator side * Refactor prospective parachains mode request (#6179) * Extract prospective parachains mode into util * Skip activations depending on the mode * backing: don't send backed candidate to provisioner (#6185) * backing: introduce `CanSecond` request for advertisements filtering (#6225) * Drop BoundToRelayParent * draft changes * fix backing tests * Fix genesis ancestry * Fix validator side tests * more tests * cargo generate-lockfile * Implement `StagingValidityConstraints` Runtime API method (#6258) * Implement StagingValidityConstraints * spellcheck * fix ump params * Update hrmp comment * Introduce ump per candidate limit * hypothetical earliest block * refactor primitives usage * hypothetical earliest block number test * fix build * Prepare the Runtime for asynchronous backing upgrade (#6287) * Introduce async backing params to runtime config * fix cumulus config * use config * finish runtimes * Introduce new staging API * Update collator protocol * Update provisioner * Update prospective parachains * Update backing * Move async backing params lower in the config * make naming consistent * misc * Use real prospective parachains subsystem (#6407) * Backport `HypotheticalFrontier` into the feature branch (#6605) * implement more general HypotheticalFrontier * fmt * drop unneeded request Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Resolve todo about legacy leaf activation (#6447) * fix bug/warning in handling membership answers * Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521) * Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest` * Update tests * Fix (removed wrong docstring) * Fix can_second request * Patch some dead_code errors --------- Co-authored-by: Chris Sosnin <chris125_@live.com> * Async Backing: Send Statement Distribution "Backed" messages (#6634) * Backing: Send Statement Distribution "Backed" messages Closes #6590. **TODO:** - [ ] Adjust tests * Fix compile errors * (Mostly) fix tests * Fix comment * Fix test and compile error * Test that `StatementDistributionMessage::Backed` is sent * Fix compile error * Fix some clippy errors * Add prospective parachains subsystem tests (#6454) * Add prospective parachains subsystem test * Add `should_do_no_work_if_async_backing_disabled_for_leaf` test * Implement `activate_leaf` helper, up to getting ancestry * Finish implementing `activate_leaf` * Small refactor in `activate_leaf` * Get `CandidateSeconded` working * Finish `send_candidate_and_check_if_found` test * Refactor; send more leaves & candidates * Refactor test * Implement `check_candidate_parent_leaving_view` test * Start work on `check_candidate_on_multiple_forks` test * Don’t associate specific parachains with leaf * Finish `correctly_updates_leaves` test * Fix cycle due to reused head data * Fix `check_backable_query` test * Fix `check_candidate_on_multiple_forks` test * Add `check_depth_and_pvd_queries` test * Address review comments * Remove TODO * add a new index for output head data to candidate storage * Resolve test TODOs * Fix compile errors * test candidate storage pruning, make sure new index is cleaned up --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Node-side metrics for asynchronous backing (#6549) * Add metrics for `prune_view_candidate_storage` * Add metrics for `request_unblocked_collations` * Fix docstring * Couple fixes from review comments * Fix `check_depth_query` test * inclusion-emulator: mirror advancement rule check (#6361) * inclusion-emulator: mirror advancement rule check * fix build * prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649) * Introduce `backed_in_path_only` flag for depth request * fmt * update doc comment * fmt * Add async-backing zombienet tests (#6314) * Async backing: impl guide for statement distribution (#6738) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> * Asynchronous backing statement distribution: Take III (#5999) * add notification types for v2 statement-distribution * improve protocol docs * add empty vstaging module * fmt * add backed candidate packet request types * start putting down structure of new logic * handle activated leaf * some sanity-checking on outbound statements * fmt * update vstaging share to use statements with PVD * tiny refactor, candidate_hash location * import local statements * refactor statement import * first stab at broadcast logic * fmt * fill out some TODOs * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * fmt, fix grid test after topology change * typo Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * address review * adjust comment, make easier to understand * Fix typo --------- Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * miscellaneous fixes to make asynchronous backing work (#6791) * propagate network-protocol-staging feature * add feature to adder-collator as well * allow collation-generation of occupied cores * prospective parachains: special treatment for pending availability candidates * runtime: fetch candidates pending availability * lazily construct PVD for pending candidates * fix fallout in prospective parachains hypothetical/select_child * runtime: enact candidates when creating paras-inherent * make tests compile * test pending availability in the scope * add prospective parachains test * fix validity constraints leftovers * drop prints * Fix typos --------- Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Marcin S <marcin@realemail.net> * Remove restart from test (#6840) * Async Backing: Statement Distribution Tests (#6755) * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * Hook up request sender * Add `valid_statement_without_prior_seconded_is_ignored` test * Fix `valid_statement_without_prior_seconded_is_ignored` test * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Remove obsolete test * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * First draft of `ensure_seconding_limit_is_respected` test * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * Fix `ensure_seconding_limit_is_respected` test * Start `backed_candidate_leads_to_advertisement` test * fmt, fix grid test after topology change * Send Backed notification * Finish `backed_candidate_leads_to_advertisement` test * Finish `peer_reported_for_duplicate_statements` test * Finish `received_advertisement_before_confirmation_leads_to_request` * Add `advertisements_rejected_from_incorrect_peers` test * Add `manifest_rejected_*` tests * Add `manifest_rejected_when_group_does_not_match_para` test * Add `local_node_sanity_checks_incoming_requests` test * Add `local_node_respects_statement_mask` test * Add tests where peer is reported for providing invalid signatures * Add `cluster_peer_allowed_to_send_incomplete_statements` test * Add `received_advertisement_after_backing_leads_to_acknowledgement` * Add `received_advertisement_after_confirmation_before_backing` test * peer_reported_for_advertisement_conflicting_with_confirmed_candidate * Add `peer_reported_for_not_enough_statements` test * Add `peer_reported_for_providing_statements_meant_to_be_masked_out` * Add `additional_statements_are_shared_after_manifest_exchange` * Add `grid_statements_imported_to_backing` test * Add `relay_parent_entering_peer_view_leads_to_advertisement` test * Add `advertisement_not_re_sent_when_peer_re_enters_view` test * Update node/network/statement-distribution/src/vstaging/tests/grid.rs Co-authored-by: asynchronous rob <rphmeier@gmail.com> * Resolve TODOs, update test * Address unused code * Add check after every test for unhandled requests * Refactor (`make_dummy_leaf` and `handle_sent_request`) * Refactor (`make_dummy_topology`) * Minor refactor --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * Fix some clippy lints in tests * Async backing: minor fixes (#6920) * bitfield-distribution test * implicit view tests * Refactor parameters -> params * scheduler: update storage migration (#6963) * update scheduler migration * Adjust weight to account for storage read * Statement Distribution Guide Edits (#7025) * Statement distribution guide edits * Addressed Marcin's comments * Add attested candidate request retry timeouts (#6833) Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Fix async backing statement distribution tests (#6621) Resolve some todos in async backing statement-distribution branch (#6482) Fix clippy errors in statement distribution branch (#6720) * Async backing: add Prospective Parachains impl guide (#6933) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> * Updates to Provisioner Guide for Async Backing (#7106) * Initial corrections and clarifications * Partial first draft * Finished first draft * Adding back wrongly removed test bit * fmt * Update roadmap/implementers-guide/src/node/utility/provisioner.md Co-authored-by: Marcin S. <marcin@realemail.net> * Addressing comments * Reorganization * fmt --------- Co-authored-by: Marcin S. <marcin@realemail.net> * fmt * Renaming Parathread Mentions (#7287) * Renaming parathreads * Renaming module to pallet * More updates * PVF: Refactor workers into separate crates, remove host dependency (#7253) * PVF: Refactor workers into separate crates, remove host dependency * Fix compile error * Remove some leftover code * Fix compile errors * Update Cargo.lock * Remove worker main.rs files I accidentally copied these from the other PR. This PR isn't intended to introduce standalone workers yet. * Address review comments * cargo fmt * Update a couple of comments * Update log targets * Update quote to 1.0.27 (#7280) Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: parity-processbot <> * pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271) * pallets: implement Default for GenesisConfig in no_std This change is follow-up of: https://github.com/paritytech/substrate/pull/14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334 * Cargo.lock updated * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * cli: enable BEEFY by default on test networks (#7293) We consider BEEFY mature enough to run by default on all nodes for test networks (Rococo/Wococo/Versi). Right now, most nodes are not running it since it's opt-in using --beefy flag. Switch to an opt-out model for test networks. Replace --beefy flag from CLI with --no-beefy and have BEEFY client start by default on test networks. Signed-off-by: acatangiu <adrian@parity.io> * runtime: past session slashing runtime API (#6667) * runtime/vstaging: unapplied_slashes runtime API * runtime/vstaging: key_ownership_proof runtime API * runtime/ParachainHost: submit_report_dispute_lost * fix key_ownership_proof API * runtime: submit_report_dispute_lost runtime API * nits * Update node/subsystem-types/src/messages.rs Co-authored-by: Marcin S. <marcin@bytedude.com> * revert unrelated fmt changes * post merge fixes * fix compilation --------- Co-authored-by: Marcin S. <marcin@bytedude.com> * Correcting git mishap * Document usage of `gum` crate (#7294) * Document usage of gum crate * Small fix * Add some more basic info * Update node/gum/src/lib.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Update target docs --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * XCM: Fix issue with RequestUnlock (#7278) * XCM: Fix issue with RequestUnlock * Leave API changes for v4 * Fix clippy errors * Fix tests --------- Co-authored-by: parity-processbot <> * Companion for Substrate#14228 (#7295) * Companion for Substrate#14228 https://github.com/paritytech/substrate/pull/14228 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * Companion for #14237: Use latest sp-crates (#7300) * To revert: Update substrate branch to "lexnv/bump_sp_crates" Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates"" This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008. * Update cargo lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * bounded-collections bump to 0.1.7 (#7305) * bounded-collections bump to 0.1.7 Companion for: paritytech/substrate#14225 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * bump to quote 1.0.28 (#7306) * `RollingSessionWindow` cleanup (#7204) * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit * Fix tests in import * Fix the rest of the tests * Remove dead code * Fix todos * Simplify session caching * Comments for `SessionInfoProvider` * Separate `SessionInfoProvider` from `State` * `cache_session_info_for_head` becomes freestanding function * Remove unneeded `mut` usage * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider` * Fix SessionInfo retrieval * Code cleanup * Don't wrap `SessionInfoProvider` in an `Option` * Remove `earliest_session()` * Remove pre-caching -> wip * Fix some tests and code cleanup * Fix all tests * Fixes in tests * Fix comments, variable names and small style changes * Fix a warning * impl From<SessionWindowSize> for NonZeroUsize * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG * Code review feedback * Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db * Remove `col_session_data` usages * Storage migration clearing columns w/o removing them * Remove session data column usages from `approval-voting` and `dispute-coordinator` tests * Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests * Fix formatting in initialized.rs * Fix a corner case in `SessionInfo` caching for `dispute-coordinator` * Remove `RollingSessionWindow` ;( * Revert "Fix formatting in initialized.rs" This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc. * v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it * Fix `NUM_COLUMNS` in `approval-voting` * Use `columns::v3::NUM_COLUMNS` when opening db * Update node/service/src/parachains_db/upgrade.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3` * Fix `NUM+COLUMNS` in approval_voting * Fix formatting * Fix columns usage * Clarification comments about the different db versions --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * pallet-para-config: Remove remnant WeightInfo functions (#7308) * pallet-para-config: Remove remnant WeightInfo functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * set_config_with_weight begone Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> * XCM: PayOverXcm config (#6900) * Move XCM query functionality to trait * Fix tests * Add PayOverXcm implementation * fix the PayOverXcm trait to compile * moved doc comment out of trait implmeentation and to the trait * PayOverXCM documentation * Change documentation a bit * Added empty benchmark methods implementation and changed docs * update PayOverXCM to convert AccountIds to MultiLocations * Implement benchmarking method * Change v3 to latest * Descend origin to an asset sender (#6970) * descend origin to an asset sender * sender as tuple of dest and sender * Add more variants to the QueryResponseStatus enum * Change Beneficiary to Into<[u8; 32]> * update PayOverXcm to return concrete errors and use AccountId as sender * use polkadot-primitives for AccountId * fix dependency to use polkadot-core-primitives * force Unpaid instruction to the top of the instructions list * modify report_outcome to accept interior argument * use new_query directly for building final xcm query, instead of report_outcome * fix usage of new_query to use the XcmQueryHandler * fix usage of new_query to use the XcmQueryHandler * tiny method calling fix * xcm query handler (#7198) * drop redundant query status * rename ReportQueryStatus to OuterQueryStatus * revert rename of QueryResponseStatus * update mapping * Update xcm/xcm-builder/src/pay.rs Co-authored-by: Gavin Wood <gavin@parity.io> * Updates * Docs * Fix benchmarking stuff * Destination can be determined based on asset_kind * Tweaking API to minimise clones * Some repotting and docs --------- Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * Companion for #14265 (#7307) * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: parity-processbot <> * bump serde to 1.0.163 (#7315) * bump serde to 1.0.163 * bump ci * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * fmt * Updated fmt * Removing changes accidentally pulled from master * fix another master pull issue * Another master pull fix * fmt * Fixing implementers guide build * Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread" This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Marcin S. <marcin@bytedude.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * fix bitfield distribution test * approval distribution tests * fix bridge tests * update Cargo.lock * [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457) * Optimize collator-protocol validator-side request fetching * address feedback: replace tuples with structs * feedback: add doc comments * move collation types to subfolder --------- Signed-off-by: alindima <alin@parity.io> * Update collation generation for asynchronous backing (#7405) * break candidate receipt construction and distribution into own function * update implementers' guide to include SubmitCollation * implement SubmitCollation for collation-generation * fmt * fix test compilation & remove unnecessary submodule * add some TODOs for a test suite. * Update roadmap/implementers-guide/src/types/overseer-protocol.md Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * add new test harness and first test * refactor to avoid requiring background sender * ensure collation gets packaged and distributed * tests for the fallback case with no hint * add parent rp-number hint tests * fmt * update uses of CollationGenerationConfig * fix remaining test * address review comments * use subsystemsender for background tasks * fmt * remove ValidationCodeHashHint and related tests --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * fix some more fallout from merge * fmt * remove staging APIs from Rococo & Westend (#7513) * send network messages on main protocol name (#7515) * misc async backing improvements for allowed ancestry blocks (#7532) * shared: fix acquire_info * backwards-compat test for prospective parachains * same relay parent is allowed * provisioner: request candidate receipt by relay parent (#7527) * return candidates hash from prospective parachains * update provisioner * update tests * guide changes * send a single message to backing * fix test * revert to old `handle_new_activations` logic in some cases (#7514) * revert to old `handle_new_activations` logic * gate sending messages on scheduled cores to max_depth >= 2 * fmt * 2->1 * Omnibus asynchronous backing bugfix PR (#7529) * fix a bug in backing * add some more logs * prospective parachains: take ancestry only up to session bounds * add test * fix zombienet tests (#7614) Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix runtime compilation * make bitfield distribution tests compile * attempt to fix zombienet disputes (#7618) * update metric name * update some metric names * avoid cycles when creating fake candidates * make undying collator more friendly to malformed parents * fix a bug in malus * fmt * clippy * add RUN_IN_CONTAINER to new ZombieNet tests (#7631) * remove duplicated migration happened because of master-merge --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: alindima <alin@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Mattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com> Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Alin Dima <alin@parity.io>
2026-06-14 05:11:09 +00:00 · 2023-08-18 11:11:56 -05:00
parent ad539f0e41
commit 5174b9d2d7
175 changed files with 40882 additions and 6462 deletions
@@ -19,10 +19,10 @@ use futures::channel::{mpsc, oneshot};

 use polkadot_node_subsystem::{
 	messages::{StoreAvailableDataError, ValidationFailed},
-	SubsystemError,
+	RuntimeApiError, SubsystemError,
 };
-use polkadot_node_subsystem_util::Error as UtilError;
-use polkadot_primitives::BackedCandidate;
+use polkadot_node_subsystem_util::{runtime, Error as UtilError};
+use polkadot_primitives::{BackedCandidate, ValidationCodeHash};

 use crate::LOG_TARGET;

@@ -33,6 +33,18 @@ pub type FatalResult<T> = std::result::Result<T, FatalError>;
 #[allow(missing_docs)]
 #[fatality::fatality(splitable)]
 pub enum Error {
+	#[fatal]
+	#[error("Failed to spawn background task")]
+	FailedToSpawnBackgroundTask,
+
+	#[fatal(forward)]
+	#[error("Error while accessing runtime information")]
+	Runtime(#[from] runtime::Error),
+
+	#[fatal]
+	#[error(transparent)]
+	BackgroundValidationMpsc(#[from] mpsc::SendError),
+
 	#[error("Candidate is not found")]
 	CandidateNotFound,

@@ -45,16 +57,27 @@ pub enum Error {
 	#[error("FetchPoV failed")]
 	FetchPoV,

-	#[fatal]
-	#[error("Failed to spawn background task")]
-	FailedToSpawnBackgroundTask,
+	#[error("Fetching validation code by hash failed {0:?}, {1:?}")]
+	FetchValidationCode(ValidationCodeHash, RuntimeApiError),

-	#[error("ValidateFromChainState channel closed before receipt")]
-	ValidateFromChainState(#[source] oneshot::Canceled),
+	#[error("Fetching Runtime API version failed {0:?}")]
+	FetchRuntimeApiVersion(RuntimeApiError),
+
+	#[error("No validation code {0:?}")]
+	NoValidationCode(ValidationCodeHash),
+
+	#[error("Candidate rejected by prospective parachains subsystem")]
+	RejectedByProspectiveParachains,
+
+	#[error("ValidateFromExhaustive channel closed before receipt")]
+	ValidateFromExhaustive(#[source] oneshot::Canceled),

 	#[error("StoreAvailableData channel closed before receipt")]
 	StoreAvailableDataChannel(#[source] oneshot::Canceled),

+	#[error("RuntimeAPISubsystem channel closed before receipt")]
+	RuntimeApiUnavailable(#[source] oneshot::Canceled),
+
 	#[error("a channel was closed before receipt in try_join!")]
 	JoinMultiple(#[source] oneshot::Canceled),

@@ -64,10 +87,6 @@ pub enum Error {
 	#[error(transparent)]
 	ValidationFailed(#[from] ValidationFailed),

-	#[fatal]
-	#[error(transparent)]
-	BackgroundValidationMpsc(#[from] mpsc::SendError),
-
 	#[error(transparent)]
 	UtilError(#[from] UtilError),

@@ -136,8 +136,7 @@ fn make_candidate_receipt(relay_parent: Hash) -> CandidateReceipt {
 		para_head: zeros,
 		validation_code_hash: zeros.into(),
 	};
-	let candidate = CandidateReceipt { descriptor, commitments_hash: zeros };
-	candidate
+	CandidateReceipt { descriptor, commitments_hash: zeros }
 }

 /// Get a dummy `ActivatedLeaf` for a given block number.
@@ -0,0 +1,29 @@
+[package]
+name = "polkadot-node-core-prospective-parachains"
+version = "0.9.16"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+
+[dependencies]
+futures = "0.3.19"
+gum = { package = "tracing-gum", path = "../../gum" }
+parity-scale-codec = "2"
+thiserror = "1.0.30"
+fatality = "0.0.6"
+bitvec = "1"
+
+polkadot-primitives = { path = "../../../primitives" }
+polkadot-node-primitives = { path = "../../primitives" }
+polkadot-node-subsystem = { path = "../../subsystem" }
+polkadot-node-subsystem-util = { path = "../../subsystem-util" }
+
+[dev-dependencies]
+assert_matches = "1"
+polkadot-node-subsystem-test-helpers = { path = "../../subsystem-test-helpers" }
+polkadot-node-subsystem-types = { path = "../../subsystem-types" }
+polkadot-primitives-test-helpers = { path = "../../../primitives/test-helpers" }
+sp-core = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sc-keystore = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-application-crypto = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-keyring = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-keystore = { git = "https://github.com/paritytech/substrate", branch = "master" }
@@ -0,0 +1,87 @@
+// Copyright 2022 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Error types.
+
+use futures::channel::oneshot;
+
+use polkadot_node_subsystem::{
+	errors::{ChainApiError, RuntimeApiError},
+	SubsystemError,
+};
+use polkadot_node_subsystem_util::runtime;
+
+use crate::LOG_TARGET;
+use fatality::Nested;
+
+#[allow(missing_docs)]
+#[fatality::fatality(splitable)]
+pub enum Error {
+	#[fatal]
+	#[error("SubsystemError::Context error: {0}")]
+	SubsystemContext(String),
+
+	#[fatal]
+	#[error("Spawning a task failed: {0}")]
+	SpawnFailed(SubsystemError),
+
+	#[fatal]
+	#[error("Participation worker receiver exhausted.")]
+	ParticipationWorkerReceiverExhausted,
+
+	#[fatal]
+	#[error("Receiving message from overseer failed: {0}")]
+	SubsystemReceive(#[source] SubsystemError),
+
+	#[error("Error while accessing runtime information")]
+	Runtime(#[from] runtime::Error),
+
+	#[error(transparent)]
+	RuntimeApi(#[from] RuntimeApiError),
+
+	#[error(transparent)]
+	ChainApi(#[from] ChainApiError),
+
+	#[error(transparent)]
+	Subsystem(SubsystemError),
+
+	#[error("Request to chain API subsystem dropped")]
+	ChainApiRequestCanceled(oneshot::Canceled),
+
+	#[error("Request to runtime API subsystem dropped")]
+	RuntimeApiRequestCanceled(oneshot::Canceled),
+}
+
+/// General `Result` type.
+pub type Result<R> = std::result::Result<R, Error>;
+/// Result for non-fatal only failures.
+pub type JfyiErrorResult<T> = std::result::Result<T, JfyiError>;
+/// Result for fatal only failures.
+pub type FatalResult<T> = std::result::Result<T, FatalError>;
+
+/// Utility for eating top level errors and log them.
+///
+/// We basically always want to try and continue on error. This utility function is meant to
+/// consume top-level errors by simply logging them
+pub fn log_error(result: Result<()>, ctx: &'static str) -> FatalResult<()> {
+	match result.into_nested()? {
+		Ok(()) => Ok(()),
+		Err(jfyi) => {
+			gum::debug!(target: LOG_TARGET, error = ?jfyi, ctx);
+			Ok(())
+		},
+	}
+}
@@ -0,0 +1,939 @@
+// Copyright 2022-2023 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Implementation of the Prospective Parachains subsystem - this tracks and handles
+//! prospective parachain fragments and informs other backing-stage subsystems
+//! of work to be done.
+//!
+//! This is the main coordinator of work within the node for the collation and
+//! backing phases of parachain consensus.
+//!
+//! This is primarily an implementation of "Fragment Trees", as described in
+//! [`polkadot_node_subsystem_util::inclusion_emulator::staging`].
+//!
+//! This subsystem also handles concerns such as the relay-chain being forkful and session changes.
+
+use std::{
+	borrow::Cow,
+	collections::{HashMap, HashSet},
+};
+
+use futures::{channel::oneshot, prelude::*};
+
+use polkadot_node_subsystem::{
+	messages::{
+		ChainApiMessage, FragmentTreeMembership, HypotheticalCandidate,
+		HypotheticalFrontierRequest, IntroduceCandidateRequest, ProspectiveParachainsMessage,
+		ProspectiveValidationDataRequest, RuntimeApiMessage, RuntimeApiRequest,
+	},
+	overseer, ActiveLeavesUpdate, FromOrchestra, OverseerSignal, SpawnedSubsystem, SubsystemError,
+};
+use polkadot_node_subsystem_util::{
+	inclusion_emulator::staging::{Constraints, RelayChainBlockInfo},
+	request_session_index_for_child,
+	runtime::{prospective_parachains_mode, ProspectiveParachainsMode},
+};
+use polkadot_primitives::vstaging::{
+	BlockNumber, CandidateHash, CandidatePendingAvailability, CommittedCandidateReceipt, CoreState,
+	Hash, HeadData, Header, Id as ParaId, PersistedValidationData,
+};
+
+use crate::{
+	error::{FatalError, FatalResult, JfyiError, JfyiErrorResult, Result},
+	fragment_tree::{
+		CandidateStorage, CandidateStorageInsertionError, FragmentTree, Scope as TreeScope,
+	},
+};
+
+mod error;
+mod fragment_tree;
+#[cfg(test)]
+mod tests;
+
+mod metrics;
+use self::metrics::Metrics;
+
+const LOG_TARGET: &str = "parachain::prospective-parachains";
+
+struct RelayBlockViewData {
+	// Scheduling info for paras and upcoming paras.
+	fragment_trees: HashMap<ParaId, FragmentTree>,
+	pending_availability: HashSet<CandidateHash>,
+}
+
+struct View {
+	// Active or recent relay-chain blocks by block hash.
+	active_leaves: HashMap<Hash, RelayBlockViewData>,
+	candidate_storage: HashMap<ParaId, CandidateStorage>,
+}
+
+impl View {
+	fn new() -> Self {
+		View { active_leaves: HashMap::new(), candidate_storage: HashMap::new() }
+	}
+}
+
+/// The prospective parachains subsystem.
+#[derive(Default)]
+pub struct ProspectiveParachainsSubsystem {
+	metrics: Metrics,
+}
+
+impl ProspectiveParachainsSubsystem {
+	/// Create a new instance of the `ProspectiveParachainsSubsystem`.
+	pub fn new(metrics: Metrics) -> Self {
+		Self { metrics }
+	}
+}
+
+#[overseer::subsystem(ProspectiveParachains, error = SubsystemError, prefix = self::overseer)]
+impl<Context> ProspectiveParachainsSubsystem
+where
+	Context: Send + Sync,
+{
+	fn start(self, ctx: Context) -> SpawnedSubsystem {
+		SpawnedSubsystem {
+			future: run(ctx, self.metrics)
+				.map_err(|e| SubsystemError::with_origin("prospective-parachains", e))
+				.boxed(),
+			name: "prospective-parachains-subsystem",
+		}
+	}
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn run<Context>(mut ctx: Context, metrics: Metrics) -> FatalResult<()> {
+	let mut view = View::new();
+	loop {
+		crate::error::log_error(
+			run_iteration(&mut ctx, &mut view, &metrics).await,
+			"Encountered issue during run iteration",
+		)?;
+	}
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn run_iteration<Context>(
+	ctx: &mut Context,
+	view: &mut View,
+	metrics: &Metrics,
+) -> Result<()> {
+	loop {
+		match ctx.recv().await.map_err(FatalError::SubsystemReceive)? {
+			FromOrchestra::Signal(OverseerSignal::Conclude) => return Ok(()),
+			FromOrchestra::Signal(OverseerSignal::ActiveLeaves(update)) => {
+				handle_active_leaves_update(&mut *ctx, view, update, metrics).await?;
+			},
+			FromOrchestra::Signal(OverseerSignal::BlockFinalized(..)) => {},
+			FromOrchestra::Communication { msg } => match msg {
+				ProspectiveParachainsMessage::IntroduceCandidate(request, tx) =>
+					handle_candidate_introduced(&mut *ctx, view, request, tx).await?,
+				ProspectiveParachainsMessage::CandidateSeconded(para, candidate_hash) =>
+					handle_candidate_seconded(view, para, candidate_hash),
+				ProspectiveParachainsMessage::CandidateBacked(para, candidate_hash) =>
+					handle_candidate_backed(&mut *ctx, view, para, candidate_hash).await?,
+				ProspectiveParachainsMessage::GetBackableCandidate(
+					relay_parent,
+					para,
+					required_path,
+					tx,
+				) => answer_get_backable_candidate(&view, relay_parent, para, required_path, tx),
+				ProspectiveParachainsMessage::GetHypotheticalFrontier(request, tx) =>
+					answer_hypothetical_frontier_request(&view, request, tx),
+				ProspectiveParachainsMessage::GetTreeMembership(para, candidate, tx) =>
+					answer_tree_membership_request(&view, para, candidate, tx),
+				ProspectiveParachainsMessage::GetMinimumRelayParents(relay_parent, tx) =>
+					answer_minimum_relay_parents_request(&view, relay_parent, tx),
+				ProspectiveParachainsMessage::GetProspectiveValidationData(request, tx) =>
+					answer_prospective_validation_data_request(&view, request, tx),
+			},
+		}
+	}
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn handle_active_leaves_update<Context>(
+	ctx: &mut Context,
+	view: &mut View,
+	update: ActiveLeavesUpdate,
+	metrics: &Metrics,
+) -> JfyiErrorResult<()> {
+	// 1. clean up inactive leaves
+	// 2. determine all scheduled para at new block
+	// 3. construct new fragment tree for each para for each new leaf
+	// 4. prune candidate storage.
+
+	for deactivated in &update.deactivated {
+		view.active_leaves.remove(deactivated);
+	}
+
+	let mut temp_header_cache = HashMap::new();
+	for activated in update.activated.into_iter() {
+		let hash = activated.hash;
+
+		let mode = prospective_parachains_mode(ctx.sender(), hash)
+			.await
+			.map_err(JfyiError::Runtime)?;
+
+		let ProspectiveParachainsMode::Enabled { max_candidate_depth, allowed_ancestry_len } = mode
+		else {
+			gum::trace!(
+				target: LOG_TARGET,
+				block_hash = ?hash,
+				"Skipping leaf activation since async backing is disabled"
+			);
+
+			// Not a part of any allowed ancestry.
+			return Ok(())
+		};
+
+		let mut pending_availability = HashSet::new();
+		let scheduled_paras =
+			fetch_upcoming_paras(&mut *ctx, hash, &mut pending_availability).await?;
+
+		let block_info: RelayChainBlockInfo =
+			match fetch_block_info(&mut *ctx, &mut temp_header_cache, hash).await? {
+				None => {
+					gum::warn!(
+						target: LOG_TARGET,
+						block_hash = ?hash,
+						"Failed to get block info for newly activated leaf block."
+					);
+
+					// `update.activated` is an option, but we can use this
+					// to exit the 'loop' and skip this block without skipping
+					// pruning logic.
+					continue
+				},
+				Some(info) => info,
+			};
+
+		let ancestry =
+			fetch_ancestry(&mut *ctx, &mut temp_header_cache, hash, allowed_ancestry_len).await?;
+
+		// Find constraints.
+		let mut fragment_trees = HashMap::new();
+		for para in scheduled_paras {
+			let candidate_storage =
+				view.candidate_storage.entry(para).or_insert_with(CandidateStorage::new);
+
+			let backing_state = fetch_backing_state(&mut *ctx, hash, para).await?;
+
+			let (constraints, pending_availability) = match backing_state {
+				Some(c) => c,
+				None => {
+					// This indicates a runtime conflict of some kind.
+
+					gum::debug!(
+						target: LOG_TARGET,
+						para_id = ?para,
+						relay_parent = ?hash,
+						"Failed to get inclusion backing state."
+					);
+
+					continue
+				},
+			};
+
+			let pending_availability = preprocess_candidates_pending_availability(
+				ctx,
+				&mut temp_header_cache,
+				constraints.required_parent.clone(),
+				pending_availability,
+			)
+			.await?;
+			let mut compact_pending = Vec::with_capacity(pending_availability.len());
+
+			for c in pending_availability {
+				let res = candidate_storage.add_candidate(c.candidate, c.persisted_validation_data);
+				let candidate_hash = c.compact.candidate_hash;
+				compact_pending.push(c.compact);
+
+				match res {
+					Ok(_) | Err(CandidateStorageInsertionError::CandidateAlreadyKnown(_)) => {
+						// Anything on-chain is guaranteed to be backed.
+						candidate_storage.mark_backed(&candidate_hash);
+					},
+					Err(err) => {
+						gum::warn!(
+							target: LOG_TARGET,
+							?candidate_hash,
+							para_id = ?para,
+							?err,
+							"Scraped invalid candidate pending availability",
+						);
+					},
+				}
+			}
+
+			let scope = TreeScope::with_ancestors(
+				para,
+				block_info.clone(),
+				constraints,
+				compact_pending,
+				max_candidate_depth,
+				ancestry.iter().cloned(),
+			)
+			.expect("ancestors are provided in reverse order and correctly; qed");
+
+			let tree = FragmentTree::populate(scope, &*candidate_storage);
+
+			fragment_trees.insert(para, tree);
+		}
+
+		view.active_leaves
+			.insert(hash, RelayBlockViewData { fragment_trees, pending_availability });
+	}
+
+	if !update.deactivated.is_empty() {
+		// This has potential to be a hotspot.
+		prune_view_candidate_storage(view, metrics);
+	}
+
+	Ok(())
+}
+
+fn prune_view_candidate_storage(view: &mut View, metrics: &Metrics) {
+	metrics.time_prune_view_candidate_storage();
+
+	let active_leaves = &view.active_leaves;
+	let mut live_candidates = HashSet::new();
+	let mut live_paras = HashSet::new();
+	for sub_view in active_leaves.values() {
+		for (para_id, fragment_tree) in &sub_view.fragment_trees {
+			live_candidates.extend(fragment_tree.candidates());
+			live_paras.insert(*para_id);
+		}
+
+		live_candidates.extend(sub_view.pending_availability.iter().cloned());
+	}
+
+	view.candidate_storage.retain(|para_id, storage| {
+		if !live_paras.contains(&para_id) {
+			return false
+		}
+
+		storage.retain(|h| live_candidates.contains(&h));
+
+		// Even if `storage` is now empty, we retain.
+		// This maintains a convenient invariant that para-id storage exists
+		// as long as there's an active head which schedules the para.
+		true
+	})
+}
+
+struct ImportablePendingAvailability {
+	candidate: CommittedCandidateReceipt,
+	persisted_validation_data: PersistedValidationData,
+	compact: crate::fragment_tree::PendingAvailability,
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn preprocess_candidates_pending_availability<Context>(
+	ctx: &mut Context,
+	cache: &mut HashMap<Hash, Header>,
+	required_parent: HeadData,
+	pending_availability: Vec<CandidatePendingAvailability>,
+) -> JfyiErrorResult<Vec<ImportablePendingAvailability>> {
+	let mut required_parent = required_parent;
+
+	let mut importable = Vec::new();
+	let expected_count = pending_availability.len();
+
+	for (i, pending) in pending_availability.into_iter().enumerate() {
+		let relay_parent =
+			match fetch_block_info(ctx, cache, pending.descriptor.relay_parent).await? {
+				None => {
+					gum::debug!(
+						target: LOG_TARGET,
+						?pending.candidate_hash,
+						?pending.descriptor.para_id,
+						index = ?i,
+						?expected_count,
+						"Had to stop processing pending candidates early due to missing info.",
+					);
+
+					break
+				},
+				Some(b) => b,
+			};
+
+		let next_required_parent = pending.commitments.head_data.clone();
+		importable.push(ImportablePendingAvailability {
+			candidate: CommittedCandidateReceipt {
+				descriptor: pending.descriptor,
+				commitments: pending.commitments,
+			},
+			persisted_validation_data: PersistedValidationData {
+				parent_head: required_parent,
+				max_pov_size: pending.max_pov_size,
+				relay_parent_number: relay_parent.number,
+				relay_parent_storage_root: relay_parent.storage_root,
+			},
+			compact: crate::fragment_tree::PendingAvailability {
+				candidate_hash: pending.candidate_hash,
+				relay_parent,
+			},
+		});
+
+		required_parent = next_required_parent;
+	}
+
+	Ok(importable)
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn handle_candidate_introduced<Context>(
+	_ctx: &mut Context,
+	view: &mut View,
+	request: IntroduceCandidateRequest,
+	tx: oneshot::Sender<FragmentTreeMembership>,
+) -> JfyiErrorResult<()> {
+	let IntroduceCandidateRequest {
+		candidate_para: para,
+		candidate_receipt: candidate,
+		persisted_validation_data: pvd,
+	} = request;
+
+	// Add the candidate to storage.
+	// Then attempt to add it to all trees.
+	let storage = match view.candidate_storage.get_mut(&para) {
+		None => {
+			gum::warn!(
+				target: LOG_TARGET,
+				para_id = ?para,
+				candidate_hash = ?candidate.hash(),
+				"Received seconded candidate for inactive para",
+			);
+
+			let _ = tx.send(Vec::new());
+			return Ok(())
+		},
+		Some(storage) => storage,
+	};
+
+	let candidate_hash = match storage.add_candidate(candidate, pvd) {
+		Ok(c) => c,
+		Err(CandidateStorageInsertionError::CandidateAlreadyKnown(c)) => {
+			// Candidate known - return existing fragment tree membership.
+			let _ = tx.send(fragment_tree_membership(&view.active_leaves, para, c));
+			return Ok(())
+		},
+		Err(CandidateStorageInsertionError::PersistedValidationDataMismatch) => {
+			// We can't log the candidate hash without either doing more ~expensive
+			// hashing but this branch indicates something is seriously wrong elsewhere
+			// so it's doubtful that it would affect debugging.
+
+			gum::warn!(
+				target: LOG_TARGET,
+				para = ?para,
+				"Received seconded candidate had mismatching validation data",
+			);
+
+			let _ = tx.send(Vec::new());
+			return Ok(())
+		},
+	};
+
+	let mut membership = Vec::new();
+	for (relay_parent, leaf_data) in &mut view.active_leaves {
+		if let Some(tree) = leaf_data.fragment_trees.get_mut(&para) {
+			tree.add_and_populate(candidate_hash, &*storage);
+			if let Some(depths) = tree.candidate(&candidate_hash) {
+				membership.push((*relay_parent, depths));
+			}
+		}
+	}
+
+	if membership.is_empty() {
+		storage.remove_candidate(&candidate_hash);
+	}
+
+	let _ = tx.send(membership);
+
+	Ok(())
+}
+
+fn handle_candidate_seconded(view: &mut View, para: ParaId, candidate_hash: CandidateHash) {
+	let storage = match view.candidate_storage.get_mut(&para) {
+		None => {
+			gum::warn!(
+				target: LOG_TARGET,
+				para_id = ?para,
+				?candidate_hash,
+				"Received instruction to second unknown candidate",
+			);
+
+			return
+		},
+		Some(storage) => storage,
+	};
+
+	if !storage.contains(&candidate_hash) {
+		gum::warn!(
+			target: LOG_TARGET,
+			para_id = ?para,
+			?candidate_hash,
+			"Received instruction to second unknown candidate",
+		);
+
+		return
+	}
+
+	storage.mark_seconded(&candidate_hash);
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn handle_candidate_backed<Context>(
+	_ctx: &mut Context,
+	view: &mut View,
+	para: ParaId,
+	candidate_hash: CandidateHash,
+) -> JfyiErrorResult<()> {
+	let storage = match view.candidate_storage.get_mut(&para) {
+		None => {
+			gum::warn!(
+				target: LOG_TARGET,
+				para_id = ?para,
+				?candidate_hash,
+				"Received instruction to back unknown candidate",
+			);
+
+			return Ok(())
+		},
+		Some(storage) => storage,
+	};
+
+	if !storage.contains(&candidate_hash) {
+		gum::warn!(
+			target: LOG_TARGET,
+			para_id = ?para,
+			?candidate_hash,
+			"Received instruction to back unknown candidate",
+		);
+
+		return Ok(())
+	}
+
+	if storage.is_backed(&candidate_hash) {
+		gum::debug!(
+			target: LOG_TARGET,
+			para_id = ?para,
+			?candidate_hash,
+			"Received redundant instruction to mark candidate as backed",
+		);
+
+		return Ok(())
+	}
+
+	storage.mark_backed(&candidate_hash);
+	Ok(())
+}
+
+fn answer_get_backable_candidate(
+	view: &View,
+	relay_parent: Hash,
+	para: ParaId,
+	required_path: Vec<CandidateHash>,
+	tx: oneshot::Sender<Option<(CandidateHash, Hash)>>,
+) {
+	let data = match view.active_leaves.get(&relay_parent) {
+		None => {
+			gum::debug!(
+				target: LOG_TARGET,
+				?relay_parent,
+				para_id = ?para,
+				"Requested backable candidate for inactive relay-parent."
+			);
+
+			let _ = tx.send(None);
+			return
+		},
+		Some(d) => d,
+	};
+
+	let tree = match data.fragment_trees.get(&para) {
+		None => {
+			gum::debug!(
+				target: LOG_TARGET,
+				?relay_parent,
+				para_id = ?para,
+				"Requested backable candidate for inactive para."
+			);
+
+			let _ = tx.send(None);
+			return
+		},
+		Some(tree) => tree,
+	};
+
+	let storage = match view.candidate_storage.get(&para) {
+		None => {
+			gum::warn!(
+				target: LOG_TARGET,
+				?relay_parent,
+				para_id = ?para,
+				"No candidate storage for active para",
+			);
+
+			let _ = tx.send(None);
+			return
+		},
+		Some(s) => s,
+	};
+
+	let Some(child_hash) =
+		tree.select_child(&required_path, |candidate| storage.is_backed(candidate))
+	else {
+		let _ = tx.send(None);
+		return
+	};
+	let Some(candidate_relay_parent) = storage.relay_parent_by_candidate_hash(&child_hash) else {
+		gum::error!(
+			target: LOG_TARGET,
+			?child_hash,
+			para_id = ?para,
+			"Candidate is present in fragment tree but not in candidate's storage!",
+		);
+		let _ = tx.send(None);
+		return
+	};
+
+	let _ = tx.send(Some((child_hash, candidate_relay_parent)));
+}
+
+fn answer_hypothetical_frontier_request(
+	view: &View,
+	request: HypotheticalFrontierRequest,
+	tx: oneshot::Sender<Vec<(HypotheticalCandidate, FragmentTreeMembership)>>,
+) {
+	let mut response = Vec::with_capacity(request.candidates.len());
+	for candidate in request.candidates {
+		response.push((candidate, Vec::new()));
+	}
+
+	let required_active_leaf = request.fragment_tree_relay_parent;
+	for (active_leaf, leaf_view) in view
+		.active_leaves
+		.iter()
+		.filter(|(h, _)| required_active_leaf.as_ref().map_or(true, |x| h == &x))
+	{
+		for &mut (ref c, ref mut membership) in &mut response {
+			let fragment_tree = match leaf_view.fragment_trees.get(&c.candidate_para()) {
+				None => continue,
+				Some(f) => f,
+			};
+			let candidate_storage = match view.candidate_storage.get(&c.candidate_para()) {
+				None => continue,
+				Some(storage) => storage,
+			};
+
+			let candidate_hash = c.candidate_hash();
+			let hypothetical = match c {
+				HypotheticalCandidate::Complete { receipt, persisted_validation_data, .. } =>
+					fragment_tree::HypotheticalCandidate::Complete {
+						receipt: Cow::Borrowed(receipt),
+						persisted_validation_data: Cow::Borrowed(persisted_validation_data),
+					},
+				HypotheticalCandidate::Incomplete {
+					parent_head_data_hash,
+					candidate_relay_parent,
+					..
+				} => fragment_tree::HypotheticalCandidate::Incomplete {
+					relay_parent: *candidate_relay_parent,
+					parent_head_data_hash: *parent_head_data_hash,
+				},
+			};
+
+			let depths = fragment_tree.hypothetical_depths(
+				candidate_hash,
+				hypothetical,
+				candidate_storage,
+				request.backed_in_path_only,
+			);
+
+			if !depths.is_empty() {
+				membership.push((*active_leaf, depths));
+			}
+		}
+	}
+
+	let _ = tx.send(response);
+}
+
+fn fragment_tree_membership(
+	active_leaves: &HashMap<Hash, RelayBlockViewData>,
+	para: ParaId,
+	candidate: CandidateHash,
+) -> FragmentTreeMembership {
+	let mut membership = Vec::new();
+	for (relay_parent, view_data) in active_leaves {
+		if let Some(tree) = view_data.fragment_trees.get(&para) {
+			if let Some(depths) = tree.candidate(&candidate) {
+				membership.push((*relay_parent, depths));
+			}
+		}
+	}
+	membership
+}
+
+fn answer_tree_membership_request(
+	view: &View,
+	para: ParaId,
+	candidate: CandidateHash,
+	tx: oneshot::Sender<FragmentTreeMembership>,
+) {
+	let _ = tx.send(fragment_tree_membership(&view.active_leaves, para, candidate));
+}
+
+fn answer_minimum_relay_parents_request(
+	view: &View,
+	relay_parent: Hash,
+	tx: oneshot::Sender<Vec<(ParaId, BlockNumber)>>,
+) {
+	let mut v = Vec::new();
+	if let Some(leaf_data) = view.active_leaves.get(&relay_parent) {
+		for (para_id, fragment_tree) in &leaf_data.fragment_trees {
+			v.push((*para_id, fragment_tree.scope().earliest_relay_parent().number));
+		}
+	}
+
+	let _ = tx.send(v);
+}
+
+fn answer_prospective_validation_data_request(
+	view: &View,
+	request: ProspectiveValidationDataRequest,
+	tx: oneshot::Sender<Option<PersistedValidationData>>,
+) {
+	// 1. Try to get the head-data from the candidate store if known.
+	// 2. Otherwise, it might exist as the base in some relay-parent and we can find it by iterating
+	//    fragment trees.
+	// 3. Otherwise, it is unknown.
+	// 4. Also try to find the relay parent block info by scanning fragment trees.
+	// 5. If head data and relay parent block info are found - success. Otherwise, failure.
+
+	let storage = match view.candidate_storage.get(&request.para_id) {
+		None => {
+			let _ = tx.send(None);
+			return
+		},
+		Some(s) => s,
+	};
+
+	let mut head_data =
+		storage.head_data_by_hash(&request.parent_head_data_hash).map(|x| x.clone());
+	let mut relay_parent_info = None;
+	let mut max_pov_size = None;
+
+	for fragment_tree in view
+		.active_leaves
+		.values()
+		.filter_map(|x| x.fragment_trees.get(&request.para_id))
+	{
+		if head_data.is_some() && relay_parent_info.is_some() && max_pov_size.is_some() {
+			break
+		}
+		if relay_parent_info.is_none() {
+			relay_parent_info =
+				fragment_tree.scope().ancestor_by_hash(&request.candidate_relay_parent);
+		}
+		if head_data.is_none() {
+			let required_parent = &fragment_tree.scope().base_constraints().required_parent;
+			if required_parent.hash() == request.parent_head_data_hash {
+				head_data = Some(required_parent.clone());
+			}
+		}
+		if max_pov_size.is_none() {
+			let contains_ancestor = fragment_tree
+				.scope()
+				.ancestor_by_hash(&request.candidate_relay_parent)
+				.is_some();
+			if contains_ancestor {
+				// We are leaning hard on two assumptions here.
+				// 1. That the fragment tree never contains allowed relay-parents whose session for
+				//    children is different from that of the base block's.
+				// 2. That the max_pov_size is only configurable per session.
+				max_pov_size = Some(fragment_tree.scope().base_constraints().max_pov_size);
+			}
+		}
+	}
+
+	let _ = tx.send(match (head_data, relay_parent_info, max_pov_size) {
+		(Some(h), Some(i), Some(m)) => Some(PersistedValidationData {
+			parent_head: h,
+			relay_parent_number: i.number,
+			relay_parent_storage_root: i.storage_root,
+			max_pov_size: m as _,
+		}),
+		_ => None,
+	});
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn fetch_backing_state<Context>(
+	ctx: &mut Context,
+	relay_parent: Hash,
+	para_id: ParaId,
+) -> JfyiErrorResult<Option<(Constraints, Vec<CandidatePendingAvailability>)>> {
+	let (tx, rx) = oneshot::channel();
+	ctx.send_message(RuntimeApiMessage::Request(
+		relay_parent,
+		RuntimeApiRequest::StagingParaBackingState(para_id, tx),
+	))
+	.await;
+
+	Ok(rx
+		.await
+		.map_err(JfyiError::RuntimeApiRequestCanceled)??
+		.map(|s| (From::from(s.constraints), s.pending_availability)))
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn fetch_upcoming_paras<Context>(
+	ctx: &mut Context,
+	relay_parent: Hash,
+	pending_availability: &mut HashSet<CandidateHash>,
+) -> JfyiErrorResult<Vec<ParaId>> {
+	let (tx, rx) = oneshot::channel();
+
+	// This'll have to get more sophisticated with parathreads,
+	// but for now we can just use the `AvailabilityCores`.
+	ctx.send_message(RuntimeApiMessage::Request(
+		relay_parent,
+		RuntimeApiRequest::AvailabilityCores(tx),
+	))
+	.await;
+
+	let cores = rx.await.map_err(JfyiError::RuntimeApiRequestCanceled)??;
+	let mut upcoming = HashSet::new();
+	for core in cores {
+		match core {
+			CoreState::Occupied(occupied) => {
+				pending_availability.insert(occupied.candidate_hash);
+
+				if let Some(next_up_on_available) = occupied.next_up_on_available {
+					upcoming.insert(next_up_on_available.para_id);
+				}
+				if let Some(next_up_on_time_out) = occupied.next_up_on_time_out {
+					upcoming.insert(next_up_on_time_out.para_id);
+				}
+			},
+			CoreState::Scheduled(scheduled) => {
+				upcoming.insert(scheduled.para_id);
+			},
+			CoreState::Free => {},
+		}
+	}
+
+	Ok(upcoming.into_iter().collect())
+}
+
+// Fetch ancestors in descending order, up to the amount requested.
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn fetch_ancestry<Context>(
+	ctx: &mut Context,
+	cache: &mut HashMap<Hash, Header>,
+	relay_hash: Hash,
+	ancestors: usize,
+) -> JfyiErrorResult<Vec<RelayChainBlockInfo>> {
+	if ancestors == 0 {
+		return Ok(Vec::new())
+	}
+
+	let (tx, rx) = oneshot::channel();
+	ctx.send_message(ChainApiMessage::Ancestors {
+		hash: relay_hash,
+		k: ancestors,
+		response_channel: tx,
+	})
+	.await;
+
+	let hashes = rx.map_err(JfyiError::ChainApiRequestCanceled).await??;
+	let required_session = request_session_index_for_child(relay_hash, ctx.sender())
+		.await
+		.await
+		.map_err(JfyiError::RuntimeApiRequestCanceled)??;
+
+	let mut block_info = Vec::with_capacity(hashes.len());
+	for hash in hashes {
+		let info = match fetch_block_info(ctx, cache, hash).await? {
+			None => {
+				gum::warn!(
+					target: LOG_TARGET,
+					relay_hash = ?hash,
+					"Failed to fetch info for hash returned from ancestry.",
+				);
+
+				// Return, however far we got.
+				break
+			},
+			Some(info) => info,
+		};
+
+		// The relay chain cannot accept blocks backed from previous sessions, with
+		// potentially previous validators. This is a technical limitation we need to
+		// respect here.
+
+		let session = request_session_index_for_child(hash, ctx.sender())
+			.await
+			.await
+			.map_err(JfyiError::RuntimeApiRequestCanceled)??;
+
+		if session == required_session {
+			block_info.push(info);
+		} else {
+			break
+		}
+	}
+
+	Ok(block_info)
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn fetch_block_header_with_cache<Context>(
+	ctx: &mut Context,
+	cache: &mut HashMap<Hash, Header>,
+	relay_hash: Hash,
+) -> JfyiErrorResult<Option<Header>> {
+	if let Some(h) = cache.get(&relay_hash) {
+		return Ok(Some(h.clone()))
+	}
+
+	let (tx, rx) = oneshot::channel();
+
+	ctx.send_message(ChainApiMessage::BlockHeader(relay_hash, tx)).await;
+	let header = rx.map_err(JfyiError::ChainApiRequestCanceled).await??;
+	if let Some(ref h) = header {
+		cache.insert(relay_hash, h.clone());
+	}
+	Ok(header)
+}
+
+#[overseer::contextbounds(ProspectiveParachains, prefix = self::overseer)]
+async fn fetch_block_info<Context>(
+	ctx: &mut Context,
+	cache: &mut HashMap<Hash, Header>,
+	relay_hash: Hash,
+) -> JfyiErrorResult<Option<RelayChainBlockInfo>> {
+	let header = fetch_block_header_with_cache(ctx, cache, relay_hash).await?;
+
+	Ok(header.map(|header| RelayChainBlockInfo {
+		hash: relay_hash,
+		number: header.number,
+		storage_root: header.state_root,
+	}))
+}
@@ -0,0 +1,52 @@
+// Copyright 2023 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+use polkadot_node_subsystem_util::metrics::{self, prometheus};
+
+#[derive(Clone)]
+pub(crate) struct MetricsInner {
+	pub(crate) prune_view_candidate_storage: prometheus::Histogram,
+}
+
+/// Candidate backing metrics.
+#[derive(Default, Clone)]
+pub struct Metrics(pub(crate) Option<MetricsInner>);
+
+impl Metrics {
+	/// Provide a timer for handling `prune_view_candidate_storage` which observes on drop.
+	pub fn time_prune_view_candidate_storage(
+		&self,
+	) -> Option<metrics::prometheus::prometheus::HistogramTimer> {
+		self.0
+			.as_ref()
+			.map(|metrics| metrics.prune_view_candidate_storage.start_timer())
+	}
+}
+
+impl metrics::Metrics for Metrics {
+	fn try_register(registry: &prometheus::Registry) -> Result<Self, prometheus::PrometheusError> {
+		let metrics = MetricsInner {
+			prune_view_candidate_storage: prometheus::register(
+				prometheus::Histogram::with_opts(prometheus::HistogramOpts::new(
+					"polkadot_parachain_prospective_parachains_prune_view_candidate_storage",
+					"Time spent within `prospective_parachains::prune_view_candidate_storage`",
+				))?,
+				registry,
+			)?,
+		};
+		Ok(Metrics(Some(metrics)))
+	}
+}
@@ -28,6 +28,10 @@ pub type Result<T> = std::result::Result<T, Error>;
 #[allow(missing_docs)]
 #[fatality::fatality(splitable)]
 pub enum Error {
+	#[fatal(forward)]
+	#[error("Error while accessing runtime information")]
+	Runtime(#[from] util::runtime::Error),
+
 	#[error(transparent)]
 	Util(#[from] util::Error),

@@ -46,11 +50,14 @@ pub enum Error {
 	#[error("failed to get votes on dispute")]
 	CanceledCandidateVotes(#[source] oneshot::Canceled),

+	#[error("failed to get backable candidate from prospective parachains")]
+	CanceledBackableCandidate(#[source] oneshot::Canceled),
+
 	#[error(transparent)]
 	ChainApi(#[from] ChainApiError),

 	#[error(transparent)]
-	Runtime(#[from] RuntimeApiError),
+	RuntimeApi(#[from] RuntimeApiError),

 	#[error("failed to send message to ChainAPI")]
 	ChainApiMessageSend(#[source] mpsc::SendError),
@@ -28,18 +28,20 @@ use futures_timer::Delay;
 use polkadot_node_subsystem::{
 	jaeger,
 	messages::{
-		CandidateBackingMessage, ChainApiMessage, ProvisionableData, ProvisionerInherentData,
-		ProvisionerMessage, RuntimeApiMessage, RuntimeApiRequest,
+		CandidateBackingMessage, ChainApiMessage, ProspectiveParachainsMessage, ProvisionableData,
+		ProvisionerInherentData, ProvisionerMessage, RuntimeApiMessage, RuntimeApiRequest,
 	},
 	overseer, ActivatedLeaf, ActiveLeavesUpdate, FromOrchestra, LeafStatus, OverseerSignal,
 	PerLeafSpan, RuntimeApiError, SpawnedSubsystem, SubsystemError,
 };
 use polkadot_node_subsystem_util::{
-	request_availability_cores, request_persisted_validation_data, TimeoutExt,
+	request_availability_cores, request_persisted_validation_data,
+	runtime::{prospective_parachains_mode, ProspectiveParachainsMode},
+	TimeoutExt,
 };
 use polkadot_primitives::{
-	BackedCandidate, BlockNumber, CandidateReceipt, CoreState, Hash, OccupiedCoreAssumption,
-	SignedAvailabilityBitfield, ValidatorIndex,
+	BackedCandidate, BlockNumber, CandidateHash, CandidateReceipt, CoreState, Hash, Id as ParaId,
+	OccupiedCoreAssumption, SignedAvailabilityBitfield, ValidatorIndex,
 };
 use std::collections::{BTreeMap, HashMap};

@@ -79,6 +81,7 @@ impl ProvisionerSubsystem {
 pub struct PerRelayParent {
 	leaf: ActivatedLeaf,
 	backed_candidates: Vec<CandidateReceipt>,
+	prospective_parachains_mode: ProspectiveParachainsMode,
 	signed_bitfields: Vec<SignedAvailabilityBitfield>,
 	is_inherent_ready: bool,
 	awaiting_inherent: Vec<oneshot::Sender<ProvisionerInherentData>>,
@@ -86,12 +89,13 @@ pub struct PerRelayParent {
 }

 impl PerRelayParent {
-	fn new(leaf: ActivatedLeaf) -> Self {
+	fn new(leaf: ActivatedLeaf, prospective_parachains_mode: ProspectiveParachainsMode) -> Self {
 		let span = PerLeafSpan::new(leaf.span.clone(), "provisioner");

 		Self {
 			leaf,
 			backed_candidates: Vec::new(),
+			prospective_parachains_mode,
 			signed_bitfields: Vec::new(),
 			is_inherent_ready: false,
 			awaiting_inherent: Vec::new(),
@@ -147,7 +151,7 @@ async fn run_iteration<Context>(
 				// Map the error to ensure that the subsystem exits when the overseer is gone.
 				match from_overseer.map_err(Error::OverseerExited)? {
 					FromOrchestra::Signal(OverseerSignal::ActiveLeaves(update)) =>
-						handle_active_leaves_update(update, per_relay_parent, inherent_delays),
+						handle_active_leaves_update(ctx.sender(), update, per_relay_parent, inherent_delays).await?,
 					FromOrchestra::Signal(OverseerSignal::BlockFinalized(..)) => {},
 					FromOrchestra::Signal(OverseerSignal::Conclude) => return Ok(()),
 					FromOrchestra::Communication { msg } => {
@@ -175,11 +179,12 @@ async fn run_iteration<Context>(
 	}
 }

-fn handle_active_leaves_update(
+async fn handle_active_leaves_update(
+	sender: &mut impl overseer::ProvisionerSenderTrait,
 	update: ActiveLeavesUpdate,
 	per_relay_parent: &mut HashMap<Hash, PerRelayParent>,
 	inherent_delays: &mut InherentDelays,
-) {
+) -> Result<(), Error> {
 	gum::trace!(target: LOG_TARGET, "Handle ActiveLeavesUpdate");
 	for deactivated in &update.deactivated {
 		per_relay_parent.remove(deactivated);
@@ -187,10 +192,13 @@ fn handle_active_leaves_update(

 	if let Some(leaf) = update.activated {
 		gum::trace!(target: LOG_TARGET, leaf_hash=?leaf.hash, "Adding delay");
+		let prospective_parachains_mode = prospective_parachains_mode(sender, leaf.hash).await?;
 		let delay_fut = Delay::new(PRE_PROPOSE_TIMEOUT).map(move |_| leaf.hash).boxed();
-		per_relay_parent.insert(leaf.hash, PerRelayParent::new(leaf));
+		per_relay_parent.insert(leaf.hash, PerRelayParent::new(leaf, prospective_parachains_mode));
 		inherent_delays.push(delay_fut);
 	}
+
+	Ok(())
 }

 #[overseer::contextbounds(Provisioner, prefix = self::overseer)]
@@ -244,6 +252,7 @@ async fn send_inherent_data_bg<Context>(
 	let leaf = per_relay_parent.leaf.clone();
 	let signed_bitfields = per_relay_parent.signed_bitfields.clone();
 	let backed_candidates = per_relay_parent.backed_candidates.clone();
+	let mode = per_relay_parent.prospective_parachains_mode;
 	let span = per_relay_parent.span.child("req-inherent-data");

 	let mut sender = ctx.sender().clone();
@@ -262,6 +271,7 @@ async fn send_inherent_data_bg<Context>(
 			&leaf,
 			&signed_bitfields,
 			&backed_candidates,
+			mode,
 			return_senders,
 			&mut sender,
 			&metrics,
@@ -290,7 +300,6 @@ async fn send_inherent_data_bg<Context>(
 				gum::debug!(
 					target: LOG_TARGET,
 					signed_bitfield_count = signed_bitfields.len(),
-					backed_candidates_count = backed_candidates.len(),
 					leaf_hash = ?leaf.hash,
 					"inherent data sent successfully"
 				);
@@ -325,7 +334,7 @@ fn note_provisionable_data(
 				.child("provisionable-backed")
 				.with_candidate(candidate_hash)
 				.with_para_id(backed_candidate.descriptor().para_id);
-			per_relay_parent.backed_candidates.push(backed_candidate)
+			per_relay_parent.backed_candidates.push(backed_candidate);
 		},
 		// We choose not to punish these forms of misbehavior for the time being.
 		// Risks from misbehavior are sufficiently mitigated at the protocol level
@@ -373,6 +382,7 @@ async fn send_inherent_data(
 	leaf: &ActivatedLeaf,
 	bitfields: &[SignedAvailabilityBitfield],
 	candidates: &[CandidateReceipt],
+	prospective_parachains_mode: ProspectiveParachainsMode,
 	return_senders: Vec<oneshot::Sender<ProvisionerInherentData>>,
 	from_job: &mut impl overseer::ProvisionerSenderTrait,
 	metrics: &Metrics,
@@ -424,8 +434,16 @@ async fn send_inherent_data(
 		relay_parent = ?leaf.hash,
 		"Selected bitfields"
 	);
-	let candidates =
-		select_candidates(&availability_cores, &bitfields, candidates, leaf.hash, from_job).await?;
+
+	let candidates = select_candidates(
+		&availability_cores,
+		&bitfields,
+		candidates,
+		prospective_parachains_mode,
+		leaf.hash,
+		from_job,
+	)
+	.await?;

 	gum::trace!(
 		target: LOG_TARGET,
@@ -532,15 +550,16 @@ fn select_availability_bitfields(
 	selected.into_values().collect()
 }

-/// Determine which cores are free, and then to the degree possible, pick a candidate appropriate to
-/// each free core.
-async fn select_candidates(
+/// Selects candidates from tracked ones to note in a relay chain block.
+///
+/// Should be called when prospective parachains are disabled.
+async fn select_candidate_hashes_from_tracked(
 	availability_cores: &[CoreState],
 	bitfields: &[SignedAvailabilityBitfield],
 	candidates: &[CandidateReceipt],
 	relay_parent: Hash,
 	sender: &mut impl overseer::ProvisionerSenderTrait,
-) -> Result<Vec<BackedCandidate>, Error> {
+) -> Result<Vec<(CandidateHash, Hash)>, Error> {
 	let block_number = get_block_number_under_construction(relay_parent, sender).await?;

 	let mut selected_candidates =
@@ -611,18 +630,112 @@ async fn select_candidates(
 				"Selected candidate receipt",
 			);

-			selected_candidates.push(candidate_hash);
+			selected_candidates.push((candidate_hash, candidate.descriptor.relay_parent));
 		}
 	}

+	Ok(selected_candidates)
+}
+
+/// Requests backable candidates from Prospective Parachains subsystem
+/// based on core states.
+///
+/// Should be called when prospective parachains are enabled.
+async fn request_backable_candidates(
+	availability_cores: &[CoreState],
+	bitfields: &[SignedAvailabilityBitfield],
+	relay_parent: Hash,
+	sender: &mut impl overseer::ProvisionerSenderTrait,
+) -> Result<Vec<(CandidateHash, Hash)>, Error> {
+	let block_number = get_block_number_under_construction(relay_parent, sender).await?;
+
+	let mut selected_candidates = Vec::with_capacity(availability_cores.len());
+
+	for (core_idx, core) in availability_cores.iter().enumerate() {
+		let (para_id, required_path) = match core {
+			CoreState::Scheduled(scheduled_core) => {
+				// The core is free, pick the first eligible candidate from
+				// the fragment tree.
+				(scheduled_core.para_id, Vec::new())
+			},
+			CoreState::Occupied(occupied_core) => {
+				if bitfields_indicate_availability(core_idx, bitfields, &occupied_core.availability)
+				{
+					if let Some(ref scheduled_core) = occupied_core.next_up_on_available {
+						// The candidate occupying the core is available, choose its
+						// child in the fragment tree.
+						//
+						// TODO: doesn't work for on-demand parachains. We lean hard on the
+						// assumption that cores are fixed to specific parachains within a session.
+						// https://github.com/paritytech/polkadot/issues/5492
+						(scheduled_core.para_id, vec![occupied_core.candidate_hash])
+					} else {
+						continue
+					}
+				} else {
+					if occupied_core.time_out_at != block_number {
+						continue
+					}
+					if let Some(ref scheduled_core) = occupied_core.next_up_on_time_out {
+						// Candidate's availability timed out, practically same as scheduled.
+						(scheduled_core.para_id, Vec::new())
+					} else {
+						continue
+					}
+				}
+			},
+			CoreState::Free => continue,
+		};
+
+		let response = get_backable_candidate(relay_parent, para_id, required_path, sender).await?;
+
+		match response {
+			Some((hash, relay_parent)) => selected_candidates.push((hash, relay_parent)),
+			None => {
+				gum::debug!(
+					target: LOG_TARGET,
+					leaf_hash = ?relay_parent,
+					core = core_idx,
+					"No backable candidate returned by prospective parachains",
+				);
+			},
+		}
+	}
+
+	Ok(selected_candidates)
+}
+
+/// Determine which cores are free, and then to the degree possible, pick a candidate appropriate to
+/// each free core.
+async fn select_candidates(
+	availability_cores: &[CoreState],
+	bitfields: &[SignedAvailabilityBitfield],
+	candidates: &[CandidateReceipt],
+	prospective_parachains_mode: ProspectiveParachainsMode,
+	relay_parent: Hash,
+	sender: &mut impl overseer::ProvisionerSenderTrait,
+) -> Result<Vec<BackedCandidate>, Error> {
 	gum::trace!(target: LOG_TARGET,
 		leaf_hash=?relay_parent,
 		"before GetBackedCandidates");

+	let selected_candidates = match prospective_parachains_mode {
+		ProspectiveParachainsMode::Enabled { .. } =>
+			request_backable_candidates(availability_cores, bitfields, relay_parent, sender).await?,
+		ProspectiveParachainsMode::Disabled =>
+			select_candidate_hashes_from_tracked(
+				availability_cores,
+				bitfields,
+				&candidates,
+				relay_parent,
+				sender,
+			)
+			.await?,
+	};
+
 	// now get the backed candidates corresponding to these candidate receipts
 	let (tx, rx) = oneshot::channel();
 	sender.send_unbounded_message(CandidateBackingMessage::GetBackedCandidates(
-		relay_parent,
 		selected_candidates.clone(),
 		tx,
 	));
@@ -638,7 +751,7 @@ async fn select_candidates(
 	// checking them in order, we can ensure that the backed candidates are also in order.
 	let mut backed_idx = 0;
 	for selected in selected_candidates {
-		if selected ==
+		if selected.0 ==
 			candidates.get(backed_idx).ok_or(Error::BackedCandidateOrderingProblem)?.hash()
 		{
 			backed_idx += 1;
@@ -689,6 +802,27 @@ async fn get_block_number_under_construction(
 	}
 }

+/// Requests backable candidate from Prospective Parachains based on
+/// the given path in the fragment tree.
+async fn get_backable_candidate(
+	relay_parent: Hash,
+	para_id: ParaId,
+	required_path: Vec<CandidateHash>,
+	sender: &mut impl overseer::ProvisionerSenderTrait,
+) -> Result<Option<(CandidateHash, Hash)>, Error> {
+	let (tx, rx) = oneshot::channel();
+	sender
+		.send_message(ProspectiveParachainsMessage::GetBackableCandidate(
+			relay_parent,
+			para_id,
+			required_path,
+			tx,
+		))
+		.await;
+
+	rx.await.map_err(Error::CanceledBackableCandidate)
+}
+
 /// The availability bitfield for a given core is the transpose
 /// of a set of signed availability bitfields. It goes like this:
 ///
@@ -19,6 +19,8 @@ use ::test_helpers::{dummy_candidate_descriptor, dummy_hash};
 use bitvec::bitvec;
 use polkadot_primitives::{OccupiedCore, ScheduledCore};

+const MOCK_GROUP_SIZE: usize = 5;
+
 pub fn occupied_core(para_id: u32) -> CoreState {
 	CoreState::Occupied(OccupiedCore {
 		group_responsible: para_id.into(),
@@ -46,8 +48,8 @@ where
 	CoreState::Occupied(core)
 }

-pub fn default_bitvec(n_cores: usize) -> CoreAvailability {
-	bitvec![u8, bitvec::order::Lsb0; 0; n_cores]
+pub fn default_bitvec(size: usize) -> CoreAvailability {
+	bitvec![u8, bitvec::order::Lsb0; 0; size]
 }

 pub fn scheduled_core(id: u32) -> ScheduledCore {
@@ -237,7 +239,7 @@ pub(crate) mod common {
 mod select_candidates {
 	use super::{
 		super::*, build_occupied_core, common::test_harness, default_bitvec, occupied_core,
-		scheduled_core,
+		scheduled_core, MOCK_GROUP_SIZE,
 	};
 	use ::test_helpers::{dummy_candidate_descriptor, dummy_hash};
 	use futures::channel::mpsc;
@@ -248,6 +250,7 @@ mod select_candidates {
 		},
 	};
 	use polkadot_node_subsystem_test_helpers::TestSubsystemSender;
+	use polkadot_node_subsystem_util::runtime::ProspectiveParachainsMode;
 	use polkadot_primitives::{
 		BlockNumber, CandidateCommitments, CommittedCandidateReceipt, PersistedValidationData,
 	};
@@ -333,10 +336,17 @@ mod select_candidates {
 	async fn mock_overseer(
 		mut receiver: mpsc::UnboundedReceiver<AllMessages>,
 		expected: Vec<BackedCandidate>,
+		prospective_parachains_mode: ProspectiveParachainsMode,
 	) {
 		use ChainApiMessage::BlockNumber;
 		use RuntimeApiMessage::Request;

+		let mut candidates_iter = expected
+			.iter()
+			.map(|candidate| (candidate.hash(), candidate.descriptor().relay_parent));
+
+		let mut backed_iter = expected.clone().into_iter();
+
 		while let Some(from_job) = receiver.next().await {
 			match from_job {
 				AllMessages::ChainApi(BlockNumber(_relay_parent, tx)) =>
@@ -348,11 +358,28 @@ mod select_candidates {
 				AllMessages::RuntimeApi(Request(_parent_hash, AvailabilityCores(tx))) =>
 					tx.send(Ok(mock_availability_cores())).unwrap(),
 				AllMessages::CandidateBacking(CandidateBackingMessage::GetBackedCandidates(
-					_,
-					_,
+					hashes,
 					sender,
 				)) => {
-					let _ = sender.send(expected.clone());
+					let response: Vec<BackedCandidate> =
+						backed_iter.by_ref().take(hashes.len()).collect();
+					let expected_hashes: Vec<(CandidateHash, Hash)> = response
+						.iter()
+						.map(|candidate| (candidate.hash(), candidate.descriptor().relay_parent))
+						.collect();
+
+					assert_eq!(expected_hashes, hashes);
+
+					let _ = sender.send(response);
+				},
+				AllMessages::ProspectiveParachains(
+					ProspectiveParachainsMessage::GetBackableCandidate(.., tx),
+				) => match prospective_parachains_mode {
+					ProspectiveParachainsMode::Enabled { .. } => {
+						let _ = tx.send(candidates_iter.next());
+					},
+					ProspectiveParachainsMode::Disabled =>
+						panic!("unexpected prospective parachains request"),
 				},
 				_ => panic!("Unexpected message: {:?}", from_job),
 			}
@@ -362,9 +389,19 @@ mod select_candidates {
 	#[test]
 	fn can_succeed() {
 		test_harness(
-			|r| mock_overseer(r, Vec::new()),
+			|r| mock_overseer(r, Vec::new(), ProspectiveParachainsMode::Disabled),
 			|mut tx: TestSubsystemSender| async move {
-				select_candidates(&[], &[], &[], Default::default(), &mut tx).await.unwrap();
+				let prospective_parachains_mode = ProspectiveParachainsMode::Disabled;
+				select_candidates(
+					&[],
+					&[],
+					&[],
+					prospective_parachains_mode,
+					Default::default(),
+					&mut tx,
+				)
+				.await
+				.unwrap();
 			},
 		)
 	}
@@ -375,7 +412,6 @@ mod select_candidates {
 	#[test]
 	fn selects_correct_candidates() {
 		let mock_cores = mock_availability_cores();
-		let n_cores = mock_cores.len();

 		let empty_hash = PersistedValidationData::<Hash, BlockNumber>::default().hash();

@@ -415,6 +451,7 @@ mod select_candidates {
 		// why those particular indices? see the comments on mock_availability_cores()
 		let expected_candidates: Vec<_> =
 			[1, 4, 7, 8, 10].iter().map(|&idx| candidates[idx].clone()).collect();
+		let prospective_parachains_mode = ProspectiveParachainsMode::Disabled;

 		let expected_backed = expected_candidates
 			.iter()
@@ -424,17 +461,23 @@ mod select_candidates {
 					commitments: Default::default(),
 				},
 				validity_votes: Vec::new(),
-				validator_indices: default_bitvec(n_cores),
+				validator_indices: default_bitvec(MOCK_GROUP_SIZE),
 			})
 			.collect();

 		test_harness(
-			|r| mock_overseer(r, expected_backed),
+			|r| mock_overseer(r, expected_backed, prospective_parachains_mode),
 			|mut tx: TestSubsystemSender| async move {
-				let result =
-					select_candidates(&mock_cores, &[], &candidates, Default::default(), &mut tx)
-						.await
-						.unwrap();
+				let result = select_candidates(
+					&mock_cores,
+					&[],
+					&candidates,
+					prospective_parachains_mode,
+					Default::default(),
+					&mut tx,
+				)
+				.await
+				.unwrap();

 				result.into_iter().for_each(|c| {
 					assert!(
@@ -450,15 +493,16 @@ mod select_candidates {
 	#[test]
 	fn selects_max_one_code_upgrade() {
 		let mock_cores = mock_availability_cores();
-		let n_cores = mock_cores.len();

 		let empty_hash = PersistedValidationData::<Hash, BlockNumber>::default().hash();

 		// why those particular indices? see the comments on mock_availability_cores()
 		// the first candidate with code is included out of [1, 4, 7, 8, 10].
-		let cores = [1, 7, 10];
+		let cores = [1, 4, 7, 8, 10];
 		let cores_with_code = [1, 4, 8];

+		let expected_cores = [1, 7, 10];
+
 		let committed_receipts: Vec<_> = (0..mock_cores.len())
 			.map(|i| {
 				let mut descriptor = dummy_candidate_descriptor(dummy_hash());
@@ -478,27 +522,173 @@ mod select_candidates {
 			})
 			.collect();

+		// Input to select_candidates
 		let candidates: Vec<_> = committed_receipts.iter().map(|r| r.to_plain()).collect();
-
-		let expected_candidates: Vec<_> =
-			cores.iter().map(|&idx| candidates[idx].clone()).collect();
-
-		let expected_backed: Vec<_> = cores
+		// Build possible outputs from select_candidates
+		let backed_candidates: Vec<_> = committed_receipts
 			.iter()
-			.map(|&idx| BackedCandidate {
-				candidate: committed_receipts[idx].clone(),
+			.map(|committed_receipt| BackedCandidate {
+				candidate: committed_receipt.clone(),
 				validity_votes: Vec::new(),
-				validator_indices: default_bitvec(n_cores),
+				validator_indices: default_bitvec(MOCK_GROUP_SIZE),
+			})
+			.collect();
+
+		// First, provisioner will request backable candidates for each scheduled core.
+		// Then, some of them get filtered due to new validation code rule.
+		let expected_backed: Vec<_> =
+			cores.iter().map(|&idx| backed_candidates[idx].clone()).collect();
+		let expected_backed_filtered: Vec<_> =
+			expected_cores.iter().map(|&idx| candidates[idx].clone()).collect();
+
+		let prospective_parachains_mode = ProspectiveParachainsMode::Disabled;
+
+		test_harness(
+			|r| mock_overseer(r, expected_backed, prospective_parachains_mode),
+			|mut tx: TestSubsystemSender| async move {
+				let result = select_candidates(
+					&mock_cores,
+					&[],
+					&candidates,
+					prospective_parachains_mode,
+					Default::default(),
+					&mut tx,
+				)
+				.await
+				.unwrap();
+
+				assert_eq!(result.len(), 3);
+
+				result.into_iter().for_each(|c| {
+					assert!(
+						expected_backed_filtered.iter().any(|c2| c.candidate.corresponds_to(c2)),
+						"Failed to find candidate: {:?}",
+						c,
+					)
+				});
+			},
+		)
+	}
+
+	#[test]
+	fn request_from_prospective_parachains() {
+		let mock_cores = mock_availability_cores();
+		let empty_hash = PersistedValidationData::<Hash, BlockNumber>::default().hash();
+
+		let mut descriptor_template = dummy_candidate_descriptor(dummy_hash());
+		descriptor_template.persisted_validation_data_hash = empty_hash;
+		let candidate_template = CandidateReceipt {
+			descriptor: descriptor_template,
+			commitments_hash: CandidateCommitments::default().hash(),
+		};
+
+		let candidates: Vec<_> = std::iter::repeat(candidate_template)
+			.take(mock_cores.len())
+			.enumerate()
+			.map(|(idx, mut candidate)| {
+				candidate.descriptor.para_id = idx.into();
+				candidate
+			})
+			.collect();
+
+		// why those particular indices? see the comments on mock_availability_cores()
+		let expected_candidates: Vec<_> =
+			[1, 4, 7, 8, 10].iter().map(|&idx| candidates[idx].clone()).collect();
+		// Expect prospective parachains subsystem requests.
+		let prospective_parachains_mode =
+			ProspectiveParachainsMode::Enabled { max_candidate_depth: 0, allowed_ancestry_len: 0 };
+
+		let expected_backed = expected_candidates
+			.iter()
+			.map(|c| BackedCandidate {
+				candidate: CommittedCandidateReceipt {
+					descriptor: c.descriptor.clone(),
+					commitments: Default::default(),
+				},
+				validity_votes: Vec::new(),
+				validator_indices: default_bitvec(MOCK_GROUP_SIZE),
 			})
 			.collect();

 		test_harness(
-			|r| mock_overseer(r, expected_backed),
+			|r| mock_overseer(r, expected_backed, prospective_parachains_mode),
 			|mut tx: TestSubsystemSender| async move {
-				let result =
-					select_candidates(&mock_cores, &[], &candidates, Default::default(), &mut tx)
-						.await
-						.unwrap();
+				let result = select_candidates(
+					&mock_cores,
+					&[],
+					&[],
+					prospective_parachains_mode,
+					Default::default(),
+					&mut tx,
+				)
+				.await
+				.unwrap();
+
+				result.into_iter().for_each(|c| {
+					assert!(
+						expected_candidates.iter().any(|c2| c.candidate.corresponds_to(c2)),
+						"Failed to find candidate: {:?}",
+						c,
+					)
+				});
+			},
+		)
+	}
+
+	#[test]
+	fn request_receipts_based_on_relay_parent() {
+		let mock_cores = mock_availability_cores();
+		let empty_hash = PersistedValidationData::<Hash, BlockNumber>::default().hash();
+
+		let mut descriptor_template = dummy_candidate_descriptor(dummy_hash());
+		descriptor_template.persisted_validation_data_hash = empty_hash;
+		let candidate_template = CandidateReceipt {
+			descriptor: descriptor_template,
+			commitments_hash: CandidateCommitments::default().hash(),
+		};
+
+		let candidates: Vec<_> = std::iter::repeat(candidate_template)
+			.take(mock_cores.len())
+			.enumerate()
+			.map(|(idx, mut candidate)| {
+				candidate.descriptor.para_id = idx.into();
+				candidate.descriptor.relay_parent = Hash::repeat_byte(idx as u8);
+				candidate
+			})
+			.collect();
+
+		// why those particular indices? see the comments on mock_availability_cores()
+		let expected_candidates: Vec<_> =
+			[1, 4, 7, 8, 10].iter().map(|&idx| candidates[idx].clone()).collect();
+		// Expect prospective parachains subsystem requests.
+		let prospective_parachains_mode =
+			ProspectiveParachainsMode::Enabled { max_candidate_depth: 0, allowed_ancestry_len: 0 };
+
+		let expected_backed = expected_candidates
+			.iter()
+			.map(|c| BackedCandidate {
+				candidate: CommittedCandidateReceipt {
+					descriptor: c.descriptor.clone(),
+					commitments: Default::default(),
+				},
+				validity_votes: Vec::new(),
+				validator_indices: default_bitvec(MOCK_GROUP_SIZE),
+			})
+			.collect();
+
+		test_harness(
+			|r| mock_overseer(r, expected_backed, prospective_parachains_mode),
+			|mut tx: TestSubsystemSender| async move {
+				let result = select_candidates(
+					&mock_cores,
+					&[],
+					&[],
+					prospective_parachains_mode,
+					Default::default(),
+					&mut tx,
+				)
+				.await
+				.unwrap();

 				result.into_iter().for_each(|c| {
 					assert!(
@@ -68,6 +68,9 @@ pub(crate) struct RequestResultCache {
 		LruCache<Hash, Vec<(SessionIndex, CandidateHash, vstaging::slashing::PendingSlashes)>>,
 	key_ownership_proof:
 		LruCache<(Hash, ValidatorId), Option<vstaging::slashing::OpaqueKeyOwnershipProof>>,
+
+	staging_para_backing_state: LruCache<(Hash, ParaId), Option<vstaging::BackingState>>,
+	staging_async_backing_params: LruCache<Hash, vstaging::AsyncBackingParams>,
 }

 impl Default for RequestResultCache {
@@ -97,6 +100,9 @@ impl Default for RequestResultCache {
 			disputes: LruCache::new(DEFAULT_CACHE_CAP),
 			unapplied_slashes: LruCache::new(DEFAULT_CACHE_CAP),
 			key_ownership_proof: LruCache::new(DEFAULT_CACHE_CAP),
+
+			staging_para_backing_state: LruCache::new(DEFAULT_CACHE_CAP),
+			staging_async_backing_params: LruCache::new(DEFAULT_CACHE_CAP),
 		}
 	}
 }
@@ -430,6 +436,36 @@ impl RequestResultCache {
 	) -> Option<&Option<()>> {
 		None
 	}
+
+	pub(crate) fn staging_para_backing_state(
+		&mut self,
+		key: (Hash, ParaId),
+	) -> Option<&Option<vstaging::BackingState>> {
+		self.staging_para_backing_state.get(&key)
+	}
+
+	pub(crate) fn cache_staging_para_backing_state(
+		&mut self,
+		key: (Hash, ParaId),
+		value: Option<vstaging::BackingState>,
+	) {
+		self.staging_para_backing_state.put(key, value);
+	}
+
+	pub(crate) fn staging_async_backing_params(
+		&mut self,
+		key: &Hash,
+	) -> Option<&vstaging::AsyncBackingParams> {
+		self.staging_async_backing_params.get(key)
+	}
+
+	pub(crate) fn cache_staging_async_backing_params(
+		&mut self,
+		key: Hash,
+		value: vstaging::AsyncBackingParams,
+	) {
+		self.staging_async_backing_params.put(key, value);
+	}
 }

 pub(crate) enum RequestResult {
@@ -476,4 +512,7 @@ pub(crate) enum RequestResult {
 		vstaging::slashing::OpaqueKeyOwnershipProof,
 		Option<()>,
 	),
+
+	StagingParaBackingState(Hash, ParaId, Option<vstaging::BackingState>),
+	StagingAsyncBackingParams(Hash, vstaging::AsyncBackingParams),
 }
@@ -163,6 +163,12 @@ where
 				.requests_cache
 				.cache_key_ownership_proof((relay_parent, validator_id), key_ownership_proof),
 			SubmitReportDisputeLost(_, _, _, _) => {},
+
+			StagingParaBackingState(relay_parent, para_id, constraints) => self
+				.requests_cache
+				.cache_staging_para_backing_state((relay_parent, para_id), constraints),
+			StagingAsyncBackingParams(relay_parent, params) =>
+				self.requests_cache.cache_staging_async_backing_params(relay_parent, params),
 		}
 	}

@@ -288,6 +294,13 @@ where
 						Request::SubmitReportDisputeLost(dispute_proof, key_ownership_proof, sender)
 					},
 				),
+
+			Request::StagingParaBackingState(para, sender) =>
+				query!(staging_para_backing_state(para), sender)
+					.map(|sender| Request::StagingParaBackingState(para, sender)),
+			Request::StagingAsyncBackingParams(sender) =>
+				query!(staging_async_backing_params(), sender)
+					.map(|sender| Request::StagingAsyncBackingParams(sender)),
 		}
 	}

@@ -538,5 +551,22 @@ where
 			ver = Request::SUBMIT_REPORT_DISPUTE_LOST_RUNTIME_REQUIREMENT,
 			sender
 		),
+
+		Request::StagingParaBackingState(para, sender) => {
+			query!(
+				StagingParaBackingState,
+				staging_para_backing_state(para),
+				ver = Request::STAGING_BACKING_STATE,
+				sender
+			)
+		},
+		Request::StagingAsyncBackingParams(sender) => {
+			query!(
+				StagingAsyncBackingParams,
+				staging_async_backing_params(),
+				ver = Request::STAGING_BACKING_STATE,
+				sender
+			)
+		},
 	}
 }
@@ -249,6 +249,21 @@ impl RuntimeApiSubsystemClient for MockSubsystemClient {
 	async fn authorities(&self, _: Hash) -> Result<Vec<AuthorityDiscoveryId>, ApiError> {
 		Ok(self.authorities.clone())
 	}
+
+	async fn staging_async_backing_params(
+		&self,
+		_: Hash,
+	) -> Result<vstaging::AsyncBackingParams, ApiError> {
+		todo!("Not required for tests")
+	}
+
+	async fn staging_para_backing_state(
+		&self,
+		_: Hash,
+		_: ParaId,
+	) -> Result<Option<vstaging::BackingState>, ApiError> {
+		todo!("Not required for tests")
+	}
 }

 #[test]