33 Commits

Author	SHA1	Message	Date
asynchronous rob	5174b9d2d7	Asynchronous Backing MegaPR (#5022) ... * inclusion emulator logic for asynchronous backing (#4790) * initial stab at candidate_context * fmt * docs & more TODOs * some cleanups * reframe as inclusion_emulator * documentations yes * update types * add constraint modifications * watermark * produce modifications * v2 primitives: re-export all v1 for consistency * vstaging primitives * emulator constraints: handle code upgrades * produce outbound HRMP modifications * stack. * method for applying modifications * method just for sanity-checking modifications * fragments produce modifications, not prospectives * make linear * add some TODOs * remove stacking; handle code upgrades * take `fragment` private * reintroduce stacking. * fragment constructor * add TODO * allow validating fragments against future constraints * docs * relay-parent number and min code size checks * check code upgrade restriction * check max hrmp per candidate * fmt * remove GoAhead logic because it wasn't helpful * docs on code upgrade failure * test stacking * test modifications against constraints * fmt * test fragments * descending or duplicate test * fmt * remove unused imports in vstaging * wrong primitives * spellcheck * Runtime changes for Asynchronous Backing (#4786) * inclusion: utility for allowed relay-parents * inclusion: use prev number instead of prev hash * track most recent context of paras * inclusion: accept previous relay-parents * update dmp advancement rule for async backing * fmt * add a comment about validation outputs * clean up a couple of TODOs * weights * fix weights * fmt * Resolve dmp todo * Restore inclusion tests * Restore paras_inherent tests * MostRecentContext test * Benchmark for new paras dispatchable * Prepare check_validation_outputs for upgrade * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs * Implementers guide changes * More tests for allowed relay parents * Add a github issue link * Compute group index based on relay parent * Storage migration * Move allowed parents tracker to shared * Compile error * Get group assigned to core at the next block * Test group assignment * fmt * Error instead of panic * Update guide * Extend doc-comment * Update runtime/parachains/src/shared.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Prospective Parachains Subsystem (#4913) * docs and skeleton * subsystem skeleton * main loop * fragment tree basics & fmt * begin fragment trees & view * flesh out more of view update logic * further flesh out update logic * some refcount functions for fragment trees * add fatal/non-fatal errors * use non-fatal results * clear up some TODOs * ideal format for scheduling info * add a bunch of TODOs * some more fluff * extract fragment graph to submodule * begin fragment graph API * trees, not graphs * improve docs * scope and constructor for trees * add some test TODOs * limit max ancestors and store constraints * constructor * constraints: fix bug in HRMP watermarks * fragment tree population logic * set::retain * extract population logic * implement add_and_populate * fmt * add some TODOs in tests * implement child-selection * strip out old stuff based on wrong assumptions * use fatality * implement pruning * remove unused ancestor constraints * fragment tree instantiation * remove outdated comment * add message/request types and skeleton for handling * fmt * implement handle_candidate_seconded * candidate storage: handle backed * implement handle_candidate_backed * implement answer_get_backable_candidate * remove async where not needed * implement fetch_ancestry * add logic for run_iteration * add some docs * remove global allow(unused), fix warnings * make spellcheck happy (despite English) * fmt * bump Cargo.lock * replace tracing with gum * introduce PopulateFrom trait * implement GetHypotheticalDepths * revise docs slightly * first fragment tree scope test * more scope tests * test add_candidate * fmt * test retain * refactor test code * test populate is recursive * test contiguity of depth 0 is maintained * add_and_populate tests * cycle tests * remove PopulateFrom trait * fmt * test hypothetical depths (non-recursive) * have CandidateSeconded return membership * tree membership requests * Add a ProspectiveParachainsSubsystem struct * add a staging API for base constraints * add a `From` impl * add runtime API for staging_validity_constraints * implement fetch_base_constraints * implement `fetch_upcoming_paras` * remove reconstruction of candidate receipt; no obvious usecase * fmt * export message to broader module * remove last TODO * correctly export * fix compilation and add GetMinimumRelayParent request * make provisioner into a real subsystem with proper mesage bounds * fmt * fix ChannelsOut in overseer test * fix overseer tests * fix again * fmt * Integrate prospective parachains subsystem into backing: Part 1 (#5557) * BEGIN ASYNC candidate-backing CHANGES * rename & document modes * answer prospective validation data requests * GetMinimumRelayParents request is now plural * implement an implicit view utility for backing subsystems * implicit-view: get allowed relay parents * refactorings and improvements to implicit view * add some TODOs for tests * split implicit view updates into 2 functions * backing: define State to prepare for functional refactor * add some docs * backing: implement bones of new leaf activation logic * backing: create per-relay-parent-states * use new handle_active_leaves_update * begin extracting logic from CandidateBackingJob * mostly extract statement import from job logic * handle statement imports outside of job logic * do some TODO planning for prospective parachains integration * finish rewriting backing subsystem in functional style * add prospective parachains mode to relay parent entries * fmt * add a RejectedByProspectiveParachains error * notify prospective parachains of seconded and backed candidates * always validate candidates exhaustively in backing. * return persisted_validation_data from validation * handle rejections by prospective parachains * implement seconding sanity check * invoke validate_and_second * Alter statement table to allow multiple seconded messages per validator * refactor backing to have statements carry PVD * clean up all warnings * Add tests for implicit view * Improve doc comments * Prospective parachains mode based on Runtime API version * Add a TODO * Rework seconding_sanity_check * Iterate over responses * Update backing tests * collator-protocol: load PVD from runtime * Fix validator side tests * Update statement-distribution to fetch PVD * Fix statement-distribution tests * Backing tests with prospective paras #1 * fix per_relay_parent pruning in backing * Test multiple leaves * Test seconding sanity check * Import statement order Before creating an entry in `PerCandidateState` map wait for the approval from the prospective parachains * Add a test for correct state updates * Second multiple candidates per relay parent test * Add backing tests with prospective paras * Second more than one test without prospective paras * Add a test for prospective para blocks * Update malus * typos Co-authored-by: Chris Sosnin <chris125_@live.com> * Track occupied depth in backing per parachain (#5778) * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * provisioner: async backing changes (#5711) * Provisioner changes for async backing * Select candidates based on prospective paras mode * Revert naming * Update tests * Update TODO comment * review * fmt * Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991) * BEGIN STATEMENT DISTRIBUTION WORK create a vstaging network protocol which is the same as v1 * mostly make network bridge amenable to vstaging * network-bridge: fully adapt to vstaging * add some TODOs for tests * fix fallout in bitfield-distribution * bitfield distribution tests + TODOs * fix fallout in gossip-support * collator-protocol: fix message fallout * collator-protocol: load PVD from runtime * add TODO for vstaging tests * make things compile * set used network protocol version using a feature * fmt * get approval-distribution building * fix approval-distribution tests * spellcheck * nits * approval distribution net protocol test * bitfield distribution net protocol test * Revert "collator-protocol: fix message fallout" This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1. * Network bridge tests Co-authored-by: Chris Sosnin <chris125_@live.com> * remove max_pov_size requirement from prospective pvd request (#6014) * remove max_pov_size requirement from prospective pvd request * fmt * Extract legacy statement distribution to its own module (#6026) * add compatibility type to v2 statement distribution message * warning cleanup * handle compatibility layer for v2 * clean up an unimplemented!() block * circulate statements based on version * extract legacy v1 code into separate module * remove unimplemented * clean up naming of from_requester/responder * remove TODOs * have backing share seconded statements with PVD * fmt * fix warning * Quick fix unused warning for not yet implemented/used staging messages. * Fix network bridge test * Fix wrong merge. We now have 23 subsystems (network bridge split + prospective parachains) Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> * Version 3 is already live. * Fix tests (#6055) * Fix backing tests * Fix warnings. * fmt * collator-protocol: asynchronous backing changes (#5740) * Draft collator side changes * Start working on collations management * Handle peer's view change * Versioning on advertising * Versioned collation fetching request * Handle versioned messages * Improve docs for collation requests * Add spans * Add request receiver to overseer * Fix collator side tests * Extract relay parent mode to lib * Validator side draft * Add more checks for advertisement * Request pvd based on async backing mode * review * Validator side improvements * Make old tests green * More fixes * Collator side tests draft * Send collation test * fmt * Collator side network protocol versioning * cleanup * merge artifacts * Validator side net protocol versioning * Remove fragment tree membership request * Resolve todo * Collator side core state test * Improve net protocol compatibility * Validator side tests * more improvements * style fixes * downgrade log * Track implicit assignments * Limit the number of seconded candidates per para * Add a sanity check * Handle fetched candidate * fix tests * Retry fetch * Guard against dequeueing while already fetching * Reintegrate connection management * Timeout on advertisements * fmt * spellcheck * update tests after merge * validator assignment fixes for backing and collator protocol (#6158) * Rename depth->ancestry len in tests * Refactor group assignments * Remove implicit assignments * backing: consider occupied core assignments * Track a single para on validator side * Refactor prospective parachains mode request (#6179) * Extract prospective parachains mode into util * Skip activations depending on the mode * backing: don't send backed candidate to provisioner (#6185) * backing: introduce `CanSecond` request for advertisements filtering (#6225) * Drop BoundToRelayParent * draft changes * fix backing tests * Fix genesis ancestry * Fix validator side tests * more tests * cargo generate-lockfile * Implement `StagingValidityConstraints` Runtime API method (#6258) * Implement StagingValidityConstraints * spellcheck * fix ump params * Update hrmp comment * Introduce ump per candidate limit * hypothetical earliest block * refactor primitives usage * hypothetical earliest block number test * fix build * Prepare the Runtime for asynchronous backing upgrade (#6287) * Introduce async backing params to runtime config * fix cumulus config * use config * finish runtimes * Introduce new staging API * Update collator protocol * Update provisioner * Update prospective parachains * Update backing * Move async backing params lower in the config * make naming consistent * misc * Use real prospective parachains subsystem (#6407) * Backport `HypotheticalFrontier` into the feature branch (#6605) * implement more general HypotheticalFrontier * fmt * drop unneeded request Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Resolve todo about legacy leaf activation (#6447) * fix bug/warning in handling membership answers * Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521) * Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest` * Update tests * Fix (removed wrong docstring) * Fix can_second request * Patch some dead_code errors --------- Co-authored-by: Chris Sosnin <chris125_@live.com> * Async Backing: Send Statement Distribution "Backed" messages (#6634) * Backing: Send Statement Distribution "Backed" messages Closes #6590. **TODO:** - [ ] Adjust tests * Fix compile errors * (Mostly) fix tests * Fix comment * Fix test and compile error * Test that `StatementDistributionMessage::Backed` is sent * Fix compile error * Fix some clippy errors * Add prospective parachains subsystem tests (#6454) * Add prospective parachains subsystem test * Add `should_do_no_work_if_async_backing_disabled_for_leaf` test * Implement `activate_leaf` helper, up to getting ancestry * Finish implementing `activate_leaf` * Small refactor in `activate_leaf` * Get `CandidateSeconded` working * Finish `send_candidate_and_check_if_found` test * Refactor; send more leaves & candidates * Refactor test * Implement `check_candidate_parent_leaving_view` test * Start work on `check_candidate_on_multiple_forks` test * Don’t associate specific parachains with leaf * Finish `correctly_updates_leaves` test * Fix cycle due to reused head data * Fix `check_backable_query` test * Fix `check_candidate_on_multiple_forks` test * Add `check_depth_and_pvd_queries` test * Address review comments * Remove TODO * add a new index for output head data to candidate storage * Resolve test TODOs * Fix compile errors * test candidate storage pruning, make sure new index is cleaned up --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Node-side metrics for asynchronous backing (#6549) * Add metrics for `prune_view_candidate_storage` * Add metrics for `request_unblocked_collations` * Fix docstring * Couple fixes from review comments * Fix `check_depth_query` test * inclusion-emulator: mirror advancement rule check (#6361) * inclusion-emulator: mirror advancement rule check * fix build * prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649) * Introduce `backed_in_path_only` flag for depth request * fmt * update doc comment * fmt * Add async-backing zombienet tests (#6314) * Async backing: impl guide for statement distribution (#6738) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> * Asynchronous backing statement distribution: Take III (#5999) * add notification types for v2 statement-distribution * improve protocol docs * add empty vstaging module * fmt * add backed candidate packet request types * start putting down structure of new logic * handle activated leaf * some sanity-checking on outbound statements * fmt * update vstaging share to use statements with PVD * tiny refactor, candidate_hash location * import local statements * refactor statement import * first stab at broadcast logic * fmt * fill out some TODOs * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * fmt, fix grid test after topology change * typo Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * address review * adjust comment, make easier to understand * Fix typo --------- Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * miscellaneous fixes to make asynchronous backing work (#6791) * propagate network-protocol-staging feature * add feature to adder-collator as well * allow collation-generation of occupied cores * prospective parachains: special treatment for pending availability candidates * runtime: fetch candidates pending availability * lazily construct PVD for pending candidates * fix fallout in prospective parachains hypothetical/select_child * runtime: enact candidates when creating paras-inherent * make tests compile * test pending availability in the scope * add prospective parachains test * fix validity constraints leftovers * drop prints * Fix typos --------- Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Marcin S <marcin@realemail.net> * Remove restart from test (#6840) * Async Backing: Statement Distribution Tests (#6755) * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * Hook up request sender * Add `valid_statement_without_prior_seconded_is_ignored` test * Fix `valid_statement_without_prior_seconded_is_ignored` test * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Remove obsolete test * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Update node/network/protocol/src/request_response/mod.rs Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> * Mark receiver with `vstaging` * First draft of `ensure_seconding_limit_is_respected` test * validate grid senders based on manifest kind * fix mask_seconded/valid * fix unwanted-mask check * fix build * resolve todo on leaf mode * Unify protocol naming to vstaging * Fix `ensure_seconding_limit_is_respected` test * Start `backed_candidate_leads_to_advertisement` test * fmt, fix grid test after topology change * Send Backed notification * Finish `backed_candidate_leads_to_advertisement` test * Finish `peer_reported_for_duplicate_statements` test * Finish `received_advertisement_before_confirmation_leads_to_request` * Add `advertisements_rejected_from_incorrect_peers` test * Add `manifest_rejected_*` tests * Add `manifest_rejected_when_group_does_not_match_para` test * Add `local_node_sanity_checks_incoming_requests` test * Add `local_node_respects_statement_mask` test * Add tests where peer is reported for providing invalid signatures * Add `cluster_peer_allowed_to_send_incomplete_statements` test * Add `received_advertisement_after_backing_leads_to_acknowledgement` * Add `received_advertisement_after_confirmation_before_backing` test * peer_reported_for_advertisement_conflicting_with_confirmed_candidate * Add `peer_reported_for_not_enough_statements` test * Add `peer_reported_for_providing_statements_meant_to_be_masked_out` * Add `additional_statements_are_shared_after_manifest_exchange` * Add `grid_statements_imported_to_backing` test * Add `relay_parent_entering_peer_view_leads_to_advertisement` test * Add `advertisement_not_re_sent_when_peer_re_enters_view` test * Update node/network/statement-distribution/src/vstaging/tests/grid.rs Co-authored-by: asynchronous rob <rphmeier@gmail.com> * Resolve TODOs, update test * Address unused code * Add check after every test for unhandled requests * Refactor (`make_dummy_leaf` and `handle_sent_request`) * Refactor (`make_dummy_topology`) * Minor refactor --------- Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Chris Sosnin <chris125_@live.com> * Fix some clippy lints in tests * Async backing: minor fixes (#6920) * bitfield-distribution test * implicit view tests * Refactor parameters -> params * scheduler: update storage migration (#6963) * update scheduler migration * Adjust weight to account for storage read * Statement Distribution Guide Edits (#7025) * Statement distribution guide edits * Addressed Marcin's comments * Add attested candidate request retry timeouts (#6833) Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Chris Sosnin <chris125_@live.com> Fix async backing statement distribution tests (#6621) Resolve some todos in async backing statement-distribution branch (#6482) Fix clippy errors in statement distribution branch (#6720) * Async backing: add Prospective Parachains impl guide (#6933) Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> * Updates to Provisioner Guide for Async Backing (#7106) * Initial corrections and clarifications * Partial first draft * Finished first draft * Adding back wrongly removed test bit * fmt * Update roadmap/implementers-guide/src/node/utility/provisioner.md Co-authored-by: Marcin S. <marcin@realemail.net> * Addressing comments * Reorganization * fmt --------- Co-authored-by: Marcin S. <marcin@realemail.net> * fmt * Renaming Parathread Mentions (#7287) * Renaming parathreads * Renaming module to pallet * More updates * PVF: Refactor workers into separate crates, remove host dependency (#7253) * PVF: Refactor workers into separate crates, remove host dependency * Fix compile error * Remove some leftover code * Fix compile errors * Update Cargo.lock * Remove worker main.rs files I accidentally copied these from the other PR. This PR isn't intended to introduce standalone workers yet. * Address review comments * cargo fmt * Update a couple of comments * Update log targets * Update quote to 1.0.27 (#7280) Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: parity-processbot <> * pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271) * pallets: implement Default for GenesisConfig in no_std This change is follow-up of: https://github.com/paritytech/substrate/pull/14108 It is a step towards: https://github.com/paritytech/substrate/issues/13334 * Cargo.lock updated * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * cli: enable BEEFY by default on test networks (#7293) We consider BEEFY mature enough to run by default on all nodes for test networks (Rococo/Wococo/Versi). Right now, most nodes are not running it since it's opt-in using --beefy flag. Switch to an opt-out model for test networks. Replace --beefy flag from CLI with --no-beefy and have BEEFY client start by default on test networks. Signed-off-by: acatangiu <adrian@parity.io> * runtime: past session slashing runtime API (#6667) * runtime/vstaging: unapplied_slashes runtime API * runtime/vstaging: key_ownership_proof runtime API * runtime/ParachainHost: submit_report_dispute_lost * fix key_ownership_proof API * runtime: submit_report_dispute_lost runtime API * nits * Update node/subsystem-types/src/messages.rs Co-authored-by: Marcin S. <marcin@bytedude.com> * revert unrelated fmt changes * post merge fixes * fix compilation --------- Co-authored-by: Marcin S. <marcin@bytedude.com> * Correcting git mishap * Document usage of `gum` crate (#7294) * Document usage of gum crate * Small fix * Add some more basic info * Update node/gum/src/lib.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Update target docs --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * XCM: Fix issue with RequestUnlock (#7278) * XCM: Fix issue with RequestUnlock * Leave API changes for v4 * Fix clippy errors * Fix tests --------- Co-authored-by: parity-processbot <> * Companion for Substrate#14228 (#7295) * Companion for Substrate#14228 https://github.com/paritytech/substrate/pull/14228 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * Companion for #14237: Use latest sp-crates (#7300) * To revert: Update substrate branch to "lexnv/bump_sp_crates" Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates"" This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008. * Update cargo lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * bounded-collections bump to 0.1.7 (#7305) * bounded-collections bump to 0.1.7 Companion for: paritytech/substrate#14225 * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * bump to quote 1.0.28 (#7306) * `RollingSessionWindow` cleanup (#7204) * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit * Fix tests in import * Fix the rest of the tests * Remove dead code * Fix todos * Simplify session caching * Comments for `SessionInfoProvider` * Separate `SessionInfoProvider` from `State` * `cache_session_info_for_head` becomes freestanding function * Remove unneeded `mut` usage * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider` * Fix SessionInfo retrieval * Code cleanup * Don't wrap `SessionInfoProvider` in an `Option` * Remove `earliest_session()` * Remove pre-caching -> wip * Fix some tests and code cleanup * Fix all tests * Fixes in tests * Fix comments, variable names and small style changes * Fix a warning * impl From<SessionWindowSize> for NonZeroUsize * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG * Code review feedback * Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db * Remove `col_session_data` usages * Storage migration clearing columns w/o removing them * Remove session data column usages from `approval-voting` and `dispute-coordinator` tests * Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests * Fix formatting in initialized.rs * Fix a corner case in `SessionInfo` caching for `dispute-coordinator` * Remove `RollingSessionWindow` ;( * Revert "Fix formatting in initialized.rs" This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc. * v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it * Fix `NUM_COLUMNS` in `approval-voting` * Use `columns::v3::NUM_COLUMNS` when opening db * Update node/service/src/parachains_db/upgrade.rs Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3` * Fix `NUM+COLUMNS` in approval_voting * Fix formatting * Fix columns usage * Clarification comments about the different db versions --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * pallet-para-config: Remove remnant WeightInfo functions (#7308) * pallet-para-config: Remove remnant WeightInfo functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * set_config_with_weight begone Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> * XCM: PayOverXcm config (#6900) * Move XCM query functionality to trait * Fix tests * Add PayOverXcm implementation * fix the PayOverXcm trait to compile * moved doc comment out of trait implmeentation and to the trait * PayOverXCM documentation * Change documentation a bit * Added empty benchmark methods implementation and changed docs * update PayOverXCM to convert AccountIds to MultiLocations * Implement benchmarking method * Change v3 to latest * Descend origin to an asset sender (#6970) * descend origin to an asset sender * sender as tuple of dest and sender * Add more variants to the QueryResponseStatus enum * Change Beneficiary to Into<[u8; 32]> * update PayOverXcm to return concrete errors and use AccountId as sender * use polkadot-primitives for AccountId * fix dependency to use polkadot-core-primitives * force Unpaid instruction to the top of the instructions list * modify report_outcome to accept interior argument * use new_query directly for building final xcm query, instead of report_outcome * fix usage of new_query to use the XcmQueryHandler * fix usage of new_query to use the XcmQueryHandler * tiny method calling fix * xcm query handler (#7198) * drop redundant query status * rename ReportQueryStatus to OuterQueryStatus * revert rename of QueryResponseStatus * update mapping * Update xcm/xcm-builder/src/pay.rs Co-authored-by: Gavin Wood <gavin@parity.io> * Updates * Docs * Fix benchmarking stuff * Destination can be determined based on asset_kind * Tweaking API to minimise clones * Some repotting and docs --------- Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * Companion for #14265 (#7307) * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update Cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: parity-processbot <> * bump serde to 1.0.163 (#7315) * bump serde to 1.0.163 * bump ci * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <> * fmt * Updated fmt * Removing changes accidentally pulled from master * fix another master pull issue * Another master pull fix * fmt * Fixing implementers guide build * Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread" This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Marcin S. <marcin@bytedude.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> * fix bitfield distribution test * approval distribution tests * fix bridge tests * update Cargo.lock * [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457) * Optimize collator-protocol validator-side request fetching * address feedback: replace tuples with structs * feedback: add doc comments * move collation types to subfolder --------- Signed-off-by: alindima <alin@parity.io> * Update collation generation for asynchronous backing (#7405) * break candidate receipt construction and distribution into own function * update implementers' guide to include SubmitCollation * implement SubmitCollation for collation-generation * fmt * fix test compilation & remove unnecessary submodule * add some TODOs for a test suite. * Update roadmap/implementers-guide/src/types/overseer-protocol.md Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * add new test harness and first test * refactor to avoid requiring background sender * ensure collation gets packaged and distributed * tests for the fallback case with no hint * add parent rp-number hint tests * fmt * update uses of CollationGenerationConfig * fix remaining test * address review comments * use subsystemsender for background tasks * fmt * remove ValidationCodeHashHint and related tests --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> * fix some more fallout from merge * fmt * remove staging APIs from Rococo & Westend (#7513) * send network messages on main protocol name (#7515) * misc async backing improvements for allowed ancestry blocks (#7532) * shared: fix acquire_info * backwards-compat test for prospective parachains * same relay parent is allowed * provisioner: request candidate receipt by relay parent (#7527) * return candidates hash from prospective parachains * update provisioner * update tests * guide changes * send a single message to backing * fix test * revert to old `handle_new_activations` logic in some cases (#7514) * revert to old `handle_new_activations` logic * gate sending messages on scheduled cores to max_depth >= 2 * fmt * 2->1 * Omnibus asynchronous backing bugfix PR (#7529) * fix a bug in backing * add some more logs * prospective parachains: take ancestry only up to session bounds * add test * fix zombienet tests (#7614) Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix runtime compilation * make bitfield distribution tests compile * attempt to fix zombienet disputes (#7618) * update metric name * update some metric names * avoid cycles when creating fake candidates * make undying collator more friendly to malformed parents * fix a bug in malus * fmt * clippy * add RUN_IN_CONTAINER to new ZombieNet tests (#7631) * remove duplicated migration happened because of master-merge --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: alindima <alin@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Chris Sosnin <chris125_@live.com> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com> Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Marcin S <marcin@bytedude.com> Co-authored-by: Marcin S <marcin@realemail.net> Co-authored-by: Mattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com> Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com> Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com> Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Alin Dima <alin@parity.io>	2023-08-18 16:11:56 +00:00
eskimor	eaf057c5ed	Parathreads Feature Branch (#6969) ... * First baby steps * Split scheduler into several modules * Towards a more modular approach for scheduling * move free_cores; IntoInterator -> BTreeMap * Move clear() * Move more functions out of scheduler * Change weight composition * More abstraction * Further refactor * clippy * fmt * fix test-runtime * Add parathreads pallet to construct_runtime! * Make all runtimes use (Parachains, Parathreads) scheduling * Delete commented out code * Remove parathreads scheduler from westend, rococo, and kusama * fix rococo, westend, and kusama config * Revert "fix rococo, westend, and kusama config" This reverts commit 59e4de380d5c7d17eaaba5e2c2b81405de3465e3. * Revert "Remove parathreads scheduler from westend, rococo, and kusama" This reverts commit 4c44255296083ac5670560790ed77104917890a4. * Remove CoreIndex from free_cores * Remove unnecessary struct for parathreads * parathreads provider take 1 * Comment out parathread tests * Pop into lookahead * fmt * Fill lookahead with two entries for parachains * fmt * Current stage * Towards ab parathreads * no AB use * Make tests typecheck * quick hack to set scheduling lookahead to 1 * Fix scheduler tests * fix paras_inherent tests * misc * Update more of a test * cfg(test) * some cleanup * Undo paras_inherent changes * Adjust paras inherent tests * Undo changes to v2 primitives * Undo v2 mod changes to tests * minor * Remove parathreads assigner and pallet * minor * minor * more cleanup * fmt * minor * minor * minor * Remove on_new_session from assignment provider * Make adder collator integration test pass * disable failing unit tests * minor * minor * re-enable one unit test * minor * handle retries, add concluded para to pop interface * comment out unused code * Remove core_para from interface * Remove first claimqueue element on clear if None instead removing all Nones * Move claimqueue get out of loop * Use VecDeque instead of Ved in ClaimQueue * Make occupied() AB ready(?) * handle freed disputed in clear_and_fill_claimqueue * clear_and_fill_claimqueue returns scheduled Vec * Rename and minor refactor * return position of assignment taken from claimqueue * minor * Fix session boundary parachains number change + extended test * Fix runtimes * Fix polkadot runtime * Remove polkadot pallet from benchmarks * fix test runtime * Add storage migration * Minor refactor * Minor * migratin typechecks * Add migration to runtimes * Towards modular scheduling II (#6568) * Add post migration check * pebkac * Disable migrations but mine * Revert "Disable migrations but mine" This reverts commit 4fa5c5a370c199944a7e0926f50b08626bfbad4c. * Move scheduler migration * Revert "Move scheduler migration" This reverts commit a16b1659a907950bae048a9f7010f2aa76e02b6d. * Fix migration * cleanup * Don't lose retries value anymore * comment out test function * Remove retries value from Assignment again * minor * Make collator for parathreads optional * data type refactor * update scheduler tests * Change test function cfg * comment out test function * Try cfg(test) only * fix cfg flags * Add get_max_retries function to provider interface (#7047) * Fix merge commit * pebkac * fix merge * update cargo.lock * fix merge * fix merge * Use btreemap instead of vec, fix scheduler calls. * Use imported `ScheduledCore` * Remove unused import in inclusion tests * Use keys() instead of mapping over a BTreeMap * Fix migrations for parachains scheduler * Use BlockNumberFor<T> everywhere in scheduler * Add on demand assignment provider pallet (#7110) * Address some PR comments * minor * more cleanup * find_map and timeout availability fixes * Change default scheduling_lookahead to 1 * Add on demand assignment provider pallet * Move test-runtime to new assignment provider * Run cargo format on scheduler tests * minor * Mutate cores in single loop * timeout predicate simplification * claimqueue desired size fix * Replace expect by ok_or * More improvements * Fix push back order and next_up_on_timeout * minor * session change docs * Add pre_new_session call to hand pre session updates * Remove sc_network dependency and PeerId from unnecessary data structures * Remove unnecessary peer_ids * Add OnDemandOrdering proxy (#7156) * Add OnDemandBidding proxy * Fix names * OnDemandAssigner for rococo only * Check PeerId in collator protocol before fetching collation * On occupied, remove non occupied cores from the claimqueue front and refill * Add missing docs * Comment out unused field * fix ScheduledCore in tests * Fix the fix * pebkac * fmt * Fix occupied dropping * Remove double import * ScheduledCore fixes * Readd sc-network dep * pebkac * OpaquePeerId -> PeerId in can_collate interface * Cargo.lock update for interface change * Remove checks not needed anymore? * Drop occupied core on session change if it would time out after the new session * Add on demand assignment provider pallet * Move test-runtime to new assignment provider * Run cargo format on scheduler tests * Add OnDemandOrdering proxy (#7156) * Add OnDemandBidding proxy * Fix names * OnDemandAssigner for rococo only * Remove unneeded config values * Update comments * Use and_then for queue position * Return the max size of the spot queue on error * Add comments to add_parathread_entry * Add module comments * Add log for when can_collate fails * Change assigner queue type to `Assignment` * Update assignment provider tests * More logs * Remove unused keyring import * disable can_collate * comment out can_collate * Can collate first checks set if empty * Move can_collate call to collation advertisement * Fix backing test * map to loop * Remove obsolete check * Move invalid collation test from backing to collator-protocol * fix unused imports * fix test * fix Debug derivation * Increase time limit on zombienet predicates * Increase zombienet timeout * Minor * Address some PR comments * Address PR comments * Comment out failing assert due to on-demand assigner missing * remove collator_restrictions info from backing * Move can_collate to ActiveParas * minor * minor * Update weight information for on demand config * Add ttl to parasentry * Fix tests missing parasentry ttl * Adjust scheduler tests to use ttl default values * Use match instead of if let for ttl drop * Use RuntimeDebug trait for `ParasEntry` fields * Add comments to on demand assignment pallet * Fix spot traffic calculation * Revert runtimedebug changes to primitives * Remove runtimedebug derivation from `ParasEntry` * Mention affinity in pallet level docs * Use RuntimeDebug trait for ParasEntry child types * Remove collator restrictions * Fix primitive versioning and other merge issues * Fix tests post merge * Fix node side tests * Edit parascheduler migration for clarity * Move parascheduler migration up to next release * Remove vestiges from merge * Fix tests * Refactor ttl handling * Remove unused things from scheduler tests * Move on demand assigner to own directory * Update documentation * Remove unused sc-network dependency in primitives Was used for collator restrictions * Remove unused import * Reenable scheduler test * Remove unused storage value * Enable timeout predicate test and fix fn Turns out that the issue with the compiler is fixed and we can now use impl Trait in the manner used here. * Remove unused imports * Add benchmarking entry for perbill in config * Correct typo * Address review comments * Log out errors when calculating spot traffic. * Change parascheduler's log target name * Update scheduler_common documentation * Use mutate for affinity fns, add tests * Add another on demand affinity test * Unify parathreads and parachains in HostConfig (take 2) (#7452) * Unify parathreads and parachains in HostConfig * Fixed missed occurences * Remove commented out lines * `HostConfiguration v7` * Fix version check * Add `MigrateToV7` to `Unreleased` * fmt * fmt * Fix compilation errors after the rebase * Update runtime/parachains/src/scheduler/tests.rs Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com> * Update runtime/parachains/src/scheduler/tests.rs Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com> * fmt * Fix migration test * Fix tests * Remove unneeded assert from tests * parathread_cores -> on_demand_cores; parathread_retries -> on_demand_retries * Fix a compilation error in tests * Remove unused `use` * update colander image version --------- Co-authored-by: alexgparity <alex.gremm@parity.io> Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com> Co-authored-by: Javier Viola <javier@parity.io> * Fix branch after merge with master * Refactor out duplicate checks into a helper fn * Fix tests post merge * Rename add_parathread_assignment, add test * Update docs * Remove unused on_finalize function * Add weight info to on demand pallet * Update runtime/parachains/src/configuration.rs Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> * Update runtime/parachains/src/scheduler_common/mod.rs Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> * Update runtime/parachains/src/assigner_on_demand/mod.rs Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> * Add benchmarking to on demand pallet * Make place_order test check for success * Add on demand benchmarks * Add local test weights to rococo runtime * Modify TTL drop behaviour to not skip claims Previous behaviour would jump a new claim from the assignment provider ahead in the claimqueue, assuming lookahead is larger than 1. * Refactor ttl test to test claimqueue order * Disable place_order ext. when no on_demand cores * Use default genesis config for benchmark tests * Refactor config builder param * Move lifecycle test from scheduler to on demand * Remove unneeded lifecycle test Paras module via the parachain assignment provider doesn't provide new assignments if a parachain loses it's lease. The on demand assignment provider doesn't provide an assignment that is not a parathread. * Re enable validator shuffle test * More realistic weights for place_order * Remove redundant import * Fix backwards compatibility (hopefully) * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand * Fix tests. * Fix off-by-one. * Re enable claimqueue fills test * Re enable schedule_rotates_groups test * Fix fill_claimqueue_fills test * Re enable next_up_on_timeout test, move fn * Do not pop from assignment provider when retrying * Fix tests missing collator in scheduledcore * Add comment about timeout predicate. * Rename parasentry retries to availability timeouts * Re enable schedule_schedules... test * Refactor prune retried test to new scheduler * Have all scheduler tests use genesis_cfg fn * Update docs * Update copyright notices on new files * Rename is_parachain_core to is_bulk_core * Remove erroneous TODO * Simplify import * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::configuration * Revert AdvertiseCollation order shuffle * Refactor place_order into keepalive and allowdeath * Revert rename of hrmp max inbound channels parachain encompasses both on demand and slot auction / bulk. * Restore availability_timeout_predicate function * Clean up leftover comments * Update runtime/parachains/src/scheduler/tests.rs Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::configuration --------- Co-authored-by: alexgparity <alex.gremm@parity.io> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: eskimor <eskimor@no-such-url.com> Co-authored-by: command-bot <> * On Demand - update weights and small nits (#7605) * Remove collator restriction test in inclusion On demand parachains won't have collator restrictions implemented in this way but will instead use a preferred collator registered to a `ParaId` in `paras_registrar`. * Remove redundant config guard for test fns * Update weights * Update WeightInfo for on_demand assigner * Unify assignment provider parameters into one call (#7606) * Combine assignmentprovider params into one fn call * Move scheduler_common to a module under scheduler * Fix ttl handling in benchmark builder * Run cargo format * Remove obsolete test. * Small improvement. * Use same migration pattern as config module * Remove old TODO * Change log target name for assigner on demand * Fix migration * Fix clippy warnings * Add HostConfiguration storage migration to V8 * Add `MigrateToV8` to unreleased migrations for all runtimes * Fix storage version check for config v8 * Set `StorageVersion` to 8 in `MigrateToV8` * Remove dups. * Update primitives/src/v5/mod.rs Co-authored-by: Bastian Köcher <git@kchr.de> --------- Co-authored-by: alexgparity <alex.gremm@parity.io> Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com> Co-authored-by: antonva <anton.asgeirsson@parity.io> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: eskimor <eskimor@no-such-url.com> Co-authored-by: Bastian Köcher <git@kchr.de>	2023-08-17 12:52:23 +00:00
Gavin Wood	400864c352	Use Message Queue pallet for UMP dispatch (#6271) ... * Add ProcessXcmMessage struct * Migrate away from weights in host config * New well-known key to report UMPQ capacity * Add missing file * Fixes * Remove original UMP files * Docs * Update runtime/parachains/src/inclusion/mod.rs Co-authored-by: asynchronous rob <rphmeier@gmail.com> * Add benchmarking * Benchmarks * Mock example of using the QueueChangeHandler to update the WKK * Use master Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Merge remote-tracking branch 'origin/master' into gav-message-queue * Update Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update remove-weight migration The migration got touched on master; just resolving conflicts here. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add message- to dispatch-origin conversion for XCM processing Just using the `impl Into<MultiLocation>` was a bit inflexible. Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * DNM: Temporarily comment code since XCMv3 is not merged yet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use u64 for queue-wide limits on UmpAcceptanceCheckErr Using u32 here was one audit finding for the queue pallet. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Define one sub-queue per *MP queue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Harden check_upward_messages Using safe math and casts. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add type-safe well_known_keys Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add message-queue weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Deploy MessageQueue to Polkadot Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update Cargo.toml Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Migrate to parachain config V5 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update UMP tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert messed up merge 🤦 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update remove-weight migration The migration got touched on master; just resolving conflicts here. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add message- to dispatch-origin conversion for XCM processing Just using the `impl Into<MultiLocation>` was a bit inflexible. Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * DNM: Temporarily comment code since XCMv3 is not merged yet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use u64 for queue-wide limits on UmpAcceptanceCheckErr Using u32 here was one audit finding for the queue pallet. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Define one sub-queue per *MP queue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Harden check_upward_messages Using safe math and casts. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add type-safe well_known_keys Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add message-queue weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Deploy MessageQueue to Polkadot Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update Cargo.toml Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Migrate to parachain config V5 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update UMP tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make stuff compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move DMP and HRMP messages to the MessageQueue It currently does not compile in the CIbecause of some local tweaks to Substrate. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Diener for CI Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * diener update cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert wrong changes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "DNM: Temporarily comment code since XCMv3 is not merged yet" This reverts commit 820aa235cb21dd1d2621843607f7682bf035434e. * Make compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup runtimes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Define benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use master Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Lockfile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add AggregateMessageOrigin This enum currently only holds one value, but having it will make it easier in the future to extend. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Forbid UMP for off-boarding paras - Reject candidates with UMP messages for off-boarding paras - Forbid scheduling off-boarding when a para has unprocess UMPs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Delete stupid test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use BoundedVec for upward messages Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add weights and fix MessageProcessor Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bound receive_upward_messages and check bound in configuration pallet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bound Debug impl Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test runtime Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix xcm-simulator Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Properly fix xcm-simulator and fuzzer Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * cargo update -p sp-io Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Adapt to upstream Substrate changes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix ProcesseMessage impls Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Some tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use master Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * cargo update -p sp-io Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use new MQ API Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add UMP while Para offboarding tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use Mocked message processor for benchmarking Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use variables for constants Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add MQ pallet weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use MQ pallet weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Configure QueueChangeHandler Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add config test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix MQ serive weight Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup outgoing UMP dispatch queues Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use Master Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update Cargo.lock Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Weight mul is not const Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove merge marker Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update runtime/parachains/src/inclusion/mod.rs Co-authored-by: Gavin Wood <gavin@parity.io> * Update runtime/kusama/src/lib.rs Co-authored-by: Gavin Wood <gavin@parity.io> * Use lowercase UMP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clarify comment Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use Weight::from_parts Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix doc Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Emit event after the fact Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add defensive_proof to receive_upward_messages Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Reapply "Remove original UMP files" Looks like they came back from the dead. Re-apply commit cf6d316f0 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove old files Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename MaxUmpMessageLen -> MaxUmpMessageLenOf Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test defensive message dropping of receive_upward_messages Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup imports Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update implementors guide Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove FAIL-CI mark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Delete unused code Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add another test for MQ change hook Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Imports Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Keep Kusama runtime formatting Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Delete unused code" This reverts commit dd76bca5025b7e1ef846a9539c3607eed185f16a. * Feature gate mock functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review: Use saturating_add Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test RelayDispatchQueueSize storage key Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move migration to own file to avoid merge conflicts Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Migration in own file Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rococo: configure MQ pallet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Rococo: configure MQ pallet" Going to do this as follow up, since it needs Substrate changes and i dont want to stall this MR any longer. This reverts commit b9c15e8a8339c4e877d654ee3f09903af4210736. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Fixup tests" This reverts commit 88f1cbe20774d20e5e1e554e798960ae39437af1. * Fixup migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix CI Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix other migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bump MAX_CODE_SIZE to 10MiB Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add ForceUpdateUmpLimits migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * imports Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use defensive instead of defensive_proof 'defensive_proof' also prints the 'self', which spams the console too much when running the tests. Just the length is enough. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename to ScheduleConfigUpdate Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup migration checks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add MAX_CODE_SIZE to ScheduleConfigUpdate Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Set MAX_CODE_SIZE to 4MiB Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix benchmark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix formatting Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Add MAX_CODE_SIZE to ScheduleConfigUpdate" This reverts commit 7caffb09e83083b57affd548215e45b25c3d64dc. * Revert "Set MAX_CODE_SIZE to 4MiB" This reverts commit 103ffbaf686487d2fbe0082a16826af17cacc1a1. * Revert "Bump MAX_CODE_SIZE to 10MiB" This reverts commit 530734b7b0da5b7680054e0242348fcc79a666fe. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove consistency check from migration Re-addig these checks is blocked on https://github.com/paritytech/polkadot/issues/7108 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix constants Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bump MAX_UPWARD_MESSAGE_SIZE_BOUND for Westend Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix migrations Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use old nightly for fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixes * cargo fmt * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixes * Add MQ pallet to fuzzer Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix XMC simulator example Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove runtime-benchmarks from fuzzers Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Remove runtime-benchmarks from fuzzers" This reverts commit e1f2bb01b6dea2dd465539d3658719895b58b557. * Fix example simulator Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add V6 migration and remove old ones Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Actually make old migrations reusable Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Dont delete old migrations Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Future proof AggregateMessageOrigin and review fixes There are indications that Loopback and Bridged will be needed soon. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fix benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix fuzzer build Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review Co-authored-by: muharem <ismailov.m.h@gmail.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove old migration Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Set MQ service weight to 20% Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tabs in Markdown Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>	2023-05-19 16:14:13 +00:00
Keith Yeung	023d459857	XCM: Properly set the pricing for the DMP router (#6843) ... * Properly set the pricing for the DMP router * Publicize price types * Use FixedU128 instead of Percent * Add sp-arithmetic as a dependency for rococo runtime * Add sp-arithmetic as a dependency to all runtimes * Remove duplicate import * Add missing import * Fix tests * Create an appropriate QueueDownwardMessageError variant * Recalculate delivery fee factor based on past queue sizes * Remove unused error variant * Fixes * Fixes * Remove unused imports * Rewrite fee factor update mechanism * Remove unused imports * Fixes * Update runtime/parachains/src/dmp.rs Co-authored-by: Squirrel <gilescope@gmail.com> * Make DeliveryFeeFactor be a StorageMap keyed on ParaIds * Fixes * introduce limit for fee increase on dmp queue * add message_size based fee factor to increment_fee_factor * change message_size fee rate to correct value * fix div by 0 error * bind limit to variable * fix message_size_factor and add DeliveryFeeFactor test * add test for ExponentialPrice implementation * make test formula based * make delivery fee factor test formula based * add max value test for DeliveryFeeFactor and move limit to config * change threshold back to dynamic value and fix tests * fmt * suggested changes and fmt * small stylistic change * fmt * change to tokenlocation * small fixes * fmt * remove sp_arithmetic dependency * Update runtime/parachains/src/dmp.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> --------- Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: Just van Stam <just.van.stam@gmail.com> Co-authored-by: Just van Stam <vstam1@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>	2023-04-20 11:04:33 +00:00
s0me0ne-unkn0wn	64660ee8d2	Remove years from copyright notes (#7034) ... * Happy New Year! * Remove year entierly Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove years from copyright notice in the entire repo --------- Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>	2023-04-08 20:38:35 +00:00
s0me0ne-unkn0wn	1cb1d03c08	Re-export current primitives in crate root (#6487) ... * Re-export current primitives in crate root * Add missing exports * restart CI	2023-01-11 11:28:12 +00:00
Shawn Tabrizi	bccffcad12	Manual Para Lock (#5451) ... * remove para lock check for now * fmt * manual para lock * expose schedule_code_upgrade and set_current_head * extrinsics and benchmarks * use zero * add weights * fix variable name * add and fix comments * fix weights * add back default lock Co-authored-by: parity-processbot <>	2022-10-11 19:22:38 +00:00
Robert Habermeier	49f7e5cce4	Finish migration to v2 primitives (#5037) ... * remove v0 primitives from polkadot-primitives * first pass: remove v0 * fix fallout in erasure-coding * remove v1 primitives, consolidate to v2 * the great import update * update runtime_api_impl_v1 to v2 as well * guide: add `Version` request for runtime API * add version query to runtime API * reintroduce OldV1SessionInfo in a limited way	2022-03-09 14:01:13 -06:00
sandreim	b93c6e68ab	Parachain runtime metrics followup (#4602) ... * initial changes Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fmt Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * remove file Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Remove pallet Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix copyright year Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Remove metric registration op Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Register runtime metrics in client Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fmt Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix build without `runtime-metrics` Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * reduce visibility Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * remove metric prefix logic, use hardcoded Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * review feedback Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Split CounterVec api so it doesn't need mutability Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Const beautify Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Fix Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * fix docs Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> * Merge web ui feedback. Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>	2022-01-05 15:33:02 +02:00
Sergei Shulepov	bce0e5ac7f	Impose new restrictions on paras init and cleanup (#4360) ... * Impose new restrictions on paras init and cleanup For upcoming PVF pre-checking feature we will need to impose a couple of new restrictions for: - `schedule_para_initialize`. - `schedule_para_cleanup`. Specifically, for the former we do not want to allow registration of wasm blob that is empty, i.e. 0 bytes. While that currently already does not make a lot of sense, it allows us to simplify the PVF pre-checking logic: if this PR is deployed before the following changes for PVF prechecking then we can be sure that no paras onboarding have to have to go through the PVF pre-checking. In case, we deploy it altogether this property will allow us to distingush paras that came in before PVF pre-checking. For `schedule_para_cleanup` we do not want to allow offboarding of paras that are undergoing the upgrade process. While this is not a harsh restriction this change allows us to avoid making the PVF prechecking more complicated than it has to be. * Add a test for schedule_para_initialize * Link to `ParaLifecycle::is_stable` in docs. * `schedule_para_{init,cleanup}` docs Now they link to their original declarations in the pallet for more details.	2021-11-26 12:53:23 +01:00
Bernhard Schuster	1aa6a4aba4	move paras inherent filtering to runtime (#4028) ... * move things around, add filter methods * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com> Co-authored-by: Zeke Mostov <z.mostov@gmail.com> Co-authored-by: Bastian Köcher <info@kchr.de>	2021-11-16 18:39:39 +00:00
Keith Yeung	706f142516	Add benchmarking for parachain runtime configuration pallet (#3862) ... * Add benchmarking for parachain runtime configuration pallet * cargo fmt * Add WeightInfo trait * Specify missing WeightInfo associated type in mocks * Fix typo * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_configuration.rs * Fix compilation errors * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_configuration.rs * Condense the number of WeightInfo methods * Fixes * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_configuration.rs * Make use of weights generated from kusama benchmarking * Use a better dispatch function for weighing set_config_with_block_number * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_configuration.rs * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_configuration.rs Co-authored-by: Parity Bot <admin@parity.io>	2021-09-18 09:57:50 +00:00
Shawn Tabrizi	ff5d56fb76	cargo +nightly fmt (#3540) ... * cargo +nightly fmt * add cargo-fmt check to ci * update ci * fmt * fmt * skip macro * ignore bridges	2021-08-02 10:47:33 +00:00
Robert Habermeier	ec98120f0e	Disputes runtime (#2947) ... * disputes module skeleton and storage * implement dispute module initialization logic * implement disputes session change logic * provide dispute skeletons * deduplication & ancient check * fix a couple of warnings * begin provide_dispute_data impl * flesh out statement set import somewhat * move ApprovalVote to shared primitives * add a signing-payload API to explicit dispute statements * implement statement signature checking * some bitflags glue for observing changes in disputes * implement dispute vote import logic * flesh out everything except slashing * guide: tweaks * declare and use punishment trait * punish validators for inconclusive disputes * guide: tiny fix * guide: update docs * add disputes getter fn * guide: small change to spam slots handling * improve spam slots handling and fix some bugs * finish API of disputes runtime * define and deposit `RevertTo` log * begin integrating disputes into para_inherent * use precomputed slash_for/against * return candidate hash from process_bitfields * implement inclusion::collect_disputed * finish integration into rest of runtime * add Disputes to initializer * address suggestions * use pallet macro * fix typo * Update runtime/parachains/src/disputes.rs * add test: fix pruning * document specific behavior * deposit events on dispute changes * add an allow(unused) on fn disputes * add a dummy PunishValidators implementation * add disputes module to Rococo * add disputes module to westend runtime * add disputes module to test runtime * add disputes module to kusama runtime * guide: prepare for runtime API for checking frozenness * remove revert digests in favor of state variable * merge reversions * Update runtime/parachains/src/disputes.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Update runtime/parachains/src/disputes.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Update runtime/parachains/src/disputes.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * add byzantine_threshold and supermajority_threshold utilities to primitives * use primitive helpers * deposit revert event when freezing chain * deposit revert log when freezing chain * test revert event and log are generated when freezing * add trait to decouple disputes handling from paras inherent handling * runtime: fix compilation and setup dispute handler * disputes: add hook for filtering out dispute statements * disputes: add initializer hooks to DisputesHandler * runtime: remove disputes pallet from all runtimes * tag TODOs * don't import any dispute statements just yet... * address grumbles * fix spellcheck, hopefully * maybe now? * last spellcheck round * fix runtime tests * fix test-runtime Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: André Silva <andrerfosilva@gmail.com>	2021-07-19 16:49:19 +00:00
ferrell-code	39969ca536	paras.rs to FRAME V2 (#3403) ... * to v2 macro * line width * fix mock * actually compile moxk * compile tests * add hooks * origin back in mocks * assimilate_storage * add generic * maybe mock compiles now * origin between parachain and system causing problem * change origin * type alias as origin * keep origin * add aliases to tests * remove unnecessary imports * Parachain -> Paras * paras test * nvm * use genesis build in mock	2021-07-06 00:24:27 +00:00
Robert Habermeier	0794f69306	Add dispute types and change InclusionInherent to ParasInherent (#2791) ... * dispute types * add Debug to dispute primitives in std and InherentData * use ParachainsInherentData on node-side * change inclusion_inherent to paras_inherent * RuntimeDebug * add type parameter to PersistedValidationData users * fix test client * spaces * fix collation-generation test * fix provisioner tests * remove references to inclusion inherent	2021-04-01 18:23:27 +02:00
Gavin Wood	edd81bf6a5	New slots/auctions architecture (#2294) ... * TODOs * Add auctions.rs, comment on changes needed. * Remove cruft from slots * Remove more from auctions.rs * More logic drafting in slots. * More logic in slots.rs * patch some errors * more fixes * last nit * Cleanups in slots.rs * Cleanups in slots.rs * patches * make build * crowdloan to new api * auction compile * Use ParaId instead of FundIndex in Crowdloan (#2303) * use paraid instead of fundindex * Update crowdloan.rs * check caller is manager * Auction tests and fix build warnings. * Configurable origin for initiating auctions * Remove on_finalize * #2303 (manual merge) * Tests for Slots * some registrar tests * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update runtime/common/src/slots.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Slots uses Registrar for CurrentChains * swap works test * on swap impl * traitify parachain cleanup * explicit lifecycle tracking for paras * initial implementation of lifecycles and upgrades * clean up a bit * Update runtime/common/src/slots.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * fix doc comment * more rigid lifecycle checks * include paras which are transitioning, and lifecycle query * format guide * update api * update guide * explicit outgoing state, fix genesis * handle outgoing with transitioning paras * Revert "explicit lifecycle tracking for paras" This reverts commit 4177af7ba473bbd9c26bccd861793f25265b6657. * remove lifecycle tracking from registrar * do not include transitioning paras in identifier * Update paras_registrar.rs * final patches to registrar * Fix test * use noop in test * clean up pending swap on deregistration * finish registrar tests * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Apply suggestions from code review * Use matches macro * Correct terms * Apply suggestions from code review * Remove direct need for Slots and Registrar from Crowdloan * Rejig things slightly * actions queue * Revert "actions queue" This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73. * Traitify Auction interface. * Mockups and initial code for Crowdloan testing * One test... * collapse onboarding state * fix some crowdloan tests * one more * start benchmarks for auctions * benchmark bid * fix more crowdloan tests * onboard and begin retirement no longer exist * Revert "onboard and begin retirement no longer exist" This reverts commit 2e100fd94e3540bff5f172328b5d917896f1c6fc. * Simplify crowdloan and make it work. * Fixes * fix some * finish merge fixes * fix refund bug in auctions * Add traits to Registrar for tests and benchmarks * fix more auction benchmarks * Fix TestAuctioneer * finish crowdloan benchmarks * start setting up full integration tests * expand integration tests * finish basic integration test * add more integration tests * begin slots benchmarks * start paras registrar benchmarks * fix merge * fix tests * clean up paras registrar * remove println * remove outdated cleanup config * update benchmarks * Add WeightInfo * enable runtime-benchmarks feature flag * complete swap benchmark * add parachains and onboarding into westend * add benchmarks and genesis * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=slots --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * fix benchmark execution * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * Use `new_raise_len` in crowdloan on_initialize * Update paras_registrar.rs * fix westend merge * impl on_swap for crowdloan * Check fund exists before create * update for crowdloan sig * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * slots on_initialize * use integration tests environment for benchmarks * fix hrmp event * auction on_initialize * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * fix storage name in auctions * add auction_index to winning data * winning data takes into account current auction index * remove println * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=slots --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * Revert "add auction_index to winning data" * PastRandomness. * Fixes * Use new randomness * fix use of randomness in auctions and runtime config * expose consts * fix auction test * add deposit per byte for para registration * basic swap integration test * make swap test more comprehensive * Add WinningVec for easier retrieval in the front-end. * clean up `WinningVec` at the end * Add event for when a new best bid comes in * Fix propagation of winners in ending period * fix benchmarks, refund weight in dissolve * fix unused * remove some TODOs * setup opaque keys for paras in westend * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * remove unused * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/ * back to regular runtime config * use saturating math where user input can be * better first slot check * Update runtime/common/src/claims.rs * update westend onswap impl Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io>	2021-03-14 17:39:41 -04:00
Shawn Tabrizi	a5defa7c7f	Session Delayed Para Changes / Actions Queue (#2406) ... * initial implementation of lifecycles and upgrades * clean up a bit * fix doc comment * more rigid lifecycle checks * include paras which are transitioning, and lifecycle query * format guide * update api * update guide * explicit outgoing state, fix genesis * handle outgoing with transitioning paras * do not include transitioning paras in identifier * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Apply suggestions from code review * Use matches macro * Correct terms * Apply suggestions from code review * actions queue * Revert "actions queue" This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73. * collapse onboarding state * starting actions queue * consolidate actions queue * schedule para initialize result * more actions queue for upgrade/downgrade * clean up with fully implemented actions queue * fix tests * fix scheduler tests * fix hrmp tests * fix test * doc fixes * fix hrmp test w/ valid para * Update paras.md * fix paras registrar * Update propose_parachain.rs * fix merge * Introduce "shared" module * fix rococo build * fix up and use shared * guide updates * add shared config to common tests * add shared to test-runtime * remove println * fix note Co-authored-by: Gavin Wood <gavin@parity.io>	2021-02-18 21:20:18 -06:00
Shawn Tabrizi	eb0159f288	Initializer + Paras Clean Up Messages When Offboarding (#2413) ... * initial hack in * finish up * use notification to pass outgoing paras * move outgoing paras from notifications * missing comma * update guides * clean up	2021-02-15 17:48:25 +00:00
Shawn Tabrizi	2b533ddc77	Explicit Para Lifecycle w/ Upgrades and Downgrades (#2354) ... * initial implementation of lifecycles and upgrades * clean up a bit * fix doc comment * more rigid lifecycle checks * include paras which are transitioning, and lifecycle query * format guide * update api * update guide * explicit outgoing state, fix genesis * handle outgoing with transitioning paras * do not include transitioning paras in identifier * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Update roadmap/implementers-guide/src/runtime/paras.md * Apply suggestions from code review * Use matches macro * Correct terms * Apply suggestions from code review * actions queue * Revert "actions queue" This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73. * collapse onboarding state Co-authored-by: Gavin Wood <gavin@parity.io>	2021-02-04 11:05:44 -06:00
Robert Habermeier	f4e930529b	Reward validators for participating in parachains (#2089) ... * plumbing for rewarding backers * give validators reward points for participating * fix tests * add bitfield rewarding * add mocks for backing rewards * add testing for backing & availability rewards * implement RewardValidators on top of staking * add to test-runtime and rococo * add to test-runtime & rococo * point to source on rewards values * fix common tests * do not reward availability anymore	2020-12-16 12:28:46 -05:00
郭光华	9014232903	Useless modules and misspellings (#2068)	2020-12-04 12:16:14 +00:00
Guillaume Thiolliere	2d4aa3a42e	Companion: Rename pallet trait Trait to Config (#2014) ... * rename Trait -> Config * revert diener changes * rename HostConfig to ActiveConfig as more meaningful * fix merge * "Update Substrate" * cargo update -p sp-io Co-authored-by: parity-processbot <>	2020-11-30 15:13:43 +00:00
Andronik Ordian	39a12b68f6	past-session validator discovery APIs (#2009) ... * guide: fix formatting for SessionInfo module * primitives: SessionInfo type * punt on approval keys * ah, revert the type alias * session info runtime module skeleton * update the guide * runtime/configuration: sync with the guide * runtime/configuration: setters for newly added fields * runtime/configuration: set codec indexes * runtime/configuration: update test * primitives: fix SessionInfo definition * runtime/session_info: initial impl * runtime/session_info: use initializer for session handling (wip) * runtime/session_info: mock authority discovery trait * guide: update the initializer's order * runtime/session_info: tests skeleton * runtime/session_info: store n_delay_tranches in Configuration * runtime/session_info: punt on approval keys * runtime/session_info: add some basic tests * Update primitives/src/v1.rs * small fixes * remove codec index annotation on structs * fix off-by-one error * validator_discovery: accept a session index * runtime: replace validator_discovery api with session_info * Update runtime/parachains/src/session_info.rs Co-authored-by: Sergei Shulepov <sergei@parity.io> * runtime/session_info: add a comment about missing entries * runtime/session_info: define the keys * util: expose connect_to_past_session_validators * util: allow session_info requests for jobs * runtime-api: add mock test for session_info * collator-protocol: add session_index to test state * util: fix error message for runtime error * fix compilation * fix tests after merge with master Co-authored-by: Sergei Shulepov <sergei@parity.io>	2020-11-26 11:02:50 +00:00
Sergei Shulepov	c69e5766db	Breakdown the Router module on Dmp, Ump, Hrmp modules (#1939) ... * Guide: Split router module in guide. Now we have: DMP, UMP and Router module. * Add a glossary entry for what used to be called Router * Extract DMP * Extract UMP * Extract HRMP * Switch over to new modules * Router: goodbye sweet prince * Link to messaging overview for details. * Update missed rococo and test runtimes. * Commit destroyed by rebase changes * Don't deprecate Router but rather make it a meta-project Co-authored-by: Bernhard Schuster <bernhard@ahoi.io> * Fix typos suggestion Co-authored-by: Bernhard Schuster <bernhard@ahoi.io> * Fix repetition in the impl guide * Clarify that processed_downward_messages has the u32 type * Remove the router subdir. * Deabbreviate DMP,UMP,HRMP Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>	2020-11-16 14:02:01 +00:00
Sergei Shulepov	a532315737	Parachains: Introduce a dummy module to include the Origin. (#1749) ... That's because `construct_runtime` doesn't support a free-standing Origin. See the comment in parachains/src/origin.rs	2020-09-24 15:13:23 +00:00
Sergei Shulepov	9bb374fb90	Skeleton of the Router module (#1726) ... The purpose of this commit is to establish a foundation of the router module, it paves the way for the implementation of other features.	2020-09-17 12:27:40 +02:00
Robert Habermeier	262574fc49	Implement validation data refactor (#1585) ... * update primitives * correct parent_head field * make hrmp field pub * refactor validation data: runtime * refactor validation data: messages * add arguments to full_validation_data runtime API * port runtime API * mostly port over candidate validation * remove some parameters from ValidationParams * guide: update candidate validation * update candidate outputs * update ValidationOutputs in primitives * port over candidate validation * add a new test for no-transient behavior * update util runtime API wrappers * candidate backing * fix missing imports * change some fields of validation data around * runtime API impl * update candidate validation * fix backing tests * grumbles from review * fix av-store tests * fix some more crates * fix provisioner tests * fix availability distribution tests * port collation-generation to new validation data * fix overseer tests * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com> Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>	2020-08-18 14:41:40 +02:00
Fedor Sakharov	bd85b3c7d3	Add an Origin to parachains v1 (#1542)	2020-08-05 22:24:20 +03:00
Fedor Sakharov	047e2fdb21	Parachains-runtime (#1505) ... * Initial commit * v0 to v1 primitives and remove attestations * Review fixes * implement candidate_events * remove dead code	2020-07-31 23:42:10 +03:00
Robert Habermeier	dddde219a2	Implement Runtime APIs (#1411) ... * create a README on Runtime APIs * add ParaId type * write up runtime APIs * more preamble * rename * rejig runtime APIs * add occupied_since to `BlockNumber` * skeleton crate for runtime API subsystem * improve group_for_core * improve docs on availability cores runtime API * guide: freed -> free * add primitives for runtime APIs * create a v1 ParachainHost API trait * guide: make validation code return `Option`al. * skeleton runtime API helpers * make parachain-host runtime-generic * skeleton for most runtime API implementation functions * guide: add runtime API helper methods * implement new helpers of the inclusion module * guide: remove retries check, as it is unneeded * implement helpers for scheduler module for Runtime APIs * clean up `validator_groups` implementation * implement next_rotation_at and last_rotation_at * guide: more helpers on GroupRotationInfo * almost finish implementing runtime APIs * add explicit block parameter to runtime API fns * guide: generalize number parameter * guide: add group_responsible to occupied-core * update primitives due to guide changes * finishing touches on runtime API implementation; squash warnings * break out runtime API impl to separate file * add tests for next_up logic * test group rotation info * point to filed TODO * remove unused TODO [now] * indentation * guide: para -> para_id * rename para field to para_id for core meta * remove reference to outdated AvailabilityCores type * add an event in `inclusion` for candidates being included or timing out * guide: candidate events * guide: adjust language * Candidate events type from guide and adjust inclusion event * implement `candidate_events` runtime API * fix runtime test compilation * max -> min * fix typos * guide: add `RuntimeAPIRequest::CandidateEvents`	2020-07-18 16:01:51 -04:00
Robert Habermeier	879892d3f9	Inclusion Module (#1242) ... * add availability bitfield types to primitives * begin inclusion module * use GitHub issue link for limitation * fix some compiler errors * integrate validators into initializer * add generic signing context * make signing-context more generic * fix issues with inclusion module * add TODO * guide: add validators and session index to inclusion * guide: add session index to change notification * implement session change logic * add BackedCandidate type * guide: refine inclusion pipeline * guide: rename group_on to group_validators * guide: add check about collator for parathread * guide: add last_code_upgrade to paras and use in inclusion * implement Paras::last_code_upgrade * implement most checks in process_candidates * make candidate receipt structs more generic * make BackedCandidate struct more generic * use hash param, not block number * check that candidate is in context of the parent block * include inclusion module in initializer * implement enact-candidate * check that only occupied cores have bits set * finish implementing bitfield processing * restructure consistency checks on candidates * make some more primitives generic * signature checking logic for backed candidates * finish implementing process_candidates * implement collect_pending * add some trait implementations to primitives * implement InclusionInherent and squash warnings * test bitfield signing checks * rename parachain head to para_head * fix note_new_head bug in paras * test bitfield enactment in inclusion * helpers for candidate checks * add test for most candidate checks * add test for backing setting storage * test session change logic * remove extraneous type parameter * remove some allow(unused)s * extract threshold computation to const fn * remove some more allow(unused)s * improve doc * add debug assertion * fix primitive test compilation * tag unanimous variant as unused	2020-06-18 19:38:07 -04:00
Robert Habermeier	bd2304ec98	New parachain runtime skeleton (#1158) ... * file structure and initializer skeleton * ensure session changes happen before initialization * add a couple tests for initializer flow * integrate with session handling * configuration update logic * configuration methods * move test mock to its own module * integrate configuration into initializer * add note about initialization order * integrate configuration module into mock * add some tests for config module * paras module storage * implement paras session change operation * amend past code pruning to fully cover acceptance period * update guide again * do pruning of historical validation code * add weight to initialization * integrate into mock & leave notes for next session * clean up un-ended sentence * alter test to account for double index in past code meta * port over code-at logic test * clarify checking for conflicting code upgrades * add genesis for paras, include in mock, ensure incoming paras are processed * note on return value of `validation_code_at` * implement paras routines from implementor's guide * bring over some existing tests and begin porting * port over code upgrade tests * test parachain registration * test code_at with intermediate block * fix warnings * clean up docs and extract to separate struct * adjust implementor's guide to include replacementtimes * kill stray println * rename expected_at to applied_after * rewrite ParaPastCodeMeta to avoid reversal * clarify and test interface of validation_code_at * make FutureCode optional * rename do_old_code_pruning * add comment on Option<()> to answer FAQ * address some more grumbles	2020-06-02 12:34:07 -04:00