re:
https://forum.polkadot.network/t/blocknumber-vs-timestamps-should-we-abandon-blocktimes-altogether/4077
This exposes the `LastRelayChainBlockNumber` storage member of
`cumulus-pallet-parachain-system` with a getter and alters the behavior
of this storage item to only be updated in `on_finalize` to ensure a
consistent value throughout `on_initialize` and within transactions.
Parachains, especially with features such as asynchronous backing and
agile coretime, should not use the parachain block number as a clock.
Any feature of Polkadot intended to optimize core utilization and
parachain coretime consumption is likely to worsen this clock as it is
practically applied.
* inclusion emulator logic for asynchronous backing (#4790)
* initial stab at candidate_context
* fmt
* docs & more TODOs
* some cleanups
* reframe as inclusion_emulator
* documentations yes
* update types
* add constraint modifications
* watermark
* produce modifications
* v2 primitives: re-export all v1 for consistency
* vstaging primitives
* emulator constraints: handle code upgrades
* produce outbound HRMP modifications
* stack.
* method for applying modifications
* method just for sanity-checking modifications
* fragments produce modifications, not prospectives
* make linear
* add some TODOs
* remove stacking; handle code upgrades
* take `fragment` private
* reintroduce stacking.
* fragment constructor
* add TODO
* allow validating fragments against future constraints
* docs
* relay-parent number and min code size checks
* check code upgrade restriction
* check max hrmp per candidate
* fmt
* remove GoAhead logic because it wasn't helpful
* docs on code upgrade failure
* test stacking
* test modifications against constraints
* fmt
* test fragments
* descending or duplicate test
* fmt
* remove unused imports in vstaging
* wrong primitives
* spellcheck
* Runtime changes for Asynchronous Backing (#4786)
* inclusion: utility for allowed relay-parents
* inclusion: use prev number instead of prev hash
* track most recent context of paras
* inclusion: accept previous relay-parents
* update dmp advancement rule for async backing
* fmt
* add a comment about validation outputs
* clean up a couple of TODOs
* weights
* fix weights
* fmt
* Resolve dmp todo
* Restore inclusion tests
* Restore paras_inherent tests
* MostRecentContext test
* Benchmark for new paras dispatchable
* Prepare check_validation_outputs for upgrade
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs
* Implementers guide changes
* More tests for allowed relay parents
* Add a github issue link
* Compute group index based on relay parent
* Storage migration
* Move allowed parents tracker to shared
* Compile error
* Get group assigned to core at the next block
* Test group assignment
* fmt
* Error instead of panic
* Update guide
* Extend doc-comment
* Update runtime/parachains/src/shared.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* Prospective Parachains Subsystem (#4913)
* docs and skeleton
* subsystem skeleton
* main loop
* fragment tree basics & fmt
* begin fragment trees & view
* flesh out more of view update logic
* further flesh out update logic
* some refcount functions for fragment trees
* add fatal/non-fatal errors
* use non-fatal results
* clear up some TODOs
* ideal format for scheduling info
* add a bunch of TODOs
* some more fluff
* extract fragment graph to submodule
* begin fragment graph API
* trees, not graphs
* improve docs
* scope and constructor for trees
* add some test TODOs
* limit max ancestors and store constraints
* constructor
* constraints: fix bug in HRMP watermarks
* fragment tree population logic
* set::retain
* extract population logic
* implement add_and_populate
* fmt
* add some TODOs in tests
* implement child-selection
* strip out old stuff based on wrong assumptions
* use fatality
* implement pruning
* remove unused ancestor constraints
* fragment tree instantiation
* remove outdated comment
* add message/request types and skeleton for handling
* fmt
* implement handle_candidate_seconded
* candidate storage: handle backed
* implement handle_candidate_backed
* implement answer_get_backable_candidate
* remove async where not needed
* implement fetch_ancestry
* add logic for run_iteration
* add some docs
* remove global allow(unused), fix warnings
* make spellcheck happy (despite English)
* fmt
* bump Cargo.lock
* replace tracing with gum
* introduce PopulateFrom trait
* implement GetHypotheticalDepths
* revise docs slightly
* first fragment tree scope test
* more scope tests
* test add_candidate
* fmt
* test retain
* refactor test code
* test populate is recursive
* test contiguity of depth 0 is maintained
* add_and_populate tests
* cycle tests
* remove PopulateFrom trait
* fmt
* test hypothetical depths (non-recursive)
* have CandidateSeconded return membership
* tree membership requests
* Add a ProspectiveParachainsSubsystem struct
* add a staging API for base constraints
* add a `From` impl
* add runtime API for staging_validity_constraints
* implement fetch_base_constraints
* implement `fetch_upcoming_paras`
* remove reconstruction of candidate receipt; no obvious usecase
* fmt
* export message to broader module
* remove last TODO
* correctly export
* fix compilation and add GetMinimumRelayParent request
* make provisioner into a real subsystem with proper mesage bounds
* fmt
* fix ChannelsOut in overseer test
* fix overseer tests
* fix again
* fmt
* Integrate prospective parachains subsystem into backing: Part 1 (#5557)
* BEGIN ASYNC candidate-backing CHANGES
* rename & document modes
* answer prospective validation data requests
* GetMinimumRelayParents request is now plural
* implement an implicit view utility for backing subsystems
* implicit-view: get allowed relay parents
* refactorings and improvements to implicit view
* add some TODOs for tests
* split implicit view updates into 2 functions
* backing: define State to prepare for functional refactor
* add some docs
* backing: implement bones of new leaf activation logic
* backing: create per-relay-parent-states
* use new handle_active_leaves_update
* begin extracting logic from CandidateBackingJob
* mostly extract statement import from job logic
* handle statement imports outside of job logic
* do some TODO planning for prospective parachains integration
* finish rewriting backing subsystem in functional style
* add prospective parachains mode to relay parent entries
* fmt
* add a RejectedByProspectiveParachains error
* notify prospective parachains of seconded and backed candidates
* always validate candidates exhaustively in backing.
* return persisted_validation_data from validation
* handle rejections by prospective parachains
* implement seconding sanity check
* invoke validate_and_second
* Alter statement table to allow multiple seconded messages per validator
* refactor backing to have statements carry PVD
* clean up all warnings
* Add tests for implicit view
* Improve doc comments
* Prospective parachains mode based on Runtime API version
* Add a TODO
* Rework seconding_sanity_check
* Iterate over responses
* Update backing tests
* collator-protocol: load PVD from runtime
* Fix validator side tests
* Update statement-distribution to fetch PVD
* Fix statement-distribution tests
* Backing tests with prospective paras #1
* fix per_relay_parent pruning in backing
* Test multiple leaves
* Test seconding sanity check
* Import statement order
Before creating an entry in `PerCandidateState` map
wait for the approval from the prospective parachains
* Add a test for correct state updates
* Second multiple candidates per relay parent test
* Add backing tests with prospective paras
* Second more than one test without prospective paras
* Add a test for prospective para blocks
* Update malus
* typos
Co-authored-by: Chris Sosnin <chris125_@live.com>
* Track occupied depth in backing per parachain (#5778)
* provisioner: async backing changes (#5711)
* Provisioner changes for async backing
* Select candidates based on prospective paras mode
* Revert naming
* Update tests
* Update TODO comment
* review
* provisioner: async backing changes (#5711)
* Provisioner changes for async backing
* Select candidates based on prospective paras mode
* Revert naming
* Update tests
* Update TODO comment
* review
* fmt
* Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991)
* BEGIN STATEMENT DISTRIBUTION WORK
create a vstaging network protocol which is the same as v1
* mostly make network bridge amenable to vstaging
* network-bridge: fully adapt to vstaging
* add some TODOs for tests
* fix fallout in bitfield-distribution
* bitfield distribution tests + TODOs
* fix fallout in gossip-support
* collator-protocol: fix message fallout
* collator-protocol: load PVD from runtime
* add TODO for vstaging tests
* make things compile
* set used network protocol version using a feature
* fmt
* get approval-distribution building
* fix approval-distribution tests
* spellcheck
* nits
* approval distribution net protocol test
* bitfield distribution net protocol test
* Revert "collator-protocol: fix message fallout"
This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1.
* Network bridge tests
Co-authored-by: Chris Sosnin <chris125_@live.com>
* remove max_pov_size requirement from prospective pvd request (#6014)
* remove max_pov_size requirement from prospective pvd request
* fmt
* Extract legacy statement distribution to its own module (#6026)
* add compatibility type to v2 statement distribution message
* warning cleanup
* handle compatibility layer for v2
* clean up an unimplemented!() block
* circulate statements based on version
* extract legacy v1 code into separate module
* remove unimplemented
* clean up naming of from_requester/responder
* remove TODOs
* have backing share seconded statements with PVD
* fmt
* fix warning
* Quick fix unused warning for not yet implemented/used staging messages.
* Fix network bridge test
* Fix wrong merge.
We now have 23 subsystems (network bridge split + prospective
parachains)
Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
* Version 3 is already live.
* Fix tests (#6055)
* Fix backing tests
* Fix warnings.
* fmt
* collator-protocol: asynchronous backing changes (#5740)
* Draft collator side changes
* Start working on collations management
* Handle peer's view change
* Versioning on advertising
* Versioned collation fetching request
* Handle versioned messages
* Improve docs for collation requests
* Add spans
* Add request receiver to overseer
* Fix collator side tests
* Extract relay parent mode to lib
* Validator side draft
* Add more checks for advertisement
* Request pvd based on async backing mode
* review
* Validator side improvements
* Make old tests green
* More fixes
* Collator side tests draft
* Send collation test
* fmt
* Collator side network protocol versioning
* cleanup
* merge artifacts
* Validator side net protocol versioning
* Remove fragment tree membership request
* Resolve todo
* Collator side core state test
* Improve net protocol compatibility
* Validator side tests
* more improvements
* style fixes
* downgrade log
* Track implicit assignments
* Limit the number of seconded candidates per para
* Add a sanity check
* Handle fetched candidate
* fix tests
* Retry fetch
* Guard against dequeueing while already fetching
* Reintegrate connection management
* Timeout on advertisements
* fmt
* spellcheck
* update tests after merge
* validator assignment fixes for backing and collator protocol (#6158)
* Rename depth->ancestry len in tests
* Refactor group assignments
* Remove implicit assignments
* backing: consider occupied core assignments
* Track a single para on validator side
* Refactor prospective parachains mode request (#6179)
* Extract prospective parachains mode into util
* Skip activations depending on the mode
* backing: don't send backed candidate to provisioner (#6185)
* backing: introduce `CanSecond` request for advertisements filtering (#6225)
* Drop BoundToRelayParent
* draft changes
* fix backing tests
* Fix genesis ancestry
* Fix validator side tests
* more tests
* cargo generate-lockfile
* Implement `StagingValidityConstraints` Runtime API method (#6258)
* Implement StagingValidityConstraints
* spellcheck
* fix ump params
* Update hrmp comment
* Introduce ump per candidate limit
* hypothetical earliest block
* refactor primitives usage
* hypothetical earliest block number test
* fix build
* Prepare the Runtime for asynchronous backing upgrade (#6287)
* Introduce async backing params to runtime config
* fix cumulus config
* use config
* finish runtimes
* Introduce new staging API
* Update collator protocol
* Update provisioner
* Update prospective parachains
* Update backing
* Move async backing params lower in the config
* make naming consistent
* misc
* Use real prospective parachains subsystem (#6407)
* Backport `HypotheticalFrontier` into the feature branch (#6605)
* implement more general HypotheticalFrontier
* fmt
* drop unneeded request
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Resolve todo about legacy leaf activation (#6447)
* fix bug/warning in handling membership answers
* Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521)
* Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest`
* Update tests
* Fix (removed wrong docstring)
* Fix can_second request
* Patch some dead_code errors
---------
Co-authored-by: Chris Sosnin <chris125_@live.com>
* Async Backing: Send Statement Distribution "Backed" messages (#6634)
* Backing: Send Statement Distribution "Backed" messages
Closes#6590.
**TODO:**
- [ ] Adjust tests
* Fix compile errors
* (Mostly) fix tests
* Fix comment
* Fix test and compile error
* Test that `StatementDistributionMessage::Backed` is sent
* Fix compile error
* Fix some clippy errors
* Add prospective parachains subsystem tests (#6454)
* Add prospective parachains subsystem test
* Add `should_do_no_work_if_async_backing_disabled_for_leaf` test
* Implement `activate_leaf` helper, up to getting ancestry
* Finish implementing `activate_leaf`
* Small refactor in `activate_leaf`
* Get `CandidateSeconded` working
* Finish `send_candidate_and_check_if_found` test
* Refactor; send more leaves & candidates
* Refactor test
* Implement `check_candidate_parent_leaving_view` test
* Start work on `check_candidate_on_multiple_forks` test
* Don’t associate specific parachains with leaf
* Finish `correctly_updates_leaves` test
* Fix cycle due to reused head data
* Fix `check_backable_query` test
* Fix `check_candidate_on_multiple_forks` test
* Add `check_depth_and_pvd_queries` test
* Address review comments
* Remove TODO
* add a new index for output head data to candidate storage
* Resolve test TODOs
* Fix compile errors
* test candidate storage pruning, make sure new index is cleaned up
---------
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Node-side metrics for asynchronous backing (#6549)
* Add metrics for `prune_view_candidate_storage`
* Add metrics for `request_unblocked_collations`
* Fix docstring
* Couple fixes from review comments
* Fix `check_depth_query` test
* inclusion-emulator: mirror advancement rule check (#6361)
* inclusion-emulator: mirror advancement rule check
* fix build
* prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649)
* Introduce `backed_in_path_only` flag for depth request
* fmt
* update doc comment
* fmt
* Add async-backing zombienet tests (#6314)
* Async backing: impl guide for statement distribution (#6738)
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
* Asynchronous backing statement distribution: Take III (#5999)
* add notification types for v2 statement-distribution
* improve protocol docs
* add empty vstaging module
* fmt
* add backed candidate packet request types
* start putting down structure of new logic
* handle activated leaf
* some sanity-checking on outbound statements
* fmt
* update vstaging share to use statements with PVD
* tiny refactor, candidate_hash location
* import local statements
* refactor statement import
* first stab at broadcast logic
* fmt
* fill out some TODOs
* start on handling incoming
* split off session info into separate map
* start in on a knowledge tracker
* address some grumbles
* format
* missed comment
* some docs for direct
* add note on slashing
* amend
* simplify 'direct' code
* finish up the 'direct' logic
* add a bunch of tests for the direct-in-group logic
* rename 'direct' to 'cluster', begin a candidate_entry module
* distill candidate_entry
* start in on a statement-store module
* some utilities for the statement store
* rewrite 'send_statement_direct' using new tools
* filter sending logic on peers which have the relay-parent in their view.
* some more logic for handling incoming statements
* req/res: BackedCandidatePacket -> AttestedCandidate + tweaks
* add a `validated_in_group` bitfield to BackedCandidateInventory
* BackedCandidateInventory -> Manifest
* start in on requester module
* add outgoing request for attested candidate
* add a priority mechanism for requester
* some request dispatch logic
* add seconded mask to tagged-request
* amend manifest to hold group index
* handle errors and set up scaffold for response validation
* validate attested candidate responses
* requester -> requests
* add some utilities for manipulating requests
* begin integrating requester
* start grid module
* tiny
* refactor grid topology to expose more info to subsystems
* fix grid_topology test
* fix overseer test
* implement topology group-based view construction logic
* fmt
* flesh out grid slightly more
* add indexed groups utility
* integrate Groups into per-session info
* refactor statement store to borrow Groups
* implement manifest knowledge utility
* add a test for topology setup
* don't send to group members
* test for conflicting manifests
* manifest knowledge tests
* fmt
* rename field
* garbage collection for grid tracker
* routines for finding correct/incorrect advertisers
* add manifest import logic
* tweak naming
* more tests for manifest import
* add comment
* rework candidates into a view-wide tracker
* fmt
* start writing boilerplate for grid sending
* fmt
* some more group boilerplate
* refactor handling of topology and authority IDs
* fmt
* send statements directly to grid peers where possible
* send to cluster only if statement belongs to cluster
* improve handling of cluster statements
* handle incoming statements along the grid
* API for introduction of candidates into the tree
* backing: use new prospective parachains API
* fmt prospective parachains changes
* fmt statement-dist
* fix condition
* get ready for tracking importable candidates
* prospective parachains: add Cow logic
* incomplete and complete hypothetical candidates
* remove keep_if_unneeded
* fmt
* implement more general HypotheticalFrontier
* fmt, cleanup
* add a by_parent_hash index to candidate tracker
* more framework for future code
* utilities for getting all hypothetical candidates for frontier
* track origin in statement store
* fmt
* requests should return peer
* apply post-confirmation reckoning
* flesh out import/announce/circulate logic on new statements
* adjust
* adjust TODO comment
* fix backing tests
* update statement-distribution to use new indexedvec
* fmt
* query hypothetical candidates
* implement `note_importable_under`
* extract common utility of fragment tree updates
* add a helper function for getting statements unknown by backing
* import fresh statements to backing
* send announcements and acknowledgements over grid
* provide freshly importable statements
also avoid tracking backed candidates in statement distribution
* do not issue requests on newly importable candidates
* add TODO for later when confirming candidate
* write a routine for handling backed candidate notifications
* simplify grid substantially
* add some test TODOs
* handle confirmed candidates & grid announcements
* finish implementing manifest handling, including follow up statements
* send follow-up statements when acknowledging freshly backed
* fmt
* handle incoming acknowledgements
* a little DRYing
* wire up network messages to handlers
* fmt
* some skeleton code for peer view update handling
* more peer view skeleton stuff
* Fix async backing statement distribution tests (#6621)
* Fix compile errors in tests
* Cargo fmt
* Resolve some todos in async backing statement-distribution branch (#6482)
* Implement `remove_by_relay_parent`
* Extract `minimum_votes` to shared primitives.
* Add `can_send_statements_received_with_prejudice` test
* Fix test
* Update docstrings
* Cargo fmt
* Fix compile error
* Fix compile errors in tests
* Cargo fmt
* Add module docs; write `test_priority_ordering` (first draft)
* Fix `test_priority_ordering`
* Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`
* Address review comments
* Remove `Entry::get_mut`
* fix test compilation
* add a TODO for a test
* clean up a couple of TODOs
* implement sending pending cluster statements
* refactor utility function for sending acknowledgement and statements
* mostly implement catching peers up via grid
* Fix clippy error
* alter grid to track all pending statements
* fix more TODOs and format
* tweak a TODO in requests
* some logic for dispatching requests
* fmt
* skeleton for response receiving
* Async backing statement distribution: cluster tests (#6678)
* Add `pending_statements_set_when_receiving_fresh_statements`
* Add `pending_statements_updated_when_sending_statements` test
* fix up
* fmt
* update TODO
* rework seconded mask in requests
* change doc
* change unhandledresponse not to borrow request manager
* only accept responses sufficient to back
* finish implementing response handling
* extract statement filter to protocol crate
* rework requests: use statement filter in network protocol
* dispatch cluster requests correctly
* rework cluster statement sending
* implement request answering
* fmt
* only send confirmed candidate statement messages on unified relay-parent
* Fix Tests In Statement Distribution Branch
* Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)
* Integrate `handle_active_leaves_update`
* Integrate `share_local_statement`/`handle_backed_candidate_message`
* Start hooking up request/response flow
* Finish hooking up request/response flow
* Limit number of parallel requests in responder
* Fix test compilation errors
* Fix missing check for prospective parachains mode
* Fix some more compile errors
* clean up some review comments
* clean up warnings
* Async backing statement distribution: grid tests (#6673)
* Add `manifest_import_returns_ok_true` test
* cargo fmt
* Add pending_communication_receiving_manifest_on_confirmed_candidate
* Add `senders_can_provide_manifests_in_acknowledgement` test
* Add a couple of tests for pending statements
* Add `pending_statements_cleared_when_sending` test
* Add `pending_statements_respect_remote_knowledge` test
* Refactor group creation in tests
* Clarify docs
* Address some review comments
* Make some clarifications
* Fix post-merge errors
* Clarify test `senders_can_provide_manifests_in_acknowledgement`
* Try writing `pending_statements_are_updated_after_manifest_exchange`
* Document "seconding limit" and `reject_overflowing_manifests` test
* Test that seconding counts are not updated for validators on error
* Fix tests
* Fix manifest exchange test
* Add more tests in `requests.rs` (#6707)
This resolves remaining TODOs in this file.
* remove outdated inventory terminology
* Async backing statement distribution: `Candidates` tests (#6658)
* Async Backing: Fix clippy errors in statement distribution branch (#6720)
* Integrate `handle_active_leaves_update`
* Integrate `share_local_statement`/`handle_backed_candidate_message`
* Start hooking up request/response flow
* Finish hooking up request/response flow
* Limit number of parallel requests in responder
* Fix test compilation errors
* Fix missing check for prospective parachains mode
* Fix some more compile errors
* Async Backing: Fix clippy errors in statement distribution branch
* Fix some more clippy lints
* add tests module
* fix warnings in existing tests
* create basic test harness
* create a test state struct
* fmt
* create empty cluster & grid modules for tests
* some TODOs for cluster test suite
* describe test-suite for grid logic
* describe request test suite
* fix seconding-limit bug
* Remove extraneous `pub`
This somehow made it into my clippy PR.
* Fix some test compile warnings
* Remove some unneeded `allow`s
* adapt some new test helpers from Marcin
* add helper for activating a gossip topology
* add utility for signing statements
* helpers for connecting/disconnecting peers
* round out network utilities
* fmt
* fix bug in initializing validator-meta
* fix compilation
* implement first cluster test
* TODOs for incoming request tests
* Remove unneeded `make_committed_candidate` helper
* fmt
* some more tests for cluster
* add a TODO about grid senders
* integrate inbound req/res into test harness
* polish off initial cluster test suite
* keep introduce candidate request
* fix tests after introduce candidate request
* fmt
* Add grid protocol to module docs
* Fix comments
* Test `backed_in_path_only: true`
* Update node/network/protocol/src/lib.rs
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* Update node/network/protocol/src/request_response/mod.rs
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* Mark receiver with `vstaging`
* validate grid senders based on manifest kind
* fix mask_seconded/valid
* fix unwanted-mask check
* fix build
* resolve todo on leaf mode
* Unify protocol naming to vstaging
* fmt, fix grid test after topology change
* typo
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* address review
* adjust comment, make easier to understand
* Fix typo
---------
Co-authored-by: Marcin S <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
* miscellaneous fixes to make asynchronous backing work (#6791)
* propagate network-protocol-staging feature
* add feature to adder-collator as well
* allow collation-generation of occupied cores
* prospective parachains: special treatment for pending availability candidates
* runtime: fetch candidates pending availability
* lazily construct PVD for pending candidates
* fix fallout in prospective parachains hypothetical/select_child
* runtime: enact candidates when creating paras-inherent
* make tests compile
* test pending availability in the scope
* add prospective parachains test
* fix validity constraints leftovers
* drop prints
* Fix typos
---------
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Marcin S <marcin@realemail.net>
* Remove restart from test (#6840)
* Async Backing: Statement Distribution Tests (#6755)
* start on handling incoming
* split off session info into separate map
* start in on a knowledge tracker
* address some grumbles
* format
* missed comment
* some docs for direct
* add note on slashing
* amend
* simplify 'direct' code
* finish up the 'direct' logic
* add a bunch of tests for the direct-in-group logic
* rename 'direct' to 'cluster', begin a candidate_entry module
* distill candidate_entry
* start in on a statement-store module
* some utilities for the statement store
* rewrite 'send_statement_direct' using new tools
* filter sending logic on peers which have the relay-parent in their view.
* some more logic for handling incoming statements
* req/res: BackedCandidatePacket -> AttestedCandidate + tweaks
* add a `validated_in_group` bitfield to BackedCandidateInventory
* BackedCandidateInventory -> Manifest
* start in on requester module
* add outgoing request for attested candidate
* add a priority mechanism for requester
* some request dispatch logic
* add seconded mask to tagged-request
* amend manifest to hold group index
* handle errors and set up scaffold for response validation
* validate attested candidate responses
* requester -> requests
* add some utilities for manipulating requests
* begin integrating requester
* start grid module
* tiny
* refactor grid topology to expose more info to subsystems
* fix grid_topology test
* fix overseer test
* implement topology group-based view construction logic
* fmt
* flesh out grid slightly more
* add indexed groups utility
* integrate Groups into per-session info
* refactor statement store to borrow Groups
* implement manifest knowledge utility
* add a test for topology setup
* don't send to group members
* test for conflicting manifests
* manifest knowledge tests
* fmt
* rename field
* garbage collection for grid tracker
* routines for finding correct/incorrect advertisers
* add manifest import logic
* tweak naming
* more tests for manifest import
* add comment
* rework candidates into a view-wide tracker
* fmt
* start writing boilerplate for grid sending
* fmt
* some more group boilerplate
* refactor handling of topology and authority IDs
* fmt
* send statements directly to grid peers where possible
* send to cluster only if statement belongs to cluster
* improve handling of cluster statements
* handle incoming statements along the grid
* API for introduction of candidates into the tree
* backing: use new prospective parachains API
* fmt prospective parachains changes
* fmt statement-dist
* fix condition
* get ready for tracking importable candidates
* prospective parachains: add Cow logic
* incomplete and complete hypothetical candidates
* remove keep_if_unneeded
* fmt
* implement more general HypotheticalFrontier
* fmt, cleanup
* add a by_parent_hash index to candidate tracker
* more framework for future code
* utilities for getting all hypothetical candidates for frontier
* track origin in statement store
* fmt
* requests should return peer
* apply post-confirmation reckoning
* flesh out import/announce/circulate logic on new statements
* adjust
* adjust TODO comment
* fix backing tests
* update statement-distribution to use new indexedvec
* fmt
* query hypothetical candidates
* implement `note_importable_under`
* extract common utility of fragment tree updates
* add a helper function for getting statements unknown by backing
* import fresh statements to backing
* send announcements and acknowledgements over grid
* provide freshly importable statements
also avoid tracking backed candidates in statement distribution
* do not issue requests on newly importable candidates
* add TODO for later when confirming candidate
* write a routine for handling backed candidate notifications
* simplify grid substantially
* add some test TODOs
* handle confirmed candidates & grid announcements
* finish implementing manifest handling, including follow up statements
* send follow-up statements when acknowledging freshly backed
* fmt
* handle incoming acknowledgements
* a little DRYing
* wire up network messages to handlers
* fmt
* some skeleton code for peer view update handling
* more peer view skeleton stuff
* Fix async backing statement distribution tests (#6621)
* Fix compile errors in tests
* Cargo fmt
* Resolve some todos in async backing statement-distribution branch (#6482)
* Implement `remove_by_relay_parent`
* Extract `minimum_votes` to shared primitives.
* Add `can_send_statements_received_with_prejudice` test
* Fix test
* Update docstrings
* Cargo fmt
* Fix compile error
* Fix compile errors in tests
* Cargo fmt
* Add module docs; write `test_priority_ordering` (first draft)
* Fix `test_priority_ordering`
* Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`
* Address review comments
* Remove `Entry::get_mut`
* fix test compilation
* add a TODO for a test
* clean up a couple of TODOs
* implement sending pending cluster statements
* refactor utility function for sending acknowledgement and statements
* mostly implement catching peers up via grid
* Fix clippy error
* alter grid to track all pending statements
* fix more TODOs and format
* tweak a TODO in requests
* some logic for dispatching requests
* fmt
* skeleton for response receiving
* Async backing statement distribution: cluster tests (#6678)
* Add `pending_statements_set_when_receiving_fresh_statements`
* Add `pending_statements_updated_when_sending_statements` test
* fix up
* fmt
* update TODO
* rework seconded mask in requests
* change doc
* change unhandledresponse not to borrow request manager
* only accept responses sufficient to back
* finish implementing response handling
* extract statement filter to protocol crate
* rework requests: use statement filter in network protocol
* dispatch cluster requests correctly
* rework cluster statement sending
* implement request answering
* fmt
* only send confirmed candidate statement messages on unified relay-parent
* Fix Tests In Statement Distribution Branch
* Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)
* Integrate `handle_active_leaves_update`
* Integrate `share_local_statement`/`handle_backed_candidate_message`
* Start hooking up request/response flow
* Finish hooking up request/response flow
* Limit number of parallel requests in responder
* Fix test compilation errors
* Fix missing check for prospective parachains mode
* Fix some more compile errors
* clean up some review comments
* clean up warnings
* Async backing statement distribution: grid tests (#6673)
* Add `manifest_import_returns_ok_true` test
* cargo fmt
* Add pending_communication_receiving_manifest_on_confirmed_candidate
* Add `senders_can_provide_manifests_in_acknowledgement` test
* Add a couple of tests for pending statements
* Add `pending_statements_cleared_when_sending` test
* Add `pending_statements_respect_remote_knowledge` test
* Refactor group creation in tests
* Clarify docs
* Address some review comments
* Make some clarifications
* Fix post-merge errors
* Clarify test `senders_can_provide_manifests_in_acknowledgement`
* Try writing `pending_statements_are_updated_after_manifest_exchange`
* Document "seconding limit" and `reject_overflowing_manifests` test
* Test that seconding counts are not updated for validators on error
* Fix tests
* Fix manifest exchange test
* Add more tests in `requests.rs` (#6707)
This resolves remaining TODOs in this file.
* remove outdated inventory terminology
* Async backing statement distribution: `Candidates` tests (#6658)
* Async Backing: Fix clippy errors in statement distribution branch (#6720)
* Integrate `handle_active_leaves_update`
* Integrate `share_local_statement`/`handle_backed_candidate_message`
* Start hooking up request/response flow
* Finish hooking up request/response flow
* Limit number of parallel requests in responder
* Fix test compilation errors
* Fix missing check for prospective parachains mode
* Fix some more compile errors
* Async Backing: Fix clippy errors in statement distribution branch
* Fix some more clippy lints
* add tests module
* fix warnings in existing tests
* create basic test harness
* create a test state struct
* fmt
* create empty cluster & grid modules for tests
* some TODOs for cluster test suite
* describe test-suite for grid logic
* describe request test suite
* fix seconding-limit bug
* Remove extraneous `pub`
This somehow made it into my clippy PR.
* Fix some test compile warnings
* Remove some unneeded `allow`s
* adapt some new test helpers from Marcin
* add helper for activating a gossip topology
* add utility for signing statements
* helpers for connecting/disconnecting peers
* round out network utilities
* fmt
* fix bug in initializing validator-meta
* fix compilation
* implement first cluster test
* TODOs for incoming request tests
* Remove unneeded `make_committed_candidate` helper
* fmt
* Hook up request sender
* Add `valid_statement_without_prior_seconded_is_ignored` test
* Fix `valid_statement_without_prior_seconded_is_ignored` test
* some more tests for cluster
* add a TODO about grid senders
* integrate inbound req/res into test harness
* polish off initial cluster test suite
* keep introduce candidate request
* fix tests after introduce candidate request
* fmt
* Add grid protocol to module docs
* Remove obsolete test
* Fix comments
* Test `backed_in_path_only: true`
* Update node/network/protocol/src/lib.rs
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* Update node/network/protocol/src/request_response/mod.rs
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* Mark receiver with `vstaging`
* First draft of `ensure_seconding_limit_is_respected` test
* validate grid senders based on manifest kind
* fix mask_seconded/valid
* fix unwanted-mask check
* fix build
* resolve todo on leaf mode
* Unify protocol naming to vstaging
* Fix `ensure_seconding_limit_is_respected` test
* Start `backed_candidate_leads_to_advertisement` test
* fmt, fix grid test after topology change
* Send Backed notification
* Finish `backed_candidate_leads_to_advertisement` test
* Finish `peer_reported_for_duplicate_statements` test
* Finish `received_advertisement_before_confirmation_leads_to_request`
* Add `advertisements_rejected_from_incorrect_peers` test
* Add `manifest_rejected_*` tests
* Add `manifest_rejected_when_group_does_not_match_para` test
* Add `local_node_sanity_checks_incoming_requests` test
* Add `local_node_respects_statement_mask` test
* Add tests where peer is reported for providing invalid signatures
* Add `cluster_peer_allowed_to_send_incomplete_statements` test
* Add `received_advertisement_after_backing_leads_to_acknowledgement`
* Add `received_advertisement_after_confirmation_before_backing` test
* peer_reported_for_advertisement_conflicting_with_confirmed_candidate
* Add `peer_reported_for_not_enough_statements` test
* Add `peer_reported_for_providing_statements_meant_to_be_masked_out`
* Add `additional_statements_are_shared_after_manifest_exchange`
* Add `grid_statements_imported_to_backing` test
* Add `relay_parent_entering_peer_view_leads_to_advertisement` test
* Add `advertisement_not_re_sent_when_peer_re_enters_view` test
* Update node/network/statement-distribution/src/vstaging/tests/grid.rs
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
* Resolve TODOs, update test
* Address unused code
* Add check after every test for unhandled requests
* Refactor (`make_dummy_leaf` and `handle_sent_request`)
* Refactor (`make_dummy_topology`)
* Minor refactor
---------
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
* Fix some clippy lints in tests
* Async backing: minor fixes (#6920)
* bitfield-distribution test
* implicit view tests
* Refactor parameters -> params
* scheduler: update storage migration (#6963)
* update scheduler migration
* Adjust weight to account for storage read
* Statement Distribution Guide Edits (#7025)
* Statement distribution guide edits
* Addressed Marcin's comments
* Add attested candidate request retry timeouts (#6833)
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
Fix async backing statement distribution tests (#6621)
Resolve some todos in async backing statement-distribution branch (#6482)
Fix clippy errors in statement distribution branch (#6720)
* Async backing: add Prospective Parachains impl guide (#6933)
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
* Updates to Provisioner Guide for Async Backing (#7106)
* Initial corrections and clarifications
* Partial first draft
* Finished first draft
* Adding back wrongly removed test bit
* fmt
* Update roadmap/implementers-guide/src/node/utility/provisioner.md
Co-authored-by: Marcin S. <marcin@realemail.net>
* Addressing comments
* Reorganization
* fmt
---------
Co-authored-by: Marcin S. <marcin@realemail.net>
* fmt
* Renaming Parathread Mentions (#7287)
* Renaming parathreads
* Renaming module to pallet
* More updates
* PVF: Refactor workers into separate crates, remove host dependency (#7253)
* PVF: Refactor workers into separate crates, remove host dependency
* Fix compile error
* Remove some leftover code
* Fix compile errors
* Update Cargo.lock
* Remove worker main.rs files
I accidentally copied these from the other PR. This PR isn't intended to
introduce standalone workers yet.
* Address review comments
* cargo fmt
* Update a couple of comments
* Update log targets
* Update quote to 1.0.27 (#7280)
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>
* pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271)
* pallets: implement Default for GenesisConfig in no_std
This change is follow-up of: https://github.com/paritytech/substrate/pull/14108
It is a step towards: https://github.com/paritytech/substrate/issues/13334
* Cargo.lock updated
* update lockfile for {"substrate"}
---------
Co-authored-by: parity-processbot <>
* cli: enable BEEFY by default on test networks (#7293)
We consider BEEFY mature enough to run by default on all nodes
for test networks (Rococo/Wococo/Versi).
Right now, most nodes are not running it since it's opt-in using
--beefy flag. Switch to an opt-out model for test networks.
Replace --beefy flag from CLI with --no-beefy and have BEEFY
client start by default on test networks.
Signed-off-by: acatangiu <adrian@parity.io>
* runtime: past session slashing runtime API (#6667)
* runtime/vstaging: unapplied_slashes runtime API
* runtime/vstaging: key_ownership_proof runtime API
* runtime/ParachainHost: submit_report_dispute_lost
* fix key_ownership_proof API
* runtime: submit_report_dispute_lost runtime API
* nits
* Update node/subsystem-types/src/messages.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* revert unrelated fmt changes
* post merge fixes
* fix compilation
---------
Co-authored-by: Marcin S. <marcin@bytedude.com>
* Correcting git mishap
* Document usage of `gum` crate (#7294)
* Document usage of gum crate
* Small fix
* Add some more basic info
* Update node/gum/src/lib.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Update target docs
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* XCM: Fix issue with RequestUnlock (#7278)
* XCM: Fix issue with RequestUnlock
* Leave API changes for v4
* Fix clippy errors
* Fix tests
---------
Co-authored-by: parity-processbot <>
* Companion for Substrate#14228 (#7295)
* Companion for Substrate#14228
https://github.com/paritytech/substrate/pull/14228
* update lockfile for {"substrate"}
---------
Co-authored-by: parity-processbot <>
* Companion for #14237: Use latest sp-crates (#7300)
* To revert: Update substrate branch to "lexnv/bump_sp_crates"
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates""
This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008.
* Update cargo lock
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update cargo.lock
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update cargo.lock
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* bounded-collections bump to 0.1.7 (#7305)
* bounded-collections bump to 0.1.7
Companion for: paritytech/substrate#14225
* update lockfile for {"substrate"}
---------
Co-authored-by: parity-processbot <>
* bump to quote 1.0.28 (#7306)
* `RollingSessionWindow` cleanup (#7204)
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
* Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db
* Remove `col_session_data` usages
* Storage migration clearing columns w/o removing them
* Remove session data column usages from `approval-voting` and `dispute-coordinator` tests
* Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests
* Fix formatting in initialized.rs
* Fix a corner case in `SessionInfo` caching for `dispute-coordinator`
* Remove `RollingSessionWindow` ;(
* Revert "Fix formatting in initialized.rs"
This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.
* v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it
* Fix `NUM_COLUMNS` in `approval-voting`
* Use `columns::v3::NUM_COLUMNS` when opening db
* Update node/service/src/parachains_db/upgrade.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`
* Fix `NUM+COLUMNS` in approval_voting
* Fix formatting
* Fix columns usage
* Clarification comments about the different db versions
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* pallet-para-config: Remove remnant WeightInfo functions (#7308)
* pallet-para-config: Remove remnant WeightInfo functions
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* set_config_with_weight begone
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
* XCM: PayOverXcm config (#6900)
* Move XCM query functionality to trait
* Fix tests
* Add PayOverXcm implementation
* fix the PayOverXcm trait to compile
* moved doc comment out of trait implmeentation and to the trait
* PayOverXCM documentation
* Change documentation a bit
* Added empty benchmark methods implementation and changed docs
* update PayOverXCM to convert AccountIds to MultiLocations
* Implement benchmarking method
* Change v3 to latest
* Descend origin to an asset sender (#6970)
* descend origin to an asset sender
* sender as tuple of dest and sender
* Add more variants to the QueryResponseStatus enum
* Change Beneficiary to Into<[u8; 32]>
* update PayOverXcm to return concrete errors and use AccountId as sender
* use polkadot-primitives for AccountId
* fix dependency to use polkadot-core-primitives
* force Unpaid instruction to the top of the instructions list
* modify report_outcome to accept interior argument
* use new_query directly for building final xcm query, instead of report_outcome
* fix usage of new_query to use the XcmQueryHandler
* fix usage of new_query to use the XcmQueryHandler
* tiny method calling fix
* xcm query handler (#7198)
* drop redundant query status
* rename ReportQueryStatus to OuterQueryStatus
* revert rename of QueryResponseStatus
* update mapping
* Update xcm/xcm-builder/src/pay.rs
Co-authored-by: Gavin Wood <gavin@parity.io>
* Updates
* Docs
* Fix benchmarking stuff
* Destination can be determined based on asset_kind
* Tweaking API to minimise clones
* Some repotting and docs
---------
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* Companion for #14265 (#7307)
* Update Cargo.lock
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update Cargo.lock
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: parity-processbot <>
* bump serde to 1.0.163 (#7315)
* bump serde to 1.0.163
* bump ci
* update lockfile for {"substrate"}
---------
Co-authored-by: parity-processbot <>
* fmt
* Updated fmt
* Removing changes accidentally pulled from master
* fix another master pull issue
* Another master pull fix
* fmt
* Fixing implementers guide build
* Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread"
This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing
changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: acatangiu <adrian@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* fix bitfield distribution test
* approval distribution tests
* fix bridge tests
* update Cargo.lock
* [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457)
* Optimize collator-protocol validator-side request fetching
* address feedback: replace tuples with structs
* feedback: add doc comments
* move collation types to subfolder
---------
Signed-off-by: alindima <alin@parity.io>
* Update collation generation for asynchronous backing (#7405)
* break candidate receipt construction and distribution into own function
* update implementers' guide to include SubmitCollation
* implement SubmitCollation for collation-generation
* fmt
* fix test compilation & remove unnecessary submodule
* add some TODOs for a test suite.
* Update roadmap/implementers-guide/src/types/overseer-protocol.md
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* add new test harness and first test
* refactor to avoid requiring background sender
* ensure collation gets packaged and distributed
* tests for the fallback case with no hint
* add parent rp-number hint tests
* fmt
* update uses of CollationGenerationConfig
* fix remaining test
* address review comments
* use subsystemsender for background tasks
* fmt
* remove ValidationCodeHashHint and related tests
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* fix some more fallout from merge
* fmt
* remove staging APIs from Rococo & Westend (#7513)
* send network messages on main protocol name (#7515)
* misc async backing improvements for allowed ancestry blocks (#7532)
* shared: fix acquire_info
* backwards-compat test for prospective parachains
* same relay parent is allowed
* provisioner: request candidate receipt by relay parent (#7527)
* return candidates hash from prospective parachains
* update provisioner
* update tests
* guide changes
* send a single message to backing
* fix test
* revert to old `handle_new_activations` logic in some cases (#7514)
* revert to old `handle_new_activations` logic
* gate sending messages on scheduled cores to max_depth >= 2
* fmt
* 2->1
* Omnibus asynchronous backing bugfix PR (#7529)
* fix a bug in backing
* add some more logs
* prospective parachains: take ancestry only up to session bounds
* add test
* fix zombienet tests (#7614)
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
* fix runtime compilation
* make bitfield distribution tests compile
* attempt to fix zombienet disputes (#7618)
* update metric name
* update some metric names
* avoid cycles when creating fake candidates
* make undying collator more friendly to malformed parents
* fix a bug in malus
* fmt
* clippy
* add RUN_IN_CONTAINER to new ZombieNet tests (#7631)
* remove duplicated migration
happened because of master-merge
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: acatangiu <adrian@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: alindima <alin@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Marcin S <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Mattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Alin Dima <alin@parity.io>
* aura: add feature-flagged explicit slot duration type
* fmt
* add some comments
* have node-template use new explicit feature
* fix mock
* fmt
* use the experimental feature flag instead
* checkout master Cargo.lock
* rough draft of potential parent search
* get things compiling
* fmt
* add new function to all RelayChainInterface implementations
* fix compilation
* set slot and timestamp based on relay parent, prepare for find-parent
* skeleton of new aura logic
* fmt
* introduce a collator module in the Aura crate
* extract different implementations into own modules
* make interface more convenient
* docs and todos for lookahead
* refactor basic collator to use new collator utility
* some more refactoring
* finish most of the control flow for new aura
* introduce backend as parameter
* fix compilation
* fix a couple more TODOs
* add an `announce_block` function to collator service
* announce with barrier
* rename block announcement validator to be more specific
* fmt
* clean up unused import errors
* update references to BlockAnnounceValidator
* rename unstable_reimpl
* add AuraUnincludedSegmentApi
* finish rename
* integrate AuraUnincludedSegmentApi
* add a new block announcement validator for backwards compatibility
* add some naive equivocation defenses
* rustfmt
* clean up remaining TODO [now]s
* fmt
* try to fix inprocess-interface
* actually fix compilation
* ignored -> rejected rephrase
* fix test compilation
* fmt
* clippy
* Update all uses of pallet-aura to disallow multiple blocks per slot
* use ConstBool
* update lockfile for {"substrate", "polkadot"}
---------
Co-authored-by: parity-processbot <>
* implement a proposer utility for consensus
* tidy up deps of new proposer crate
* implement a collator-service crate
* rewrite cumulus-collator to use new service struct
* implement a module for relay-chain-driven collators
* adapt start_collator to use the new relay_chain_driven module
* move collator-service to a public submodule
* create an interface trait for the proposer
* begin aura reimplementation
* address review comments
* update substrrate git ref
* update polkadot-primitives refs
* rough draft of aura collation using standalone fns
* add a ServiceInterface
* port aura reimpl to use new service trait
* add an import queue utility crate
* remove import queue crate in favor of module in common
* implement new verification queue for aura
* implement remaining behaviors
* split 'collate' into smaller functions that could be pub
* add telemetry
* fix doc job?
* Specify async-trait patch version
Co-authored-by: Bastian Köcher <git@kchr.de>
* remove 'fn@' in doc string.
Co-authored-by: Bastian Köcher <git@kchr.de>
* update variable names to be more readable
* refactor proposer errors to anyhow/thiserror
* remove manual span instrumentation
Co-authored-by: Bastian Köcher <git@kchr.de>
* make slot_claim private
* fix unused import
* fmt
* fmt
* make clippy happy
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* add digest item for relay-parent to primitives
* add a relay-parent-storage-root digest as a workaround
* more docs
* deposit log in pallet-parachain-system
* even more docs
* fix duplicate imports after botched mertge
* fix hyperlinks in docs
* clean up match
Co-authored-by: Bastian Köcher <git@kchr.de>
* improve docs
* fix typo
* add number to the digest item
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* Extract most aura logic out to standalone module, make use of these
* Update client/consensus/aura/src/standalone.rs
improve docs
Co-authored-by: Bastian Köcher <git@kchr.de>
* add slot_duration_at
* ".git/.scripts/commands/fmt/fmt.sh"
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
* explicitly tag network requests with version
* fmt
* make PeerSet more aware of versioning
* some generalization of the network bridge to support upgrades
* walk back some renaming
* walk back some version stuff
* extract version from fallback
* remove V1 from NetworkBridgeUpdate
* add accidentally-removed timer
* implement focusing for versioned messages
* fmt
* fix up network bridge & tests
* remove inaccurate version check in bridge
* remove some TODO [now]s
* fix fallout in statement distribution
* fmt
* fallout in gossip-support
* fix fallout in collator-protocol
* fix fallout in bitfield-distribution
* fix fallout in approval-distribution
* fmt
* use never!
* fmt
* gossip-support: be explicit about dimensions
* some guide updates
* update network-bridge to distinguish x and y dimensions
* get everything to compile
* beginnings
* some TODOs
* polkadot runtime: use relevant_authorities
* make gossip topologies per-session
* better formatting
* gossip support: use current session validators
* expand in comment
* adjust tests and fix index bug
* add past/present/future connection test and clean up code
* fmt
* network bridge: updated types
* update protocols to new gossip topology message
* guide updates
* add session to BlockApprovalMeta
* add session to block info
* refactor knowledge and remove most unify logic
* start replacing gossip_peers with new SessionTopologies
* add routing information to message state
* add some utilities to SessionTopology
* implement new gossip topology logic
* re-implement unify_with_peer
* distribute assignments according to topology
* finish grid topology implementation
* refactor network bridge slightly
* issue connection requests on all past/present/future
* fmt
* address grumbles
* tighten invariants in unify_with_peer
* implement random propagation
* refactor: extract required routing adjustment logic
* some block-age logic
* aggressively propagate messages when finality is slow
* overhaul aggression system to have 3 levels
* add aggression metrics
* remove aggression L3
* reduce random circulation
* remove PeerData
* get approval tests compiling
* use btree_map in known_by to make deterministic
* Revert "use btree_map in known_by to make deterministic"
This reverts commit 330d65343a7bb6fe4dd0f24bd8dbc15c0cbdbd9d.
* test XY grid propagation
* remove stray println
* test unshared dimension propagation
* add random gossip check
* test unify_with_peer better
* test sending after getting gossip topology
* test L1 aggression on originator
* test L1 aggression for non-originators
* test non-originator aggression L2
* fnt
* ~spellcheck
* fix statement-distribution tests
* fix flaky test
* fix metrics typo
* re-send periodically
* test resending
* typo
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* add more metrics about apd messages
* add back unify_with_peer logs
* make Resend an enum
* be more explicit when resending
* fmt
* fix error
* add a TODO for refactoring
* remove debug metrics
* add some guide stuff
* fmt
* update runtime API in test-runtim
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* remove v0 primitives from polkadot-primitives
* first pass: remove v0
* fix fallout in erasure-coding
* remove v1 primitives, consolidate to v2
* the great import update
* update runtime_api_impl_v1 to v2 as well
* guide: add `Version` request for runtime API
* add version query to runtime API
* reintroduce OldV1SessionInfo in a limited way
* Revert "collator-protocol: fix wrong warning (#4909)"
This reverts commit 128421b5dd.
* Revert "collator-protocol: short-term fixes for connectivity (#4640)"
This reverts commit aff88a864a.
* make the slots great again
Co-authored-by: Andronik <write@reusable.software>
* alter currently-checking-set to launch work only on new candidates
* fmt
* fix compilation
* address review
* Introduce approvals cache test that ensures approval work is only triggered once for each Candidate Hash
* Fix formatting
* Address Feedback
* Move final message await into handle function
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* guide: extract free_cores in scheduler
* scheduler: extract free cores to a separate function
* guide: remove disputed cores from scheduler first
* free disputed cores in scheduler before processing bitfields
* spellcheck is mostly right but sometimes stupid
* add comment and fmt
* pvf: make execution timeout configurable
* guide: add timeouts to candidate validation params
* add timeouts to candidate validation messages
* fmt
* port backing to use the backing pvf timeout
* port approval-voting to use the execution timeout
* port dispute participation to use the correct timeout
* fmt
* address grumbles & test failure
* participate in disputes only if haven't voted already
* Add tests for dispute participation on local import.
* Fixes.
* cargo fmt
* Get rid of redundant dependency.
Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
* DB skeleton
* key formats
* lexicographic test
* custom types for DB
* implement backend for db-v1
* remove VoidBackend and integrate with real DbBackend
* detect stagnant blocks on in interval
* fix tests
* add tests for stagnant
* send ChainSelectionMessage::Approved
* tests for DB backend
* unused import
* upgrade kvdb-memorydb
Co-authored-by: Andronik Ordian <write@reusable.software>
* stubbed SelectRelayChain
* disconnected overseer handlers
* add is_disconnected
* add fallback in case overseer is disconnected
* fall back on fallback
* fetch leaves by calling into chain-selection subsystem
* implement best_chain
* mostly implement finality_target
* chain constrain
* metrics and maximum safeguard
* remove review comment after review
* crate skeleton and type definitions
* add ChainSelectionMessage
* add error type
* run loop
* fix overseer
* simplify determine_new_blocks API
* write an overlay struct and fetch new blocks
* add new function to overlay
* more flow
* add leaves to overlay and add a strong type around leaves-set
* add is_parent_viable
* implement block import, ignoring reversions
* add stagnant-at to overlay
* add stagnant
* add revert consensus log
* flow for reversions
* extract and import block reversions
* recursively update viability
* remove redundant parameter from WriteBlockEntry
* do some removal of viable leaves
* address grumbles
* refactor
* address grumbles
* add comment about non-monotonicity
* extract backend to submodule
* begin the hunt for viable leaves
* viability pivots for updating the active leaves
* remove LeafSearchFrontier
* partially -> explicitly viable and untwist some booleans
* extract tree to submodule
* implement block finality update
* Implement block approval routine
* implement stagnant detection
* ensure blocks pruned on finality are removed from the active leaves set
* write down some planned test cases
* floww
* leaf loading
* implement best_leaf_containing
* write down a few more tests to do
* remove dependence of tree on header
* guide: ChainApiMessage::BlockWeight
* node: BlockWeight ChainAPI
* fix compile issue
* note a few TODOs for the future
* fetch block weight using new BlockWeight ChainAPI
* implement unimplemented
* sort leaves by block number after weight
* remove warnings and add more TODOs
* create test module
* storage for test backend
* wrap inner in mutex
* add write waker query to test backend
* Add OverseerSignal -> FromOverseer conversion
* add test harnes
* add no-op test
* add some more test helpers
* the first test
* more progress on tests
* test two subtrees
* determine-new-blocks: cleaner genesis avoidance and tighter ancestry requests
* don't make ancestry requests when asking for one block
* add a couple more tests
* add to AllMessages in guide
* remove bad spaces from bridge
* compact iterator
* test import with gaps
* more reversion tests
* test finalization pruning subtrees
* fixups
* test clobbering and fix bug in overlay
* exhaustive backend state after finalizaiton tested
* more finality tests
* leaf tests
* test approval
* test ChainSelectionMessage::Leaves thoroughly
* remove TODO
* avoid Ordering::is_ne so CI can build
* comment algorithmic complexity
* Update node/core/chain-selection/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* skeleton for dispute-coordinator
* add coordinator and participation message types
* begin dispute-coordinator DB
* functions for loading
* implement strongly-typed DB transaction
* add some tests for DB transaction
* core logic for pruning
* guide: update candidate-votes key for coordinator
* update candidate-votes key
* use big-endian encoding for session, and implement upper bound generator
* finish implementing pruning
* add a test for note_current_session
* define state of the subsystem itself
* barebones subsystem definition
* control flow
* more control flow
* implement session-updating logic
* trace
* control flow for message handling
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update node/subsystem/src/messages.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* some more control flow
* guide: remove overlay
* more control flow
* implement some DB getters
* make progress on importing statements
* add SignedDisputeStatement struct
* move ApprovalVote to shared primitives
* add a signing-payload API to explicit dispute statements
* add signing-payload to CompactStatement
* add relay-parent hash to seconded/valid dispute variatns
* correct import
* type-safe wrapper around dispute statements
* use checked dispute statement in message type
* extract rolling session window cache to subsystem-util
* extract session window tests
* approval-voting: use rolling session info cache
* reduce dispute window to match runtime in practice
* add byzantine_threshold and supermajority_threshold utilities to primitives
* integrate rolling session window
* Add PartialOrd to CandidateHash
* add Ord to CandidateHash
* implement active dispute update
* add dispute messages to AllMessages
* add dispute stubs to overseer
* inform dispute participation to participate
* implement issue_local_statement
* implement `determine_undisputed_chain`
* fix warnings
* test harness for dispute coordinator tests
* add more helpers to test harness
* add some more helpers
* some tests for dispute coordinator
* ignore wrong validator indices
* test finality voting rule constraint
* add more tests
* add variants to network bridge
* fix test compilation
* remove most dispute coordinator functionality
as of #3222 we can do most of the work within the approval voting subsystem
* Revert "remove most dispute coordinator functionality"
This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.
* Use thiserror
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* extract tests to separate module
* address nit
* adjust run_iteration API
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* guide: reversion safety
* guide: manage reversion safety in subsystems
* add leaf status to ActivatedLeaf
* add an LRU-cache to overseer for staleness detection
* update ActivatedLeaf usages in tests to contain status field
* add variant where missed accidentally
* add some helpers to LeafStatus
* address grumbles
* fix approval-checking GRANDPA voting rule
a `None` return value implies to vote on the best, not to vote on the base.
this explicitly changes the logic to vote on the base
* refactor logic out and test
with the MaxValidators parameter set, this meant that only validators participating in parachain consensus would be able to author relay-chain blocks containing backed candidates
* use compressed blob in candidate-validation
* add some tests for compressed code blobs
* remove CompressedPoV and apply compression in collation-generation
* decompress BlockData before executing
* don't produce oversized collations
* add test for PoV decompression failure
* fix tests and clean up
* fix test
* address review and fix CI
* take this )
* remove real-overseer
* overseer: only activate leaves which support parachains
* integrate HeadSupportsParachains into service
* remove unneeded line
