Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,
## Overall idea
When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.
This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:
- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.
## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.
## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor @burdges, let me know what you think.
- [x] More and more testing.
- [x] Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742
- [x] Final versi burn-in before merging
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Hey guys, as discussed I've changed the name to a more general one
`PvfExecKind`, is this good or too general?
Creating this as a draft, I still have to fix the tests.
Closes#1585
Kusama address: FkB6QEo8VnV3oifugNj5NeVG3Mvq1zFbrUu4P5YwRoe5mQN
---------
Co-authored-by: command-bot <>
Co-authored-by: Marcin S <marcin@realemail.net>
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_**
This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.
There are still a few things that need to be done:
- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802
- [x] Fix existing approval-distribution and approval-voting tests
- [x] Fix bitfield-distribution and statement-distribution tests
- [x] Fix network bridge tests
- [x] Implement todos in the code
- [x] Add tests to cover new code
- [x] Update metrics
- [x] Remove the approval distribution aggression levels: TBD PR
- [x] Parachains DB migration
- [x] Test network protocol upgrade on Versi
- [x] Versi Load test
- [x] Add Zombienet test
- [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
- [x] Fix AcceptedDuplicate
- [x] Fix DB migration so that we can still keep old data.
- [x] Final Versi burn in
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* polkadot: propagate UnpinHandle to ActiveLeafUpdate
Also extract the leaf creation for tests
into a common function.
* dispute-coordinator: try pinned blocks for slashin
* apparently 1.72 is smarter than 1.70
* address nits
* rename fresh_leaf to new_leaf
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
* Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db
* Remove `col_session_data` usages
* Storage migration clearing columns w/o removing them
* Remove session data column usages from `approval-voting` and `dispute-coordinator` tests
* Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests
* Fix formatting in initialized.rs
* Fix a corner case in `SessionInfo` caching for `dispute-coordinator`
* Remove `RollingSessionWindow` ;(
* Revert "Fix formatting in initialized.rs"
This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.
* v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it
* Fix `NUM_COLUMNS` in `approval-voting`
* Use `columns::v3::NUM_COLUMNS` when opening db
* Update node/service/src/parachains_db/upgrade.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`
* Fix `NUM+COLUMNS` in approval_voting
* Fix formatting
* Fix columns usage
* Clarification comments about the different db versions
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
* Happy New Year!
* Remove year entierly
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove years from copyright notice in the entire repo
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Don't import backing statements directly
into the dispute coordinator. This also gets rid of a redundant
signature check. Both should have some impact on backing performance.
In general this PR should make us scale better in the number of parachains.
Reasoning (aka why this is fine):
For the signature check: As mentioned, it is a redundant check. The
signature has already been checked at this point. This is even made
obvious by the used types. The smart constructor is not perfect as
discussed [here](https://github.com/paritytech/polkadot/issues/3455),
but is still a reasonable security.
For not importing to the dispute-coordinator: This should be good as the
dispute coordinator does scrape backing votes from chain. This suffices
in practice as a super majority of validators must have seen a backing
fork in order for a candidate to get included and only included
candidates pose a threat to our system. The import from chain is
preferable over direct import of backing votes for two reasons:
1. The import is batched, greatly improving import performance. All
backing votes for a candidate are imported with a single import.
And indeed we were able to see in metrics that importing votes
from chain is fast.
2. We do less work in general as not every candidate for which
statements are gossiped might actually make it on a chain. The
dispute coordinator as with the current implementation would still
import and keep those votes around for six sessions.
While redundancy is good for reliability in the event of bugs, this also
comes at a non negligible cost. The dispute-coordinator right now is the
subsystem with the highest load, despite the fact that it should not be
doing much during mormal operation and it is only getting worse
with more parachains as the load is a direct function of the number of statements.
We'll see on Versi how much of a performance improvement this PR
* Get rid of dead code.
* Dont send approval vote
* Make it pass CI
* Bring back tests for fixing them later.
* Explicit signature check.
* Resurrect approval-voting tests (not fixed yet)
* Send out approval votes in dispute-distribution.
Use BTreeMap for ordered dispute votes.
* Bring back an important warning.
* Fix approval voting tests.
* Don't send out dispute message on import + test
+ Some cleanup.
* Guide changes.
Note that the introduced complexity is actually redundant.
* WIP: guide changes.
* Finish guide changes about dispute-coordinator
conceputally. Requires more proof read still.
Also removed obsolete implementation details, where the code is better
suited as the source of truth.
* Finish guide changes for now.
* Remove own approval vote import logic.
* Implement logic for retrieving approval-votes
into approval-voting and approval-distribution subsystems.
* Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
* Review feedback.
In particular: Add note about disputes of non included candidates.
* Incorporate Review Remarks
* Get rid of superfluous space.
* Tidy up import logic a bit.
Logical vote import is now separated, making the code more readable and
maintainable.
Also: Accept import if there is at least one invalid signer that has not
exceeded its spam slots, instead of requiring all of them to not exceed
their limits. This is more correct and a preparation for vote batching.
* We don't need/have empty imports.
* Fix tests and bugs.
* Remove error prone redundancy.
* Import approval votes on dispute initiated/concluded.
* Add test for approval vote import.
* Make guide checker happy (hopefully)
* Another sanity check + better logs.
* Reasoning about boundedness.
* Use `CandidateIndex` as opposed to `CoreIndex`.
* Remove redundant import.
* Review remarks.
* Add metric for calls to request signatures
* More review remarks.
* Add metric on imported approval votes.
* Include candidate hash in logs.
* More trace log
* Break cycle.
* Add some tracing.
* Cleanup allowed messages.
* fmt
* Tracing + timeout for get inherent data.
* Better error.
* Break cycle in all places.
* Clarified comment some more.
* Typo.
* Break cycle approval-distribution - approval-voting.
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
* Revert approval-voting subsystem
* Approval voting revert encapsulated within 'ops' module
* use 'get_stored_blocks' to get lower block height
* Fix error message
* Optionally shrink/delete stored blocks range
* range end number is last block number plus 1
* Apply code review suggestions
* Use tristate enum for block range in backend overlay
* Add clarification comment
* Add comments to private struct
* Don't wait for dispute coordinator
in backing and approval-voting - we are single threaded there, so this
is blocking everything.
* Add missing import.
* Don't warn on dropped receiver.
* remove v0 primitives from polkadot-primitives
* first pass: remove v0
* fix fallout in erasure-coding
* remove v1 primitives, consolidate to v2
* the great import update
* update runtime_api_impl_v1 to v2 as well
* guide: add `Version` request for runtime API
* add version query to runtime API
* reintroduce OldV1SessionInfo in a limited way
* parity db subsystem without cache and no splitted column
* fmt
* fix path (auto from parity-db fail)
* lru cache for db column with cache
* Revert "lru cache for db column with cache"
This reverts commit ae177bc5e107a075eff6a21f651218ada6599b74.
* Write_lock mutex
* theoric code for bridges
* revert changes
* Revert bridge changes
* fix spec_version
* update parity db
* test purge-db
* Use specific ordered collection with paritydb.
* Revert "Use specific ordered collection with paritydb."
This reverts commit 8b66d0a4ae914cba1af0f44050d45dd6d9327c6b.
* fix chain selection tests.
* remove patch
* fix auto.
* Remove useless exists directory method
* purge chain without parity-db removal
* spellcheck
* renamings and filtering.
* fix assertion
* format
* update parity-db and fmt
* Auto keep using rocksdb when it exists.
* Revert "Auto keep using rocksdb when it exists."
This reverts commit cea49b32ae590bdce31fed5c45f3c028ae0c7564.
* Update kvdb version.
* alter currently-checking-set to launch work only on new candidates
* fmt
* fix compilation
* address review
* Introduce approvals cache test that ensures approval work is only triggered once for each Candidate Hash
* Fix formatting
* Address Feedback
* Move final message await into handle function
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* remove Default from CandidateHash
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: fmt
* remove backed candidate default
* Partial migration away from CandidateReceipt::default
* Remove more CandidateReceipt defaults
* fmt
* Mostly remove CommittedCandidateReceipt default usage
* Remove CommittedCandidateReceipt
* Remove more Defaults from polakdot primitives v1 + fmt
* Remove more Default from polkadot primites v1
* WIP trying to get overseer example + tests to compile
* feat: add primitives test helpers
* reduce deps of helper
* update primitive helpers
* make candidate validation compile
* fixup cargo lock
* make av-store compile
* fixup disputes coordinator tests
* test: fixup backing
* test: fixup approval voting
* fixup bitfield signing
* test: fixup runtime-api
* test: fixup availability dist
* foxi[ pverseer test]
* remove some Defaults, remove bounds from `dummy`
All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.
* it's a test helper, so always use std
* test: fixup parachains runtime tests
Excluding benches.
* fix keyring
* fix paras runtime properly, no more default
* Remove fn dummy() usage from approval voting
* Move TestCandidateBuilder out of av store to test helpers
* Make candidate validation tests pass
* Make most dispute coirdinator tests pass
* Make provisioner tests work
* Make availability recovery tests work with test helpers
* Update polkadot-collator-protocol tests
* Update statement distribution tests
* Update polkadot overseer examples and tests
* Derive default for validation code so we don't break unrelated things
* Make para runtime test pass (no bench)
* Some more work
* chore: cargo fmt
* cargo fix
* avoid some Default::default
* fixup dispute coordinator test
* remove unused crate deps
* remove Default::default wherever possible, replace by dummy_* for the most part
* chore: cargo fmt
* Remove some warnings
* Remove CommittedCandidateReceipt dummy
* Remove CandidateReceipt dummy
* Remove CandidateDescriptor dummy
* Remove commented out code
* Fix para runtime tests
* chore: nightly
* Some updates to the builder
* Dynamically adjust mock head data size
* Make dispute cooridinator tests work
* Fix test candidate_backing_reorders_votes work
* +nightly-2021-10-29 fmt
* Spelling and remove a default use in builder
* Various clean up
* More small updates
* fmt
* More small updates
* Doc comments for test helpers
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs
* Update lib.rs
* review comments
* fix warnings
* fix test by using correct candidate receipt relay parent
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* node/approval-voting: Introduce Backend trait and Overlaybackend
This commit introduces a Backend trait and attempts to move away
from the Action model via an OverlayBackend as in the ChainSelection
subsystem.
* node/approval-voting: Add WriteOps for StoredBlockRange and BlocksAtHeight
* node/approval-voting: Add load_all_blocks to overlay
* node/approval-voting: Get all module tests to pass.
This commit modifies all tests to ensure tests are passing.
* node/approval-voting: Address oversights in the previous commit
This commit addresses some oversights in the prior commit.
1. Inner errors in backend.write were swallowed
2. One-off write functions removed to avoid useless abstraction
3. Touch-ups in general
* node/approval-voting: Move from TestDB to dyn KeyValueDB
This commit removes the TestDB from tests.rs and replaces it with
an in-memory kvdb.
* node/approval-voting: Address feedback
* node/approval-voting: Add license to ops.rs
* node/approval-voting: Address second-pass feedback
* Add TODO
* node/approval-voting: Bump spec_version
* node/approval-voting: Address final comments.
* node/approval-voting: Introduce framework for full subsystem tests
* node/approval-voting: Introduce basic tests to attempt to provide
coverage via full subsystem tests
* node/approval-voting: Introduce Chainbuilder
* node/approval-voting: Introduce Backend trait and Overlaybackend
This commit introduces a Backend trait and attempts to move away
from the Action model via an OverlayBackend as in the ChainSelection
subsystem.
* node/approval-voting: Add WriteOps for StoredBlockRange and BlocksAtHeight
* node/approval-voting: Add load_all_blocks to overlay
* node/approval-voting: Get all module tests to pass.
This commit modifies all tests to ensure tests are passing.
* node/approval-voting: Address oversights in the previous commit
This commit addresses some oversights in the prior commit.
1. Inner errors in backend.write were swallowed
2. One-off write functions removed to avoid useless abstraction
3. Touch-ups in general
* node/approval-voting: Move from TestDB to dyn KeyValueDB
This commit removes the TestDB from tests.rs and replaces it with
an in-memory kvdb.
* node/approval-voting: Address feedback
* node/approval-voting: Add license to ops.rs
* node/approval-voting: Address second-pass feedback
* Add TODO
* node/approval-voting: Bump spec_version
* node/approval-voting: Address final comments.
* DB skeleton
* key formats
* lexicographic test
* custom types for DB
* implement backend for db-v1
* remove VoidBackend and integrate with real DbBackend
* detect stagnant blocks on in interval
* fix tests
* add tests for stagnant
* send ChainSelectionMessage::Approved
* tests for DB backend
* unused import
* upgrade kvdb-memorydb
Co-authored-by: Andronik Ordian <write@reusable.software>
* skeleton for dispute-coordinator
* add coordinator and participation message types
* begin dispute-coordinator DB
* functions for loading
* implement strongly-typed DB transaction
* add some tests for DB transaction
* core logic for pruning
* guide: update candidate-votes key for coordinator
* update candidate-votes key
* use big-endian encoding for session, and implement upper bound generator
* finish implementing pruning
* add a test for note_current_session
* define state of the subsystem itself
* barebones subsystem definition
* control flow
* more control flow
* implement session-updating logic
* trace
* control flow for message handling
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update node/subsystem/src/messages.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* some more control flow
* guide: remove overlay
* more control flow
* implement some DB getters
* make progress on importing statements
* add SignedDisputeStatement struct
* move ApprovalVote to shared primitives
* add a signing-payload API to explicit dispute statements
* add signing-payload to CompactStatement
* add relay-parent hash to seconded/valid dispute variatns
* correct import
* type-safe wrapper around dispute statements
* use checked dispute statement in message type
* extract rolling session window cache to subsystem-util
* extract session window tests
* approval-voting: use rolling session info cache
* reduce dispute window to match runtime in practice
* add byzantine_threshold and supermajority_threshold utilities to primitives
* integrate rolling session window
* Add PartialOrd to CandidateHash
* add Ord to CandidateHash
* implement active dispute update
* add dispute messages to AllMessages
* add dispute stubs to overseer
* inform dispute participation to participate
* implement issue_local_statement
* implement `determine_undisputed_chain`
* fix warnings
* test harness for dispute coordinator tests
* add more helpers to test harness
* add some more helpers
* some tests for dispute coordinator
* ignore wrong validator indices
* test finality voting rule constraint
* add more tests
* add variants to network bridge
* fix test compilation
* remove most dispute coordinator functionality
as of #3222 we can do most of the work within the approval voting subsystem
* Revert "remove most dispute coordinator functionality"
This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.
* Use thiserror
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* extract tests to separate module
* address nit
* adjust run_iteration API
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* extract database from av-store itself
* generalize approval-voting over database type
* modes (without handling) and pruning old wakeups
* rework approval importing
* add our_approval_sig to ApprovalEntry
* import assignment
* guide updates for check-full-approval changes
* some aux functions
* send messages when becoming active.
* guide: network bridge sends view updates only when done syncing
* network bridge: send view updates only when done syncing
* tests for new network-bridge behavior
* add a test for updating approval entry with sig
* fix some warnings
* test load-all-blocks
* instantiate new parachains DB
* fix network-bridge empty view updates
* tweak
* fix wasm build, i think
* Update node/core/approval-voting/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some versioning to parachains_db
* warnings
* fix merge changes
* remove versioning again
Co-authored-by: Andronik Ordian <write@reusable.software>
Alexandru Gheorghe