* explicitly tag network requests with version
* fmt
* make PeerSet more aware of versioning
* some generalization of the network bridge to support upgrades
* walk back some renaming
* walk back some version stuff
* extract version from fallback
* remove V1 from NetworkBridgeUpdate
* add accidentally-removed timer
* implement focusing for versioned messages
* fmt
* fix up network bridge & tests
* remove inaccurate version check in bridge
* remove some TODO [now]s
* fix fallout in statement distribution
* fmt
* fallout in gossip-support
* fix fallout in collator-protocol
* fix fallout in bitfield-distribution
* fix fallout in approval-distribution
* fmt
* use never!
* fmt
* Move `trait ParachainHost` to a separate version independent module
`trait ParachainHost` is no longer part of a specific primitives
version. Instead there is a single trait for stable and staging api
versions. The trait contains stable AND staging methods. The latter are
explicitly marked as unstable.
* Fix `use` primitives
`polkadot_primitives::v2` becomes `polkadot_primitives::runtime_api`
* Staging API declaration and stubs
Introduces the concept for 'staging functions' in runtime API. These
functions are still in testing and they are meant to be used only
within test networks (Westend).
They coexist with the stable calls for technical reasons - maintaining
different runtime APIs for different networks is hard to implement.
Check the doc comments in source files for more details how the staging
API should be used.
* Add new staging method - get_session_disputes()
Add `staging_get_session_disputes` to `ParachainHost` as the first
method of the staging API.
* Hide vstaging runtime api implementations behind feature flag
* Fix test runtime
* fn staging_get_session_disputes() is renamed to fn staging_get_disputes()
* split metrics from bitfield signing
* cleanup all logging
* add a unit test for subset generation
* chore: add one more test to assert need is properly represented
* u8 as usize
* chore: overseer fixin
* fix test
* Update node/network/bitfield-distribution/src/metrics.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Update node/network/bitfield-distribution/src/metrics.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* fallout from suggested rename
* consistency
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* remove v0 primitives from polkadot-primitives
* first pass: remove v0
* fix fallout in erasure-coding
* remove v1 primitives, consolidate to v2
* the great import update
* update runtime_api_impl_v1 to v2 as well
* guide: add `Version` request for runtime API
* add version query to runtime API
* reintroduce OldV1SessionInfo in a limited way
This commit implements the last major piece of #3211: the subsystem that
tracks PVFs that require voting, issues pre-check requests to
candidate-validation and makes sure that the votes are submitted to the
chain.
The function `has_api` checks that the api + version matches, which
isn't true anymore after bumping the version. The fix is to just compare
the runtime api version being at least `1`.
* Companion PR for removing Prometheus metrics prefix
* Was missing some metrics
* Fix missing renames
* Fix test
* Fixes
* Update test
* Update Substrate
* Second time
* remove prefix from intergration test for zombienet
* update zombienet image
* Update Substrate
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Javier Viola <pepoviola@gmail.com>
* remove Default from CandidateHash
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: fmt
* remove backed candidate default
* Partial migration away from CandidateReceipt::default
* Remove more CandidateReceipt defaults
* fmt
* Mostly remove CommittedCandidateReceipt default usage
* Remove CommittedCandidateReceipt
* Remove more Defaults from polakdot primitives v1 + fmt
* Remove more Default from polkadot primites v1
* WIP trying to get overseer example + tests to compile
* feat: add primitives test helpers
* reduce deps of helper
* update primitive helpers
* make candidate validation compile
* fixup cargo lock
* make av-store compile
* fixup disputes coordinator tests
* test: fixup backing
* test: fixup approval voting
* fixup bitfield signing
* test: fixup runtime-api
* test: fixup availability dist
* foxi[ pverseer test]
* remove some Defaults, remove bounds from `dummy`
All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.
* it's a test helper, so always use std
* test: fixup parachains runtime tests
Excluding benches.
* fix keyring
* fix paras runtime properly, no more default
* Remove fn dummy() usage from approval voting
* Move TestCandidateBuilder out of av store to test helpers
* Make candidate validation tests pass
* Make most dispute coirdinator tests pass
* Make provisioner tests work
* Make availability recovery tests work with test helpers
* Update polkadot-collator-protocol tests
* Update statement distribution tests
* Update polkadot overseer examples and tests
* Derive default for validation code so we don't break unrelated things
* Make para runtime test pass (no bench)
* Some more work
* chore: cargo fmt
* cargo fix
* avoid some Default::default
* fixup dispute coordinator test
* remove unused crate deps
* remove Default::default wherever possible, replace by dummy_* for the most part
* chore: cargo fmt
* Remove some warnings
* Remove CommittedCandidateReceipt dummy
* Remove CandidateReceipt dummy
* Remove CandidateDescriptor dummy
* Remove commented out code
* Fix para runtime tests
* chore: nightly
* Some updates to the builder
* Dynamically adjust mock head data size
* Make dispute cooridinator tests work
* Fix test candidate_backing_reorders_votes work
* +nightly-2021-10-29 fmt
* Spelling and remove a default use in builder
* Various clean up
* More small updates
* fmt
* More small updates
* Doc comments for test helpers
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs
* Update lib.rs
* review comments
* fix warnings
* fix test by using correct candidate receipt relay parent
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* Mostly notes.
* Better error messages.
* Introduce Fatal/NonFatal + drop back channel participation
- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.
* Better error messages.
* Utility function for receiving `CandidateEvent`s.
* Ordering module typechecks.
* cargo fmt
* Prepare spam slots module.
* Implement SpamSlots mechanism.
* Implement queues.
* cargo fmt
* Participation.
* Participation taking shape.
* Finish participation.
* cargo fmt
* Cleanup.
* WIP: Cleanup + Integration.
* Make `RollingSessionWindow` initialized by default.
* Make approval voting typecheck.
* Get rid of lazy_static & fix approval voting tests
* Move `SessionWindowSize` to node primitives.
* Implement dispute coordinator initialization.
* cargo fmt
* Make queues return error instead of boolean.
* Initialized: WIP
* Introduce chain api for getting finalized block.
* Fix ordering to only prune candidates on finalized events.
* Pruning of old sessions in spam slots.
* New import logic.
* Make everything typecheck.
* Fix warnings.
* Get rid of obsolete dispute-participation.
* Fixes.
* Add back accidentelly deleted Cargo.lock
* Deliver disputes in an ordered fashion.
* Add module docs for errors
* Use type synonym.
* hidden docs.
* Fix overseer tests.
* Ordering provider taking `CandidateReceipt`.
... To be kicked on one next commit.
* Fix ordering to use relay_parent
as included block is not unique per candidate.
* Add comment in ordering.rs.
* Take care of duplicate entries in queues.
* Better spam slots.
* Review remarks + docs.
* Fix db tests.
* Participation tests.
* Also scrape votes on first leaf for good measure.
* Make tests typecheck.
* Spelling.
* Only participate in actual disputes, not on every import.
* Don't account backing votes to spam slots.
* Fix more tests.
* Don't participate if we don't have keys.
* Fix tests, typos and warnings.
* Fix merge error.
* Spelling fixes.
* Add missing docs.
* Queue tests.
* More tests.
* Add metrics + don't short circuit import.
* Basic test for ordering provider.
* Import fix.
* Remove dead link.
* One more dead link.
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* pvf: make execution timeout configurable
* guide: add timeouts to candidate validation params
* add timeouts to candidate validation messages
* fmt
* port backing to use the backing pvf timeout
* port approval-voting to use the execution timeout
* port dispute participation to use the correct timeout
* fmt
* address grumbles & test failure
* Remove incorrect proof about Jemalloc
The truth is that Jemalloc is not always the default allocator. This is
only true for the polkadot binary.
* Fmt
* Rephrase
* overseer: remove mut in connector
* rename SelectRelayChainWFallback -> SelectRelayChain
* split Basics
* introduce the OverseerConnector, use it
* introduce is_relay_chain to RelayChainSelection
* chore: rename var
* avoid dummy import in subsystem
* actually remove Disconnecte/Connected enum
* extract DummySubsystem into mod dummy.
* Handle::Connected -> Handle::new
* chore: fmt
* fix test
* select relay chain takes no arg, simplification
* fmt
* Update node/service/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: improve malus tests
* avoid the deferred setting of `is_relay_chain` in `RelayChainSelection`
* positive assertion is not mandated, only the negative one, to avoid a stall
* chore: fmt
* assure the `RelayChainSelection` is not used before the overseer is up and running
Co-authored-by: Andronik Ordian <write@reusable.software>
* Attempt to add log stats to gossip-support.
* WIP: Keep track of connected validators.
* Clarify metric.
* WIP: Make gossip support report connectivity.
* WIP: Fixing tests.
* Fix network bridge + integrate in overseer.
* Consistent naming.
* Fix logic error
* cargo fmt
* Pretty logs.
* cargo fmt
* Use `Delay` to trigger periodic checks.
* fmt
* Fix warning for authority set size of 1.
* More correct ratio report if there are no resolved validators.
* Prettier rendering of empty set.
* Fix typo.
* Another typo.
* Don't check on every leaf update.
* Make compatible with older rustc.
* Fix tests.
* Demote warning.
* remove connected disconnected state from overseer
* foo
* split new partial
* fix
* refactor init code to not require a `OverseerHandle` when we don't have an overseer
* intermediate
* fixins
* X
* fixup
* foo
* fixup
* docs
* conditional
* Update node/service/src/lib.rs
* review by ladi
* feat/overseer: introduce closure init
Enables removal of the connected/disconnected overseer state.
* feat/overseer: allow replacement logic to access the original
Allows to re-use init-once types, which would otherwise error.
* feat/overseer: introduce external connector
Preparation for removal of `AllSubsystems`
which is another prerequisite for removing
the connect/disconnect state.
* fix/test: replace needs closure
* fixup
* simplify
* mea culpa
* all-subsystems-gen test
* add jemalloc memory statistics tracking
* chore: move Metronome in a separate file
* add meta flag spellcheck
* adjust metrics names
* account for new metrics in test
* WIP: Get rid of request multiplexer.
* WIP
* Receiver for handling of incoming requests.
* Get rid of useless `Fault` abstraction.
The things the type system let us do are not worth getting abstracted in
its own type. Instead error handling is going to be merely a pattern.
* Make most things compile again.
* Port availability distribution away from request multiplexer.
* Formatting.
* Port dispute distribution over.
* Fixup statement distribution.
* Handle request directly in collator protocol.
+ Only allow fatal errors at top level.
* Use direct request channel for availability recovery.
* Finally get rid of request multiplexer
Fixes#2842 and paves the way for more back pressure possibilities.
* Fix overseer and statement distribution tests.
* Fix collator protocol and network bridge tests.
* Fix tests in availability recovery.
* Fix availability distribution tests.
* Fix dispute distribution tests.
* Add missing dependency
* Typos.
* Review remarks.
* More remarks.
* initial integration and migration code
* fix tests
* fix counting test
* assume the current version on missing file
* use SelectRelayChain
* remove duplicate metric
* Update node/service/src/lib.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* remove ApprovalCheckingVotingRule
* address my concern
* never mode for StagnantCheckInterval
* REVERTME: some logs
* w00t
* it's ugly but it works
* Revert "REVERTME: some logs"
This reverts commit e210505a2e83e31c381394924500b69277bb042e.
* it's handle, not handler
* fix a few typos
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* CI: add spellcheck
* revert me
* CI: explicit command for spellchecker
* spellcheck: edit misspells
* CI: run spellcheck on diff
* spellcheck: edits
* spellcheck: edit misspells
* spellcheck: add rules
* spellcheck: mv configs
* spellcheck: more edits
* spellcheck: chore
* spellcheck: one more thing
* spellcheck: and another one
* spellcheck: seems like it doesn't get to an end
* spellcheck: new words after rebase
* spellcheck: new words appearing out of nowhere
* chore
* review edits
* more review edits
* more edits
* wonky behavior
* wonky behavior 2
* wonky behavior 3
* change git behavior
* spellcheck: another bunch of new edits
* spellcheck: new words are koming out of nowhere
* CI: finding the master
* CI: fetching master implicitly
* CI: undebug
* new errors
* a bunch of new edits
* and some more
* Update node/core/approval-voting/src/approval_db/v1/mod.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update xcm/xcm-executor/src/assets.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* Suggestions from the code review
* CI: scan only changed files
Co-authored-by: Andronik Ordian <write@reusable.software>
* stubbed SelectRelayChain
* disconnected overseer handlers
* add is_disconnected
* add fallback in case overseer is disconnected
* fall back on fallback
* fetch leaves by calling into chain-selection subsystem
* implement best_chain
* mostly implement finality_target
* chain constrain
* metrics and maximum safeguard
* remove review comment after review
* crate skeleton and type definitions
* add ChainSelectionMessage
* add error type
* run loop
* fix overseer
* simplify determine_new_blocks API
* write an overlay struct and fetch new blocks
* add new function to overlay
* more flow
* add leaves to overlay and add a strong type around leaves-set
* add is_parent_viable
* implement block import, ignoring reversions
* add stagnant-at to overlay
* add stagnant
* add revert consensus log
* flow for reversions
* extract and import block reversions
* recursively update viability
* remove redundant parameter from WriteBlockEntry
* do some removal of viable leaves
* address grumbles
* refactor
* address grumbles
* add comment about non-monotonicity
* extract backend to submodule
* begin the hunt for viable leaves
* viability pivots for updating the active leaves
* remove LeafSearchFrontier
* partially -> explicitly viable and untwist some booleans
* extract tree to submodule
* implement block finality update
* Implement block approval routine
* implement stagnant detection
* ensure blocks pruned on finality are removed from the active leaves set
* write down some planned test cases
* floww
* leaf loading
* implement best_leaf_containing
* write down a few more tests to do
* remove dependence of tree on header
* guide: ChainApiMessage::BlockWeight
* node: BlockWeight ChainAPI
* fix compile issue
* note a few TODOs for the future
* fetch block weight using new BlockWeight ChainAPI
* implement unimplemented
* sort leaves by block number after weight
* remove warnings and add more TODOs
* create test module
* storage for test backend
* wrap inner in mutex
* add write waker query to test backend
* Add OverseerSignal -> FromOverseer conversion
* add test harnes
* add no-op test
* add some more test helpers
* the first test
* more progress on tests
* test two subtrees
* determine-new-blocks: cleaner genesis avoidance and tighter ancestry requests
* don't make ancestry requests when asking for one block
* add a couple more tests
* add to AllMessages in guide
* remove bad spaces from bridge
* compact iterator
* test import with gaps
* more reversion tests
* test finalization pruning subtrees
* fixups
* test clobbering and fix bug in overlay
* exhaustive backend state after finalizaiton tested
* more finality tests
* leaf tests
* test approval
* test ChainSelectionMessage::Leaves thoroughly
* remove TODO
* avoid Ordering::is_ne so CI can build
* comment algorithmic complexity
* Update node/core/chain-selection/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* skeleton for dispute-coordinator
* add coordinator and participation message types
* begin dispute-coordinator DB
* functions for loading
* implement strongly-typed DB transaction
* add some tests for DB transaction
* core logic for pruning
* guide: update candidate-votes key for coordinator
* update candidate-votes key
* use big-endian encoding for session, and implement upper bound generator
* finish implementing pruning
* add a test for note_current_session
* define state of the subsystem itself
* barebones subsystem definition
* control flow
* more control flow
* implement session-updating logic
* trace
* control flow for message handling
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update node/subsystem/src/messages.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* some more control flow
* guide: remove overlay
* more control flow
* implement some DB getters
* make progress on importing statements
* add SignedDisputeStatement struct
* move ApprovalVote to shared primitives
* add a signing-payload API to explicit dispute statements
* add signing-payload to CompactStatement
* add relay-parent hash to seconded/valid dispute variatns
* correct import
* type-safe wrapper around dispute statements
* use checked dispute statement in message type
* extract rolling session window cache to subsystem-util
* extract session window tests
* approval-voting: use rolling session info cache
* reduce dispute window to match runtime in practice
* add byzantine_threshold and supermajority_threshold utilities to primitives
* integrate rolling session window
* Add PartialOrd to CandidateHash
* add Ord to CandidateHash
* implement active dispute update
* add dispute messages to AllMessages
* add dispute stubs to overseer
* inform dispute participation to participate
* implement issue_local_statement
* implement `determine_undisputed_chain`
* fix warnings
* test harness for dispute coordinator tests
* add more helpers to test harness
* add some more helpers
* some tests for dispute coordinator
* ignore wrong validator indices
* test finality voting rule constraint
* add more tests
* add variants to network bridge
* fix test compilation
* remove most dispute coordinator functionality
as of #3222 we can do most of the work within the approval voting subsystem
* Revert "remove most dispute coordinator functionality"
This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.
* Use thiserror
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update node/core/dispute-coordinator/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* extract tests to separate module
* address nit
* adjust run_iteration API
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Create validator_side module
* Subsume Candidate Selection
* Add test to ensure candidate backing logic is correct
* Ensure secondings are adequately cleaned up and address test flakyness
* Address Feedback
* guide: reversion safety
* guide: manage reversion safety in subsystems
* add leaf status to ActivatedLeaf
* add an LRU-cache to overseer for staleness detection
* update ActivatedLeaf usages in tests to contain status field
* add variant where missed accidentally
* add some helpers to LeafStatus
* address grumbles
asynchronous rob