* remove Default from CandidateHash
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: fmt
* remove backed candidate default
* Partial migration away from CandidateReceipt::default
* Remove more CandidateReceipt defaults
* fmt
* Mostly remove CommittedCandidateReceipt default usage
* Remove CommittedCandidateReceipt
* Remove more Defaults from polakdot primitives v1 + fmt
* Remove more Default from polkadot primites v1
* WIP trying to get overseer example + tests to compile
* feat: add primitives test helpers
* reduce deps of helper
* update primitive helpers
* make candidate validation compile
* fixup cargo lock
* make av-store compile
* fixup disputes coordinator tests
* test: fixup backing
* test: fixup approval voting
* fixup bitfield signing
* test: fixup runtime-api
* test: fixup availability dist
* foxi[ pverseer test]
* remove some Defaults, remove bounds from `dummy`
All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.
* it's a test helper, so always use std
* test: fixup parachains runtime tests
Excluding benches.
* fix keyring
* fix paras runtime properly, no more default
* Remove fn dummy() usage from approval voting
* Move TestCandidateBuilder out of av store to test helpers
* Make candidate validation tests pass
* Make most dispute coirdinator tests pass
* Make provisioner tests work
* Make availability recovery tests work with test helpers
* Update polkadot-collator-protocol tests
* Update statement distribution tests
* Update polkadot overseer examples and tests
* Derive default for validation code so we don't break unrelated things
* Make para runtime test pass (no bench)
* Some more work
* chore: cargo fmt
* cargo fix
* avoid some Default::default
* fixup dispute coordinator test
* remove unused crate deps
* remove Default::default wherever possible, replace by dummy_* for the most part
* chore: cargo fmt
* Remove some warnings
* Remove CommittedCandidateReceipt dummy
* Remove CandidateReceipt dummy
* Remove CandidateDescriptor dummy
* Remove commented out code
* Fix para runtime tests
* chore: nightly
* Some updates to the builder
* Dynamically adjust mock head data size
* Make dispute cooridinator tests work
* Fix test candidate_backing_reorders_votes work
* +nightly-2021-10-29 fmt
* Spelling and remove a default use in builder
* Various clean up
* More small updates
* fmt
* More small updates
* Doc comments for test helpers
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs
* Update lib.rs
* review comments
* fix warnings
* fix test by using correct candidate receipt relay parent
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* bump version and spec_version
* bump transaction version
* Bump of the default value in CI
* bump tx version for westend and kusama
Co-authored-by: Wilfried Kopp <wilfried@parity.io>
* WIP: Get rid of request multiplexer.
* WIP
* Receiver for handling of incoming requests.
* Get rid of useless `Fault` abstraction.
The things the type system let us do are not worth getting abstracted in
its own type. Instead error handling is going to be merely a pattern.
* Make most things compile again.
* Port availability distribution away from request multiplexer.
* Formatting.
* Port dispute distribution over.
* Fixup statement distribution.
* Handle request directly in collator protocol.
+ Only allow fatal errors at top level.
* Use direct request channel for availability recovery.
* Finally get rid of request multiplexer
Fixes#2842 and paves the way for more back pressure possibilities.
* Fix overseer and statement distribution tests.
* Fix collator protocol and network bridge tests.
* Fix tests in availability recovery.
* Fix availability distribution tests.
* Fix dispute distribution tests.
* Add missing dependency
* Typos.
* Review remarks.
* More remarks.
* extract database from av-store itself
* generalize approval-voting over database type
* modes (without handling) and pruning old wakeups
* rework approval importing
* add our_approval_sig to ApprovalEntry
* import assignment
* guide updates for check-full-approval changes
* some aux functions
* send messages when becoming active.
* guide: network bridge sends view updates only when done syncing
* network bridge: send view updates only when done syncing
* tests for new network-bridge behavior
* add a test for updating approval entry with sig
* fix some warnings
* test load-all-blocks
* instantiate new parachains DB
* fix network-bridge empty view updates
* tweak
* fix wasm build, i think
* Update node/core/approval-voting/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some versioning to parachains_db
* warnings
* fix merge changes
* remove versioning again
Co-authored-by: Andronik Ordian <write@reusable.software>
* overseer: pass messages directly between subsystems
* test that message is held on to
* Update node/overseer/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* give every subsystem an unbounded sender too
* remove metered_channel::name
1. we don't provide good names
2. these names are never used anywhere
* unused mut
* remove unnecessary &mut
* subsystem unbounded_send
* remove unused MaybeTimer
We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.
* remove comment
* split up senders and receivers
* update metrics
* fix tests
* fix test subsystem context
* use SubsystemSender in jobs system now
* refactor of awful jobs code
* expose public `run` on JobSubsystem
* update candidate backing to new jobs & use unbounded
* bitfield signing
* candidate-selection
* provisioner
* approval voting: send unbounded for assignment/approvals
* async not needed
* begin bridge split
* split up network tasks into background worker
* port over network bridge
* Update node/network/bridge/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* rename ValidationWorkerNotifications
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* Move NetworkBridgeEvent to subsystem::messages.
It is not protocol related at all, it is in fact only part of the
subsystem communication as it gets wrapped into messages of each
subsystem.
* Request/response infrastructure is taking shape.
WIP: Does not compile.
* Multiplexer variant not supported by Rusts type system.
* request_response::request type checks.
* Cleanup.
* Minor fixes for request_response.
* Implement request sending + move multiplexer.
Request multiplexer is moved to bridge, as there the implementation is
more straight forward as we can specialize on `AllMessages` for the
multiplexing target.
Sending of requests is mostly complete, apart from a few `From`
instances. Receiving is also almost done, initializtion needs to be
fixed and the multiplexer needs to be invoked.
* Remove obsolete multiplexer.
* Initialize bridge with multiplexer.
* Finish generic request sending/receiving.
Subsystems are now able to receive and send requests and responses via
the overseer.
* Doc update.
* Fixes.
* Link issue for not yet implemented code.
* Fixes suggested by @ordian - thanks!
- start encoding at 0
- don't crash on zero protocols
- don't panic on not yet implemented request handling
* Update node/network/protocol/src/request_response/v1.rs
Use index 0 instead of 1.
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update node/network/protocol/src/request_response.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Fix existing tests.
* Better avoidance of division by zoro errors.
* Doc fixes.
* send_request -> start_request.
* Fix missing renamings.
* Update substrate.
* Pass TryConnect instead of true.
* Actually import `IfDisconnected`.
* Fix wrong import.
* Update node/network/bridge/src/lib.rs
typo
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update node/network/bridge/src/multiplexer.rs
Remove redundant import.
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Stop doing tracing from within `From` instance.
Thanks for the catch @tomaka!
* Get rid of redundant import.
* Formatting cleanup.
* Fix tests.
* Add link to issue.
* Clarify comments some more.
* Fix tests.
* Formatting fix.
* tabs
* Fix link
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Use map_err.
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Improvements inspired by suggestions by @drahnr.
- Channel size is now determined by function.
- Explicitely scope NetworkService::start_request.
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* drop in tracing to replace log
* add structured logging to trace messages
* add structured logging to debug messages
* add structured logging to info messages
* add structured logging to warn messages
* add structured logging to error messages
* normalize spacing and Display vs Debug
* add instrumentation to the various 'fn run'
* use explicit tracing module throughout
* fix availability distribution test
* don't double-print errors
* remove further redundancy from logs
* fix test errors
* fix more test errors
* remove unused kv_log_macro
* fix unused variable
* add tracing spans to collation generation
* add tracing spans to av-store
* add tracing spans to backing
* add tracing spans to bitfield-signing
* add tracing spans to candidate-selection
* add tracing spans to candidate-validation
* add tracing spans to chain-api
* add tracing spans to provisioner
* add tracing spans to runtime-api
* add tracing spans to availability-distribution
* add tracing spans to bitfield-distribution
* add tracing spans to network-bridge
* add tracing spans to collator-protocol
* add tracing spans to pov-distribution
* add tracing spans to statement-distribution
* add tracing spans to overseer
* cleanup
* stupid, but it compiles
* redo
* cleanup
* add ValidatorDiscovery to msgs
* sketch network bridge code
* ConnectToAuthorities instead of validators
* more stuff
* cleanup
* more stuff
* complete ConnectToAuthoritiesState
* Update node/network/bridge/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Collator protocol subsystem (#1659)
* WIP
* The initial implementation of the collator side.
* Improve comments
* Multiple collation requests
* Add more tests and comments to validator side
* Add comments, remove dead code
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix build after suggested changes
* Also connect to the next validator group
* Remove a Future impl and move TimeoutExt to util
* Minor nits
* Fix build
* Change FetchCollations back to FetchCollation
* Try this
* Final fixes
* Fix build
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* handle multiple in-flight connection requests
* handle cancelled requests
* Update node/core/runtime-api/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* redo it again
* more stuff
* redo it again
* update comments
* workaround Future is not Send
* fix trailing spaces
* clarify comments
* bridge: fix compilation in tests
* update more comments
* small fixes
* port collator protocol to new validator discovery api
* collator tests compile
* collator tests pass
* do not revoke a request when the stream receiver is closed
* make revoking opt-in
* fix is_fulfilled
* handle request revokation in collator
* tests
* wait for validator connections asyncronously
* fix compilation
* relabel my todos
* apply Fedor's patch
* resolve reconnection TODO
* resolve revoking TODO
* resolve channel capacity TODO
* resolve peer cloning TODO
* resolve peer disconnected TODO
* resolve PeerSet TODO
* wip tests
* more tests
* resolve Arc TODO
* rename pending to non_revoked
* one more test
* extract utility function into util crate
* fix compilation in tests
* Apply suggestions from code review
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
* revert pin_project removal
* fix while let loop
* Revert "revert pin_project removal"
This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.
* fix compilation
* Update node/subsystem/src/messages.rs
* docs on pub items
* guide updates
* remove a TODO
* small guide update
* fix a typo
* link to the issue
* validator discovery: on_request docs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* update networking types
* port over overseer-protocol message types
* Add the collation protocol to network bridge
* message sending
* stub for ConnectToValidators
* add some helper traits and methods to protocol types
* add collator protocol message
* leaves-updating
* peer connection and disconnection
* add utilities for dispatching multiple events
* implement message handling
* add an observedrole enum with equality and no sentry nodes
* derive partial-eq on network bridge event
* add PartialEq impls for network message types
* add Into implementation for observedrole
* port over existing network bridge tests
* add some more tests
* port bitfield distribution
* port over bitfield distribution tests
* add codec indices
* port PoV distribution
* port over PoV distribution tests
* port over statement distribution
* port over statement distribution tests
* update overseer and service-new
* address review comments
* port availability distribution
* port over availability distribution tests
* add ActiveLeavesUpdate, remove StartWork, StopWork
* replace StartWork, StopWork in subsystem crate tests
* mechanically update OverseerSignal in other modules
* convert overseer to take advantage of new multi-hash update abilities
Note: this does not yet convert the tests; some of the tests now freeze:
test tests::overseer_start_stop_works ... test tests::overseer_start_stop_works has been running for over 60 seconds
test tests::overseer_finalize_works ... test tests::overseer_finalize_works has been running for over 60 seconds
* fix broken overseer tests
* manually impl PartialEq for ActiveLeavesUpdate, rm trait Equivalent
This cleans up the code a bit and makes it easier in the future to
do the right thing when comparing ALUs.
* use target in all network bridge logging
* reduce spamming of and
* get conclude signal working properly; don't allocate a vector
* wip: add test suite / example / explanation for using utility subsystem
Unfortunately, the test fails right now for reasons which seem
very odd. Just have to keep poking at it.
* explicitly import everything
* fix subsystem-util test
The root problem here was two-fold:
- there was a circular dependency from subsystem -> test-helpers/subsystem ->
subsystem
- cfg(test) doesn't propagate between crates
The solution: move the subsystem test helpers into a sub-module
within subsystem. Publicly export them from the previous location
so no other code breaks.
Doing this has an additional benefit: it ensures that no production
code can ever accidentally use the subsystem helpers, as they are compile-
gated on cfg(test).
* fully commit to moving test helpers into a subsystem module
* add some more tests
* get rid of log tests in favor of real error forwarding
It's not obvious whether we'll ever really want to chase down
these errors outside a testing context, but having the capability
won't hurt.
* fix issue which caused test to hang on osx
* only require that job errors are PartialEq when testing
also fix polkadot-node-core-backing tests
* get rid of any notion of partialeq
* rethink testing
Combine tests of starting and stopping job: leaving a test executor
with a job running was pretty clearly the cause of the sometimes-hang.
Also, add a timeout so tests _can't_ hang anymore; they just fail
after a while.
* rename fwd_errors -> forward_errors
* warn on error propagation failure
* fix unused import leftover from merge
* derive eq for subsystemerror
* network bridge skeleton
* move some primitives around and add debug impls
* protocol registration glue & abstract network interface
* add send_msgs to subsystemctx
* select logic
* transform different events into actions and handle
* implement remaining network bridge state machine
* start test skeleton
* make network methods asynchronous
* extract subsystem out to subsystem crate
* port over overseer to subsystem context trait
* fix minimal example
* fix overseer doc test
* update network-bridge crate
* write a subsystem test-helpers crate
* write a network test helper for network-bridge
* set up (broken) view test
* Revamp network to be more async-friendly and not require Sync
* fix spacing
* fix test compilation
* insert side-channel for actions
* Add some more message types to AllMessages
* introduce a test harness
* add some tests
* ensure service compiles and passes tests
* fix typo
* fix service-new compilation
* Subsystem test helpers send messages synchronously
* remove smelly action inspector
* remove superfluous let binding
* fix warnings
* Update node/network/bridge/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* fix compilation
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
dependabot[bot]