* Remove signature verification in backing.
`SignedFullStatement` now signals that the signature has already been
checked.
* Remove unused check_payload function.
* Introduced unchecked signed variants.
* Fix inclusion to use unchecked variant.
* More unchecked variants.
* Use unchecked variants in protocols.
* Start fixing statement-distribution.
* Fixup statement distribution.
* Fix inclusion.
* Fix warning.
* Fix backing properly.
* Fix bitfield distribution.
* Make crypto store optional for `RuntimeInfo`.
* Factor out utility functions.
* get_group_rotation_info
* WIP: Collator cleanup + check signatures.
* Convenience signature checking functions.
* Check signature on collator-side.
* Fix warnings.
* Fix collator side tests.
* Get rid of warnings.
* Better Signed/UncheckedSigned implementation.
Also get rid of Encode/Decode for Signed! *party*
* Get rid of dead code.
* Move Signed in its own module.
* into_checked -> try_into_checked
* Fix merge.
* Simplify some Option / Result / ? operator patterns
When they identically match a combinator on those types.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
* adjust review comments
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Factor out runtime module into utils.
* Add maybe_authority information to `PeerConnected` event.
We already gather this information in authority discovery, so we might
as well share it with others.
This opens up an easy path to trigger validators differently from normal
nodes, e.g. for prioritization. This change has become more important
now, that we just connect to all validators and therefore just have a
long peer list without any information about those nodes.
* Test fix.
* Wip
* Increase proposer timeout.
* WIP.
* Better timeout values now that we are going to be connected to all nodes. (#2778)
* Better timeout values.
* Fix typo.
* Fix validator bandwidth.
* Fix compilation.
* Better and more consistent sizes.
Most importantly code size is now 5 Meg, which is the limit we currently
want to support in statement distribution.
* Introduce statement fetching request.
* WIP
* Statement cache retrieval logic.
* Review remarks by @rphmeier
* Fixes.
* Better requester logic.
* WIP: Handle requester messages.
* Missing dep.
* Fix request launching logic.
* Finish fetching logic.
* Sending logic.
* Redo code size calculations.
Now that max code size is compressed size.
* Update Cargo.lock (new dep)
* Get request receiver to statement distribution.
* Expose new functionality for responding to requests.
* Cleanup.
* Responder logic.
* Fixes + Cleanup.
* Cargo.lock
* Whitespace.
* Add lost copyright.
* Launch responder task.
* Typo.
* info -> warn
* Typo.
* Fix.
* Fix.
* Update comment.
* Doc fix.
* Better large statement heuristics.
* Fix tests.
* Fix network bridge tests.
* Add test for size estimate.
* Very simple tests that checks we get LargeStatement.
* Basic check, that fetching of large candidates is performed.
* More tests.
* Basic metrics for responder.
* More metrics.
* Use Encode::encoded_size().
* Some useful spans.
* Get rid of redundant metrics.
* Don't add peer on duplicate.
* Properly check hash
instead of relying on signatures alone.
* Preserve ordering + better flood protection.
* Get rid of redundant clone.
* Don't shutdown responder on failed query.
And add test for this.
* Smaller fixes.
* Quotes.
* Better queue size calculation.
* A bit saner response sizes.
* Fixes.
with the MaxValidators parameter set, this meant that only validators participating in parachain consensus would be able to author relay-chain blocks containing backed candidates
* Implement PVF validation host
* WIP: Diener
* Increase the alloted compilation time
* Add more comments
* Minor clean up
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix pruning artifact removal
* Fix formatting and newlines
* Fix the thread pool
* Update node/core/pvf/src/executor_intf.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove redundant test declaration
* Don't convert the path into an intermediate string
* Try to workaround the test failure
* Use the puppet_worker trick again
* Fix a blip
* Move `ensure_wasmtime_version` under the tests mod
* Add a macro for puppet_workers
* fix build for not real-overseer
* Rename the puppet worker for adder collator
* play it safe with the name of adder puppet worker
* Typo: triggered
* Add more comments
* Do not kill exec worker on every error
* Plumb Duration for timeouts
* typo: critical
* Add proofs
* Clean unused imports
* Revert "WIP: Diener"
This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.
* Sync version of wasmtime
* Update cargo.lock
* Update Substrate
* Merge fixes still
* Update wasmtime version in test
* bastifmt
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Squash spaces
* Trailing new line for testing.rs
* Remove controversial code
* comment about biasing
* Fix suggestion
* Add comments
* make it more clear why unwrap_err
* tmpfile retry
* proper proofs for claim_idle
* Remove mutex from ValidationHost
* Add some more logging
* Extract exec timeout into a constant
* Add some clarifying logging
* Use blake2_256
* Clean up the merge
Specifically the leftovers after removing real-overseer
* Update parachain/test-parachains/adder/collator/Cargo.toml
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* use compressed blob in candidate-validation
* add some tests for compressed code blobs
* remove CompressedPoV and apply compression in collation-generation
* decompress BlockData before executing
* don't produce oversized collations
* add test for PoV decompression failure
* fix tests and clean up
* fix test
* address review and fix CI
* take this )
* tests/av-store: use future::join instead of future::select
* tests/backing: use future::join instead of future::select
* tests/provisioner: use future::join instead of future::select
* tests/av-dist: use future::join instead of future::select
* tests/av-recovery: use future::join instead of future::select
* tests/bridge: use future::join instead of future::select
* tests/collator-protocol: use future::join instead of future::select
* tests/stmt-dist: use future::join instead of future::select
* fix tests
* guide: declare one para as a collator
* add ParaId to Declare messages and clean up
* fix build
* fix the testerinos
* begin adding keystore to collator-protocol
* remove request_x_ctx
* add core_for_group
* add bump_rotation
* add some more helpers to subsystem-util
* change signing_key API to take ref
* determine current and next para assignments
* disconnect collators who are not on current or next para
* add collator peer count metric
* notes for later
* some fixes
* add data & keystore to test state
* add a test utility for answering runtime API requests
* fix existing collator tests
* add new tests
* remove sc_keystore
* update cargo lock
Co-authored-by: Andronik Ordian <write@reusable.software>
* code stored in para + modify CandidateDescriptor.
* WIP: digest + some more impl
* validation_code_hash in payload + check in inclusion
* check in client + refator
* tests
* fix encoding indices
* remove old todos
* fix test
* fix test
* add test
* fetch validation code inside collation-generation from the relay-chain
* HashMismatch -> PoVHashMismatch + miscompilation
* refactor, store hash when needed
* storage rename: more specific but slightly too verbose
* do not hash on candidate validation, fetch hash instead
* better test
* fix test
* guide updates
* don't panic in runtime
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* extract database from av-store itself
* generalize approval-voting over database type
* modes (without handling) and pruning old wakeups
* rework approval importing
* add our_approval_sig to ApprovalEntry
* import assignment
* guide updates for check-full-approval changes
* some aux functions
* send messages when becoming active.
* guide: network bridge sends view updates only when done syncing
* network bridge: send view updates only when done syncing
* tests for new network-bridge behavior
* add a test for updating approval entry with sig
* fix some warnings
* test load-all-blocks
* instantiate new parachains DB
* fix network-bridge empty view updates
* tweak
* fix wasm build, i think
* Update node/core/approval-voting/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some versioning to parachains_db
* warnings
* fix merge changes
* remove versioning again
Co-authored-by: Andronik Ordian <write@reusable.software>
* dispute types
* add Debug to dispute primitives in std and InherentData
* use ParachainsInherentData on node-side
* change inclusion_inherent to paras_inherent
* RuntimeDebug
* add type parameter to PersistedValidationData users
* fix test client
* spaces
* fix collation-generation test
* fix provisioner tests
* remove references to inclusion inherent
* guide: don't request availability data from ourselves
* add QueryAllChunks message
* implement QueryAllChunks
* remove unused relay_parent from StoreChunk
* test QueryAllChunks
* fast paths make short roads
* test early exit behavior
* guide updates
* keep interactions alive until receivers drop
* retry indefinitely
* cancel approval tasks on finality
* use swap_remove instead of remove
* Remove stuff out of the runtime that does not belong there.
There might be more, but it is a start.
* White space fixes.
* Fix tests.
* Leave whitespace in ui tests alone.
* Add back zstd for no reason.
* Fix browser wasm (hopefully)
* overseer: pass messages directly between subsystems
* test that message is held on to
* Update node/overseer/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* give every subsystem an unbounded sender too
* remove metered_channel::name
1. we don't provide good names
2. these names are never used anywhere
* unused mut
* remove unnecessary &mut
* subsystem unbounded_send
* remove unused MaybeTimer
We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.
* remove comment
* split up senders and receivers
* update metrics
* fix tests
* fix test subsystem context
* use SubsystemSender in jobs system now
* refactor of awful jobs code
* expose public `run` on JobSubsystem
* update candidate backing to new jobs & use unbounded
* bitfield signing
* candidate-selection
* provisioner
* approval voting: send unbounded for assignment/approvals
* async not needed
* begin bridge split
* split up network tasks into background worker
* port over network bridge
* Update node/network/bridge/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* rename ValidationWorkerNotifications
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* Indentation fix.
* Prepare request-response for PoV fetching.
* Drop old PoV distribution.
* WIP: Fetch PoV directly from backing.
* Backing compiles.
* Runtime access and connection management for PoV distribution.
* Get rid of seemingly dead code.
* Implement PoV fetching.
Backing does not yet use it.
* Don't send `ConnectToValidators` for empty list.
* Even better - no need to check over and over again.
* PoV fetching implemented.
+ Typechecks
+ Should work
Missing:
- Guide
- Tests
- Do fallback fetching in case fetching from seconding validator fails.
* Check PoV hash upon reception.
* Implement retry of PoV fetching in backing.
* Avoid pointless validation spawning.
* Add jaeger span to pov requesting.
* Add back tracing.
* Review remarks.
* Whitespace.
* Whitespace again.
* Cleanup + fix tests.
* Log to log target in overseer.
* Fix more tests.
* Don't fail if group cannot be found.
* Simple test for PoV fetcher.
* Handle missing group membership better.
* Add test for retry functionality.
* Fix flaky test.
* Spaces again.
* Guide updates.
* Spaces.
* add number to `ActivatedLeavesUpdate`
* update subsystem util and overseer
* use new ActivatedLeaf everywhere
* sort view
* sorted and limited view in network bridge
* use live block hash only if it's newer
* grumples
Robert Habermeier