Commit Graph

1637 Commits

Author SHA1 Message Date
Andronik Ordian 8961628e7c bitfield-dist: check signatures once (#2868) 2021-04-09 17:31:34 +00:00
Robert Habermeier bc7761a3da set groups correctly even if not validator (#2863)
shows what I get for being hasty last time. i added a test this time, which would have caught this issue last time.
2021-04-09 12:21:39 +02:00
Robert Habermeier 234d0df515 no backing job early exit if not validator (#2860)
with the MaxValidators parameter set, this meant that only validators participating in parachain consensus would be able to author relay-chain blocks containing backed candidates
2021-04-08 22:35:23 +00:00
Andronik Ordian 78a46da384 silence some parachain warnings (#2859)
* silence some parachain warnings

* that was actually a bug that Rob is fixing
2021-04-09 00:25:46 +02:00
Sergei Shulepov 59b4d6511f New PVF validation host (#2710)
* Implement PVF validation host

* WIP: Diener

* Increase the alloted compilation time

* Add more comments

* Minor clean up

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix pruning artifact removal

* Fix formatting and newlines

* Fix the thread pool

* Update node/core/pvf/src/executor_intf.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove redundant test declaration

* Don't convert the path into an intermediate string

* Try to workaround the test failure

* Use the puppet_worker trick again

* Fix a blip

* Move `ensure_wasmtime_version` under the tests mod

* Add a macro for puppet_workers

* fix build for not real-overseer

* Rename the puppet worker for adder collator

* play it safe with the name of adder puppet worker

* Typo: triggered

* Add more comments

* Do not kill exec worker on every error

* Plumb Duration for timeouts

* typo: critical

* Add proofs

* Clean unused imports

* Revert "WIP: Diener"

This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.

* Sync version of wasmtime

* Update cargo.lock

* Update Substrate

* Merge fixes still

* Update wasmtime version in test

* bastifmt

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Squash spaces

* Trailing new line for testing.rs

* Remove controversial code

* comment about biasing

* Fix suggestion

* Add comments

* make it more clear why unwrap_err

* tmpfile retry

* proper proofs for claim_idle

* Remove mutex from ValidationHost

* Add some more logging

* Extract exec timeout into a constant

* Add some clarifying logging

* Use blake2_256

* Clean up the merge

Specifically the leftovers after removing real-overseer

* Update parachain/test-parachains/adder/collator/Cargo.toml

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-09 00:09:56 +02:00
Robert Habermeier 896ec8dbc3 Code, PoV compression and remove CompressedPoV struct (#2852)
* use compressed blob in candidate-validation

* add some tests for compressed code blobs

* remove CompressedPoV and apply compression in collation-generation

* decompress BlockData before executing

* don't produce oversized collations

* add test for PoV decompression failure

* fix tests and clean up

* fix test

* address review and fix CI

* take this )
2021-04-08 22:09:36 +02:00
Robert Habermeier 57038b2e46 Remove real-overseer 🎉 (#2834)
* remove real-overseer

* overseer: only activate leaves which support parachains

* integrate HeadSupportsParachains into service

* remove unneeded line
2021-04-08 20:24:06 +02:00
e.g. magical unicorn 5b3e5ff8d1 add new chainspec for rococo (#2851) 2021-04-07 23:10:53 +00:00
Robert Habermeier 8eae0fa443 check for lost race before warning (#2833) 2021-04-07 22:04:15 +02:00
Robert Habermeier 5a9e8b9956 update rococo genesis (#2830) 2021-04-07 10:08:43 +02:00
Andronik Ordian dbb09ac9f4 parachain db versioning (#2829)
* move parachains_db.rs to a module

* parachain versioning

* fixes

* more fixes

* Update node/service/src/parachains_db/upgrade.rs

* fix web-wasm check

* suggested style fixes

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/service/src/parachains_db/upgrade.rs

* compilation fixes

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-04-07 00:23:32 +02:00
Andronik Ordian 936f0d7c59 statement-distribution: do not verify signatures for duplicate statements (#2823)
* stmnt-dist: do not verify signature on duplicates

* renames

* more renames
2021-04-06 17:32:57 +02:00
Andronik Ordian 1244378d50 parachain_db: fix incorrect config cache sizes (#2827) 2021-04-06 16:47:38 +02:00
Pierre Krieger fa0142ac8f Properly remove peers from sets and merge the two Network traits (#2821)
* Properly remove peers from sets

* Actually rename all, I guess

* Merge the two Network traits

* Rename function

* Update node/network/bridge/src/network.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Fix erroneous change

* Update node/network/bridge/src/network.rs

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-05 21:46:39 +02:00
Andronik Ordian 2ff5c9b995 tests: use future::join instead of future::select (#2813)
* tests/av-store: use future::join instead of future::select

* tests/backing: use future::join instead of future::select

* tests/provisioner: use future::join instead of future::select

* tests/av-dist: use future::join instead of future::select

* tests/av-recovery: use future::join instead of future::select

* tests/bridge: use future::join instead of future::select

* tests/collator-protocol: use future::join instead of future::select

* tests/stmt-dist: use future::join instead of future::select

* fix tests
2021-04-05 18:30:27 +02:00
Robert Habermeier 5c94a5b9c7 finally get unbacked candidates to descend from root span correctly (#2819) 2021-04-05 16:37:35 +02:00
Robert Habermeier ec5ad35e14 Network bridge metrics (#2818)
* add metrics (unused) to network bridge

* fix test compilation

* trigger metrics messages

* add some more metrics

* track sent and received notifications

* restore metrics import

* integrate into service

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-05 01:07:05 +02:00
Andronik Ordian 4df29e71ab bitfield-dist: fix state update on gossip (#2817)
* bitfield-dist: fix state update on gossip

* fixes

* doc fixes

* oops

* 2 lines of code change
2021-04-04 22:25:40 +00:00
Robert Habermeier bfc8f4fcf3 Collators: Declare to all peers (#2816)
* fix tests

* add test for rejecting declares on collators

* fix bad test
2021-04-04 16:59:00 +00:00
Robert Habermeier 11b8e4c821 Collation protocol: stricter validators (#2810)
* guide: declare one para as a collator

* add ParaId to Declare messages and clean up

* fix build

* fix the testerinos

* begin adding keystore to collator-protocol

* remove request_x_ctx

* add core_for_group

* add bump_rotation

* add some more helpers to subsystem-util

* change signing_key API to take ref

* determine current and next para assignments

* disconnect collators who are not on current or next para

* add collator peer count metric

* notes for later

* some fixes

* add data & keystore to test state

* add a test utility for answering runtime API requests

* fix existing collator tests

* add new tests

* remove sc_keystore

* update cargo lock

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-03 21:48:58 +02:00
Andronik Ordian 94b0ccc8f1 approval-distribution: split peer knowledge into sent and received (#2809)
* approval-distribution: split peer knowledge into sent and received

* guide updates

* fixes

* revert doc changes
2021-04-03 04:29:15 +02:00
Robert Habermeier 12000de79a change some configuration for rococo (#2802) 2021-04-03 00:19:35 +00:00
Andronik Ordian cd9449cbef companion: update wasmtime to 0.24.0 (#2625)
* update using diener

* fix zstd upgrade

* Revert "update using diener"

This reverts commit 701fdfe3822d6beac4bfda591a1dbcb99bcadb1a.

* update zstd to 0.6

* update using diener

* Revert "update using diener"

This reverts commit fc81df6355375b1328bb6126e3227f193583a61b.

* update Substrate

Co-authored-by: parity-processbot <>
2021-04-02 20:23:29 +00:00
Robert Habermeier c54f8848d1 add disputes members to HostConfiguration (#2806)
* add disputes members to `HostConfiguration`

* revert bad rename
2021-04-02 19:59:21 +02:00
Guillaume Thiolliere beca01f118 Ease parachain candidate code fetching (#2593)
* code stored in para + modify CandidateDescriptor.

* WIP: digest + some more impl

* validation_code_hash in payload + check in inclusion

* check in client + refator

* tests

* fix encoding indices

* remove old todos

* fix test

* fix test

* add test

* fetch validation code inside collation-generation from the relay-chain

* HashMismatch -> PoVHashMismatch + miscompilation

* refactor, store hash when needed

* storage rename: more specific but slightly too verbose

* do not hash on candidate validation, fetch hash instead

* better test

* fix test

* guide updates

* don't panic in runtime

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
2021-04-01 23:55:39 +02:00
Andronik Ordian 98082c5326 gossip: move authorities request to runtime api subsystem (#2798) 2021-04-01 23:51:01 +02:00
Robert Habermeier 05a5047e40 remove training wheels (#2800) 2021-04-01 23:45:00 +02:00
Robert Habermeier 57b56770e0 Approval Voting improvements (#2781)
* extract database from av-store itself

* generalize approval-voting over database type

* modes (without handling) and pruning old wakeups

* rework approval importing

* add our_approval_sig to ApprovalEntry

* import assignment

* guide updates for check-full-approval changes

* some aux functions

* send messages when becoming active.

* guide: network bridge sends view updates only when done syncing

* network bridge: send view updates only when done syncing

* tests for new network-bridge behavior

* add a test for updating approval entry with sig

* fix some warnings

* test load-all-blocks

* instantiate new parachains DB

* fix network-bridge empty view updates

* tweak

* fix wasm build, i think

* Update node/core/approval-voting/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* add some versioning to parachains_db

* warnings

* fix merge changes

* remove versioning again

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-01 17:33:52 +00:00
Pierre Krieger 01badafba6 Companion PR for substrate#8510 (#2795)
* Companion PR for substrate#8510

* update Substrate

Co-authored-by: parity-processbot <>
2021-04-01 19:33:43 +02:00
Robert Habermeier 0794f69306 Add dispute types and change InclusionInherent to ParasInherent (#2791)
* dispute types

* add Debug to dispute primitives in std and InherentData

* use ParachainsInherentData on node-side

* change inclusion_inherent to paras_inherent

* RuntimeDebug

* add type parameter to PersistedValidationData users

* fix test client

* spaces

* fix collation-generation test

* fix provisioner tests

* remove references to inclusion inherent
2021-04-01 18:23:27 +02:00
Andronik Ordian 7a2e1ef6c1 gossip: do not try to connect if we are not validators (#2786)
* gossip: do not issue a connection request if we are not a validator

* guide updates

* use all relevant authorities when issuing a request

* use AuthorityDiscoveryApi instead

* update comments to the status quo
2021-04-01 18:11:43 +02:00
Robert Habermeier 5da762e728 Avoid querying the local validator in availability recovery (#2792)
* guide: don't request availability data from ourselves

* add QueryAllChunks message

* implement QueryAllChunks

* remove unused relay_parent from StoreChunk

* test QueryAllChunks

* fast paths make short roads

* test early exit behavior
2021-04-01 15:57:41 +02:00
Tomasz Drwięga 5b77a89874 Merkle Mountain Range & BEEFY integration (#2101)
* Switch branch.

* Implement basic MMR leaf.

* Revert "Switch branch."

This reverts commit 7f4d41c67f27ca560c53fc63fd3bd06ac182403c.

* Bump substrate.

* Integrate BEEFY.

Bump all.

Fix missing imports.

* Use beefy pallet to get authorities.

* Bump BEEFY repo.

* Use next authority set instead of the current one.

* Start BEEFY service.

* Fix BEEFY start up.

* Cache BEEFY authority set.

* Add BEEFY ValidatorSetId to MMR

* Fix code.

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Review grumbles.

* Update beefy repo.

* Work-around missing protocol.

* Revert "Work-around missing protocol."

This reverts commit 0a6257a8bccc1c67e966898cdedc408c6469ffd6.

* Add beefy peers set config.

* Expose storage of BEEFY.

* Uncompress BEEFY keys for merkle tree.

* Update ordering.

* Switch to branch.

* Bump deps.

* Switch to custom beefy.

* Add MMR RuntimeApi and custom rpc.

* Add set length details.

* Fix compilation.

* Expose MmrLeaf storage.

* Expose MmrLeaf storage.

* Don't use session handler, and rather compute & cache beefy details on call.

* Don't use session handler, and rather compute & cache beefy details on call.

* Fixes.

* Update Cargo.lock.

* Switch back to master.

* Update lockfile.

* Fix xcm print issue.

* Cargo.lock.

* Use master branch.

* Remove extra dep.

* Fix tests.

* Update Cargo.lock

* Add BEEFY & MMR to westend.

* Implement session keys migration.

* Update testnet script.

* start BEEFY for all node types

* Update Cargo.lock

* fix Cargo.toml

* resolve another merge conflict

* add Westend BEEFY keys

* Apply suggestions from code review

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Update BEEFY.

* Add Rococo BEEFY keys

* resolve merge issue

* fix pallet indices

* fix Westend OldSessionKey

* remove unused imports in Westend runtime

* Fix compilation for Westend.

* address review

* start BEEFY gadget conditionally

* address review again

* fix typo

* remove duplicate

* remove another duplicate

* well

* add missing stuff

* cleanup Cargo.toml files

- revert unnecessary changes
- add missing /std dependencies
- remove unused dependencies

* runtime: remove unused structs from rococo runtime

* node: cleanup service

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-04-01 12:50:04 +02:00
Andronik Ordian caebf642dd statement-distribution: do not use OurViewChange (#2790)
* quickfix for statement-distribution

* some logs
2021-03-31 23:28:17 +02:00
Robert Klotzner eb6786ad05 Better timeout values now that we are going to be connected to all nodes. (#2778)
* Better timeout values.

* Fix typo.

* Fix validator bandwidth.

* Fix compilation.
2021-03-31 22:34:12 +02:00
Kian Paimani c56f9e0514 proper executor/block type for benchmarks and try-runtime (#2771)
* proper executor/block type for benchmarks and try-runtime

* Fix

* update benches as well to latest work

* Remvoe dbg
2021-03-31 20:40:15 +02:00
Robert Habermeier e65cad69ec Fix future-polling loop in availability and add a better early-exit (#2779)
* onto the front

* fix early exit for waiting for requests

* add logging back
2021-03-31 17:35:17 +02:00
Andronik Ordian 9ac35d9f2b gossip: choose a random subset on send instead of limiting connections (#2776)
* gossip: choose random subset on send

* naming bikeshed
2021-03-30 20:59:53 +02:00
Andronik Ordian a3115401c3 network-bridge: elevate log level for connections (#2772) 2021-03-30 20:01:57 +02:00
Robert Habermeier 08d5b268a0 Retry availability until the receiver of the request is dropped (#2763)
* guide updates

* keep interactions alive until receivers drop

* retry indefinitely

* cancel approval tasks on finality

* use swap_remove instead of remove
2021-03-30 17:33:38 +02:00
Robert Klotzner 6514e00144 Add tags to pov-fetcher. (#2768)
* Add tags to pov-fetcher.

* Add stage as well.

* Get rid of redundant tags.
2021-03-30 15:07:07 +02:00
Andronik Ordian bdee5a3923 approval-distribution: add an assertion (#2761) 2021-03-30 14:18:34 +02:00
Bastian Köcher c047a2bde2 Companion for Substrate#8472 (#2759)
* Companion for Substrate#8472

https://github.com/paritytech/substrate/pull/8472

* update Substrate

Co-authored-by: parity-processbot <>
2021-03-30 12:15:50 +02:00
Robert Klotzner 0bc42785b4 availability-distribution: Retry failed fetches on next block. (#2762)
* availability-distribution: Retry on fail on next block.

Retry failed fetches on next block when still pending availability.

* Update node/network/availability-distribution/src/requester/fetch_task/mod.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Fix existing tests.

* Add test for trying all validators.

* Add test for testing retries.

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-03-30 00:28:43 +02:00
Robert Habermeier e906598e94 tracing for pending_known map (#2755)
* tracing for pending_known map

* fix condition in retain

* add block hash to pending tracing
2021-03-29 21:38:03 +02:00
Robert Habermeier 235c2e26b1 more logs in issue-approval (#2758) 2021-03-29 21:06:44 +02:00
Robert Habermeier 54074d2d76 send assignments even when we have an approval (#2757) 2021-03-29 16:34:14 +00:00
Robert Habermeier 7c21dbbdf4 use unbounded when notifying distribution of new blocks (#2752) 2021-03-29 14:30:20 +00:00
Robert Habermeier fc154d2ada Reduce signal channel sizes and more logging on approval-voting (#2751)
* reduce signal channel capacity

* more tracing for approval-voting
2021-03-29 15:46:12 +02:00
Robert Klotzner 0a9fe852df Move non runtime related stuff into node/primitives (#2743)
* Remove stuff out of the runtime that does not belong there.

There might be more, but it is a start.

* White space fixes.

* Fix tests.

* Leave whitespace in ui tests alone.

* Add back zstd for no reason.

* Fix browser wasm (hopefully)
2021-03-29 02:15:44 +02:00