Commit Graph

1647 Commits

Author SHA1 Message Date
Robert Klotzner 48409e5548 Request based availability distribution (#2423)
* WIP

* availability distribution, still very wip.

Work on the requesting side of things.

* Some docs on what I intend to do.

* Checkpoint of session cache implementation

as I will likely replace it with something smarter.

* More work, mostly on cache

and getting things to type check.

* Only derive MallocSizeOf and Debug for std.

* availability-distribution: Cache feature complete.

* Sketch out logic in `FetchTask` for actual fetching.

- Compile fixes.
- Cleanup.

* Format cleanup.

* More format fixes.

* Almost feature complete `fetch_task`.

Missing:

- Check for cancel
- Actual querying of peer ids.

* Finish FetchTask so far.

* Directly use AuthorityDiscoveryId in protocol and cache.

* Resolve `AuthorityDiscoveryId` on sending requests.

* Rework fetch_task

- also make it impossible to check the wrong chunk index.
- Export needed function in validator_discovery.

* From<u32> implementation for `ValidatorIndex`.

* Fixes and more integration work.

* Make session cache proper lru cache.

* Use proper lru cache.

* Requester finished.

* ProtocolState -> Requester

Also make sure to not fetch our own chunk.

* Cleanup + fixes.

* Remove unused functions

- FetchTask::is_finished
- SessionCache::fetch_session_info

* availability-distribution responding side.

* Cleanup + Fixes.

* More fixes.

* More fixes.

adder-collator is running!

* Some docs.

* Docs.

* Fix reporting of bad guys.

* Fix tests

* Make all tests compile.

* Fix test.

* Cleanup + get rid of some warnings.

* state -> requester

* Mostly doc fixes.

* Fix test suite.

* Get rid of now redundant message types.

* WIP

* Rob's review remarks.

* Fix test suite.

* core.relay_parent -> leaf for session request.

* Style fix.

* Decrease request timeout.

* Cleanup obsolete errors.

* Metrics + don't fail on non fatal errors.

* requester.rs -> requester/mod.rs

* Panic on invalid BadValidator report.

* Fix indentation.

* Use typed default timeout constant.

* Make channel size 0, as each sender gets one slot anyways.

* Fix incorrect metrics initialization.

* Fix build after merge.

* More fixes.

* Hopefully valid metrics names.

* Better metrics names.

* Some tests that already work.

* Slightly better docs.

* Some more tests.

* Fix network bridge test.
2021-02-26 11:58:07 -06:00
Andronik Ordian 241b1f12a7 make runtime_api non blocking task again (#2531) 2021-02-26 12:04:58 +01:00
Bernhard Schuster 05f74c2171 do not expect on unbounded send (#2530) 2021-02-26 10:46:16 +01:00
Bernhard Schuster 31327eb0c7 test: add unit test to catch missing distribution to subsystems faster (#2495)
* test: add unit test to catch missing distribution to subsystems faster

* add a simple count

* introduce proc macro to generate dispatch type

* refactor

* refactor

* chore: add license

* fixup unit test

* fixup merge

* better errors

* better fmt

* fix error spans

* better docs

* better error messages

* ui test foo

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/subsystem/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* fix compilation

* use find_map

* drop the silly 2, use _inner instead

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* nail deps down

* more into()

* flatten

* missing use statement

* fix messages order

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
2021-02-26 08:10:41 +00:00
Xiliang Chen 69734bb8ed Add an optional verifier to crowdloan (#2248)
* Add an optional verifier to crowdloan

* add tests

* verify signatures

* benchmark with signatures

* update crowdloan benchmark

* try to get keystore working in test

* rewrite to avoid traits

* Use MultiSignature and MultiSigner

* refactor and update benchmarks

* optimize check order

* fix no_std build

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-02-26 08:26:54 +01:00
Robert Habermeier 49705026e0 some initial spans for approval voting (#2525)
* some initial spans for approval voting

* add stage earlier
2021-02-25 17:56:50 +00:00
Andronik Ordian 4e3dba91be small cleanup after #2518 (#2527) 2021-02-25 14:00:20 +01:00
André Silva 2f6cadf359 Companion for Substrate #8148 (#2523)
* Pass shared_authority_set to request_response_config_for_chain

* "Update Substrate"

Co-authored-by: Ashley Ruglys <ashley.ruglys@gmail.com>
Co-authored-by: parity-processbot <>
2021-02-25 09:16:59 +00:00
Robert Habermeier ae218bb608 make runtime API and chain API subsystems blocking too (#2526) 2021-02-25 07:26:49 +00:00
Bastian Köcher 327a203dc7 Companion for Substrate #8185 (#2507)
* Companion for Substrate #8185

https://github.com/paritytech/substrate/pull/8185

* "Update Substrate"

Co-authored-by: parity-processbot <>
2021-02-24 22:31:54 +01:00
Robert Habermeier 3cdaa88509 spawn availability store and approval voting subsystems as blocking tasks (#2521)
* spawn availability store and approval voting subsystems as blocking tasks

* refactor
2021-02-24 14:22:48 -06:00
Bastian Köcher c3ef81ab30 Make sure we generate a valid slot in tests (#2520)
Currently the first last timestamp is `0` and that leads to the first
slot being `0` which is reserved for genesis. There is some debug assert
in BABE that complains about this in Cumulus :D
2021-02-24 20:07:03 +00:00
Martin Pugh 5ffb8cb463 [release] Prep for v0.8.29 (#2497)
* bump version and substrate

* update weights

* bump substrate

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
2021-02-24 20:32:34 +01:00
Robert Habermeier 4675a784a8 Revert "bump Substrate"
This reverts commit 3a13ef6ca8.
2021-02-24 12:03:55 -06:00
Robert Habermeier 3a13ef6ca8 bump Substrate 2021-02-24 11:52:27 -06:00
Robert Habermeier 0fac609a61 merge approval-checking feature back into real-overseer (#2518) 2021-02-24 11:28:32 -06:00
dependabot[bot] edbbe27b28 Bump wasm-bindgen from 0.2.69 to 0.2.70 (#2324)
Bumps [wasm-bindgen](https://github.com/rustwasm/wasm-bindgen) from 0.2.69 to 0.2.70.
- [Release notes](https://github.com/rustwasm/wasm-bindgen/releases)
- [Changelog](https://github.com/rustwasm/wasm-bindgen/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rustwasm/wasm-bindgen/compare/0.2.69...0.2.70)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-02-24 16:17:44 +01:00
Kian Paimani 4f83b3e247 Properly use weights for the new election pallet (#2508)
* Revert some weights

* Add features

* Update lock file

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* Use read weights

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* export stuff

* Update substrate

Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
2021-02-24 14:38:22 +00:00
Kian Paimani 2398bbfaef Migration testing CLI (#2447)
* Template

* Midway..

* Make all 3 runtimes work

* Update node/service/Cargo.toml

* undo brach updates

* fix with latest version

* Update runtime/kusama/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update runtime/rococo/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Final nits

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2021-02-24 14:38:05 +00:00
Martin Pugh 6d46fc1d85 Companion for #8184 (#2506)
* remove weights for suicide

* "Update Substrate"

* Update Cargo.lock

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
2021-02-23 22:07:45 +00:00
Robert Habermeier 5683f11628 fix master: use view.iter() instead of view.heads (#2510) 2021-02-23 21:02:16 +00:00
Robert Habermeier 3300b53306 Approval Checking Improvements Omnibus (#2480)
* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound
2021-02-23 14:12:28 -06:00
Parth 3c4ed7b234 Override RuntimeApi::authorities() to include past sessions (#2494)
* override authorities runtime call

* addressing feedback

* addressing feedback and restoring semantics

* updated helper function

* naming convention + comment

Co-authored-by: Parity <parity2305@gmail.com>
2021-02-23 13:49:46 -06:00
Kian Paimani eb09db2aef Enable two phase election provider in runtimes in 'DryRun' mode (#2476)
* Apply some changes

* Make the runtimes build

* Master.into()

* Undo branch updates.

* Undo unwatned changes

* Fix deps one last time

* Fix election lookahead

* Revert deps

* Fix :P

Co-authored-by: Bastian Köcher <info@kchr.de>
2021-02-23 16:58:00 +01:00
Bernhard Schuster e3f776abed feat/view: assure heads in a view are sorted (#2493)
* feat/view: assure heads in a view are sorted

Allows O(n) comparisons, adds an alternate equiv relation
which takes O(n^2) for integrity verification.

Ref #2133

* revert: remove custom PartialEq impl, there are no duplicates

* fix: do not sort the live_heads, that alters the local view

* refactor/view: heads should not be public

* chore/spellcheck: add unfinalized

* fix/view: add missing len() and is_empty() fns

* quirk

* vec is not view

* Update node/network/approval-distribution/src/tests.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/network/protocol/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* fixup comment

* fix botched test

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-02-23 15:39:57 +00:00
Kian Paimani 571651e326 Change a few fee/weight parameters (#2481)
* Change a few fee parameters

* Update runtime/common/src/lib.rs
2021-02-23 13:34:29 +00:00
Martin Pugh ece23588d2 Remove v0.8.28 runtime migrations (#2496)
* remove v0.8.28 runtime migrations

* Update runtime/westend/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* commit suggestions

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-02-23 12:28:41 +00:00
Shaun Wang 893c9a60ed Fix XCM encoding in HRMP pallet. (#2491) 2021-02-22 23:21:17 +01:00
Shawn Tabrizi 5b57f02e80 Companion for 8166 (Update storage_kill return value) (#2488)
* New return value

* update kill outcome

* Update name

* "Update Substrate"

Co-authored-by: parity-processbot <>
2021-02-22 15:53:20 -04:00
Pierre Krieger c120e7d83a Update thread_local past the memory leak (#2498) 2021-02-22 17:59:47 +00:00
Bastian Köcher 2584c121fb Substrate companion for #8163 (#2492)
* Substrate companion for #8163

https://github.com/paritytech/substrate/pull/8163

* "Update Substrate"

Co-authored-by: parity-processbot <>
2021-02-22 14:52:43 +00:00
Bastian Köcher 44519d1d2c Companion for Substrate#8155 (#2478)
* Companion for Substrate#8155

https://github.com/paritytech/substrate/pull/8155

* :D

* Update the Substrate reference
2021-02-19 20:59:13 +01:00
Bernhard Schuster 49c6aa9a76 feat/jaeger: more spans, more stages (#2477)
* feat/jaeger: more spans, more stages

Stage numbers are still arbitrarily picked.

* feat/jaeger: additional spans

* chore/spellcheck: improve the dictionary

* fix/jaeger JaegerSpan -> jaeger::Span
2021-02-19 14:19:43 +00:00
Shawn Tabrizi a5defa7c7f Session Delayed Para Changes / Actions Queue (#2406)
* initial implementation of lifecycles and upgrades

* clean up a bit

* fix doc comment

* more rigid lifecycle checks

* include paras which are transitioning, and lifecycle query

* format guide

* update api

* update guide

* explicit outgoing state, fix genesis

* handle outgoing with transitioning paras

* do not include transitioning paras in identifier

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Apply suggestions from code review

* Use matches macro

* Correct terms

* Apply suggestions from code review

* actions queue

* Revert "actions queue"

This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.

* collapse onboarding state

* starting actions queue

* consolidate actions queue

* schedule para initialize result

* more actions queue for upgrade/downgrade

* clean up with fully implemented actions queue

* fix tests

* fix scheduler tests

* fix hrmp tests

* fix test

* doc fixes

* fix hrmp test w/ valid para

* Update paras.md

* fix paras registrar

* Update propose_parachain.rs

* fix merge

* Introduce "shared" module

* fix rococo build

* fix up and use shared

* guide updates

* add shared config to common tests

* add shared to test-runtime

* remove println

* fix note

Co-authored-by: Gavin Wood <gavin@parity.io>
2021-02-18 21:20:18 -06:00
Robert Habermeier 006602eff2 Replace AuxStore with custom RocksDB (#2471)
* Use KeyValueDB in approval-voting

* use KVDB instead of AuxStore

* add rocksdb to cargo toml

* add a Config struct

* create new DB in service

* fix dep for regular node

* make optional

* post merge fix

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-02-18 09:24:46 -06:00
Bernhard Schuster 85489ceb36 [jaeger] unify all used tags, introduce builder pattern, additional… (#2473)
* feat/jaeger: unify all used tags, introduce builder pattern, additional candidate annotations

* chores

* fixes, incomplete fn rename

* another fix

* more fixes

* silly doctests
2021-02-18 15:07:17 +01:00
Robert Habermeier fb0106a00e gate approval-checking logic (#2470) 2021-02-18 08:05:13 +01:00
Robert Habermeier b7aac51341 A fast-path for requesting AvailableData from backing validators (#2453)
* guide changes for a fast-path requesting from backing validators

* add backing group to availability recovery message

* add new phase to interaction

* typos

* add full data messages

* handle new network messages

* dispatch full data requests

* cleanup

* check chunk index

* test for invalid recovery

* tests

* Typos.

* fix some grumbles

* be more explicit about error handling and control flow

* fast-path param

* use with_chunks_only in Service

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
2021-02-17 13:51:50 -06:00
Robert Habermeier 4a5e5f13ae Full chunk messages for availability-recovery (#2466)
* add full data messages

* handle new messages

* flip condition
2021-02-17 19:01:39 +00:00
Bastian Köcher 64691eefba Track the collation status in the collator protocol (#2465)
* Track the collation status in the collator protocol

This pr changes the collator protocol to track the status of a
collation. This is mainly used to log the status of a collation when it
is removed to inform the user if a collation maybe never reached a validator.

* Update node/network/collator-protocol/src/collator_side.rs

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Moare logging

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
2021-02-17 18:26:30 +00:00
Robert Habermeier 59e2a810bb remove unused RequestBlockAuthorshipData (#2455) 2021-02-17 11:54:51 -06:00
Andronik Ordian 4004217059 make approval voting work on a small testnet (#2421)
* insta-approval for low-node testnets

* fix

* handle 0 needed_approvals and add some logs

* downgrade logs to debug, per block

* fix a warning

* more useful logs

* test

* finish test 🎉

* not so fast

* the test passes, but is it enough?
2021-02-17 11:35:26 -06:00
Andronik Ordian 03a8566cba fix earliest_stored_session (#2461) 2021-02-17 10:43:15 -06:00
Robert Habermeier 9e525e296d Allow at most one candidate with a code upgrade in a block (#2456)
* guide: max one candidate with upgrade per block

* max one candidate with upgrade per block

* ignore on runtime level too
2021-02-17 10:42:06 -06:00
Bernhard Schuster 1e2161258b refactor/reputation: unify the values used (#2462)
* refactor/reputation: unify the values used

* chore/rep: rename Annoy* to Cost*, make duplicate message Cost*Repeated

* fix/reputation: lost and found, convert at the boundary to substrate

* refactor/rep: move conversion to base reputation one level down, left conversions

* fix/rep: order of magnitude adjustments

Thanks pierre!

* remove spaces

* chore/rep: give rationale for order of magnitude

* refactor/rep: move UnifiedReputationChange to separate file

* fix/rep: order of magnitudes correction
2021-02-17 17:18:13 +01:00
Martin Pugh 62c5896592 [CI] Fix check-tx-version Gitlab job (#2449)
* set -e for tx job

* dont set -e while we investigate

* fetch latest release polkadot

* add jq to node:15 image

* Add logic for timing out if nodes never start listening

* remove trap
2021-02-17 16:25:44 +01:00
Cecile Tonglet 0771537a0b Polkadot companion for #8043 (#2382)
* WIP

* WIP

* WIP

* Revert branch change

* CLEANUP

* "Update Substrate"

* Fix build

* Revert "Fix build"

This reverts commit 2dcc3f2a14d28febb84c2c35d682243bc03523a8.

* cargo update -p sp-io

Co-authored-by: parity-processbot <>
2021-02-17 10:12:34 +00:00
Bastian Köcher 27c7347b9b Make sure the adder collator fails when something is wrong (#2450)
As futures most of the time are catching panics and we don't check this,
it could happen that we have some statement that isn't correct but the
test succeeds successfully.
2021-02-16 19:27:50 +00:00
Robert Habermeier a8d3aca13d Disputes High-level rewrite & Disputes runtime (#2424)
* REVERT: comment out graphviz

* rewrite most of protocol-disputes

* write about conclusion and  chain selection

* tie back in overview

* basic disputes module

* guide: InclusionInherent -> ParaInherent

* language

* add ParaInherentData type

* plug parainherentdata into provisioner

* provide_multi_dispute

* tweak

* inclusion pipeline logic for disputes

* be clearer about signature checking

* reject backing of disputed blocks

* some type rejigging

* known-disputes runtime API

* wire up inclusion

* Revert "REVERT: comment out graphviz"

This reverts commit 66203e362f7872cb413d258f74634a0aad70302b.

* timeouts

* include in initialization order

* address grumbles
2021-02-16 18:50:14 +00:00
Guillaume Thiolliere 8734cf62b2 companion: Babe remove inherent (#2438) 2021-02-16 19:43:38 +01:00