Commit Graph
91 Commits
Author SHA1 Message Date
Andronik Ordian bd66121324 validator-discovery: some extra logs (#3157)
* remove duplicate logging

* validator-discovery: add extra logging
2021-06-03 14:17:24 +00:00
Andronik Ordian 44704a57e1 dependabot: ignore beefy git updates (#3111)
It should rather be updated manually with proper `Cargo.lock` updates at least until we switch to crates.io versions. See #3109 as an example of that (bad) update.
2021-05-26 16:47:32 +02:00
Andronik Ordian 701be9aa03 validator_discovery: small tweak in retrying logic (#3102)
* validator_discovery: small tweak in retrying logic

* validator_discovery: use timeouts instead
2021-05-25 20:59:14 +02:00
Andronik Ordian 44d02faa62 network-bridge: downgrade log level of benefit rep change (#3068)
* network-bridge: downgrade log level of benefit rep change

* remove it as we log it at higher level
2021-05-21 19:16:20 -05:00
2e70f4ea08 validator-discovery: basic retrying logic (#3059)
* validator_discovery: less flexible, but simpler design

* fix test

* remove unused struct

* smol optimization

* validator_discovery: basic retrying logic

* add a test

* add more tests

* update the guide

* more test logic

* Require at least 2/3 connectivity.

* Fix test.

* Update node/network/gossip-support/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update node/network/gossip-support/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-05-20 10:05:44 +00:00
Andronik Ordian 98c06f5b57 validator_discovery: less flexible, but simpler design (#3052)
* validator_discovery: less flexible, but simpler design

* fix test

* remove unused struct

* smol optimization
2021-05-19 18:54:13 +02:00
Andronik OrdianandVladimir Istyufeev 87264b7b0b bump spec versions in kusama, polkadot and westend again II (#3021)
* bump spec versions in kusama, polkadot and westend again II

* Don't build artifacts for the downstream simnet job as it's disabled now

* uncomment rococo-v1 publishing job

Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
2021-05-13 16:13:01 +02:00
Andronik Ordian 60fbca3c2a validator_discovery: simplification (#3009)
* validator_discovery: simplification

* compilation fixes

* compilation fixes II

* compilation fixes III

* compilation fixes IV
2021-05-13 11:31:15 +02:00
Andronik Ordian 6c0cb7f8d0 CI: bot allow simnet failure (#2993) 2021-05-07 12:08:42 -05:00
Andronik Ordian 6fe5b46f8d bump rocksdb to 6.17.3 (#2899) 2021-04-20 08:07:52 +02:00
Andronik Ordian e62939ad7a distribution: handle sqrt peer view updates (#2871)
* distribution: handle sqrt peer view updates

* someone please put rustc into my brain

* guide updates
2021-04-10 00:45:25 +02:00
Andronik Ordian 8961628e7c bitfield-dist: check signatures once (#2868) 2021-04-09 17:31:34 +00:00
Andronik Ordian 78a46da384 silence some parachain warnings (#2859)
* silence some parachain warnings

* that was actually a bug that Rob is fixing
2021-04-09 00:25:46 +02:00
Andronik Ordian a345e2a83e fix fuzz tests (#2828) 2021-04-08 18:52:00 +02:00
Andronik Ordian 1e4517800b fix failing test (#2844) 2021-04-07 16:22:04 +00:00
Andronik OrdianandBastian Köcher dbb09ac9f4 parachain db versioning (#2829)
* move parachains_db.rs to a module

* parachain versioning

* fixes

* more fixes

* Update node/service/src/parachains_db/upgrade.rs

* fix web-wasm check

* suggested style fixes

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/service/src/parachains_db/upgrade.rs

* compilation fixes

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-04-07 00:23:32 +02:00
Andronik Ordian 936f0d7c59 statement-distribution: do not verify signatures for duplicate statements (#2823)
* stmnt-dist: do not verify signature on duplicates

* renames

* more renames
2021-04-06 17:32:57 +02:00
Andronik Ordian 1244378d50 parachain_db: fix incorrect config cache sizes (#2827) 2021-04-06 16:47:38 +02:00
Andronik Ordian 2ff5c9b995 tests: use future::join instead of future::select (#2813)
* tests/av-store: use future::join instead of future::select

* tests/backing: use future::join instead of future::select

* tests/provisioner: use future::join instead of future::select

* tests/av-dist: use future::join instead of future::select

* tests/av-recovery: use future::join instead of future::select

* tests/bridge: use future::join instead of future::select

* tests/collator-protocol: use future::join instead of future::select

* tests/stmt-dist: use future::join instead of future::select

* fix tests
2021-04-05 18:30:27 +02:00
Andronik Ordian 4df29e71ab bitfield-dist: fix state update on gossip (#2817)
* bitfield-dist: fix state update on gossip

* fixes

* doc fixes

* oops

* 2 lines of code change
2021-04-04 22:25:40 +00:00
Andronik Ordian 94b0ccc8f1 approval-distribution: split peer knowledge into sent and received (#2809)
* approval-distribution: split peer knowledge into sent and received

* guide updates

* fixes

* revert doc changes
2021-04-03 04:29:15 +02:00
Andronik Ordianandparity-processbot cd9449cbef companion: update wasmtime to 0.24.0 (#2625)
* update using diener

* fix zstd upgrade

* Revert "update using diener"

This reverts commit 701fdfe3822d6beac4bfda591a1dbcb99bcadb1a.

* update zstd to 0.6

* update using diener

* Revert "update using diener"

This reverts commit fc81df6355375b1328bb6126e3227f193583a61b.

* update Substrate

Co-authored-by: parity-processbot <>
2021-04-02 20:23:29 +00:00
Andronik Ordian 98082c5326 gossip: move authorities request to runtime api subsystem (#2798) 2021-04-01 23:51:01 +02:00
Andronik Ordian 7a2e1ef6c1 gossip: do not try to connect if we are not validators (#2786)
* gossip: do not issue a connection request if we are not a validator

* guide updates

* use all relevant authorities when issuing a request

* use AuthorityDiscoveryApi instead

* update comments to the status quo
2021-04-01 18:11:43 +02:00
Andronik Ordian caebf642dd statement-distribution: do not use OurViewChange (#2790)
* quickfix for statement-distribution

* some logs
2021-03-31 23:28:17 +02:00
Andronik Ordian 9ac35d9f2b gossip: choose a random subset on send instead of limiting connections (#2776)
* gossip: choose random subset on send

* naming bikeshed
2021-03-30 20:59:53 +02:00
Andronik Ordian a3115401c3 network-bridge: elevate log level for connections (#2772) 2021-03-30 20:01:57 +02:00
Andronik Ordian bdee5a3923 approval-distribution: add an assertion (#2761) 2021-03-30 14:18:34 +02:00
Andronik OrdianandDavid 6f464a360f approval-distribution: limit the amount of assignments on unify (#2737)
* approval-distribution: limit the amount of packets on unify

* guide: fix a typo

* compilation fix

* grammar

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: David <dvdplm@gmail.com>

* more grammar

* propagate only local assignments/approvals after a certain depth

* increase the threshold

* guides update

Co-authored-by: David <dvdplm@gmail.com>
2021-03-28 23:30:06 +02:00
Andronik Ordian 171fc69961 approval-voting: more spans and metrics (#2742)
* approval-voting: more spans and metrics

* s/db/approval db
2021-03-28 23:28:49 +02:00
Andronik Ordian dce20644c8 approval-distribution: moar metrics (#2734) 2021-03-28 00:23:32 +01:00
Andronik Ordian 71f1985172 approval-distribution: moar logs (#2732) 2021-03-27 23:21:25 +01:00
+6 9862c8172b CI: initial fuzzer jobs (#2616)
* ci: initial fuzzer job

* erasure-coding: update fuzzer Cargo.lock

* syntax fix

* try this first

* install honggfuzz deps

* try not

* try if else

* try SIGINT

* ignore hfuzz dirs

* ???

* bash is growing on me

* decouple builds from running

* fix a typo

* try copying dirs

* fix indentation

* try using absolute paths

* another try

* caching is not worth it

* remove outdated needs

* cleanup and add futher TODOs

* Update .github/workflows/honggfuzz.yml

* more diagnostic logs for approval-voting (#2618)

* Backing and collator protocol traces including para-id (#2620)

* improve backing/provisioner spans

* span for collation requests

* add para_id to unbacked candidate spans

* differentiate validation-construction and find-assignment in selection

* better find-assignment spans

* organize unbacked-candidate spans directly under job root

* Update node/core/provisioner/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>

* add tracing when no assignment in candidate selection (#2623)

* New slots/auctions architecture (#2294)

* TODOs

* Add auctions.rs, comment on changes needed.

* Remove cruft from slots

* Remove more from auctions.rs

* More logic drafting in slots.

* More logic in slots.rs

* patch some errors

* more fixes

* last nit

* Cleanups in slots.rs

* Cleanups in slots.rs

* patches

* make build

* crowdloan to new api

* auction compile

* Use ParaId instead of FundIndex in Crowdloan (#2303)

* use paraid instead of fundindex

* Update crowdloan.rs

* check caller is manager

* Auction tests and fix build warnings.

* Configurable origin for initiating auctions

* Remove on_finalize

* #2303 (manual merge)

* Tests for Slots

* some registrar tests

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update runtime/common/src/slots.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Slots uses Registrar for CurrentChains

* swap works test

* on swap impl

* traitify parachain cleanup

* explicit lifecycle tracking for paras

* initial implementation of lifecycles and upgrades

* clean up a bit

* Update runtime/common/src/slots.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* fix doc comment

* more rigid lifecycle checks

* include paras which are transitioning, and lifecycle query

* format guide

* update api

* update guide

* explicit outgoing state, fix genesis

* handle outgoing with transitioning paras

* Revert "explicit lifecycle tracking for paras"

This reverts commit 4177af7ba473bbd9c26bccd861793f25265b6657.

* remove lifecycle tracking from registrar

* do not include transitioning paras in identifier

* Update paras_registrar.rs

* final patches to registrar

* Fix test

* use noop in test

* clean up pending swap on deregistration

* finish registrar tests

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Apply suggestions from code review

* Use matches macro

* Correct terms

* Apply suggestions from code review

* Remove direct need for Slots and Registrar from Crowdloan

* Rejig things slightly

* actions queue

* Revert "actions queue"

This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.

* Traitify Auction interface.

* Mockups and initial code for Crowdloan testing

* One test...

* collapse onboarding state

* fix some crowdloan tests

* one more

* start benchmarks for auctions

* benchmark bid

* fix more crowdloan tests

* onboard and begin retirement no longer exist

* Revert "onboard and begin retirement no longer exist"

This reverts commit 2e100fd94e3540bff5f172328b5d917896f1c6fc.

* Simplify crowdloan and make it work.

* Fixes

* fix some

* finish merge fixes

* fix refund bug in auctions

* Add traits to Registrar for tests and benchmarks

* fix more auction benchmarks

* Fix TestAuctioneer

* finish crowdloan benchmarks

* start setting up full integration tests

* expand integration tests

* finish basic integration test

* add more integration tests

* begin slots benchmarks

* start paras registrar benchmarks

* fix merge

* fix tests

* clean up paras registrar

* remove println

* remove outdated cleanup config

* update benchmarks

* Add WeightInfo

* enable runtime-benchmarks feature flag

* complete swap benchmark

* add parachains and onboarding into westend

* add benchmarks and genesis

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=slots --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* fix benchmark execution

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* Use `new_raise_len` in crowdloan on_initialize

* Update paras_registrar.rs

* fix westend merge

* impl on_swap for crowdloan

* Check fund exists before create

* update for crowdloan sig

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* slots on_initialize

* use integration tests environment for benchmarks

* fix hrmp event

* auction on_initialize

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* fix storage name in auctions

* add auction_index to winning data

* winning data takes into account current auction index

* remove println

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=slots --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* Revert "add auction_index to winning data"

* PastRandomness.

* Fixes

* Use new randomness

* fix use of randomness in auctions and runtime config

* expose consts

* fix auction test

* add deposit per byte for para registration

* basic swap integration test

* make swap test more comprehensive

* Add WinningVec for easier retrieval in the front-end.

* clean up `WinningVec` at the end

* Add event for when a new best bid comes in

* Fix propagation of winners in ending period

* fix benchmarks, refund weight in dissolve

* fix unused

* remove some TODOs

* setup opaque keys for paras in westend

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=crowdloan --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* remove unused

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=auctions --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* back to regular runtime config

* use saturating math where user input can be

* better first slot check

* Update runtime/common/src/claims.rs

* update westend onswap impl

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>

* update ring to 0.16.20 (#2626)

* Remove TODO from substrate#2986 (#2628)

* Companion for sub/8176 (#2622)

* Merge

* Fixes

* Fix build

* remove dep.

* undo dep.

* upadte substrate

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* Fix lock

* revert lock; cargo update -p sp-io

* from_rational_approx -> from_rational

* Silence more warnings

Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>

* Add /data symlink to Docker containers (#2627)

* add /data symlink to Docker

* fix comments

* Fix wrong deposit amount in council voters. (#2562)

* Fix wrong deposit amount in council voters.

* Fix some build

* make it all compile.. so far.

* Fix

* break build

* Okay fix it again

* re benchmark  (#2630)

* Change something

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

Co-authored-by: Parity Benchmarking Bot <admin@parity.io>

* Reduce number of active leaves at startup (#2631)

Currently we will take all leaves and give that to the overseer on
startup, but this is a bad idea when the finality is lagging for
example. There can be many of unfinalized leaves, we don't even need to
look at anymore. To solve this, the pr adds a maximum of 4 leaves we
forward to the overseer and the pr also checks that we only pass uncles
of the best block.

* yet another set of logging improvements (#2638)

* Polkadot companion for Substrate PR #7640 (Store multiple Justifications per block) (#2358)

* service: update for substrate PR #7640

* update substrate

* Add Pallet Babe to Integration Tests Runtime

Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Companion for Substrate#8386 (#2634)

* Companion for Substrate#8386

https://github.com/paritytech/substrate/pull/8386

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Request based collation fetching (#2621)

* Introduce collation fetching protocol

also move to mod.rs

* Allow `PeerId`s in requests to network bridge.

* Fix availability distribution tests.

* Move CompressedPoV to primitives.

* Request based collator protocol: validator side

- Missing: tests
- Collator side
- don't connect, if not connected

* Fixes.

* Basic request based collator side.

* Minor fix on collator side.

* Don't connect in requests in collation protocol.

Also some cleanup.

* Fix PoV distribution

* Bump substrate

* Add back metrics + whitespace fixes.

* Add back missing spans.

* More cleanup.

* Guide update.

* Fix tests

* Handle results in tests.

* Fix weird compilation issue.

* Add missing )

* Get rid of dead code.

* Get rid of redundant import.

* Fix runtime build.

* Cleanup.

* Fix wasm build.

* Format fixes.

Thanks @andronik !

* Companion for #8372 (Replace 'Module' with 'Pallet' in construct_runtime macro) (#2629)

* Replace 'Module' with 'Pallet'.

* "Update Substrate"

* fix babe usage

* fix benchmark

Co-authored-by: parity-processbot <>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>

* integrate faster erasure code (#2608)

Breaks compatibility for distributing PoV and PersistentValidationData between validators.

Ref #2442

* Update for the new substrate client API (#2570)

* Update for the new substrate client API

* Code review suggestions

* Update substrate

* Improve the logging (#2645)

* Don't accept incoming connections for collators (#2644)

* Don't accept incoming connections for collators

on the `Collation` peer set.

* Better docs.

* fix reconstruct fuzzer name

* make script more robust

* REVERTME: test run

* REVERTME: test run II

* Revert "REVERTME: test run II"

This reverts commit 58375df3882b9dff4c207f58fd38ccb35bc90c50.

* Revert "REVERTME: test run"

This reverts commit 9759cb61da6169de27d8ee7be0edef75a11f0e7f.

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Martin Pugh <pugh@s3kr.it>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Shaun Wang <spxwang@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2021-03-24 18:00:35 +01:00
Andronik Ordian 0dba55b507 update ring to 0.16.20 (#2626) 2021-03-15 08:14:33 +01:00
Andronik Ordian a543b1d6c3 availability distribution: don't early return on runtime errors (#2606)
* availability distribution: don't early return on runtime errors

* log error

* extract runtime api error from Error

* uh

* oh
2021-03-11 12:47:56 -06:00
Andronik Ordian baa691deb1 prefix parachain log targets with parachain:: (#2600)
* prefix parachain log targets with parachain::

* even more consistent
2021-03-10 17:07:56 +01:00
Andronik Ordian 287604cf7e approval-voting metrics (#2483)
* approval-voting metrics

* metric: approvals produced
2021-03-09 15:32:31 -06:00
Andronik Ordian 1bec8cfed9 bump spec versions in kusama, polkadot and westend again (#2557) 2021-03-04 14:28:14 +01:00
Andronik Ordian 4c1de66d5d subsystem for issuing background connection requests (#2538)
* initial subsystem for issuing connection requests

* finish the initial impl

* integrate with the overseer

* rename to gossip-support

* fix renamings leftover

* remove run_inner

* fix compilation

* random subset of sqrt
2021-03-02 10:40:06 +00:00
Andronik Ordian 7530a0c432 validator_discovery: some logs to help identify authority discovery issues (#2533) 2021-02-26 12:06:23 -06:00
Andronik Ordian 241b1f12a7 make runtime_api non blocking task again (#2531) 2021-02-26 12:04:58 +01:00
Andronik Ordian 4e3dba91be small cleanup after #2518 (#2527) 2021-02-25 14:00:20 +01:00
Andronik Ordian 4004217059 make approval voting work on a small testnet (#2421)
* insta-approval for low-node testnets

* fix

* handle 0 needed_approvals and add some logs

* downgrade logs to debug, per block

* fix a warning

* more useful logs

* test

* finish test 🎉

* not so fast

* the test passes, but is it enough?
2021-02-17 11:35:26 -06:00
Andronik Ordian 03a8566cba fix earliest_stored_session (#2461) 2021-02-17 10:43:15 -06:00
Andronik Ordian b28e1b4e74 fix approval import tests and some bugs (#2452)
* tests: use future::join

* fix panic in cache_session_info_for_head

* fix test assertion

* fix infinite loop in determine_new_blocks

* fix ordering in determine_new_blocks

* fix expected ancestry in tests
2021-02-16 19:08:35 +01:00
Andronik Ordian 89e8e81336 validator_discovery: cache by (Hash, ParaId) (#2402) 2021-02-09 14:42:46 -06:00
Andronik OrdianandRobert Klotzner 6981a1c366 validator_discovery: pass PeerSet to the request (#2372)
* validator_discovery: pass PeerSet to the request

* validator_discovery: track PeerSet of connected peers

* validator_discovery: fix tests

* validator_discovery: fix long line

* some fixes

* some validator_discovery logs

* log validator discovery request

* Also connect to validators on `DistributePoV`.

* validator_discovery: store the whole state per peer_set

* bump spec versions in kusama, polkadot and westend

* Correcting doc.

* validator_discovery: bump channel capacity

* pov-distribution: some cleanup

* this should fix the test, but it does not

* I just got some brain damage while fixing this

Why are you even reading this???

* wrap long line

* address some review nits

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
2021-02-08 08:57:59 +01:00
Andronik Ordian 0ffee4f6ef bump spec versions in kusama, polkadot and westend (#2391) 2021-02-05 17:20:13 +01:00
Andronik Ordian 3f1e1a6ff7 impl approval distribution (#2160)
* initial impl approval distribution

* initial tests and fixes

* batching seems difficult: different peers have different needs

* bridge: fix test after merge

* some guide updates

* only send assignments to peers who know about the block

* fix a test, add approvals test

* simplify

* do not send assignment to peers for finalized blocks

* guide: protocol input and output

* one more test

* more comments, logs, initial metrics

* fix a typo

* one more thing: early return when reimporting a thing locally
2021-01-25 18:14:32 -05:00
Andronik OrdianandPeter Goodspeed-Niklaus deff43fd30 small cleanup (#2267)
* session_info: fix authorities docstring

* overseer: more consistent metrics naming

* session_info: mention ordering

* use correct bucket sizes

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2021-01-14 10:41:19 +00:00
Andronik Ordianandparity-processbot e1773be63d Companion for substrate#7892 (#2262)
* session_info: use correct authorities set

* bump rococo spec_version to 19

* "Update Substrate"

Co-authored-by: parity-processbot <>
2021-01-13 19:02:03 +00:00
Andronik Ordianandparity-processbot a0a5f4cf88 upgrade a few dependencies (companion for 7831) (#2205)
* upgrade a few dependencies

* "Update Substrate"

Co-authored-by: parity-processbot <>
2021-01-06 20:38:25 +01:00
Andronik Ordian 3f5156e866 refactor View to include finalized_number (#2128)
* refactor View to include finalized_number

* guide: update the NetworkBridge on BlockFinalized

* av-store: fix the tests

* actually fix tests

* grumbles

* ignore macro doctest

* use Hash::repeat_bytes more consistently

* broadcast empty leaves updates as well

* fix issuing view updates on empty leaves updates
2020-12-17 12:50:58 -05:00
Andronik Ordian 1aa2554000 guide: minor fixes approval distribution (#2129)
* guide: add missing ApprovalDistributionMessage variant

* guide: deduplicate sensence
2020-12-17 15:02:04 +00:00
Andronik OrdianandBastian Köcher 04e9489da6 session_ info: small fixes (#2106)
* session_info: use proper timeout in test

* Revert "fix off-by-one error"

This reverts commit 35cb56305a19134acd8a8f881f3aabf999a09d74.

* session_info: use correct EarliestStoredSession when introduced on a live chain

* use saturating_sub

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* session_info: revert the timeout test

* session_info: rust is dumb

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-12-10 15:34:17 -06:00
Andronik Ordian be3e5b3b3b session_info: store the assigment ids (#2102)
* session_info: store the assigment ids

* how about this?
2020-12-10 15:02:36 -06:00
Andronik Ordian 6f1590b253 runtime/session_info: avoid heavy loop when introduced on a live chain (#2099)
* session_info: a heavy loop test

* session_info: fix a typo

* session_info: fix heavy loop

* session_info: crank the iterations all the way up
2020-12-09 09:33:58 -06:00
Andronik Ordian 87e0b6fe65 remove unused code (#2058) 2020-12-02 14:18:18 +00:00
Andronik Ordian bbb036e304 util: implement FusedStream for Jobs (#2031) 2020-11-28 23:14:37 +01:00
Andronik OrdianandSergei Shulepov f8181fa1f2 erasure-coding: do not panic on 1 validator (#2022)
* erasure-coding: do not panic on 1 validator

* tabify fuzz tests

* remove num_validators length check

* remove unused import

* move erasure_coding_fuzzer to fuzzer

* change the authors to admit (at) parity.io

* Apply suggestions from code review

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-11-27 16:49:00 -05:00
Andronik OrdianandSergei Shulepov 39a12b68f6 past-session validator discovery APIs (#2009)
* guide: fix formatting for SessionInfo module

* primitives: SessionInfo type

* punt on approval keys

* ah, revert the type alias

* session info runtime module skeleton

* update the guide

* runtime/configuration: sync with the guide

* runtime/configuration: setters for newly added fields

* runtime/configuration: set codec indexes

* runtime/configuration: update test

* primitives: fix SessionInfo definition

* runtime/session_info: initial impl

* runtime/session_info: use initializer for session handling (wip)

* runtime/session_info: mock authority discovery trait

* guide: update the initializer's order

* runtime/session_info: tests skeleton

* runtime/session_info: store n_delay_tranches in Configuration

* runtime/session_info: punt on approval keys

* runtime/session_info: add some basic tests

* Update primitives/src/v1.rs

* small fixes

* remove codec index annotation on structs

* fix off-by-one error

* validator_discovery: accept a session index

* runtime: replace validator_discovery api with session_info

* Update runtime/parachains/src/session_info.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* runtime/session_info: add a comment about missing entries

* runtime/session_info: define the keys

* util: expose connect_to_past_session_validators

* util: allow session_info requests for jobs

* runtime-api: add mock test for session_info

* collator-protocol: add session_index to test state

* util: fix error message for runtime error

* fix compilation

* fix tests after merge with master

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-11-26 11:02:50 +00:00
Andronik OrdianandPeter Goodspeed-Niklaus 69b103b1d5 overseer: send_msg should not return an error (#1995)
* send_message should not return an error

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* s/send_logging_error/send_and_log_error

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-11-23 12:42:14 +01:00
Andronik Ordian 97f5bd9047 cleanup validator discovery (#1992)
* use snake_case for log targets

* remove unused continue

* validator_discovery: when disconnecting, use all addresses

* validator_discovery: simplify request revokation

* fix a typo
2020-11-20 18:34:57 +00:00
Andronik Ordian 0a8a607a58 update most of the dependencies (#1946)
* update tiny-keccak to 0.2

* update deps except bitvec and shared_memory

* fix some warning after futures upgrade

* remove useless package rename caused by bug in cargo-upgrade

* revert parity-util-mem *

* remove unused import

* cargo update

* remove all renames on parity-scale-codec

* remove the leftovers

* remove unused dep
2020-11-17 11:16:31 +01:00
Andronik Ordian 2cde7732da more resilient subsystems (#1908)
* backing: extract log target

* bitfield-signing: extract log target

* utils: fix a typo

* provisioner: extract log target

* candidate selection: remove unused error variant

* bitfield-distribution: change the return type of run

* pov-distribution: extract log target

* collator-protocol: simplify runtime request

* collation-generation: do not exit early on error

* collation-generation: do not exit on double init

* collator-protocol: do not exit on errors and rename LOG_TARGET

* collator-protocol: a workaround for ununused imports warning

* Update node/network/bitfield-distribution/src/lib.rs

* collation-generation: elevate warn! to error!

* collator-protocol: fix imports

* post merge fix

* fix compilation
2020-11-05 14:22:41 +01:00
Andronik Ordian bcec9f9a99 more resilient subsystems: av-store (#1888)
* utils: remove unused error

* av-store: do not exit early on errors

* av-store: revert logging change on master

* av-store: add a test
2020-11-02 12:55:32 -06:00
Andronik OrdianandAndrew Plaza fa1881458f fix compilation on nightly (#1850)
* runtime: annonate some types

* cargo update -p sp-io

* add wasm overwrites to Configuration in test-service

* rename overwrite to override

Co-authored-by: Andrew Plaza <andrew.plaza@parity.io>
2020-10-26 17:25:49 +00:00
Andronik Ordian 979b9c5556 av-store: create a db dir if missing (#1819) 2020-10-15 15:09:48 +02:00
Andronik Ordianandparity-processbot e62b300f47 Companion PR for #7247 (incremental priority group updates) (#1800)
* validator discovery: use incremental updates for priority_group

* validator discovery: fix compilation

* validator discovery: remove Sync bound on Net

* "Update Substrate"

Co-authored-by: parity-processbot <>
2020-10-09 14:34:17 +00:00
Andronik Ordian d00bdfef08 chain-api subsystem: implement BlockHeader messsage (#1778)
* chain-api subsystem: implement BlockHeader messsage

* update the guide
2020-10-06 16:46:54 +02:00
Andronik Ordian abf76d27dc collator: fix a typo (#1788)
* collator: fix a typo

* collator: fix more typos

* collator: fix even more typos
2020-10-06 13:00:18 +00:00
ca89e3edbe NetworkBridge: validator (authorities) discovery api (#1699)
* stupid, but it compiles

* redo

* cleanup

* add ValidatorDiscovery to msgs

* sketch network bridge code

* ConnectToAuthorities instead of validators

* more stuff

* cleanup

* more stuff

* complete ConnectToAuthoritiesState

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Collator protocol subsystem (#1659)

* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* handle multiple in-flight connection requests

* handle cancelled requests

* Update node/core/runtime-api/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* redo it again

* more stuff

* redo it again

* update comments

* workaround Future is not Send

* fix trailing spaces

* clarify comments

* bridge: fix compilation in tests

* update more comments

* small fixes

* port collator protocol to new validator discovery api

* collator tests compile

* collator tests pass

* do not revoke a request when the stream receiver is closed

* make revoking opt-in

* fix is_fulfilled

* handle request revokation in collator

* tests

* wait for validator connections asyncronously

* fix compilation

* relabel my todos

* apply Fedor's patch

* resolve reconnection TODO

* resolve revoking TODO

* resolve channel capacity TODO

* resolve peer cloning TODO

* resolve peer disconnected TODO

* resolve PeerSet TODO

* wip tests

* more tests

* resolve Arc TODO

* rename pending to non_revoked

* one more test

* extract utility function into util crate

* fix compilation in tests

* Apply suggestions from code review

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* revert pin_project removal

* fix while let loop

* Revert "revert pin_project removal"

This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.

* fix compilation

* Update node/subsystem/src/messages.rs

* docs on pub items

* guide updates

* remove a TODO

* small guide update

* fix a typo

* link to the issue

* validator discovery: on_request docs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2020-10-06 13:34:57 +02:00
Andronik Ordian 579614d127 implement remaining subsystem metrics (#1770)
* overseer metrics: messages relayed

* provisioner metrics: cosmetic changes

* candidate selection metrics: cosmetic changes

* availability bitfields metrics

* availability distribution metrics

* PoV distribution metrics

* statement-distribution: small simplification

* statement-distribution: extract log target into a const

* statement-distribution: metrics

* address review nits
2020-10-01 12:08:03 +02:00
Andronik Ordian de05bec4d6 move Metrics to utils (#1765) 2020-09-29 11:42:20 +00:00
Andronik Ordian f9b15f654b provisioner tests: remove tokio from dev-dependencies (#1745)
* provisioner: remove tokio from dev-dependencies

* provisioner: use futures_timer instead
2020-09-23 20:21:03 +02:00
Andronik Ordianandparity-processbot bcafc81d9f update kvdb-rocksdb to 0.9.1 (companion for #6963) (#1647)
* update kvdb-rocksdb to 0.9.1

* update Cargo.lock

* "Update Substrate"

* resolve merge conflicts

* resolve merge conflicts t. 2

Co-authored-by: parity-processbot <>
2020-08-29 03:53:09 +00:00
Andronik Ordian e899a3f844 jobs: don't early exit when there are no jobs (#1621)
* jobs: don't early exit when there are no jobs

* utils: fix merged test

* utils: less verbose

* utils: add an assert subsystem is running

* utils: use TimeoutExt from test-helpers

* test-helpers: use TimeoutExt
2020-08-21 16:45:39 +02:00
Andronik Ordian 44354e717b utils: handle race condition gracefully (#1583)
* utils: handle race condition gracefully

* utils: add a test

* update Cargo.lock

* utils: remove a warning

* utils: init logger in tests

* utils: update the outdated comment

* util: wait for both subsystem and test_future to finish

* Revert "util: wait for both subsystem and test_future to finish"

This reverts commit 075b39242cd7200e3f4c24ea42f702c3124ce41c.
2020-08-21 15:30:18 +02:00
Andronik Ordian 8eecdd61b8 proposer: wait for a hash to be in the active-leaves set (#1616)
* overseer: add ExternalRequest to Event

* proposer: wait for the hash to be activated

* update comments

* overseer: handle unbounded growth of listeners map

* overseer: fix compilation

* overseer: clean up dead listeners

* overseer: cosmetic changes

* overseer: cosmetic changes t.2

* overseer: add debug_assertions

* overseer: fix formatting
2020-08-20 13:43:36 +00:00
e7ead40255 initial prometheus metrics (#1536)
* service-new: cosmetic changes

* overseer: draft of prometheus metrics

* metrics: update active_leaves metrics

* metrics: extract into functions

* metrics: resolve XXX

* metrics: it's ugly, but it works

* Bump Substrate

* metrics: move a bunch of code around

* Bumb substrate again

* metrics: fix a warning

* fix a warning in runtime

* metrics: statements signed

* metrics: statements impl RegisterMetrics

* metrics: refactor Metrics trait

* metrics: add Metrics assoc type to JobTrait

* metrics: move Metrics trait to util

* metrics: fix overseer

* metrics: fix backing

* metrics: fix candidate validation

* metrics: derive Default

* metrics: docs

* metrics: add stubs for other subsystems

* metrics: add more stubs and fix compilation

* metrics: fix doctest

* metrics: move to subsystem

* metrics: fix candidate validation

* metrics: bitfield signing

* metrics: av store

* metrics: chain API

* metrics: runtime API

* metrics: stub for avad

* metrics: candidates seconded

* metrics: ok I gave up

* metrics: provisioner

* metrics: remove a clone by requiring Metrics: Sync

* metrics: YAGNI

* metrics: remove another TODO

* metrics: for later

* metrics: add parachain_ prefix

* metrics: s/signed_statement/signed_statements

* utils: add a comment for job metrics

* metrics: address review comments

* metrics: oops

* metrics: make sure to save files before commit 😅

* use _total suffix for requests metrics

Co-authored-by: Max Inden <mail@max-inden.de>

* metrics: add tests for overseer

* update Cargo.lock

* overseer: add a test for CollationGeneration

* collation-generation: impl metrics

* collation-generation: use kebab-case for name

* collation-generation: add a constructor

Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: Ashley Ruglys <ashley.ruglys@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
2020-08-18 11:18:54 +02:00
Andronik Ordian dffefac54e move AssignmentKind and CoreAssigment to scheduler (#1571) 2020-08-17 19:26:13 +02:00
Andronik Ordian ed616e4b29 overseer: fix build (#1596)
* overseer: add a test for CollationGeneration

* fix build
2020-08-17 16:24:52 +00:00
Andronik Ordian ab1a513265 Add spawn_blocking to SubsystemContext (#1570)
* subsystem: add spawn_blocking to SubsystemContext

* candidate-validation: use spawn_blocking for exhaustive tasks
2020-08-17 12:26:32 +00:00
Andronik Ordian c5f7b4d17d overseer: add a test to ensure all subsystem receive msgs (#1590)
* overseer: add a test to ensure all subsystem receive msgs

* lol
2020-08-14 21:35:00 +02:00
Andronik OrdianandPeter Goodspeed-Niklaus 07a9d2de3f Integrate ChainApi with all messages (#1533)
* propagate chain_api subsystem through various locations

* add ChainApi to AllMessages

Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
2020-08-04 12:32:42 +00:00
Andronik Ordian 596ce826e8 Chain API subsystem (#1498)
* chain-api subsystem skeleton

* chain-api subsystem: draft impl

* chain-api subsystem: mock testclient

* chain-api subsystem: impl HeaderBacked for TestClient

* chain-api subsystem: impl basic tests

* chain-api subsystem: tiny guide

* chain-api subsystem: rename ChainApiRequestMessage to ChainApiMessage

* chain-api subsystem: add the page to the ToC

* chain-api subsystem: proper error type

* chain-api subsystem: impl ancestors request

* chain-api subsystem: tests for ancestors request

* guide: fix ancestor return type

* runtime-api subsystem: remove unused dep

* fix fmt

* fix outdated comment

* chain-api subsystem: s/format/to_string

* lower-case subsystem names

* chain-api subsystem: resolve Finalized todo

* chain-api subsystem: remove TODO

* extract request errors into a module

* remove caching TODO

* fix imports
2020-07-31 16:02:16 +00:00
Andronik Ordian ea834b12de Guide: add a diagram for Inclusion Pipeline & Approval Subsystem (#1457)
* guide: optimize diagrams for readability

* guide: inclusion subsystem diagram

* guide: rename Approval Subsystem for clarity

* guide: correct the approval subsystem

* guide: clarify that validator are selected from the same set

* guide: simplify secondary checkers labels
2020-07-31 14:28:45 +00:00
Andronik Ordian e7a420be0f Scheduler: handle timeouts for no group rotation (#1439)
* runtime: rename parachain_rotation_frequency to group_rotation_frequency

* scheduler: handle timeouts for no group rotation

* scheduler: apply fixes from code review

* scheduler: remove my comments
2020-07-21 20:08:01 -04:00
Andronik OrdianandRobert Habermeier 1ea2b27403 inclusion: split CandidatePendingAvailability according to the guide (#1413)
* inclusion: split PendingAvailability storage into descriptor and commitments

* inclusion: fix tests

* implementers-guide: update CandidatePendingAvailability type

* inclusion: simplify process_candidates a bit

* implementers-guide: more updates to the inclusion module

* inclusion: fix copy-paste errors in tests

* inclusion: revert some of the changes

* inclusion: lazy commitments loading and a test

* guide: revert enact_candidate changes

* inclusion: test process_bitfield for no commitments

* Grammar

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
2020-07-16 21:27:49 -04:00
Andronik Ordian 6957847b6b Integrate all (dummy) subsystems with the Overseer (#1374)
* overseer: introduce a utility typemap

* it's ugly but it compiles

* move DummySubsystem to subsystem crate

* fix tests fallout

* use a struct for all subsystems

* more tests fallout

* add missing pov_distribution subsystem

* remove unused imports and bounds

* fix minimal-example
2020-07-09 16:25:40 -04:00
Andronik Ordian d348fed02e deduplicate parity-util-mem (#926) 2020-03-23 08:57:15 +01:00