Commit Graph
37 Commits
Author SHA1 Message Date
Andronik OrdianandBastian Köcher 04e9489da6 session_ info: small fixes (#2106)
* session_info: use proper timeout in test

* Revert "fix off-by-one error"

This reverts commit 35cb56305a19134acd8a8f881f3aabf999a09d74.

* session_info: use correct EarliestStoredSession when introduced on a live chain

* use saturating_sub

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* session_info: revert the timeout test

* session_info: rust is dumb

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-12-10 15:34:17 -06:00
Andronik Ordian be3e5b3b3b session_info: store the assigment ids (#2102)
* session_info: store the assigment ids

* how about this?
2020-12-10 15:02:36 -06:00
Andronik Ordian 6f1590b253 runtime/session_info: avoid heavy loop when introduced on a live chain (#2099)
* session_info: a heavy loop test

* session_info: fix a typo

* session_info: fix heavy loop

* session_info: crank the iterations all the way up
2020-12-09 09:33:58 -06:00
Andronik Ordian 87e0b6fe65 remove unused code (#2058) 2020-12-02 14:18:18 +00:00
Andronik Ordian bbb036e304 util: implement FusedStream for Jobs (#2031) 2020-11-28 23:14:37 +01:00
Andronik OrdianandSergei Shulepov f8181fa1f2 erasure-coding: do not panic on 1 validator (#2022)
* erasure-coding: do not panic on 1 validator

* tabify fuzz tests

* remove num_validators length check

* remove unused import

* move erasure_coding_fuzzer to fuzzer

* change the authors to admit (at) parity.io

* Apply suggestions from code review

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-11-27 16:49:00 -05:00
Andronik OrdianandSergei Shulepov 39a12b68f6 past-session validator discovery APIs (#2009)
* guide: fix formatting for SessionInfo module

* primitives: SessionInfo type

* punt on approval keys

* ah, revert the type alias

* session info runtime module skeleton

* update the guide

* runtime/configuration: sync with the guide

* runtime/configuration: setters for newly added fields

* runtime/configuration: set codec indexes

* runtime/configuration: update test

* primitives: fix SessionInfo definition

* runtime/session_info: initial impl

* runtime/session_info: use initializer for session handling (wip)

* runtime/session_info: mock authority discovery trait

* guide: update the initializer's order

* runtime/session_info: tests skeleton

* runtime/session_info: store n_delay_tranches in Configuration

* runtime/session_info: punt on approval keys

* runtime/session_info: add some basic tests

* Update primitives/src/v1.rs

* small fixes

* remove codec index annotation on structs

* fix off-by-one error

* validator_discovery: accept a session index

* runtime: replace validator_discovery api with session_info

* Update runtime/parachains/src/session_info.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* runtime/session_info: add a comment about missing entries

* runtime/session_info: define the keys

* util: expose connect_to_past_session_validators

* util: allow session_info requests for jobs

* runtime-api: add mock test for session_info

* collator-protocol: add session_index to test state

* util: fix error message for runtime error

* fix compilation

* fix tests after merge with master

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-11-26 11:02:50 +00:00
Andronik OrdianandPeter Goodspeed-Niklaus 69b103b1d5 overseer: send_msg should not return an error (#1995)
* send_message should not return an error

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* s/send_logging_error/send_and_log_error

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-11-23 12:42:14 +01:00
Andronik Ordian 97f5bd9047 cleanup validator discovery (#1992)
* use snake_case for log targets

* remove unused continue

* validator_discovery: when disconnecting, use all addresses

* validator_discovery: simplify request revokation

* fix a typo
2020-11-20 18:34:57 +00:00
Andronik Ordian 0a8a607a58 update most of the dependencies (#1946)
* update tiny-keccak to 0.2

* update deps except bitvec and shared_memory

* fix some warning after futures upgrade

* remove useless package rename caused by bug in cargo-upgrade

* revert parity-util-mem *

* remove unused import

* cargo update

* remove all renames on parity-scale-codec

* remove the leftovers

* remove unused dep
2020-11-17 11:16:31 +01:00
Andronik Ordian 2cde7732da more resilient subsystems (#1908)
* backing: extract log target

* bitfield-signing: extract log target

* utils: fix a typo

* provisioner: extract log target

* candidate selection: remove unused error variant

* bitfield-distribution: change the return type of run

* pov-distribution: extract log target

* collator-protocol: simplify runtime request

* collation-generation: do not exit early on error

* collation-generation: do not exit on double init

* collator-protocol: do not exit on errors and rename LOG_TARGET

* collator-protocol: a workaround for ununused imports warning

* Update node/network/bitfield-distribution/src/lib.rs

* collation-generation: elevate warn! to error!

* collator-protocol: fix imports

* post merge fix

* fix compilation
2020-11-05 14:22:41 +01:00
Andronik Ordian bcec9f9a99 more resilient subsystems: av-store (#1888)
* utils: remove unused error

* av-store: do not exit early on errors

* av-store: revert logging change on master

* av-store: add a test
2020-11-02 12:55:32 -06:00
Andronik OrdianandAndrew Plaza fa1881458f fix compilation on nightly (#1850)
* runtime: annonate some types

* cargo update -p sp-io

* add wasm overwrites to Configuration in test-service

* rename overwrite to override

Co-authored-by: Andrew Plaza <andrew.plaza@parity.io>
2020-10-26 17:25:49 +00:00
Andronik Ordian 979b9c5556 av-store: create a db dir if missing (#1819) 2020-10-15 15:09:48 +02:00
Andronik Ordianandparity-processbot e62b300f47 Companion PR for #7247 (incremental priority group updates) (#1800)
* validator discovery: use incremental updates for priority_group

* validator discovery: fix compilation

* validator discovery: remove Sync bound on Net

* "Update Substrate"

Co-authored-by: parity-processbot <>
2020-10-09 14:34:17 +00:00
Andronik Ordian d00bdfef08 chain-api subsystem: implement BlockHeader messsage (#1778)
* chain-api subsystem: implement BlockHeader messsage

* update the guide
2020-10-06 16:46:54 +02:00
Andronik Ordian abf76d27dc collator: fix a typo (#1788)
* collator: fix a typo

* collator: fix more typos

* collator: fix even more typos
2020-10-06 13:00:18 +00:00
ca89e3edbe NetworkBridge: validator (authorities) discovery api (#1699)
* stupid, but it compiles

* redo

* cleanup

* add ValidatorDiscovery to msgs

* sketch network bridge code

* ConnectToAuthorities instead of validators

* more stuff

* cleanup

* more stuff

* complete ConnectToAuthoritiesState

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Collator protocol subsystem (#1659)

* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* handle multiple in-flight connection requests

* handle cancelled requests

* Update node/core/runtime-api/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* redo it again

* more stuff

* redo it again

* update comments

* workaround Future is not Send

* fix trailing spaces

* clarify comments

* bridge: fix compilation in tests

* update more comments

* small fixes

* port collator protocol to new validator discovery api

* collator tests compile

* collator tests pass

* do not revoke a request when the stream receiver is closed

* make revoking opt-in

* fix is_fulfilled

* handle request revokation in collator

* tests

* wait for validator connections asyncronously

* fix compilation

* relabel my todos

* apply Fedor's patch

* resolve reconnection TODO

* resolve revoking TODO

* resolve channel capacity TODO

* resolve peer cloning TODO

* resolve peer disconnected TODO

* resolve PeerSet TODO

* wip tests

* more tests

* resolve Arc TODO

* rename pending to non_revoked

* one more test

* extract utility function into util crate

* fix compilation in tests

* Apply suggestions from code review

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* revert pin_project removal

* fix while let loop

* Revert "revert pin_project removal"

This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.

* fix compilation

* Update node/subsystem/src/messages.rs

* docs on pub items

* guide updates

* remove a TODO

* small guide update

* fix a typo

* link to the issue

* validator discovery: on_request docs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2020-10-06 13:34:57 +02:00
Andronik Ordian 579614d127 implement remaining subsystem metrics (#1770)
* overseer metrics: messages relayed

* provisioner metrics: cosmetic changes

* candidate selection metrics: cosmetic changes

* availability bitfields metrics

* availability distribution metrics

* PoV distribution metrics

* statement-distribution: small simplification

* statement-distribution: extract log target into a const

* statement-distribution: metrics

* address review nits
2020-10-01 12:08:03 +02:00
Andronik Ordian de05bec4d6 move Metrics to utils (#1765) 2020-09-29 11:42:20 +00:00
Andronik Ordian f9b15f654b provisioner tests: remove tokio from dev-dependencies (#1745)
* provisioner: remove tokio from dev-dependencies

* provisioner: use futures_timer instead
2020-09-23 20:21:03 +02:00
Andronik Ordianandparity-processbot bcafc81d9f update kvdb-rocksdb to 0.9.1 (companion for #6963) (#1647)
* update kvdb-rocksdb to 0.9.1

* update Cargo.lock

* "Update Substrate"

* resolve merge conflicts

* resolve merge conflicts t. 2

Co-authored-by: parity-processbot <>
2020-08-29 03:53:09 +00:00
Andronik Ordian e899a3f844 jobs: don't early exit when there are no jobs (#1621)
* jobs: don't early exit when there are no jobs

* utils: fix merged test

* utils: less verbose

* utils: add an assert subsystem is running

* utils: use TimeoutExt from test-helpers

* test-helpers: use TimeoutExt
2020-08-21 16:45:39 +02:00
Andronik Ordian 44354e717b utils: handle race condition gracefully (#1583)
* utils: handle race condition gracefully

* utils: add a test

* update Cargo.lock

* utils: remove a warning

* utils: init logger in tests

* utils: update the outdated comment

* util: wait for both subsystem and test_future to finish

* Revert "util: wait for both subsystem and test_future to finish"

This reverts commit 075b39242cd7200e3f4c24ea42f702c3124ce41c.
2020-08-21 15:30:18 +02:00
Andronik Ordian 8eecdd61b8 proposer: wait for a hash to be in the active-leaves set (#1616)
* overseer: add ExternalRequest to Event

* proposer: wait for the hash to be activated

* update comments

* overseer: handle unbounded growth of listeners map

* overseer: fix compilation

* overseer: clean up dead listeners

* overseer: cosmetic changes

* overseer: cosmetic changes t.2

* overseer: add debug_assertions

* overseer: fix formatting
2020-08-20 13:43:36 +00:00
e7ead40255 initial prometheus metrics (#1536)
* service-new: cosmetic changes

* overseer: draft of prometheus metrics

* metrics: update active_leaves metrics

* metrics: extract into functions

* metrics: resolve XXX

* metrics: it's ugly, but it works

* Bump Substrate

* metrics: move a bunch of code around

* Bumb substrate again

* metrics: fix a warning

* fix a warning in runtime

* metrics: statements signed

* metrics: statements impl RegisterMetrics

* metrics: refactor Metrics trait

* metrics: add Metrics assoc type to JobTrait

* metrics: move Metrics trait to util

* metrics: fix overseer

* metrics: fix backing

* metrics: fix candidate validation

* metrics: derive Default

* metrics: docs

* metrics: add stubs for other subsystems

* metrics: add more stubs and fix compilation

* metrics: fix doctest

* metrics: move to subsystem

* metrics: fix candidate validation

* metrics: bitfield signing

* metrics: av store

* metrics: chain API

* metrics: runtime API

* metrics: stub for avad

* metrics: candidates seconded

* metrics: ok I gave up

* metrics: provisioner

* metrics: remove a clone by requiring Metrics: Sync

* metrics: YAGNI

* metrics: remove another TODO

* metrics: for later

* metrics: add parachain_ prefix

* metrics: s/signed_statement/signed_statements

* utils: add a comment for job metrics

* metrics: address review comments

* metrics: oops

* metrics: make sure to save files before commit 😅

* use _total suffix for requests metrics

Co-authored-by: Max Inden <mail@max-inden.de>

* metrics: add tests for overseer

* update Cargo.lock

* overseer: add a test for CollationGeneration

* collation-generation: impl metrics

* collation-generation: use kebab-case for name

* collation-generation: add a constructor

Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: Ashley Ruglys <ashley.ruglys@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
2020-08-18 11:18:54 +02:00
Andronik Ordian dffefac54e move AssignmentKind and CoreAssigment to scheduler (#1571) 2020-08-17 19:26:13 +02:00
Andronik Ordian ed616e4b29 overseer: fix build (#1596)
* overseer: add a test for CollationGeneration

* fix build
2020-08-17 16:24:52 +00:00
Andronik Ordian ab1a513265 Add spawn_blocking to SubsystemContext (#1570)
* subsystem: add spawn_blocking to SubsystemContext

* candidate-validation: use spawn_blocking for exhaustive tasks
2020-08-17 12:26:32 +00:00
Andronik Ordian c5f7b4d17d overseer: add a test to ensure all subsystem receive msgs (#1590)
* overseer: add a test to ensure all subsystem receive msgs

* lol
2020-08-14 21:35:00 +02:00
Andronik OrdianandPeter Goodspeed-Niklaus 07a9d2de3f Integrate ChainApi with all messages (#1533)
* propagate chain_api subsystem through various locations

* add ChainApi to AllMessages

Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
2020-08-04 12:32:42 +00:00
Andronik Ordian 596ce826e8 Chain API subsystem (#1498)
* chain-api subsystem skeleton

* chain-api subsystem: draft impl

* chain-api subsystem: mock testclient

* chain-api subsystem: impl HeaderBacked for TestClient

* chain-api subsystem: impl basic tests

* chain-api subsystem: tiny guide

* chain-api subsystem: rename ChainApiRequestMessage to ChainApiMessage

* chain-api subsystem: add the page to the ToC

* chain-api subsystem: proper error type

* chain-api subsystem: impl ancestors request

* chain-api subsystem: tests for ancestors request

* guide: fix ancestor return type

* runtime-api subsystem: remove unused dep

* fix fmt

* fix outdated comment

* chain-api subsystem: s/format/to_string

* lower-case subsystem names

* chain-api subsystem: resolve Finalized todo

* chain-api subsystem: remove TODO

* extract request errors into a module

* remove caching TODO

* fix imports
2020-07-31 16:02:16 +00:00
Andronik Ordian ea834b12de Guide: add a diagram for Inclusion Pipeline & Approval Subsystem (#1457)
* guide: optimize diagrams for readability

* guide: inclusion subsystem diagram

* guide: rename Approval Subsystem for clarity

* guide: correct the approval subsystem

* guide: clarify that validator are selected from the same set

* guide: simplify secondary checkers labels
2020-07-31 14:28:45 +00:00
Andronik Ordian e7a420be0f Scheduler: handle timeouts for no group rotation (#1439)
* runtime: rename parachain_rotation_frequency to group_rotation_frequency

* scheduler: handle timeouts for no group rotation

* scheduler: apply fixes from code review

* scheduler: remove my comments
2020-07-21 20:08:01 -04:00
Andronik OrdianandRobert Habermeier 1ea2b27403 inclusion: split CandidatePendingAvailability according to the guide (#1413)
* inclusion: split PendingAvailability storage into descriptor and commitments

* inclusion: fix tests

* implementers-guide: update CandidatePendingAvailability type

* inclusion: simplify process_candidates a bit

* implementers-guide: more updates to the inclusion module

* inclusion: fix copy-paste errors in tests

* inclusion: revert some of the changes

* inclusion: lazy commitments loading and a test

* guide: revert enact_candidate changes

* inclusion: test process_bitfield for no commitments

* Grammar

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
2020-07-16 21:27:49 -04:00
Andronik Ordian 6957847b6b Integrate all (dummy) subsystems with the Overseer (#1374)
* overseer: introduce a utility typemap

* it's ugly but it compiles

* move DummySubsystem to subsystem crate

* fix tests fallout

* use a struct for all subsystems

* more tests fallout

* add missing pov_distribution subsystem

* remove unused imports and bounds

* fix minimal-example
2020-07-09 16:25:40 -04:00
Andronik Ordian d348fed02e deduplicate parity-util-mem (#926) 2020-03-23 08:57:15 +01:00