pezkuwichain/pezkuwi-subxt
1
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-10-25 21:25:03 +00:00
Code Releases Activity
1,466 Commits 132 Branches 288 Tags
cd58c02cd3333cb4f0aa7f9aeebfc1381c2fbbb6
Commit Graph
47 Commits
Author SHA1 Message Date
Peter Goodspeed-Niklaus cd58c02cd3 Add metrics timing message passing from OverseerSubsystemContext to Overseer::route_message (#2201) 
* add timing setup to OverseerSubsystemContext

* figure out how to initialize the rng

* attach a timer to a portion of the messages traveling to the Overseer

This timer only exists / logs a fraction of the time (configurable
by `MESSAGE_TIMER_METRIC_CAPTURE_RATE`). When it exists, it tracks
the span between the `OverSubsystemContext` receiving the message
and its receipt in `Overseer::run`.

* propagate message timing to the start of route_message

This should be more accurate; it ensures that the timer runs
at least as long as that function. As `route_message` is async,
it may not actually run for some time after it is called (or ever).

* fix failing test

* rand_chacha apparently implicitly has getrandom feature

* change rng initialization

The previous impl using `from_entropy` depends on the `getrandom`
crate, which uses the system entropy source, and which does not
work on `wasm32-unknown-unknown` because it wants to fall back to
a JS implementation which we can't assume exists.

This impl depends only on `rand::thread_rng`, which has no documentation
stating that it's similarly limited.

* remove randomness in favor of a simpler 1 of N procedure

This deserves a bit of explanation, as the motivating issue explicitly
requested randomness. In short, it's hard to get randomness to compile
for `wasm32-unknown-unknown` because that is explicitly intended to be
as deterministic as practical. Additionally, even though it would never
be used for consensus purposes, it still felt offputting to intentionally
introduce randomness into a node's operations. Except, it wasn't really
random, either: it was a deterministic PRNG varying only in its state,
and getting the state to work right for that target would have required
initializing from a constant.

Given that it was a deterministic sequence anyway, it seemed much simpler
and more explicit to simply select one of each N messages instead of
attempting any kind of realistic randomness.

* reinstate randomness for better statistical properties

This partially reverts commit 0ab8594c328b3f9ce1f696fe405556d4000630e9.

`oorandom` is much lighter than the previous `rand`-based implementation,
which makes this easier to work with.

This implementation gives each subsystem and each child RNG a distinct
increment, which should ensure they produce distinct streams of values.
 2021-01-06 14:25:04 +01:00
Bastian KöcherandAndronik Ordian 5be092894e Add one Jaeger span per relay parent (#2196) 
* Add one Jaeger span per relay parent

This adds one Jaeger span per relay parent, instead of always creating
new spans per relay parent. This should improve the UI view, because
subsystems are now grouped below one common span.

* Fix doc tests

* Replace `PerLeaveSpan` to `PerLeafSpan`

* More renaming

* Moare

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Skip the spans

* Increase `spec_version`

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-01-05 15:09:25 +01:00
Robert Habermeier e95be77eb6 overseer: observe stalled subsystems and shut down (#2148) 
* overseer: observe stalled subsystems and shut down

* notify on send_message failure as well
 2020-12-20 20:30:02 +00:00
Robert Klotzner 9ebb9015d3 Some typos and misspellings in docs I found, during my studies. (#2144) 
* Fix stale link to overseer docs

* Some typos and mispellings in docs/comments

I found during studying how Polkadot works.
 2020-12-18 18:31:43 -05:00
Andronik Ordian 3f5156e866 refactor View to include finalized_number (#2128) 
* refactor View to include finalized_number

* guide: update the NetworkBridge on BlockFinalized

* av-store: fix the tests

* actually fix tests

* grumbles

* ignore macro doctest

* use Hash::repeat_bytes more consistently

* broadcast empty leaves updates as well

* fix issuing view updates on empty leaves updates
 2020-12-17 12:50:58 -05:00
Bastian Köcher d7047578e9 Fix tests on master (#2080) 
Because of a bug in the test script, we didn't stopped CI when the main
tests are failed.
 2020-12-07 14:47:39 +00:00
Bastian Köcher 4ce744818c Some code cleanup in overseer (#2008) 
* Some code cleanup in overseer

- Switches to select! in the overseer run loop to be more fair about
message processing between the different sources.
- Added a check to only send `ActiveLeaves` if the update actually
contains any data.

* Move the check

* Restore old behavior

* Simplify message sending and signal sending to subsystems

* Update node/subsystem/src/lib.rs
 2020-11-25 09:27:50 +00:00
dependabot[bot] 748cbfd820 Bump tracing from 0.1.21 to 0.1.22 (#2001) 
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.21 to 0.1.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.21...tracing-0.1.22)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2020-11-24 09:44:27 +01:00
Andronik OrdianandPeter Goodspeed-Niklaus 69b103b1d5 overseer: send_msg should not return an error (#1995) 
* send_message should not return an error

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* s/send_logging_error/send_and_log_error

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-23 12:42:14 +01:00
dependabot[bot] 8cdb063f72 Bump async-trait from 0.1.41 to 0.1.42 (#1996) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.41 to 0.1.42.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.41...0.1.42)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2020-11-23 08:28:21 +00:00
Peter Goodspeed-Niklaus e49989971d Add tracing support to node (#1940) 
* drop in tracing to replace log

* add structured logging to trace messages

* add structured logging to debug messages

* add structured logging to info messages

* add structured logging to warn messages

* add structured logging to error messages

* normalize spacing and Display vs Debug

* add instrumentation to the various 'fn run'

* use explicit tracing module throughout

* fix availability distribution test

* don't double-print errors

* remove further redundancy from logs

* fix test errors

* fix more test errors

* remove unused kv_log_macro

* fix unused variable

* add tracing spans to collation generation

* add tracing spans to av-store

* add tracing spans to backing

* add tracing spans to bitfield-signing

* add tracing spans to candidate-selection

* add tracing spans to candidate-validation

* add tracing spans to chain-api

* add tracing spans to provisioner

* add tracing spans to runtime-api

* add tracing spans to availability-distribution

* add tracing spans to bitfield-distribution

* add tracing spans to network-bridge

* add tracing spans to collator-protocol

* add tracing spans to pov-distribution

* add tracing spans to statement-distribution

* add tracing spans to overseer

* cleanup
 2020-11-20 12:02:04 +01:00
Andronik Ordian 0a8a607a58 update most of the dependencies (#1946) 
* update tiny-keccak to 0.2

* update deps except bitvec and shared_memory

* fix some warning after futures upgrade

* remove useless package rename caused by bug in cargo-upgrade

* revert parity-util-mem *

* remove unused import

* cargo update

* remove all renames on parity-scale-codec

* remove the leftovers

* remove unused dep
 2020-11-17 11:16:31 +01:00
Bastian KöcherandPeter Goodspeed-Niklaus 640264f38b Make CandidateHash a real type (#1916) 
* Make `CandidateHash` a real type

This pr adds a new type `CandidateHash` that is used instead of the
opaque `Hash` type. This helps to ensure on the type system level that
we are passing the correct types.

This pr also fixes wrong usage of `relay_parent` as `candidate_hash`
when communicating with the av storage.

* Update core-primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Wrap the lines

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-05 16:28:45 +01:00
Bastian KöcherPeter Goodspeed-NiklausSergei Shulepov
 		002e1141a8 Parachain improvements (#1905) 
* Parachain improvements

- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes

* Remove unused functionality

* Return error from the runtime instead of printing it

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix test

* Revert "Update primitives/src/v1.rs"

This reverts commit 11fce2785acd1de481ca57815b8e18400f09fd52.

* Revert "Update primitives/src/v1.rs"

This reverts commit d6439fed4f954360c89fb1e12b73954902c76a41.

* Revert "Return error from the runtime instead of printing it"

This reverts commit cb4b5c0830ac516a6d54b2c24197e9354f2b98cb.

* Revert "Fix test"

This reverts commit 0c5fa1b5566d4cd3c55a55d485e707165ce7a59e.

* Update runtime/parachains/src/runtime_api_impl/v1.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
 2020-11-03 11:22:38 +00:00
Fedor Sakharov 935fcd1666 Change SpawnedSubsystem type to log subsystem errors (#1878) 
* Change SpawnedSubsystem type to log subsystem errors

* Remove clone
 2020-10-28 20:57:06 +01:00
Peter Goodspeed-NiklausBastian KöcherAndronik OrdianBastian KöcherFedor SakharovRobert HabermeierMax Inden
 		1a25c41277 start working on building the real overseer (#1795) 
* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
 2020-10-28 10:26:50 +00:00
Bernhard Schuster f345123748 introduce errors with info (#1834) 2020-10-27 08:10:03 +01:00
Bastian KöcherandAndronik Ordian f2d7b6f5ac Make AllSubsystems usage easier in tests (#1794) 
* Make `AllSubsystems` usage easier in tests

This makes the usage of `AllSubsystems` easier in tests by introducing
new methods.

- `dummy` initializes `AllSubsystems` with all systems set to dummy
- `replace_*` to replace any subsystem

Besides that this pr adds a `ForwardSubsystem` that is also useful for
tests. This subsystem will forward all incoming messages to the given channel.

* Update node/overseer/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Move ForwardSubsystem and add a test

* Break some lines

Co-authored-by: Andronik Ordian <write@reusable.software>
 2020-10-08 11:27:19 +02:00
Bastian Köcher 2b798f8571 Use Pin<Box<>> for collation future (#1792) 2020-10-07 11:29:00 +00:00
Bastian Köcher 325c24580e Pass relay parent as argument when collating (#1789) 
This pr changes the collation function to also pass the current relay
parent the parachain block should be build on.
 2020-10-06 18:37:19 +00:00
Bastian Köcher a4662104db Make collation an optional return (#1787) 
This pr changes the collator interface function to return an optional
collation instead of a collation. This is required as the parachain
itself can fail to generate a valid collation for various reason. Now if
the collation fails it will return `None`.

Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
and removes some whitespaces.
 2020-10-06 11:57:10 +02:00
Bastian Köcher 5d8ae8d024 Derive From for AllMessages and simplify send_msg (#1774) 2020-10-01 16:22:15 +00:00
Andronik Ordian 579614d127 implement remaining subsystem metrics (#1770) 
* overseer metrics: messages relayed

* provisioner metrics: cosmetic changes

* candidate selection metrics: cosmetic changes

* availability bitfields metrics

* availability distribution metrics

* PoV distribution metrics

* statement-distribution: small simplification

* statement-distribution: extract log target into a const

* statement-distribution: metrics

* address review nits
 2020-10-01 12:08:03 +02:00
Andronik Ordian de05bec4d6 move Metrics to utils (#1765) 2020-09-29 11:42:20 +00:00
Andronik Ordian 8eecdd61b8 proposer: wait for a hash to be in the active-leaves set (#1616) 
* overseer: add ExternalRequest to Event

* proposer: wait for the hash to be activated

* update comments

* overseer: handle unbounded growth of listeners map

* overseer: fix compilation

* overseer: clean up dead listeners

* overseer: cosmetic changes

* overseer: cosmetic changes t.2

* overseer: add debug_assertions

* overseer: fix formatting
 2020-08-20 13:43:36 +00:00
Robert HabermeierandPeter Goodspeed-Niklaus 262574fc49 Implement validation data refactor (#1585) 
* update primitives

* correct parent_head field

* make hrmp field pub

* refactor validation data: runtime

* refactor validation data: messages

* add arguments to full_validation_data runtime API

* port runtime API

* mostly port over candidate validation

* remove some parameters from ValidationParams

* guide: update candidate validation

* update candidate outputs

* update ValidationOutputs in primitives

* port over candidate validation

* add a new test for no-transient behavior

* update util runtime API wrappers

* candidate backing

* fix missing imports

* change some fields of validation data around

* runtime API impl

* update candidate validation

* fix backing tests

* grumbles from review

* fix av-store tests

* fix some more crates

* fix provisioner tests

* fix availability distribution tests

* port collation-generation to new validation data

* fix overseer tests

* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-08-18 14:41:40 +02:00
Andronik OrdianMax IndenGav WoodAshley Ruglys
 		e7ead40255 initial prometheus metrics (#1536) 
* service-new: cosmetic changes

* overseer: draft of prometheus metrics

* metrics: update active_leaves metrics

* metrics: extract into functions

* metrics: resolve XXX

* metrics: it's ugly, but it works

* Bump Substrate

* metrics: move a bunch of code around

* Bumb substrate again

* metrics: fix a warning

* fix a warning in runtime

* metrics: statements signed

* metrics: statements impl RegisterMetrics

* metrics: refactor Metrics trait

* metrics: add Metrics assoc type to JobTrait

* metrics: move Metrics trait to util

* metrics: fix overseer

* metrics: fix backing

* metrics: fix candidate validation

* metrics: derive Default

* metrics: docs

* metrics: add stubs for other subsystems

* metrics: add more stubs and fix compilation

* metrics: fix doctest

* metrics: move to subsystem

* metrics: fix candidate validation

* metrics: bitfield signing

* metrics: av store

* metrics: chain API

* metrics: runtime API

* metrics: stub for avad

* metrics: candidates seconded

* metrics: ok I gave up

* metrics: provisioner

* metrics: remove a clone by requiring Metrics: Sync

* metrics: YAGNI

* metrics: remove another TODO

* metrics: for later

* metrics: add parachain_ prefix

* metrics: s/signed_statement/signed_statements

* utils: add a comment for job metrics

* metrics: address review comments

* metrics: oops

* metrics: make sure to save files before commit 😅

* use _total suffix for requests metrics

Co-authored-by: Max Inden <mail@max-inden.de>

* metrics: add tests for overseer

* update Cargo.lock

* overseer: add a test for CollationGeneration

* collation-generation: impl metrics

* collation-generation: use kebab-case for name

* collation-generation: add a constructor

Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: Ashley Ruglys <ashley.ruglys@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
 2020-08-18 11:18:54 +02:00
Andronik Ordian ed616e4b29 overseer: fix build (#1596) 
* overseer: add a test for CollationGeneration

* fix build
 2020-08-17 16:24:52 +00:00
Peter Goodspeed-Niklaus 54bec3bfc0 implement collation generation subsystem (#1557) 
* start sketching out a collation generation subsystem

* invent a basic strategy for double initialization

* clean up warnings

* impl util requests from runtime assuming a context instead of a FromJob sender

* implement collation generation algorithm from guide

* update AllMessages in tests

* fix trivial review comments

* remove another redundant declaration from merge

* filter availability cores by para_id

* handle new activations each in their own async task

* update guide according to the actual current implementation

* add initialization to guide

* add general-purpose subsystem_test_harness helper

* write first handle_new_activations test

* add test that handle_new_activations filters local_validation_data requests

* add (failing) test of collation distribution message sending

* rustfmt

* broken: work on fixing sender test

Unfortunately, for reasons that are not yet clear, despite the public key
and checked data being identical, the signer is not producing an identical
signature. This commit produces this output (among more):

signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]

* fix broken test

* collation function returns commitments hash

It doesn't look like we use the actual commitments data anywhere, and
it's not obvious if there are any fields of `CandidateCommitments`
not available to the collator, so this commit just assigns them the
entire responsibility of generating the hash.

* add missing overseer impls

* calculating erasure coding is polkadot's responsibility, not cumulus

* concurrentize per-relay_parent requests
 2020-08-17 14:27:37 +02:00
Andronik Ordian ab1a513265 Add spawn_blocking to SubsystemContext (#1570) 
* subsystem: add spawn_blocking to SubsystemContext

* candidate-validation: use spawn_blocking for exhaustive tasks
 2020-08-17 12:26:32 +00:00
Andronik Ordian c5f7b4d17d overseer: add a test to ensure all subsystem receive msgs (#1590) 
* overseer: add a test to ensure all subsystem receive msgs

* lol
 2020-08-14 21:35:00 +02:00
Robert Habermeier a6b1d91d6e Network bridge refactoring impl (#1537) 
* update networking types

* port over overseer-protocol message types

* Add the collation protocol to network bridge

* message sending

* stub for ConnectToValidators

* add some helper traits and methods to protocol types

* add collator protocol message

* leaves-updating

* peer connection and disconnection

* add utilities for dispatching multiple events

* implement message handling

* add an observedrole enum with equality and no sentry nodes

* derive partial-eq on network bridge event

* add PartialEq impls for network message types

* add Into implementation for observedrole

* port over existing network bridge tests

* add some more tests

* port bitfield distribution

* port over bitfield distribution tests

* add codec indices

* port PoV distribution

* port over PoV distribution tests

* port over statement distribution

* port over statement distribution tests

* update overseer and service-new

* address review comments

* port availability distribution

* port over availability distribution tests
 2020-08-12 11:16:28 +00:00
Andronik OrdianandPeter Goodspeed-Niklaus 07a9d2de3f Integrate ChainApi with all messages (#1533) 
* propagate chain_api subsystem through various locations

* add ChainApi to AllMessages

Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
 2020-08-04 12:32:42 +00:00
Fedor Sakharov 32a20a178c Availability store subsystem (#1404) 
* Initial commit

* WIP

* Make atomic transactions

* Remove pruning code

* Fix build and add a Nop to bridge

* Fixes from review

* Move config struct around for clarity

* Rename constructor and warn on missing docs

* Fix a test and rename a message

* Fix some more reviews

* Obviously failed to rebase cleanly
 2020-07-27 14:13:02 +03:00
Peter Goodspeed-Niklaus 106bd929ce add ActiveLeavesUpdate, remove StartWork, StopWork (#1458) 
* add ActiveLeavesUpdate, remove StartWork, StopWork

* replace StartWork, StopWork in subsystem crate tests

* mechanically update OverseerSignal in other modules

* convert overseer to take advantage of new multi-hash update abilities

Note: this does not yet convert the tests; some of the tests now freeze:

test tests::overseer_start_stop_works ... test tests::overseer_start_stop_works has been running for over 60 seconds
test tests::overseer_finalize_works ... test tests::overseer_finalize_works has been running for over 60 seconds

* fix broken overseer tests

* manually impl PartialEq for ActiveLeavesUpdate, rm trait Equivalent

This cleans up the code a bit and makes it easier in the future to
do the right thing when comparing ALUs.

* use target in all network bridge logging

* reduce spamming of  and
 2020-07-27 10:39:52 +02:00
Bastian Köcherandparity-processbot fa598f176b Companion for #6726 (#1469) 
* Companion for #6726

* Spaces

* 'Update substrate'

Co-authored-by: parity-processbot <>
 2020-07-26 13:16:09 +00:00
Peter Goodspeed-NiklausandAndronik Ordian ba4bfa4dd0 implement bitfield signing subsystem (#1364) 
* update guide to reduce confusion and TODOs

* work from previous bitfield signing effort

There were large merge issues with the old bitfield signing PR, so
we're just copying all the work from that onto this and restarting.

Much of the existing work will be discarded because we now have better
tools available, but that's fine.

* start rewriting bitfield signing in terms of the util module

* implement construct_availability_bitvec

It's not an ideal implementation--we can make it much more concurrent--
but at least it compiles.

* implement the unimplemented portions of bitfield signing

* get core availability concurrently, not sequentially

* use sp-std instead of std for a parachain item

* resolve type inference failure caused by multiple From impls

* handle bitfield signing subsystem & Allmessages variant in overseer

* fix more multi-From inference issues

* more concisely handle overflow

Co-authored-by: Andronik Ordian <write@reusable.software>

* Revert "resolve type inference failure caused by multiple From impls"

This reverts commit 7fc77805de5e5074a1b01037f8d4e3919e03e0e1.

* Revert "fix more multi-From inference issues"

This reverts commit f14ffe589e20d664d8a900ed62f68b6fb844a514.

* impl From<i32> for ParaId

* handle another instance of AllSubsystems

* improve consistency when returning existing options

Co-authored-by: Andronik Ordian <write@reusable.software>
 2020-07-23 16:05:48 +02:00
Fedor Sakharov 5624bd8bf4 Use SpawnNamed instead of Spawn in Overseer (#1430) 
* Use SpawnNamed instead of Spawn in Overseer

* reexport SpawnNamed and fix doc tests

* Fix deps
 2020-07-17 20:04:02 +03:00
Robert Habermeier 3b13cd9a85 Refactor primitives (#1383) 
* create a v1 primitives module

* Improve guide on availability types

* punctuate

* new parachains runtime uses new primitives

* tests of new runtime now use new primitives

* add ErasureChunk to guide

* export erasure chunk from v1 primitives

* subsystem crate uses v1 primitives

* node-primitives uses new v1 primitives

* port overseer to new primitives

* new-proposer uses v1 primitives (no ParachainHost anymore)

* fix no-std compilation for primitives

* service-new uses v1 primitives

* network-bridge uses new primitives

* statement distribution uses v1 primitives

* PoV distribution uses v1 primitives; add PoV::hash fn

* move parachain to v0

* remove inclusion_inherent module and place into v1

* remove everything from primitives crate root

* remove some unused old types from v0 primitives

* point everything else at primitives::v0

* squanch some warns up

* add RuntimeDebug import to no-std as well

* port over statement-table and validation

* fix final errors in validation and node-primitives

* add dummy Ord impl to committed candidate receipt

* guide: update CandidateValidationMessage

* add primitive for validationoutputs

* expand CandidateValidationMessage further

* bikeshed

* add some impls to omitted-validation-data and available-data

* expand CandidateValidationMessage

* make erasure-coding generic over v1/v0

* update usages of erasure-coding

* implement commitments.hash()

* use Arc<Pov> for CandidateValidation

* improve new erasure-coding method names

* fix up candidate backing

* update docs a bit

* fix most tests and add short-circuiting to make_pov_available

* fix remainder of candidate backing tests

* squanching warns

* squanch it up

* some fallout

* overseer fallout

* free from polkadot-test-service hell
 2020-07-09 21:23:03 -04:00
Andronik Ordian 6957847b6b Integrate all (dummy) subsystems with the Overseer (#1374) 
* overseer: introduce a utility typemap

* it's ugly but it compiles

* move DummySubsystem to subsystem crate

* fix tests fallout

* use a struct for all subsystems

* more tests fallout

* add missing pov_distribution subsystem

* remove unused imports and bounds

* fix minimal-example
 2020-07-09 16:25:40 -04:00
Fedor Sakharov c119627835 CandidateBackingSubsystem (#1312) 
* Updates guide for CandidateBacking

* Move assignment types to primitives

* Initial implementation.

* More functionality

* use assert_matches

* Changes to report misbehaviors

* Some fixes after a review

* Remove a blank line

* Update guide and some types

* Adds run_job function

* Some comments and refactorings

* Fix review

* Remove warnings

* Use summary in kicking off validation

* Parallelize requests

* Validation provides local and global validation params

* Test issued validity tracking

* Nits from review
 2020-07-09 16:23:58 -04:00
Robert Habermeier 42bd096413 I am dumb and can't spell (#1366) 
* rename implementor's guide to implementer's guide

* fix typos in more places
 2020-07-07 10:10:36 -04:00
Robert Habermeier 2a3e607d14 Subsystem::start takes self by-value (#1325) 
* Subsystem::start takes self by-value

* fix doc-test compilation
 2020-06-30 15:16:37 -04:00
Robert HabermeierandPeter Goodspeed-Niklaus d16e7485d4 Implement Network Bridge (#1280) 
* network bridge skeleton

* move some primitives around and add debug impls

* protocol registration glue & abstract network interface

* add send_msgs to subsystemctx

* select logic

* transform different events into actions and handle

* implement remaining network bridge state machine

* start test skeleton

* make network methods asynchronous

* extract subsystem out to subsystem crate

* port over overseer to subsystem context trait

* fix minimal example

* fix overseer doc test

* update network-bridge crate

* write a subsystem test-helpers crate

* write a network test helper for network-bridge

* set up (broken) view test

* Revamp network to be more async-friendly and not require Sync

* fix spacing

* fix test compilation

* insert side-channel for actions

* Add some more message types to AllMessages

* introduce a test harness

* add some tests

* ensure service compiles and passes tests

* fix typo

* fix service-new compilation

* Subsystem test helpers send messages synchronously

* remove smelly action inspector

* remove superfluous let binding

* fix warnings

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* fix compilation

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-06-30 11:21:40 -04:00
Robert HabermeierPeter Goodspeed-NiklausFedor Sakharov
 		c226c4403d Add some message types from subsystem definitions (#1265) 
* introduce polkadot-node-primitives

* guide: change statement distribution message types

* guide: remove variant from `CandidateSelectionMessage`

* add a few more message types

* add TODOs

* Almost all messages

* NewBackedCandidate notification

* Formatting

* Use AttestedCandidate as BackedCandidate

* Update node/primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix the tests

* Bring in types from #1242

* Adds network bridge messages

* More message types from doc

* use fn pointer type

* Fixes from the review

* Add missing Runtime subsystem message

* rename to CandidateValidationMessage and fix tests

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-06-17 19:49:17 -04:00
Fedor SakharovandRobert Habermeier d7e17fc612 New service initial commit (#1234) 
* New service initial commit

* More separation of the new and old services

* Fix review comments

* Adds polkadot.json

* Fix browser build

* Remove unused import

* Update node/service/src/lib.rs

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* Remove duplicate json files

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
 2020-06-11 15:26:15 -04:00
Robert Habermeier 9d5eae6ea3 establish new node folder for overseer, messages, and subsystems (#1200) 
* establish new `node` folder for overseer, messages, and subsystems

* extract message types from overseer crate

* remove doc links
 2020-06-05 10:48:35 -04:00