* Parachain improvements
- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes
* Remove unused functionality
* Return error from the runtime instead of printing it
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix test
* Revert "Update primitives/src/v1.rs"
This reverts commit 11fce2785acd1de481ca57815b8e18400f09fd52.
* Revert "Update primitives/src/v1.rs"
This reverts commit d6439fed4f954360c89fb1e12b73954902c76a41.
* Revert "Return error from the runtime instead of printing it"
This reverts commit cb4b5c0830ac516a6d54b2c24197e9354f2b98cb.
* Revert "Fix test"
This reverts commit 0c5fa1b5566d4cd3c55a55d485e707165ce7a59e.
* Update runtime/parachains/src/runtime_api_impl/v1.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* fix: ensure candidate validation gets code based on occupied core assumption
* guide: runtime API for historical validation code
* add historical runtime API
* integrate into runtime API subsystem
* remove blocked TODO
* fix service build: enable notifications protocol only under real overseer
* Update node/subsystem/src/messages.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* fix compilation
Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* UMP: Update the impl guide
* UMP: Incorporate XCM related changes into the guide
* UMP: Data structures and configuration
* UMP: Initial plumbing
* UMP: Data layout
* UMP: Acceptance criteria & enactment
* UMP: Fix dispatcher bug and add the test for it
* UMP: Constrain the maximum size of an UMP message
This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869
* Fix failing test due to misconfiguration
* Make the type of RelayDispatchQueueSize be more apparent in the guide
* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`
* convert spaces to tabs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update runtime/parachains/src/router/ump.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fixes bug that collator wasn't sending `Declare` message
* Set authority discovery config
* Fixes bug that collator wasn't sending `Declare` message
* Adds real overseer feature and makes the wasm_validation fail with a
proper error
* Adds README
* Remove debug stuff
* Add feature
* Make adder collator use the correct parent when building a new block
* start working on building the real overseer
Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.
* fill in AllSubsystems internal constructors
* replace fn make_metrics with Metrics::attempt_to_register
* update to account for #1740
* remove Metrics::register, rename Metrics::attempt_to_register
* add 'static bounds to real_overseer type params
* pass authority_discovery and network_service to real_overseer
It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.
* select a proper database configuration for the availability store db
* use subdirectory for av-store database path
* apply Basti's patch which avoids needing to parameterize everything on Block
* simplify path extraction
* get all tests to compile
* Fix Prometheus double-registry error
for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
err
}),
```
That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!
* authorities must have authority discovery, but not necessarily overseer handlers
* fix broken SpawnedSubsystem impls
detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.
This is not the final fix needed to get things working properly,
but it's a good start.
* use prometheus properly
Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.
We were using them wrong, though. This pattern was repeated in a
variety of places in the code:
```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```
The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.
This commit fixes that.
* get av-store subsystem to actually run properly and not die on first signal
* typo fix: incomming -> incoming
* don't disable authority discovery in test nodes
* Fix rococo-v1 missing session keys
* Update node/core/av-store/Cargo.toml
* try dummying out av-store on non-full-nodes
* overseer and subsystems are required only for full nodes
* Reduce the amount of warnings on browser target
* Fix two more warnings
* InclusionInherent should actually have an Inherent module on rococo
* Ancestry: don't return genesis' parent hash
* Update Cargo.lock
* fix broken test
* update test script: specify chainspec as script argument
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update node/service/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* node/service/src/lib: Return error via ? operator
* post-merge blues
* add is_collator flag
* prevent occasional av-store test panic
* simplify fix; expand application
* run authority_discovery in Role::Discover when collating
* distinguish between proposer closed channel errors
* add IsCollator enum, remove is_collator CLI flag
* improve formatting
* remove nop loop
* Fix some stuff
* Adds test parachain adder collator
* Add sudo to Rococo, change session length to 30 seconds and some renaming
* Update to the latest changes on master
* Some fixes
* Fix compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Review comments
* Downgrade transaction version
* Fixes
* MOARE
* Register notification protocols
* utils: remove unused error
* av-store: more resilient to some errors
* address review nits
* address more review nits
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Sergey Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Make sure validator discovery works with a delayed peer to validator mapping
Currently the implementation checks on connect of a peer if this peer is
a validator by asking the authority discovery. It can now happen that
the authority discovery is not yet aware that a given peer is an
authority. This can for example happen on start up of the node.
This pr changes the behavior, to make it possible to later associate a
peer to a validator id. Instead of just storing the connected
validators, we now store all connected peers with a vector of associated
validator ids. When we get a request to connect to a given given set of
validators, we start by checking the connected peers. If we didn't find
a validator id in the connected peers, we ask the authority discovery
for the peerid of a given authority id. When the returned peerid is part
of our connected peers set, we cache and return the authority id.
* Update node/network/bridge/Cargo.toml
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update node/network/bridge/src/validator_discovery.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update `Cargo.lock`
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* remove pending TODO after the DMP impl merge
* DMP: Update the impl guide
* DMP: Incorporate XCM related changes into the guide
This is the DMP related part of https://github.com/paritytech/polkadot/issues/1702
* DMP: data structures and plumbing
* DMP: Implement DMP logic in the router module
DMP: Integrate DMP parts into the inclusion module
* DMP: Introduce the max size limit for the size of a downward message
* DMP: Runtime API for accessing inbound messages
* OCD
Small clean ups
* DMP: fix the naming of the error
* DMP: add caution about a non-existent recipient
* start working on building the real overseer
Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.
* fill in AllSubsystems internal constructors
* replace fn make_metrics with Metrics::attempt_to_register
* update to account for #1740
* remove Metrics::register, rename Metrics::attempt_to_register
* add 'static bounds to real_overseer type params
* pass authority_discovery and network_service to real_overseer
It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.
* select a proper database configuration for the availability store db
* use subdirectory for av-store database path
* apply Basti's patch which avoids needing to parameterize everything on Block
* simplify path extraction
* get all tests to compile
* Fix Prometheus double-registry error
for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
err
}),
```
That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!
* authorities must have authority discovery, but not necessarily overseer handlers
* fix broken SpawnedSubsystem impls
detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.
This is not the final fix needed to get things working properly,
but it's a good start.
* use prometheus properly
Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.
We were using them wrong, though. This pattern was repeated in a
variety of places in the code:
```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```
The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.
This commit fixes that.
* get av-store subsystem to actually run properly and not die on first signal
* typo fix: incomming -> incoming
* don't disable authority discovery in test nodes
* Fix rococo-v1 missing session keys
* Update node/core/av-store/Cargo.toml
* try dummying out av-store on non-full-nodes
* overseer and subsystems are required only for full nodes
* Reduce the amount of warnings on browser target
* Fix two more warnings
* InclusionInherent should actually have an Inherent module on rococo
* Ancestry: don't return genesis' parent hash
* Update Cargo.lock
* fix broken test
* update test script: specify chainspec as script argument
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update node/service/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* node/service/src/lib: Return error via ? operator
* post-merge blues
* add is_collator flag
* prevent occasional av-store test panic
* simplify fix; expand application
* run authority_discovery in Role::Discover when collating
* distinguish between proposer closed channel errors
* add IsCollator enum, remove is_collator CLI flag
* improve formatting
* remove nop loop
* Fix some stuff
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
* runtime: annonate some types
* cargo update -p sp-io
* add wasm overwrites to Configuration in test-service
* rename overwrite to override
Co-authored-by: Andrew Plaza <andrew.plaza@parity.io>
* node/service/src/lib: Do not spawn authority discovery on sentries
The notion of sentry nodes has been deprecated (see [1] for details).
Support for sentry nodes in the `client/authority-discovery` module has
been removed.
This commit adjusts the instantiation of the authority discovery worker
accordingly, only spawning the module on authority nodes.
[1] https://github.com/paritytech/substrate/issues/6845
* "Update Substrate"
Co-authored-by: parity-processbot <>
* Improve and unify testing facilities
This improves the testing facilities by making the test client easier to
use. It also removes code that is not required for the test client.
Besides that it also moves the test service and test client under
`node/test`.
* Update Cargo.lock
* Update node/test/client/src/block_builder.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Remove explicit lifetime annotation
* Fix warnings and add extra `BlockBuilderExt`
Co-authored-by: Andronik Ordian <write@reusable.software>
* annoying whitespaces
* update guide
Add `CheckValidationOutputs` runtime api and also change the
candidate-validation stuff
* promote ValidationOutputs to global primitives
i.e. move it from node specific primitives to global v1 primitives. This
will be needed when we share it later in the runtime inclusion module
* refactor acceptance checks in the inclusion module
factor out the common code to share it during the block inclusion and
for the forthcoming CheckValidationOutputs runtime api.
Also note that the acceptance criteria was updated to incorporate checks
that exist now in candidate-validation
* plumb the runtime api outside
* extract validation_data from ValidationOutputs
* use runtime-api to check validation outputs
apart from that refactor, update docs and tidy a bit
* Update the maxium code size
This is to fix a test that performs an upgrade.
* Initial commit
* Move tests to separate module
* Move timestamps to the newtype
* Change idx name
* Use Duration for consts and update chunk records
* Ordering::Equal
* Update node/core/av-store/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* put_ methods do the array sorting
* Fix get_block_number
* Change StoreChunk message type and relay parent method
* Add chunk tests
* Fix block number computation for StoreChunk
* Duration instead of u64 everywhere
* Add a clarifying comment
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Make register/deregister parachain dispatchables
This makes `register_parachain` and `deregister_parachain` of
`Registrar` dispatchables.
Besides that it brings back the functionality of the test node to
register a parachain.
* Fix tests
* PR review comments
This reverts commit a7b6c91f80.
The authority discovery module was initially enabled by default on
validator and sentry nodes with commit 59f5eb4. This change was later on
reverted in a7b6c91. With this commit the authority discovery module is
again enabled by default.
* Remove old service, 3rd try
i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"
This reverts commit 9a0f08bfe1.
Closes#1757.
We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602
As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.
* paras trait now has an Origin type
* initial work running a two node local net
* use the right incantations so the nodes produce blocks together
* improve internal documentation
Co-authored-by: Bastian Köcher <git@kchr.de>
* Make `AllSubsystems` usage easier in tests
This makes the usage of `AllSubsystems` easier in tests by introducing
new methods.
- `dummy` initializes `AllSubsystems` with all systems set to dummy
- `replace_*` to replace any subsystem
Besides that this pr adds a `ForwardSubsystem` that is also useful for
tests. This subsystem will forward all incoming messages to the given channel.
* Update node/overseer/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update node/subsystem/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update node/subsystem/src/lib.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Move ForwardSubsystem and add a test
* Break some lines
Co-authored-by: Andronik Ordian <write@reusable.software>
* Advertise to already connected validators
* Merge the loops and check the view
* Extend a test to capture new logic
* Fix a comment
* Update node/network/collator-protocol/src/collator_side.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update comment
Co-authored-by: Andronik Ordian <write@reusable.software>
* stupid, but it compiles
* redo
* cleanup
* add ValidatorDiscovery to msgs
* sketch network bridge code
* ConnectToAuthorities instead of validators
* more stuff
* cleanup
* more stuff
* complete ConnectToAuthoritiesState
* Update node/network/bridge/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Collator protocol subsystem (#1659)
* WIP
* The initial implementation of the collator side.
* Improve comments
* Multiple collation requests
* Add more tests and comments to validator side
* Add comments, remove dead code
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix build after suggested changes
* Also connect to the next validator group
* Remove a Future impl and move TimeoutExt to util
* Minor nits
* Fix build
* Change FetchCollations back to FetchCollation
* Try this
* Final fixes
* Fix build
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* handle multiple in-flight connection requests
* handle cancelled requests
* Update node/core/runtime-api/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* redo it again
* more stuff
* redo it again
* update comments
* workaround Future is not Send
* fix trailing spaces
* clarify comments
* bridge: fix compilation in tests
* update more comments
* small fixes
* port collator protocol to new validator discovery api
* collator tests compile
* collator tests pass
* do not revoke a request when the stream receiver is closed
* make revoking opt-in
* fix is_fulfilled
* handle request revokation in collator
* tests
* wait for validator connections asyncronously
* fix compilation
* relabel my todos
* apply Fedor's patch
* resolve reconnection TODO
* resolve revoking TODO
* resolve channel capacity TODO
* resolve peer cloning TODO
* resolve peer disconnected TODO
* resolve PeerSet TODO
* wip tests
* more tests
* resolve Arc TODO
* rename pending to non_revoked
* one more test
* extract utility function into util crate
* fix compilation in tests
* Apply suggestions from code review
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
* revert pin_project removal
* fix while let loop
* Revert "revert pin_project removal"
This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.
* fix compilation
* Update node/subsystem/src/messages.rs
* docs on pub items
* guide updates
* remove a TODO
* small guide update
* fix a typo
* link to the issue
* validator discovery: on_request docs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
This pr changes the collator interface function to return an optional
collation instead of a collation. This is required as the parachain
itself can fail to generate a valid collation for various reason. Now if
the collation fails it will return `None`.
Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
and removes some whitespaces.
Bastian Köcher