Commit Graph
77 Commits
Author SHA1 Message Date
Sebastian Kunert e1e0381380 Expose prometheus metrics for minimal-relay-chain node in collators (#1942)
Until now prometheus metrics were not exposed by the minimal relay chain
node.
Also slight cleanups.
2023-10-20 00:47:54 +02:00
Sebastian Kunert 21b32849db Cumulus: Allow aura to use initialized collation request receiver (#1911)
When launching our [small
network](https://github.com/paritytech/polkadot-sdk/blob/master/cumulus/zombienet/examples/small_network.toml)
for testing the node was crashing here shortly after launch:

https://github.com/paritytech/polkadot-sdk/blob/5cdd819ed295645958afd9d937d989978fd0c84e/polkadot/node/collation-generation/src/lib.rs#L140

After changes in #1788 for the asset hub collator we are waiting for
blocks of the shell runtime to pass before we initialize aura. However,
this means that we attempted to initialize the collation related relay
chain subsystems twice, leading to the error.

I modified Aura to let it optionally take an already initialized stream
of collation requests.
2023-10-19 10:07:29 +02:00
Sebastian Kunert d21113c13f Delete full db directory with purge-chain subcommand (#1786)
Closes #1767 

Until now the `purge-chain` command would only remove the `full`
subfolder of the db folder. However there is also the `parachains` db
that currently remains and can cause problems on node restart.

Example wiht old code:
```
polkadot purge-chain --database paritydb --base-path /tmp/some-folder
Are you sure to remove "/tmp/some-folder/chains/polkadot/paritydb/full"? [y/N]: y
"/tmp/some-folder/chains/polkadot/paritydb/full" removed.
```
In this case `/tmp/some-folder/chains/polkadot/paritydb/parachains`
would remain and might cause problem on node restart because of version
conflicts as described in #1767. After this PR the whole
`/tmp/some-folder/chains/polkadot/paritydb` folder will be deleted.
2023-10-05 18:44:03 +02:00
Sebastian Kunert 9485b0b492 Point documentation links to monorepo (#1741) 2023-09-29 22:33:26 +02:00
Sebastian KunertandBastian Köcher 4902db2198 Use Extensions to register offchain worker custom extensions (#1719)
Closes #1671 

Adds a `type_id` function to the `Extension` trait, allowing to properly store an retrieve
boxed `Extensions`.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-09-29 09:37:51 +02:00
Sebastian Kunert fb0fd3e624 Use correct target dir in polkadot readme (#1643)
Closes #1372 

This fixes the target directories in the polkadot readme. I verified
manually the Ubuntu based install works on a fresh cloud machine. Build
from source also worked as described.

Since #1304 landed `--dev` also works again (not on v1.1.0 though), so I
would consider the install instructions fixed.
2023-09-22 12:19:09 +02:00
817c4503db Allow integrated relay chain light client (#2270)
* Add embedded light client to cli

* Prepare for light-client-worker

* First working version

* Clean up

* Remove unwanted logs

* Simplify subscription code

* Let jsonrpsee handle rpc management

* Simplify implementation

* Reorganize crate structure

* Use relay chain arg chainspec for light-client

* Clean up command line

* Add light client worker file

* Use smoldot master to avoid wasmtime conflict

* Remove sleep

* Improve naming of cli option

* Remove conflict with `validator`

* Improve docs

* Update smoldot, remove unwanted change

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Disable collation

* Reviewer comments

* Update smoldot and tokio-platform

* Update smoldot

* Update smoldot

* Adjust to new version

* Patch substrate

* Use constants

* Add readme entry, improve zombienet tests

* Apply suggestions from code review

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Make execution mode an enum

* Update smoldot, remove substrate patch

* Update client/relay-chain-rpc-interface/src/rpc_client.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Reduce duplicate code

* Update smoldot

* Update smoldot

* Fix build

* Update smoldot

* Make platform compile

* Clean up dependencies

* Use crates.io instead of github for smoldot

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Docs

* Improve docs

* Remove `RpcFrontend`

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-08-22 15:53:32 +02:00
Sebastian Kunertandparity-processbot 3d1b31a775 Companion for substrate#14776 (#3038)
* Bump

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>
2023-08-21 15:44:50 +00:00
Sebastian Kunertandparity-processbot e6451b7232 Companion for substrate#14776 (#7648)
* Bump dalek

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-08-21 15:02:13 +00:00
5b5942ef60 Use trie-cache for validate_block (#2813)
* Simple cache

* Fix node insertion

* Switch to hashbrown hashmap

* Remove unused phantomdata

* Return error when fetch_node fails

* Remove cargo patches

* Move trie cache to extra module

* Add ReadOnceBackend

* Add readonlybackend

* Improve naming and get_or_insert

* Stylistic improvements

* Improve naming, add docs

* Revert unwanted changes

* Remove unused dependencies

* Improve docs

* Use RefCell

* lockfile

* Remove ReadOnceBackend

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Code review

* Do not use value cache when calculating storage roots

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Remove hash-db dep

* Update pallets/parachain-system/src/validate_block/trie_cache.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-07-19 11:46:58 +00:00
Sebastian KunertandBastian Köcher b6b1952fc6 Make trie-cache resettable from backend (#14516)
* Add ability to reset trie-cache

* comment

* Update client/db/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-07-18 17:05:40 +02:00
Sebastian Kunertandparity-processbot 3a10a29df5 Companion for substrate#14403 (#7435)
* Bump trie-db

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-06-28 14:27:05 +00:00
30a7e6788c Make triecache generic and work for no-std environments (#14403)
* Remove `AsLocalTrieCache` trait

* Introduce new trait AsTrieDbCache

* Use AsTrieDbCache trait

* Make it compile

* Docs

* Improve naming of associated type, implement cache usage for no-std

* Improve naming

* Improve docs

* Allow construction with optional cache

* FMT

* Remove unused variable

* Revert unwanted change

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* More comment adjustments

* Docs

* Trigger CI

* ".git/.scripts/commands/fmt/fmt.sh"

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Review comments

* Review comments

* Apply suggestions from code review

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* fmt

* Bump trie-db again

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
2023-06-28 12:57:41 +00:00
Sebastian Kunertandparity-processbot f653f4519b Companion for substrate #14464 (#7429)
* Remove unneeded feature flag

* More serde flags

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-06-27 14:06:57 +00:00
Sebastian Kunert ede49c7ae6 Remove unused dependencies (#14464) 2023-06-27 13:33:42 +00:00
c047ec900a Add benchmarks for for import/validation/production (#2579)
* Add benchmarks

* Remove endowed accounts

* Fix group

* Remove duplicate code

* Comments

* Use execute_block instead of block import

* Apply suggestions from code review

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Add missing comments

* Clippy

* Reorganize utils

* Remove unused import

* Switch to compiled again

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Refactor local_testnet

* Do not write wasm blob to file

* Remove unnecessary block

* Add comment fixes

* fmt

* lock

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-06-05 16:41:20 +02:00
Sebastian Kunert 1d42bdf664 Remove manual implementation (#14096) 2023-05-08 20:32:01 +00:00
Sebastian Kunert f79d45da09 Remove manual implementation (#14096) 2023-05-08 20:31:39 +00:00
Sebastian Kunertandparity-processbot b72ae577a0 Make zombienet tests required (#2527)
Co-authored-by: parity-processbot <>
2023-05-06 07:58:25 +02:00
Sebastian KunertandBastian Köcher d3b3a46d27 Only calculate tree route during finalization when there are multiple leaves (#14067)
* Only calculate tree route when there are multiple leaves

* Update client/service/src/client/client.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-05-03 11:41:51 +00:00
Sebastian Kunert 2d9b3131d5 Skip av-store, make consensus task blocking (#2497) 2023-05-01 10:22:56 +02:00
Sebastian KunertandMichal Kucharczyk 0ae0393042 Add option to skip av-store requests in availability-recovery-subsystem (#7131)
* Allow to skip availability-store

* Update node/network/availability-recovery/src/lib.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
2023-04-28 10:13:04 +00:00
Sebastian Kunert 36b0459d6b Parachain node should not recover blocks while syncing (#2462) 2023-04-19 16:02:35 +02:00
Sebastian Kunert 6c896b1ee6 Cleanup: Remove polkadot-service dependency from minimal node (#2430)
* Remove polkadot-service dependency from minimal-node

* Clean up error handline

* Remove unwanted changes

* Unused deps
2023-04-04 15:05:38 +00:00
Sebastian Kunert 59fb7ff59c help text examples + clean up (#2418) 2023-04-01 16:44:32 +02:00
Sebastian Kunertandparity-processbot d98f3d1fec Remove HeaderBackend from RelayChainRPCClient (#2385)
* Remove HeaderBackend from RelayChainRPCClient

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: parity-processbot <>
2023-03-28 07:37:58 +00:00
Sebastian Kunert c41eadc090 Remove HeaderBackend requirement from AuthorityDiscovery and NetworkWorker (#13730)
* Remove `HeaderBackend` requirement from `NetworkWorker`

* Remove HeaderBackend from authority-discovery
2023-03-28 07:10:50 +00:00
Sebastian Kunert 7bd3f3d762 Remove unused light-client leftover (#13687)
* Remove unused light-client leftover

* Remove more unused code
2023-03-23 11:35:39 +01:00
Sebastian Kunert ccfe5fe08c Always pass port to jsonrpsee WebSocket client (#2339)
* Always pass port to jsonrpsee

* Remove useless host check

* Do not silently drop
2023-03-20 14:05:06 +00:00
Sebastian Kunert f4f467beff ReadMe followup for #2313 (#2317)
* Readme again

* relaychain -> relay chain
2023-03-15 09:33:42 +01:00
7aa2ea2690 README: Document usage of RPC node (#2313)
* Update readme

* Improve

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Quote urls

* Apply suggestions from code review

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
2023-03-14 14:56:32 +00:00
Sebastian Kunert 15ba3d666f Cleanup dependencies + dead code (#2302) 2023-03-10 17:09:38 +01:00
Sebastian KunertandBastian Köcher 588bdad7f6 Add retry mechanism for pov-recovery, fix full-node pov-recovery (#2164)
* Increase delay for pov-recovery

* Update client/service/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Comment

* FMT

* Clear waiting_recovery when block is recovered or recovery failed

* Introduce recovery queue that preserved insertion order

* Better error logs

* Decrease slot duration

* Style improvements

* Add option to use unordered queue

* Maintain cache of finalized blocks

* Wait for one relay chain slot before recovery

* Make retries testable

* fmt

* Improve docs

* Improve docs

* Simplify RecoveryQueue

* Remove unwanted changes

* Adjust to comments

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Move recovery delay into the queue

* Check for finalized number

* Clean up

* Use timer

Co-authored-by: Bastian Köcher <git@kchr.de>

* Simplify implementation

* Revert "Use timer"

This reverts commit 3809eed840d3a09d54212f99486782ff80cdc1c9.

* Properly clear `to_recover` flag

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-09 14:18:55 +01:00
Sebastian Kunert 3af67716c3 Do not verify on state import for relay chain verifier (#2140)
* Skip verification on state import

* Actually use existing variables

* Merge checks
2023-01-26 12:27:24 +00:00
Sebastian Kunert ad453c556b Remove assumption that there is a local bootnode in RPC collator (#2091)
* Remove assumption that there is a local bootnode

* Bump zombienet to 1.3.29

* Separate "--"
2023-01-24 15:07:18 +01:00
Sebastian Kunertandparity-processbot 0f94229465 Companion for Substrate #13157 (#6582)
* Update getrandom and version_check

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2023-01-19 15:48:56 +00:00
9be655701a Notification-based block pinning (#13157)
* Worker

* Reorganize and unpin onnotification drop

* Pin in state-db, pass block number

* Pin blocks in blockchain db

* Switch to reference counted LRU

* Disable pinning when we keep all blocks

* Fix pinning hint for state-db

* Remove pinning from backend layer

* Improve readability

* Add justifications to test

* Fix justification behaviour

* Remove debug prints

* Convert channels to tracing_unbounded

* Add comments to the test

* Documentation and Cleanup

* Move task start to client

* Simplify cache

* Improve test, remove unwanted log

* Add tracing logs, remove expect for block number

* Cleanup

* Add conversion method for unpin handle to Finalitynotification

* Revert unwanted changes

* Improve naming

* Make clippy happy

* Fix docs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Use `NumberFor` instead of u64 in API

* Hand over weak reference to unpin worker task

* Unwanted

* &Hash -> Hash

* Remove number from interface, rename `_unpin_handle`, LOG_TARGET

* Move RwLock one layer up

* Apply code style suggestions

* Improve comments

* Replace lru crate by schnellru

* Only insert values for pinned items + better docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Improve comments, log target and test

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-19 15:13:16 +00:00
Sebastian Kunert b929e0282d Remove unused code in state-db (#13103) 2023-01-09 14:46:09 +01:00
Sebastian Kunert 123726f308 Remove unused finalization_displaced variable in db (#13090) 2023-01-06 21:09:57 +01:00
Sebastian Kunert cf00218f77 Remove integration tests that are covered by zombienet (#1998) 2022-12-15 17:06:02 +01:00
Sebastian Kunert 7cab12e9d2 Allow to specify multiple relay chain RPC urls for collator node (#1880)
* Allow specification of multiple urls for relay chain rpc nodes

* Add pooled RPC client basics

* Add list of clients to pooled client

* Improve

* Forward requests to dispatcher

* Switch clients on error

* Implement rotation logic

* Improve subscription handling

* Error handling cleanup

* Remove retry from rpc-client

* Improve naming

* Improve documentation

* Improve `ClientManager` abstraction

* Adjust zombienet test

* Add more comments

* fmt

* Apply reviewers comments

* Extract reconnection to extra method

* Add comment to reconnection method

* Clean up some dependencies

* Fix build

* fmt

* Provide alias for cli argument

* Apply review comments

* Rename P* to Relay*

* Improve zombienet test

* fmt

* Fix zombienet sleep

* Simplify zombienet test

* Reduce log clutter and fix starting position

* Do not distribute duplicated imported and finalized blocks

* fmt

* Apply code review suggestions

* Move building of relay chain interface to `cumulus-client-service`

* Refactoring to not push back into channel

* FMT
2022-12-15 11:42:07 +01:00
Sebastian Kunert edee3d557b Remove more protocols from minimal collator (#1927) 2022-12-01 10:35:47 +01:00
Sebastian Kunert 3f1db7bfb0 Remove av-store, av-distribution, chain-api subsystems from minimal node (#1903)
* Remove unused subsystems

* Consume requests we are not going to answer

* Respond to incoming chunk and pov requests.

* fmt

* Improve comment and logging

* Add reputation changes

* Do not register pov-chunk-request protocol

* Remove inbound chunk request channel

* Bump Polkadot
2022-11-27 14:35:51 +00:00
Sebastian Kunert 53d9a50c06 Add support for outbound only configs on request/response protocols (#6343)
* Add option ot add outbound_only configurations

* Improve comment
2022-11-26 10:22:09 +01:00
Sebastian KunertandBastian Köcher 9a3e518e5c Add example zombienet network file and instructions (#1839)
* Add simple example on how to spin up network.

* Update readme

* Remove unnecessary prefix

* Improve folder structure

* Add link to file

* Fix paths in readme

* Update README.md

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-11-08 18:26:20 +00:00
Sebastian Kunert b32fcdc96d Add RPC nodes to pov-recovery test (#1807) 2022-10-31 19:45:20 +01:00
Sebastian Kunert 9e690fe1b6 Bump substrate (#6164) 2022-10-18 17:13:28 +00:00
Sebastian Kunert 096553cb59 Use ArgAction::Set for enable-offchain-indexing flag (#12521)
* Use ArgAction::Set for enable-offchain-indexing

* Provide default value for `enable-offchain-indexing`
2022-10-18 15:41:19 +00:00
Sebastian Kunert 975403e802 Update clap to version 4 (#1745)
* Bump to clap4

* Adjust to clap 4 style

* Remove two more deprecated occurences of clap macro

* Remove "deprecated" feature from clap

* Update cargo lock

* Fix group name

* More skipped group names
2022-10-18 11:30:22 +00:00
Sebastian Kunertandparity-processbot 99f705537b Update clap to version 4 (#6128)
* Move clap to 4.0.9

* Remove "deprecated" feature flag

* Convert to default_value_t

* update lockfile for {"substrate"}

* Add group(skip) to avoid naming conflict

* More group(skip)

Co-authored-by: parity-processbot <>
2022-10-18 09:02:41 +00:00
Sebastian KunertandBastian Köcher f687db40f7 Bump clap to 4.0.x and adjust to best practices (#12381)
* Bump clap to 3.2.22

* Replace `from_os_str` with `value_parser`

* Replace `from_str` and `try_from_str` with `value_parser`

* Move possible_values to the new format

* Remove unwanted print

* Add missing match branch

* Update clap to 4.0.9 and make it compile

* Replace deprecated `clap` macro with `command` and `value`

* Move remaining `clap` attributes to `arg`

* Remove no-op value_parsers

* Adjust value_parser for state_version

* Remove "deprecated" feature flag and bump to 4.0.11

* Improve range

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions

* Trigger CI

* Fix unused error warning

* Fix doc errors

* Fix ArgGroup naming conflict

* Change default_value to default_value_t

* Use 1.. instead of 0..

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-10-18 06:52:46 +00:00
Sebastian Kunert cdcf5ddd9b Enable collation via RPC relay chain node (#1585)
* Add minimal overseer gen with dummy subsystems

* Fix dependencies

* no-compile: only client transaction pool missing

* Remove unused imports

* Continue to hack towards PoC

* Continue

* Make mini node compile

* Compiling version with blockchainevents trait

* Continue

* Check in lockfile

* Block with tokio

* update patches

* Update polkadot patches

* Use polkadot-primitives v2

* Fix build problems

* First working version

* Adjust cargo.lock

* Add integration test

* Make integration test work

* Allow startinc collator without relay-chain args

* Make OverseerRuntimeClient async

* Create separate integration test

* Remove unused ChainSelection code

* Remove unused parameters on new-mini

* Connect collator node in test to relay chain nodes

* Make BlockChainRPCClient obsolete

* Clean up

* Clean up

* Reimplement blockchain-rpc-events

* Revert "Allow startinc collator without relay-chain args"

This reverts commit f22c70e16521f375fe125df5616d48ceea926b1a.

* Add `strict_record_validation` to AuthorityDiscovery

* Move network to cumulus

* Remove BlockchainRPCEvents

* Remove `BlockIdTo` and `BlockchainEvents`

* Make AuthorityDiscovery async

* Use hash in OverseerRuntime

* Adjust naming of runtime client trait

* Implement more rpc-client methods

* Improve error handling for `ApiError`

* Extract authority-discovery creationand cleanup

* RPC -> Rpc

* Extract bitswap

* Adjust to changes on master

* Implement `hash` method

* Introduce DummyChainSync, remove ProofProvider and BlockBackend

* Remove `HeaderMetadata` from blockchain-rpc-client

* Make ChainSync work

* Implement NetworkHeaderBackend

* Cleanup

* Adjustments after master merge

* Remove ImportQueue from network parameters

* Remove cargo patches

* Eliminate warnings

* Revert to HeaderBackend

* Add zombienet test

* Implement `status()` method

* Add more comments, improve readability

* Remove patches from Cargo.toml

* Remove integration test in favor of zombienet

* Remove unused dependencies, rename minimal node crate

* Adjust to latest master changes

* fmt

* Execute zombienet test on gitlab ci

* Reuse network metrics

* Chainsync metrics

* fmt

* Feed RPC node as boot node to the relay chain minimal node

* fmt

* Add bootnodes to zombienet collators

* Allow specification of relay chain args

* Apply review suggestions

* Remove unnecessary casts

* Enable PoV recovery for rpc full nodes

* Revert unwanted changes

* Make overseerHandle non-optional

* Add availability-store subsystem

* Add AuxStore and ChainApiSubsystem

* Add availability distribution subsystem

* Improve pov-recovery logging and add RPC nodes to tests

* fmt

* Make availability config const

* lock

* Enable debug logs for pov-recovery in zombienet

* Add log filters to test binary

* Allow wss

* Address review comments

* Apply reviewer comments

* Adjust to master changes

* Apply reviewer suggestions

* Bump polkadot

* Add builder method for minimal node

* Bump substrate and polkadot

* Clean up overseer building

* Add bootnode to two in pov_recovery test

* Fix missing quote in pov recovery zombienet test

* Improve zombienet pov test

* More debug logs for pov-recovery

* Remove reserved nodes like on original test

* Revert zombienet test to master
2022-10-10 09:06:26 +02:00
Sebastian Kunert 9b669edf22 Change visibility of open_database for use in cumulus (#6034)
* Make some helper methods public

* Make availability config const.
2022-09-22 08:45:58 +00:00
Sebastian KunertandSebastian Kunert 800bc5cd8c Simplify trait bounds in network to prepare for collator-rpc (#12082)
* Hack towards PoC

* Abstract away runtime requirement

* blockchainevents

* Remove bitswap

* Remove unused sync more

* Remove unused features in network

* Re-enable bitswap change

* Remove `Chain` trait bound

* Reimplement blockchain-rpc-events

* Move network to cumulus

* Make AuthorityDiscovery async

* Remove `ProofProvider` requirement from network behaviour

* Extract bitswap

* Adjustments after merge

* Remove HeaderMetadata trait from network

* Introduce NetworkHeaderBackend

* Add comments

* Improve comments

* Move NetworkHeaderBackend to new module

* Improve naming, remove redundand send + sync

* Clean up generics

* Fix CI

* Improve comment and readability

* Remove NetworkHeaderBackend

* Fix Cargo.lock

Co-authored-by: Sebastian Kunert <skunert@Sebastians-MacBook-Pro.fritz.box>
2022-08-31 14:55:46 +02:00
Sebastian KunertandDavide Galassi 36e580a56a Introduce RPC Worker (#1462)
* Extract json-rpc-client and introduce worker

* Initial rpc worker

* Add error handling

* Use bounded channels for listeners

* Improve naming and clean up

* Use tracing channels

* Improve code readability

* Decrease channel size limit

* Remove unused dependency

* Fix docs

* RPC -> Rpc

* Start worker in initialization method

* Print error in case a distribution channel is full

* Fix docs

* Make `RpcStreamWorker` private

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Use tokio channels and add TODO item

* Remove `Option` from `to_worker_channel`

Co-authored-by: Davide Galassi <davxy@datawok.net>
2022-08-01 12:08:57 +02:00
Sebastian Kunert c831138c0a Remove retain_mut crate (#11926)
* Remove retain_mut crate

* Remove reain_mut crate from babe-consensus
2022-07-27 22:35:33 +00:00
Sebastian Kunertandalvicsam 9f409dc0b8 Prepare for rust 1.62.1 (#11903)
* Update UI test output for rust 1.62.1

* switch ci to staging image to check that everything works

* fix artifacts node-bench-regression-guard

* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning

* Fix more clippy lints

* Make clippy happy by relying on auto-deref were possible

* Add tracking issue to the comments

* pin ci image

Co-authored-by: alvicsam <alvicsam@gmail.com>
2022-07-26 14:37:05 +02:00
Sebastian Kunertandalvicsam 88d9aca202 Prepare for rust 1.62.1 (#5811)
* Update update-rust-stable script

* Update UI tests for rust-1.62.1

* Use staging CI image

* pin ci image

Co-authored-by: alvicsam <alvicsam@gmail.com>
2022-07-26 14:37:02 +02:00
Sebastian Kunert 9afc93cbd1 Bump strum to 0.24.1 (#5816) 2022-07-26 14:02:43 +02:00
Sebastian Kunertandparity-processbot 7582c735df Companion for polkadot#5782 (#1447)
* Make HeadSupportsParachain async

* Update Cargo.lock

* update lockfile for {"substrate", "polkadot"}

Co-authored-by: parity-processbot <>
2022-07-20 11:03:02 +00:00
72bde2889f Introduce async runtime calling trait for runtime-api subsystem (#5782)
* Implement OverseerRuntimeClient

* blockchainevents

* Update patches

* Finish merging rntime-api subsystem

* First version that is able to produce blocks

* Make OverseerRuntimeClient async

* Move overseer notification stream forwarding to cumulus

* Remove unused imports

* Add more logging to collator-protocol

* Lockfile

* Use hashes in OverseerRuntimeClient

* Move OverseerRuntimeClient into extra module

* Fix old session info call and make HeadSupportsParachain async

* Improve naming of trait

* Cleanup

* Remove unused From trait implementation

* Remove unwanted debug print

* Move trait to polkadot-node-subsystem-types

* Add sections to runtime client

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Reorder methods

* Fix spelling

* Fix spacing in Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove unused babe methods

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-07-20 10:23:25 +00:00
Sebastian Kunert 6e13b44694 Rename help string parachain-collator -> parachain-template-node (#1392) 2022-06-23 11:36:33 +02:00
Sebastian KunertandBastian Köcher 10e966e643 Respect cargo offline env variable in wasm builder (#11735)
* Support offline env variable in wasm builder

* Clean up

* Improve checks

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update crate docs

* Add docs to `lib.rs` and introduce helper method `offline_build`

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-06-22 15:27:46 +00:00
Sebastian Kunert 15e7db81d7 Strip 'Cargo.toml' from workspace path for patches (#1391) 2022-06-21 10:26:03 +02:00
Sebastian KunertandBastian Köcher 6a6490a699 Disable beefy for inprocess relay chain node (#1382)
* Disable beefy for inprocess relay chain node

* Update client/relay-chain-inprocess-interface/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-06-17 14:27:01 +00:00
Sebastian Kunert 2248d19163 Remove without_storage_info from pallet transaction-storage (#11668)
* Introduce BoundedVec

* Fix typos

* Add comments to the bounds

* Remove migration

* Improve bound value access syntax
2022-06-15 21:12:26 +00:00
Sebastian Kunert b1cc31027f Remove without_storage_info for membership pallet (#11591) 2022-06-07 19:49:36 +02:00
b32764ff99 Create binary for cumulus testing node #1128 (#1158)
* Create binary for cumulus testing node

* Fix call to wrong method

* Fix iterator bug

* Add real networking to polkadot node

* Allow using of file chain specs

* Implement export of wasm blob and genesis state

* Add parachain id to test service export

* Remove debug logs

* Fix main.rs

* Do not use cli from polkadot-collator

* Add tracing dependency

* Improve cli structure

* Use struct instead of enum for export commands

* Add missing license header

* Fix benchmark

* add build-spec subcommand (#19)

* add build-spec subcommand

* reorder args

* Fix formatting

* Add zombienet test (#26)

* add migrated tests

* rename test files

* uncomment args and remove extra collator node

* Fix indentation (#27)

* add migrated tests

* rename test files

* uncomment args and remove extra collator node

* fix identation

* Review comments

* Remove unwanted version changes

Co-authored-by: Sebastian Kunert <skunert@Sebastians-MacBook-Pro.fritz.box>
Co-authored-by: Javier Viola <pepoviola@gmail.com>
2022-05-18 16:41:48 +02:00
Sebastian Kunert 20a7ffbfcc Bump jsonrpsee-core to 11.0 2022-04-22 11:46:19 +00:00
Sebastian Kunert e495168051 Support running relay chain full node as non-collator (#1069)
* Run relay chain full node not as collator

* Clean up
2022-03-04 17:02:37 +01:00
Sebastian Kunert 36f735e18b Improve conflicts to properly detect helper flags (#1063) 2022-03-02 12:13:33 +01:00
Sebastian Kunert 6cf8177dac Bump jsonrpsee from 0.8.0 to 0.9.0 2022-03-02 09:57:35 +00:00
Sebastian KunertandKoute bc532724b0 Introduce rpc client for relay chain full node (#963)
* Initial network interface preparations

* Implement get_storage_by_key

* Implement `validators` and `session_index_for_child`

* Implement persisted_validation_data and candidate_pending_availability

* Fix method name for persisted_validation_data and add encoded params

* Implement `retrieve_dmq_contents` and `retrieve_all_inbound_hrmp_channel_contents`

* Implement `prove_read`

* Introduce separate RPC client, expose JsonRpSee errors

* Simplify closure in call_remote_runtime_function

* Implement import stream, upgrade JsonRpSee

* Implement finality stream

* Remove unused method from interface

* Implement `is_major_syncing`

* Implement `wait_on_block`

* Fix tests

* Unify error handling `ApiError`

* Replace WaitError with RelayChainError

* Wrap BlockChainError in RelayChainError

* Unify error handling in relay chain intefaces

* Fix return type of proof method

* Improve error handling of new methods

* Improve error handling and move logging outside of interface

* Clean up

* Remove unwanted changes, clean up

* Remove unused import

* Add format for StatemachineError and remove nused From trait

* Use 'thiserror' crate to simplify error handling

* Expose error for overseer, further simplify error handling

* Reintroduce network interface

* Implement cli option

* Adjust call_state method to use hashes

* Disable PoV recovery when RPC is used

* Add integration test for network full node

* Use Hash instead of BlockId to ensure compatibility with RPC interface

* Fix cargo check warnings

* Implement retries

* Remove `expect` statements from code

* Update jsonrpsee to 0.8.0 and make collator keys optional

* Make cli arguments conflicting

* Remove unused `block_status` method

* Add clippy fixes

* Cargo fmt

* Validate relay chain rpc url

* Clean up dependencies and add one more integration test

* Clean up

* Clean up dependencies of relay-chain-network

* Use hash instead of blockid for rpc methods

* Fix tests

* Update client/cli/src/lib.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Improve error message of cli validation

* Add rpc client constructor

* Do not use debug formatting for errors

* Improve logging for remote runtime methods

* Only retry on transport problems

* Use PHash by value, rename test

* Improve tracing, return error  on relay-chain-interface build

* Fix naming, use generics instead of deserializing manually

* Rename RelayChainLocal and RelayChainNetwork

* lock

* Format

* Use impl trait for encodable runtime payload

* Only instantiate full node in tests when we need it

* Upgrade scale-codec to 3.0.0

* Improve expect log

Co-authored-by: Koute <koute@users.noreply.github.com>
2022-03-01 12:37:51 +01:00
Sebastian Kunert dacd0aed5e Unify RelayChainInterface error handling and introduce async (#909) 2022-01-25 18:27:54 +01:00
Sebastian Kunert 57d04e0fc8 Add missing features in rococo Cargo.toml (#887) 2022-01-05 13:30:47 +00:00
Sebastian Kunert 876e594f59 Introduce interface for relay chain interaction (#835) 2021-12-22 19:02:11 +01:00
Sebastian Kunert 2691e1928e Port transaction pool benchmark (#738)
Adds a benchmark to measure transaction throughput.
2021-11-11 10:30:54 +01:00