Commit Graph

642 Commits

Author SHA1 Message Date
Bastian Köcher 941288c6d0 wasm-executor: Support growing the memory (#12520)
* As always, start with something :P

* Add support for max_heap_pages

* Add support for wasmtime

* Make it compile

* Fix compilation

* Copy wrongly merged code

* Fix compilation

* Some fixes

* Fix

* Get stuff working

* More work

* More fixes

* ...

* More

* FIXEs

* Switch wasmi to use `RuntimeBlob` like wasmtime

* Removed unused stuff

* Cleanup

* More cleanups

* Introduce `CallContext`

* Fixes

* More fixes

* Add builder for creating the `WasmExecutor`

* Adds some docs

* FMT

* First round of feedback.

* Review feedback round 2

* More fixes

* Fix try-runtime

* Update client/executor/wasmtime/src/instance_wrapper.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/runtime_blob/runtime_blob.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Feedback round 3

* FMT

* Review comments

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-02-24 11:43:01 +00:00
Koute f8e3bdad3d Rework storage iterators (#13284)
* Rework storage iterators

* Make sure storage iteration is also accounted for when benchmarking

* Use `trie-db` from crates.io

* Appease clippy

* Bump `trie-bench` to 0.35.0

* Fix tests' compilation

* Update comment to clarify how `IterArgs::start_at` works

* Add extra tests

* Fix iterators on `Client` so that they behave as before

* Add extra `unwrap`s in tests

* More clippy fixes

* Come on clippy, give me a break already

* Rename `allow_missing` to `stop_on_incomplete_database`

* Add `#[inline]` to `with_recorder_and_cache`

* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment

* Simplify code: use `with_trie_db` in `next_storage_key_from_root`

* Remove `expect`s in the benchmarking CLI

* Add extra doc comments

* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)

* Remove a TODO in tests

* Update comment for `StorageIterator::was_complete`

* Update `trie-db` to 0.25.1
2023-02-22 07:49:25 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Michal Kucharczyk 5ef88dd398 BlockId removal: BlockBuilderProvider::new_block_at (#13401)
* `BlockId` removal: `BlockBuilderProvider::new_block_at`

It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`

* fmt

* fix

* more fixes
2023-02-21 18:36:00 +00:00
Michal Kucharczyk 7a10154188 BlockId removal: runtime-api refactor (#13255)
* BlockId removal: refactor of runtime API

It changes the arguments of:
- `ApiExt` methods:  `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`

It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* BlockId removal: refactor of runtime API - tests

- tests adjusted to new runtime API,
- some tests migrated from block number to block hash

* benchmarking-cli: BlockId(0) migrated to info().genesis_hash

`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.

* trivial: api.call(BlockId) -> api.call(Hash)

- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,

* CallApiAtParams::at changed to Block::Hash

* missed doc updated

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

* BlockId removal: Benchmark::consumed_weight

Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.

* at_hash renamed

* wrong merge fixed

* beefy worker: merged with master

* beefy: tests: missing block problem fixed

* Apply review suggestion

* fix

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
2023-02-20 22:47:21 +00:00
Dmitry Markin 8d033b6dfb Use async/await instead of manual polling of NetworkWorker (#13219)
* Convert `NetworkWorker::poll()` into async `next_action()`

* Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`

* Revert "Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`"

This reverts commit 4b5d851ec864f78f9d083a18a618fbe117c896d2.

* Fix `sc-network-test` to poll `NetworkWorker::next_action`

* Fix `sc_network::service` tests to poll `NetworkWorker::next_action`

* Fix docs

* kick CI

* Factor out `next_worker_message()` & `next_swarm_event()`

* Error handling: replace `futures::pending!()` with `expect()`

* Simplify stream polling in `select!`

* Replace `NetworkWorker::next_action()` with `run()`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* minor: comment

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Print debug log when network future is shut down

* Evaluate `NetworkWorker::run()` future once before the loop

* Fix client code to match new `NetworkService` interfaces

* Make clippy happy

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Revert "Apply suggestions from code review"

This reverts commit 9fa646d0ed613e5f8623d3d37d1d59ec0a535850.

* Make `NetworkWorker::run()` consume `self`

* Terminate system RPC future if RPC rx stream has terminated.

* Rewrite with let-else

* Fix comments

* Get `best_seen_block` and call `on_block_finalized` via `ChainSync` instead of `NetworkService`

* rustfmt

* make clippy happy

* Tests: schedule wake if `next_action()` returned true

* minor: comment

* minor: fix `NetworkWorker` rustdoc

* minor: amend the rustdoc

* Fix bug that caused `on_demand_beefy_justification_sync` test to hang

* rustfmt

* Apply review suggestions

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-20 12:08:02 +00:00
Michal Kucharczyk 36480b158d BlockId removal: Client::runtime_version_at (#13393)
* BlockId removal: Client::runtime_version_at

It changes the arguments of `Client::runtime_version_at` from: `BlockId<Block>` to: `Block::Hash`

* Apply suggestions from code review

Co-authored-by: Anton <anton.kalyaev@gmail.com>

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: parity-processbot <>
2023-02-16 22:34:00 +00:00
Michal Kucharczyk 2e4ea01311 BlockId removal: CallApiAt::state_at (#13394)
* BlockId removal: CallApiAt::state_at

It changes the arguments of:
- `CallApiAt::state_at`,
- `Client::code_at`,
- `Client::state_at`

from: `BlockId<Block>` to: `Block::Hash`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
2023-02-16 22:21:15 +00:00
Sam Elamin df24729d74 add warp to target block for parachains (#12761)
* add warp to target block for parachains

* fix for failing tests

* format using  `Cargo +nightly fmt`

* Remove blocking based on PR comments and create new `WarpSync` on poll

* remove method from trait

* add tests for wait for target

* Update client/network/common/src/sync/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/common/src/sync/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* code refactor based on pr comments

* Second round of PR comments

* Third round of pr comments

* add comments to explain logic

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* code refactor based on last PR comments

* move warp sync polling before `process_outbound_requests`

Add error message if target block fails to be retreived

* Update client/network/sync/src/warp.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* fmt after code suggestions

* rebase changes

* Bring down the node if the target block fails to return

* Revert "Bring down the node if the target block fails to return"

This reverts commit c0ecb220d66dd8e7b1a5ee29831b776f4f18d024.

* Update client/network/common/src/sync/warp.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Update client/network/common/src/sync/warp.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* use matching on polling to avoid calling poll more than once

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/warp.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* fix typo on comment

* update snapshot with new folder structure

* Upload snapshot

* Bump zombienet

* bump zombienet again

* Improve test

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/test/src/sync.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* fix tests

* dummy commit to restart builds

* Converted the target block to an optional value that is set to `None` when an error occurs

* dummy commit to restart builds

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-02-14 17:46:51 +00:00
Davide Galassi d48fc58729 Fix longest chain finalization target lookup (#13289)
* Finalization target should be chosed as some ancestor of SelectChain::best_chain

* More test assertions

* Improve docs

* Removed stale docs

* Rename 'target' to 'base' in lookup method

* Fix typo

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Rename 'target_hash' to 'base_hash' in 'SelectChain::finality_target()'

* Apply suggestions from code review

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Docs improvement

* Doc fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply more code suggestions

---------

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-11 18:35:04 +01:00
Andrei Sandu f552df7bbf Add task type label to metrics (#13240)
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2023-01-26 06:22:51 +00:00
Michal Kucharczyk 14a4eed2aa service: storage monitor added (#13082)
* service: storage monitor added

Storage monitor added. It uses `notify` create to get notifications
about any changes to monitored path (which is database path).
Notifications are consumed in essential task which terminates when
available storage space drops below given threshold.

Closes: #12399

* Cargo.lock updated

* misspell

* fs events throttling added

* minor updates

* filter out non mutating events

* misspell

* ".git/.scripts/commands/fmt/fmt.sh"

* Update client/service/src/storage_monitor.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* storage-monitor crate added

* cleanup: configuration + service builder

* storage_monitor in custom service (wip)

* copy-paste bad desc fixed

* notify removed

* storage_monitor added to node

* fix for clippy

* publish = false

* Update bin/node/cli/src/command.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* crate name: storage-monitor -> sc-storage-monitor

* error handling improved

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* publish=false removed

Co-authored-by: command-bot <>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-24 13:26:04 +00:00
Michal Kucharczyk 025f9d9ba3 BlockId removal: refactor: CallExecutor trait (#13173)
* BlockId removal: refactor: CallExecutor trait

It changes the arguments of CallExecutor methods:
-  `call`, 'contextual_call', 'runtime_version', 'prove_execution'

from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
2023-01-24 12:52:01 +00:00
Bastian Köcher d20a9bdc21 Rename *-private-ipv4 to *-private-ip CLI args (#13208)
* Rename `*-private-ipv4` to `*-private-ip` CLI args

Renames the `*-private-ipv4` to `*-private-ip` in the CLI interface. The old names are staying as
alias, thus it will not break for anyone. Besides that it also fixes the naming in the rest of the code.

* FMT
2023-01-23 11:35:04 +00:00
Gavin Wood 2d7fa18e73 Make DispatchError impl MEL (#13169)
* Make DispatchError impl MEL

* Upgrade SCALE codec to support `codec(skip)` for MEL

Co-authored-by: Bastian Köcher <info@kchr.de>
2023-01-19 16:47:47 +00:00
Sebastian Kunert 9be655701a Notification-based block pinning (#13157)
* Worker

* Reorganize and unpin onnotification drop

* Pin in state-db, pass block number

* Pin blocks in blockchain db

* Switch to reference counted LRU

* Disable pinning when we keep all blocks

* Fix pinning hint for state-db

* Remove pinning from backend layer

* Improve readability

* Add justifications to test

* Fix justification behaviour

* Remove debug prints

* Convert channels to tracing_unbounded

* Add comments to the test

* Documentation and Cleanup

* Move task start to client

* Simplify cache

* Improve test, remove unwanted log

* Add tracing logs, remove expect for block number

* Cleanup

* Add conversion method for unpin handle to Finalitynotification

* Revert unwanted changes

* Improve naming

* Make clippy happy

* Fix docs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Use `NumberFor` instead of u64 in API

* Hand over weak reference to unpin worker task

* Unwanted

* &Hash -> Hash

* Remove number from interface, rename `_unpin_handle`, LOG_TARGET

* Move RwLock one layer up

* Apply code style suggestions

* Improve comments

* Replace lru crate by schnellru

* Only insert values for pinned items + better docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Improve comments, log target and test

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-19 15:13:16 +00:00
Koute 338d75d0a3 Fix the storage_size/state_getStorageSize RPC call (#13154)
* Have `KeyIterator` clone the `prefix` it receives

* Stream keys in `storage_size` RPC and add a runtime limit

* Update client/rpc/Cargo.toml

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/rpc/src/state/utils.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Rename the types to signify that the cancellation is due to a timeout

* Move the test into a `mod tests`

* Add a comment regarding `biased` in `tokio::select`

* Make the `clone` explicit when calling `KeyIterator::{new, new_child}`

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-18 06:28:27 +00:00
Bastian Köcher 3a68a7818b Return RuntimeVersion of overwritten/substituted wasm binary (#13066)
* Adds test

* Ensure we are using the runtime version of the override/substitute wasm

* Update client/service/src/client/call_executor.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-01-05 13:45:40 +01:00
Michal Kucharczyk 9c69fc1b32 BlockId removal: refactor: BlockBackend::block|block_status (#13014)
* BlockId removal: refactor: BlockBackend::block|block_status

It changes the arguments of:
-  `BlockBackend::block`
-  `BlockBackend::block_status`

method from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-obvious reworks

* doc fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
2023-01-02 09:42:05 +00:00
Bastian Köcher 0a94112c9d Print "stalled" task on shutdown (#13022)
* Print "stalled" task on shutdown

When the node is shutting down, we give the Tokio runtime 60 seconds to shutdown. If after these 60
seconds there are still running tasks, we now print these tasks. This should help debugging nodes
that have stalled tasks.

This pr introduces a `TaskRegistry` that keeps track of all running tasks. Each task registers and
unregisters itself in this `TaskRegistry`.

* Fix rustdoc

* Update client/service/src/lib.rs
2022-12-28 09:16:52 +01:00
Dmitry Markin 34eb463d99 Runtime diagnostics for leaked messages in unbounded channels (#12971) 2022-12-23 16:03:08 +03:00
Michal Kucharczyk c237b82690 BlockId removal: refactor: HeaderBackend::status (#12981)
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

Co-authored-by: parity-processbot <>
2022-12-21 10:25:16 +00:00
Michal Kucharczyk 548955a73f BlockId removal: refactor: HeaderBackend::header (#12874)
* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
2022-12-20 09:43:31 +00:00
Alexandru Vasile 46932f2b47 rpc: Implement chainHead RPC API (#12544)
* rpc/chain_head: Add event structure for serialization

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add tests for events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add API trait for `chainHead`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add RPC errors

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Manage subscription ID tracking for pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add tests for subscription management

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Constructor for the API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Placeholders for API implementation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Accept RPC subscription sink

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate the runtime API event

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `body` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `header` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `call` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `unpin` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update `Cargo.lock`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement `getGenesis` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Parse params from hex string

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Constuct API with genesis hash

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add the finalized block to reported tree route

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Export the API and events for better ergonomics

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Add test module with helper functions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test block events from the `follow` pubsub

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `genesisHash` getter

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `header` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `body` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test calling into the runtime API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test runtime for the `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Add runtime code changes for `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Remove space from rustdoc

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use the `child_key` for storage queries

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Test `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Test child trie query for `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Event serialization typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Remove subscription aliases

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add `NetworkConfig` parameter

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Named parameters as camelCase if present

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement From<ApiError> for RuntimeEvents

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Handle pruning of the best block in finalization window

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate initial block events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify that initial in-memory blocks are reported

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify the finalized event with forks and pruned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Separate logic for generating initial events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Handle stopping a subscription ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Submit events until the "Stop" event is triggered

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Separate logic for handling new and finalized blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Extend subscription logic with subId handle

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust to the new subscription mngmt API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Refuse RuntimeAPI calls without the runtime flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify RuntimeAPI calls without runtime flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add best block per subscription

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Check storage keys for prefixes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Check storage queries with invalid prefixes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Allow maximum number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test the maximum number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust to origin/master and apply clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/service: Enable the `chainHead` API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Stop subscription on client disconnect and add debug logs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix sending `Stop` on subscription exit

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Check best block is descendent of latest finalized

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Report events before pruning the best block

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Nonrecursive initial block generation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate initial events on subscription executor

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Reduce dev-dependencies for tokio

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* rpc/chain_head: Accept empty parameters

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use debug of `HexDisplay` for full format

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Enable subscription ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use jsonrpsee 16.2 camelCase feature for paramaters

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use `NonZeroUsize` for `NetworkConfig` param

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Rename `runtime_updates` to `has_runtime_updates`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2022-12-19 17:38:12 +02:00
Liu-Cheng Xu b92aa3dbdc Support custom genesis block (#12291)
* Set genesis block data using the built genesis block

* Make resolve_state_version_from_wasm a separate function and some small refactorings

Useful for the commit following.

* Introduce trait BuildGenesisBlock

Substrate users can use this trait to implement their custom genesis block when constructing the
client.

* Make call_executor test compile

* cargo +nightly fmt --all

* Fix test

* Remove unnecessary clone

* FMT

* Apply review suggestions

* Revert changes to new_full_client() and new_full_parts() signature

* Remove needless `Block` type in `resolve_state_version_from_wasm`
2022-12-19 15:26:31 +01:00
Qinxuan Chen 40c04cf11d *: update rand to v0.8.5 (#12962)
* *: update rand to v0.8.5

* *: remove useless deps from Cargo.toml

* fix pallet-session-benchmarking

* fix pallet-election-provider-support test

* remove useless rand from dev-dependencies
2022-12-19 07:38:51 +01:00
Niklas Adolfsson 84303ca75d rpc server with HTTP/WS on the same socket (#12663)
* jsonrpsee v0.16

add backwards compatibility

run old http server on http only

* cargo fmt

* update jsonrpsee 0.16.1

* less verbose cors log

* fix nit in log: WS -> HTTP

* revert needless changes in Cargo.lock

* remove unused features in tower

* fix nits; add client-core feature

* jsonrpsee v0.16.2
2022-12-12 10:32:55 +00:00
Aaro Altonen d6827185c3 Move import queue out of sc-network (#12764)
* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Fix tests

* Apply review comments

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-12-09 19:50:57 +00:00
Marcin S aa21e56744 Remove mem_info and references to parity-util-mem (#12795)
* Remove mem_info and some references to parity-util-mem

* [Draft] Finish removing references to `parity-util-mem`

* Upgrade dependencies

* Update scripts/ci/deny.toml

Co-authored-by: ordian <write@reusable.software>

* Fix Cargo.lock (remove unwanted dependency changes)

* Removed unused argument

* Run cargo fmt (didn't have pre-commit set up)

* Fix some CI errors

* Fix another CI error

* Remove unused dependency

Co-authored-by: ordian <write@reusable.software>
2022-12-06 14:55:10 +00:00
Dmitry Markin 5eb84f9cc6 Upgrade tokio to 1.22.0 and replace async-std with tokio (#12646)
* Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests
2022-12-05 08:18:46 +00:00
Bastian Köcher 0c934a9352 ed25519_verify: Support using dalek for historical blocks (#12661)
* ed25519_verify: Support using dalek for historical blocks

The switch from `ed25519-dalek` to `ed25519-zebra` was actually a breaking change. `ed25519-zebra`
is more permissive. To support historical blocks when syncing a chain this pull request introduces
an externalities extension `UseDalekExt`. This extension is just used as a signaling mechanism to
`ed25519_verify` to use `ed25519-dalek` when it is present. Together with `ExtensionBeforeBlock` it
can be used to setup a node in way to sync historical blocks that require `ed25519-dalek`, because
they included a transaction that verified differently as when using `ed25519-zebra`.

This feature can be enabled in the following way. In the chain service file, directly after the
client is created, the following code should be added:

```
use sc_client_api::ExecutorProvider;
client.execution_extensions().set_extensions_factory(
	sc_client_api::execution_extensions::ExtensionBeforeBlock::<Block, sp_io::UseDalekExt>::new(BLOCK_NUMBER_UNTIL_DALEK_SHOULD_BE_USED)
);
```

* Fix doc

* More fixes

* Update client/api/src/execution_extensions.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fix merge and warning

* Fix docs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2022-11-27 15:34:07 +00:00
Aaro Altonen 1b5d52deb2 Move block/state/warpc sync requests/responses to ChainSync (#12739)
* Move block/state/warpc sync requests/responses to `ChainSync`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply review suggestions

* cargo-fmt + doc fix

* Fix tests

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-11-22 08:19:17 +00:00
Koute 4214bc527c Remove the wasmtime feature flag (#12684)
* Remove the `wasmtime` feature flag

* rustfmt
2022-11-18 14:21:44 +01:00
Niklas Adolfsson 2b8af8cb1a release sp-core 7.0.0 and sp-runtime 7.0.0 (#12599)
* chore(release): sp-core v7.0.0

* chore(release): sp-runtime v7.0.0

* fix bad merge
2022-11-15 14:54:14 +00:00
Bastian Köcher 1e9763cf81 Do not finalize parent twice (#12653)
If the parent block is alread finalized, we don't need to do this again.
2022-11-09 09:22:34 +01:00
Michal Kucharczyk 1ed70004e7 BlockId removal: &Hash to Hash (#12626)
It changes &Block::Hash argument to Block::Hash.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
2022-11-07 21:42:16 +00:00
Michal Kucharczyk 11b6409c12 BlockId removal: refactor: Backend::block_indexed_body (#12609)
* BlockId removal: refactor: Backend::block_indexed_body

It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* trigger CI job
2022-11-03 16:57:10 +00:00
Michal Kucharczyk 84167bd7d4 BlockId removal: refactor: Backend::justifications (#12602)
* BlockId removal: refactor: Backend::justifications

It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* trigger CI job

* trigger CI job

* bug fix

* match -> if

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Co-authored-by: Adrian Catangiu <adrian@parity.io>
2022-11-02 22:15:33 +00:00
Michal Kucharczyk 76bcbd09a5 BlockId removal: refactor: Backend::body (#12587)
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

Co-authored-by: parity-processbot <>
2022-11-01 16:24:09 +00:00
Michal Kucharczyk a1a3b923ab Added test for Client::block (#12590) 2022-10-31 12:43:36 +01:00
Bastian Köcher 33b61f0e03 Aura: Adds some compatibility mode to support old chains (#12492)
* Aura: Adds some compatibility mode to support old chains

In https://github.com/paritytech/substrate/pull/9132 we changed the way how we get the authorities
from the runtime. Before this mentioned pr we would call `initialize_block` before calling the
authorities runtime function. The problem with this was that when you have a block X that would
switch the authority set, it would already be signed by an authority of the new set. This was wrong,
as a block should only be signed by the current authority set. As this change is a hard fork, this
pr brings back the possibility for users that have a chain running with this old logic to upgrade.

They will need to use:
```
CompatibilityMode::UseInitializeBlock { until: some_block_in_the_future }
```

Using this compatibility mode will make the node behave like the old nodes, aka calling
`initialize_block` before doing the actual runtime call to `authorities`. Then when the given
`until` block is being build/imported the node switches to the new behaviour of not calling
`initialize_block` before. This is a hard fork, so the `until` block should be chosen wisely as a
point where all nodes in the network have upgraded.

* Fixes

* Make docs ready

* Update client/consensus/aura/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/aura/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/aura/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* FMT

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2022-10-28 13:25:49 +00:00
yjh 9375665218 feat: generalize some functions in sp-trie (#12376)
* feat: add to_memory_db to StorageProof

* feat: add iter method and generalize iter_nodes

* fmt

* feat: generalize `encode_compact` like `decode_compact`, add to_compact_proof to StorageProof

* fix to_compact_proof

* improve by suggestions

* improve by suggestions

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-10-25 07:04:32 +00:00
Aaro Altonen a877b0ccec Make NetworkService callable for ChainSync (#12542)
Introduce a middleware called `NetworkServiceProvider` which the
`ChainSync` can use to communicate with `NetworkService`. `ChainSync` is
given a `NetworkServiceHandle` which it uses to call `NetworkServiceProvider`
which then dispatches the calls to `NetworkService` on behalf of `ChainSync`.

This change will allow `ChainSync` to disconnect and report peers and
in the future it'll be possible to send requests and notifications
through the `NetworkServiceProvider`.

`NetworkServiceProvider` is needed only until the `ChainSync` object
has been removed from `Protocol`. After that, a normal `NetworkService`
handle can be passed onto `ChainSync` and these changes can be
deprecated.

Co-authored-by: parity-processbot <>
2022-10-22 12:36:26 +00:00
Michal Kucharczyk c0e8abe1a8 BlockId removal: refactor: Backend::begin_state_operation (#12541)
It changes the arguments of `Backend::begin_state_operation`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
2022-10-21 15:05:51 +00:00
Michal Kucharczyk 42215038a3 BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block (#12535)
* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block

It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Review suggestion applied

thx to @davxy

* trigger CI job
2022-10-20 15:50:59 +00:00
Michal Kucharczyk 749bcd1949 BlockId removal: refactor: Finalizer (#12528)
* BlockId removal: refactor: Finalizer

It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* minor corrections

* failing test corrected

* minor rework
2022-10-20 06:44:04 +00:00
Michal Kucharczyk 1d5aa47bee BlockId removal: refactor: ProofProvider (#12519)
* BlockId removal: refactor: ProofProvider

It changes the arguments of methods of `ProofProvider` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* LightClientRequestHandler: excessive BlockIdTo bound removed

* imports cleanup

* formatting

* args tyeps cleanup
2022-10-19 10:13:57 +00:00
Michal Kucharczyk b16135f602 BlockId removal: refactor: StorageProvider (#12510)
* BlockId removal: refactor: StorageProvider

It changes the arguments of `Backend::StorageProvider` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/api/src/backend.rs

Co-authored-by: Dmitrii Markin <dmitry@markin.tech>

* GrandpaBlockImport::current_set_id reworked

* ExportStateCmd reworked

* trigger CI job

* trigger CI job

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Dmitrii Markin <dmitry@markin.tech>
2022-10-18 12:52:04 +00:00
Aaro Altonen ce4cad8b8b Introduce ChainSyncInterface (#12489)
* Introduce `ChainSyncInterface`

`ChainSyncInterface` provides an asynchronous interface for other
subsystems to submit calls to `ChainSync`. This allows `NetworkService`
to delegate calls to `ChainSync` while still providing the same API
for other subsystems (for now). This makes it possible to move the
syncing code in piecemeal fashion out of `protocol.rs` as the calls
are just forwarded to `ChainSync`.

* Apply review comments

* Fix tests
2022-10-17 07:25:25 +00:00
Michal Kucharczyk 532dd5ecc2 BlockId removal: refactor: Backend::state_at (#12488)
* Minor naming improved

* BlockId removal refactor: Backend::state_at

* formatting
2022-10-14 09:27:32 +00:00