Commit Graph

7 Commits

Author SHA1 Message Date
Alexandru Vasile 2e6a2ffa8a chainHead: Add support for storage pagination and cancellation (#14755)
* chainHead/api: Add `chain_head_unstable_continue` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Register operations for pagination

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Merge limits with registered operation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Expose the operation state

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/storage: Generate WaitingForContinue event

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Use the continue operation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust testing to the new storage interface

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/config: Make pagination limit configurable

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust chainHeadConfig

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Check pagination and continue method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/api: Add `chainHead_unstable_stopOperation` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Add shared atomic state for efficient alloc

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Implement operation stop

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Check that storage ops can be cancelled

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/storage: Change docs for query_storage_iter_pagination

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Fix merge conflicts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Replace `async-channel` with `tokio::sync`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Add comment about the sender/recv continue

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-08-24 14:32:30 +03:00
Alexandru Vasile 744b783a7d chainHead: Limit ongoing operations (#14699)
* chainHead/api: Make storage/body/call pure RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Add mpsc channel between RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Extract mpsc::Sender via BlockGuard

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Generate and provide the method operation ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_body` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_call` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_storage` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Propagate responses of methods to chainHead_follow

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_body` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure unique operation IDs across methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/events: Remove old method events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Add limit helper

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Expose limits to `BlockGuard`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust testing to ongoing operations

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Make limits configurable via `ChainHeadConfig`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust testing to `ChainHeadConfig`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure operation limits discards items

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Improve documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Rename `OngoingOperations` -> `LimitOperations`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Rename reserve -> reserve_at_most

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Use duration const instead of u64

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscription: Use tokio::sync::Semaphore for limits

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update client/rpc-spec-v2/src/chain_head/subscription/inner.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-08-15 15:17:41 +03:00
Alexandru Vasile 4849b6e865 chainHead: Produce method responses on chainHead_follow (#14692)
* chainHead/api: Make storage/body/call pure RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Add mpsc channel between RPC methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Extract mpsc::Sender via BlockGuard

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/subscriptions: Generate and provide the method operation ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_body` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_call` response

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Generate `chainHead_storage` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Propagate responses of methods to chainHead_follow

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_body` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Adjust `chainHead_call` responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure unique operation IDs across methods

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/events: Remove old method events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Return `InvalidBlock` error if pinning fails

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead: Wrap subscription IDs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chainHead/tests: Ensure separate operation IDs across subscriptions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: parity-processbot <>
2023-08-08 21:13:52 +03:00
Arkadiy Paronyan d6d9bd9ea3 Encryption support for the statement store (#14440)
* Added ECIES encryption

* tweaks

* fmt

* Make clippy happy

* Use local keystore

* qed
2023-07-17 18:41:41 +00:00
Bastian Köcher 5eb816d7a6 Removal of execution strategies (#14387)
* Start

* More work!

* Moar

* More changes

* More fixes

* More worrk

* More fixes

* More fixes to make it compile

* Adds `NoOffchainStorage`

* Pass the extensions

* Small basti making small progress

* Fix merge errors and remove `ExecutionContext`

* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`

Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.

* Fix compilation

* Register the global extensions inside runtime api instance

* Fixes

* Fix `generate_initial_session_keys` by passing the keystore extension

* Fix the grandpa tests

* Fix more tests

* Fix more tests

* Don't set any heap pages if there isn't an override

* Fix small fallout

* FMT

* Fix tests

* More tests

* Offchain worker custom extensions

* More fixes

* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Fixes

* Fixes

* Set offchain transaction pool in BABE before using it in the runtime

* Add the `offchain_tx_pool` to Grandpa as well

* Fix the nodes

* Print some error when using the old warnings

* Fix merge issues

* Fix compilation

* Rename `babe_link`

* Rename to `offchain_tx_pool_factory`

* Cleanup

* FMT

* Fix benchmark name

* Fix `try-runtime`

* Remove `--execution` CLI args

* Make clippy happy

* Forward bls functions

* Fix docs

* Update UI tests

* Update client/api/src/execution_extensions.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/cli/src/params/import_params.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Pass the offchain storage to the MMR RPC

* Update client/api/src/execution_extensions.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Review comments

* Fixes

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-07-11 14:21:38 +00:00
Alexandru Vasile ed499bdec1 chain_head: Rename runtimeUpdates flag to withRuntime (#14244)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-05-27 22:13:00 +02:00
Alexandru Vasile 0e28645153 rpc: Use the blocks pinning API for chainHead methods (#13233)
* rpc/chain_head: Add backend to subscription management

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Pin blocks internally and adjust testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/in_mem: Reference for the number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/tests: Check in-memory references to pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Remove unused comment

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Place subscription handle under `Arc` and unpin blocks on drop

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/tests: Check all pinned blocks are unpinned on drop

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/rpc-spec-v2/src/chain_head/subscription.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* rpc/tests: Retry fetching the pinned references for CI correctness

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/service: Use 512 as maximum number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head: Fix merging conflicts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust subscriptions to use pinning API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head/tests: Test subscription management

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust chain_head follow to the new API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust chain_head.rs to the new API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head/tests: Adjust test.rs to the new API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/builder: Use new chainHead API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/in_mem: ChainHead no longer uses `in_mem::children`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update client/rpc-spec-v2/src/chain_head/subscription.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Update client/rpc-spec-v2/src/chain_head/subscription.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Update client/rpc-spec-v2/src/chain_head/subscription.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Update client/rpc-spec-v2/src/chain_head/subscription.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* chain_head: Add block state machine

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Address feedback

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Use new_native_or_wasm_executor

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head: Remove 'static on Backend

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head: Add documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head: Lock blocks before async blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head_follower: Remove static on backend

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update client/service/src/builder.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update client/service/src/builder.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* chain_head: Add BlockHeaderAbsent to the PartialEq impl

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client: Add better documentation around pinning constants

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head: Move subscription to dedicated module

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subscription: Rename global pin / unpin functions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-05-02 18:00:34 +03:00