This PR provides the infrastructure for the pov-reclaim mechanism
discussed in #209. The goal is to provide the current proof size to the
runtime so it can be used to reclaim storage weight.
## New Host Function
- A new host function is provided
[here](https://github.com/skunert/polkadot-sdk/blob/5b317fda3be205f4136f10d4490387ccd4f9765d/cumulus/primitives/pov-reclaim/src/lib.rs#L23).
It returns the size of the current proof size to the runtime. If
recording is not enabled, it returns 0.
## Implementation Overview
- Implement option to enable proof recording during import in the
client. This is currently enabled for `polkadot-parachain`,
`parachain-template` and the cumulus test node.
- Make the proof recorder ready for no-std. It was previously only
enabled for std environments, but we need to record the proof size in
`validate_block` too.
- Provide a recorder implementation that only the records the size of
incoming nodes and does not store the nodes itself.
- Fix benchmarks that were broken by async backing changes
- Provide new externalities extension that is registered by default if
proof recording is enabled.
- I think we should discuss the naming, pov-reclaim was more intuitive
to me, but we could also go with clawback like in the issue.
## Impact of proof recording during import
With proof recording: 6.3058 Kelem/s
Without proof recording: 6.3427 Kelem/s
The measured impact on the importing performance is quite low on my
machine using the block import benchmark. With proof recording I am
seeing a performance hit of 0.585%.
---------
Co-authored-by: command-bot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
This moves the macro related re-exports to `__private` to make it more
obvious for downstream users that they are using an internal api.
---------
Co-authored-by: command-bot <>
* Start
* More work!
* Moar
* More changes
* More fixes
* More worrk
* More fixes
* More fixes to make it compile
* Adds `NoOffchainStorage`
* Pass the extensions
* Small basti making small progress
* Fix merge errors and remove `ExecutionContext`
* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`
Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.
* Fix compilation
* Register the global extensions inside runtime api instance
* Fixes
* Fix `generate_initial_session_keys` by passing the keystore extension
* Fix the grandpa tests
* Fix more tests
* Fix more tests
* Don't set any heap pages if there isn't an override
* Fix small fallout
* FMT
* Fix tests
* More tests
* Offchain worker custom extensions
* More fixes
* Make offchain tx pool creation reusable
Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.
* Fixes
* Fixes
* Set offchain transaction pool in BABE before using it in the runtime
* Add the `offchain_tx_pool` to Grandpa as well
* Fix the nodes
* Print some error when using the old warnings
* Fix merge issues
* Fix compilation
* Rename `babe_link`
* Rename to `offchain_tx_pool_factory`
* Cleanup
* FMT
* Fix benchmark name
* Fix `try-runtime`
* Remove `--execution` CLI args
* Make clippy happy
* Forward bls functions
* Fix docs
* Update UI tests
* Update client/api/src/execution_extensions.rs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/cli/src/params/import_params.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/api/src/execution_extensions.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Pass the offchain storage to the MMR RPC
* Update client/api/src/execution_extensions.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Review comments
* Fixes
---------
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Make offchain tx pool creation reusable
Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.
* Update client/transaction-pool/api/src/lib.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* rpc/chain_head: Add backend to subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Pin blocks internally and adjust testing
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: Reference for the number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check in-memory references to pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Remove unused comment
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Place subscription handle under `Arc` and unpin blocks on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check all pinned blocks are unpinned on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* rpc/tests: Retry fetching the pinned references for CI correctness
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/service: Use 512 as maximum number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Fix merging conflicts
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust subscriptions to use pinning API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Test subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head follow to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Adjust test.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/builder: Use new chainHead API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: ChainHead no longer uses `in_mem::children`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* chain_head: Add block state machine
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Address feedback
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Use new_native_or_wasm_executor
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Remove 'static on Backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Add documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Lock blocks before async blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head_follower: Remove static on backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* chain_head: Add BlockHeaderAbsent to the PartialEq impl
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client: Add better documentation around pinning constants
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Move subscription to dedicated module
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* subscription: Rename global pin / unpin functions
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Remove `Backend::apply_to_key_values_while`
* Add `IterArgs::start_at_exclusive`
* Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while`
* Remove `Backend::apply_to_keys_while`
* Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix`
* Remove unnecessary `to_vec` calls
* Fix unused method warning in no_std
* Remove unnecessary import
* Also check proof sizes in the test
* Iterate over both keys and values in `prove_range_read_with_size` and add a test
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Worker
* Reorganize and unpin onnotification drop
* Pin in state-db, pass block number
* Pin blocks in blockchain db
* Switch to reference counted LRU
* Disable pinning when we keep all blocks
* Fix pinning hint for state-db
* Remove pinning from backend layer
* Improve readability
* Add justifications to test
* Fix justification behaviour
* Remove debug prints
* Convert channels to tracing_unbounded
* Add comments to the test
* Documentation and Cleanup
* Move task start to client
* Simplify cache
* Improve test, remove unwanted log
* Add tracing logs, remove expect for block number
* Cleanup
* Add conversion method for unpin handle to Finalitynotification
* Revert unwanted changes
* Improve naming
* Make clippy happy
* Fix docs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Use `NumberFor` instead of u64 in API
* Hand over weak reference to unpin worker task
* Unwanted
* &Hash -> Hash
* Remove number from interface, rename `_unpin_handle`, LOG_TARGET
* Move RwLock one layer up
* Apply code style suggestions
* Improve comments
* Replace lru crate by schnellru
* Only insert values for pinned items + better docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Improve comments, log target and test
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Have `KeyIterator` clone the `prefix` it receives
* Stream keys in `storage_size` RPC and add a runtime limit
* Update client/rpc/Cargo.toml
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/rpc/src/state/utils.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Rename the types to signify that the cancellation is due to a timeout
* Move the test into a `mod tests`
* Add a comment regarding `biased` in `tokio::select`
* Make the `clone` explicit when calling `KeyIterator::{new, new_child}`
Co-authored-by: Bastian Köcher <git@kchr.de>
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* ed25519_verify: Support using dalek for historical blocks
The switch from `ed25519-dalek` to `ed25519-zebra` was actually a breaking change. `ed25519-zebra`
is more permissive. To support historical blocks when syncing a chain this pull request introduces
an externalities extension `UseDalekExt`. This extension is just used as a signaling mechanism to
`ed25519_verify` to use `ed25519-dalek` when it is present. Together with `ExtensionBeforeBlock` it
can be used to setup a node in way to sync historical blocks that require `ed25519-dalek`, because
they included a transaction that verified differently as when using `ed25519-zebra`.
This feature can be enabled in the following way. In the chain service file, directly after the
client is created, the following code should be added:
```
use sc_client_api::ExecutorProvider;
client.execution_extensions().set_extensions_factory(
sc_client_api::execution_extensions::ExtensionBeforeBlock::<Block, sp_io::UseDalekExt>::new(BLOCK_NUMBER_UNTIL_DALEK_SHOULD_BE_USED)
);
```
* Fix doc
* More fixes
* Update client/api/src/execution_extensions.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix merge and warning
* Fix docs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Backend::append_justification
It changes the arguments of `Backend::append_justification`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of `BlockId::Number` refactoring analysis (paritytech/substrate#11292)
* Error message improved
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* single error message in beefy::finalize
* println removed
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::begin_state_operation`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block
It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* Review suggestion applied
thx to @davxy
* trigger CI job
* BlockId removal: refactor: Finalizer
It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* minor corrections
* failing test corrected
* minor rework
* BlockId removal: refactor: ProofProvider
It changes the arguments of methods of `ProofProvider` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* LightClientRequestHandler: excessive BlockIdTo bound removed
* imports cleanup
* formatting
* args tyeps cleanup
* Remove native call
With the recent introduction of staging runtime apis the native call wasn't supported anymore. This
removes the entire support for this as it is not used anymore.
* FMT
* Fix benchmarks
* FIX ui tests
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fix leaf block removal in the backend
The fix introduced the new 'removal' method for the backend leaves set
and the improvement of the undo features.
* Update docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix docs typo
* On block block removal the new children list should be persisted.
* Align leaves set removal tests to the new interface
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Revert non-best block
This makes `revert` also revert non-best blocks.
* Update client/db/src/lib.rs
* Do not count leaves against the maximum number to revert
* Add some explanation
* Fix bug
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* sc-client-db: utils::open_database(...) — return OpenDbError so that the caller could tell the `OpenDbError::DoesNotExist` clearly
* sc-client-db: utils::open_database(..) — accept the `create: bool` argument
* sc-client-db: pruning — optional argument in the DatabaseSettings
* sc-state-db: Split `Error<E>` into separate `Error<E>` and `StateDbError`
* StateDb::open: choose the pruning-mode depending on the requested and stored values
* sc-state-db: test for different combinations of stored and requested pruning-modes
* CLI-argument: mark the unsafe-pruning as deprecated
* Fix tests
* tests: do not specify --pruning when running the substrate over the existing storage
* fix types for benches
* cargo fmt
* Check whether the pruning-mode and sync-mode are compatible
* cargo fmt
* parity-db: 0.3.11 -> 0.3.12
* sc-state-db: MetaDb::set_meta — a better doc-test
* cargo fmt
* make MetaDb read-only again!
* Remove the stray newline (and run the CI once again please)
* Last nitpicks
* A more comprehensive error message
Sebastian Kunert