While investigating some pruning issues I found some room for
improvement in the notification pin handling.
**Problem:** It was not possible to define an upper limit on
notification pins. The block pinning cache has a limit, but only handles
bodies and justifications.
After this PR, bookkeeping for notifications is managed in the pinning
worker. A limit can be defined in the worker. If that limit is crossed,
blocks that were pinned for that notification are unpinned, which now
affects the state as well as bodies and justifications. The pinned
blocks cache still has a limit, but should never be hit.
closes#19
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
- Set log to `0.4.20` everywhere
- Lift `log` to the workspace
Starting with a simpler one after seeing
https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw.
This sets the `default-features` to `false` in the root and then
overwrites that in each create to its original value. This is necessary
since otherwise the `default` features are additive and its impossible
to disable them in the crate again once they are enabled in the
workspace.
I am using a tool to do this, so its mostly a test to see that it works
as expected.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.
There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.
Dependencies:
- [x] PR that upgrades CI to use rust 1.74 is merged.
---------
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.
If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html
@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.
TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR provides the infrastructure for the pov-reclaim mechanism
discussed in #209. The goal is to provide the current proof size to the
runtime so it can be used to reclaim storage weight.
## New Host Function
- A new host function is provided
[here](https://github.com/skunert/polkadot-sdk/blob/5b317fda3be205f4136f10d4490387ccd4f9765d/cumulus/primitives/pov-reclaim/src/lib.rs#L23).
It returns the size of the current proof size to the runtime. If
recording is not enabled, it returns 0.
## Implementation Overview
- Implement option to enable proof recording during import in the
client. This is currently enabled for `polkadot-parachain`,
`parachain-template` and the cumulus test node.
- Make the proof recorder ready for no-std. It was previously only
enabled for std environments, but we need to record the proof size in
`validate_block` too.
- Provide a recorder implementation that only the records the size of
incoming nodes and does not store the nodes itself.
- Fix benchmarks that were broken by async backing changes
- Provide new externalities extension that is registered by default if
proof recording is enabled.
- I think we should discuss the naming, pov-reclaim was more intuitive
to me, but we could also go with clawback like in the issue.
## Impact of proof recording during import
With proof recording: 6.3058 Kelem/s
Without proof recording: 6.3427 Kelem/s
The measured impact on the importing performance is quite low on my
machine using the block import benchmark. With proof recording I am
seeing a performance hit of 0.585%.
---------
Co-authored-by: command-bot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
This moves the macro related re-exports to `__private` to make it more
obvious for downstream users that they are using an internal api.
---------
Co-authored-by: command-bot <>
* Start
* More work!
* Moar
* More changes
* More fixes
* More worrk
* More fixes
* More fixes to make it compile
* Adds `NoOffchainStorage`
* Pass the extensions
* Small basti making small progress
* Fix merge errors and remove `ExecutionContext`
* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`
Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.
* Fix compilation
* Register the global extensions inside runtime api instance
* Fixes
* Fix `generate_initial_session_keys` by passing the keystore extension
* Fix the grandpa tests
* Fix more tests
* Fix more tests
* Don't set any heap pages if there isn't an override
* Fix small fallout
* FMT
* Fix tests
* More tests
* Offchain worker custom extensions
* More fixes
* Make offchain tx pool creation reusable
Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.
* Fixes
* Fixes
* Set offchain transaction pool in BABE before using it in the runtime
* Add the `offchain_tx_pool` to Grandpa as well
* Fix the nodes
* Print some error when using the old warnings
* Fix merge issues
* Fix compilation
* Rename `babe_link`
* Rename to `offchain_tx_pool_factory`
* Cleanup
* FMT
* Fix benchmark name
* Fix `try-runtime`
* Remove `--execution` CLI args
* Make clippy happy
* Forward bls functions
* Fix docs
* Update UI tests
* Update client/api/src/execution_extensions.rs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/cli/src/params/import_params.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/api/src/execution_extensions.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Pass the offchain storage to the MMR RPC
* Update client/api/src/execution_extensions.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Review comments
* Fixes
---------
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Make offchain tx pool creation reusable
Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.
* Update client/transaction-pool/api/src/lib.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* rpc/chain_head: Add backend to subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Pin blocks internally and adjust testing
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: Reference for the number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check in-memory references to pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Remove unused comment
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Place subscription handle under `Arc` and unpin blocks on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check all pinned blocks are unpinned on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* rpc/tests: Retry fetching the pinned references for CI correctness
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/service: Use 512 as maximum number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Fix merging conflicts
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust subscriptions to use pinning API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Test subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head follow to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Adjust test.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/builder: Use new chainHead API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: ChainHead no longer uses `in_mem::children`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* chain_head: Add block state machine
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Address feedback
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Use new_native_or_wasm_executor
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Remove 'static on Backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Add documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Lock blocks before async blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head_follower: Remove static on backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* chain_head: Add BlockHeaderAbsent to the PartialEq impl
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client: Add better documentation around pinning constants
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Move subscription to dedicated module
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* subscription: Rename global pin / unpin functions
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Remove `Backend::apply_to_key_values_while`
* Add `IterArgs::start_at_exclusive`
* Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while`
* Remove `Backend::apply_to_keys_while`
* Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix`
* Remove unnecessary `to_vec` calls
* Fix unused method warning in no_std
* Remove unnecessary import
* Also check proof sizes in the test
* Iterate over both keys and values in `prove_range_read_with_size` and add a test
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Worker
* Reorganize and unpin onnotification drop
* Pin in state-db, pass block number
* Pin blocks in blockchain db
* Switch to reference counted LRU
* Disable pinning when we keep all blocks
* Fix pinning hint for state-db
* Remove pinning from backend layer
* Improve readability
* Add justifications to test
* Fix justification behaviour
* Remove debug prints
* Convert channels to tracing_unbounded
* Add comments to the test
* Documentation and Cleanup
* Move task start to client
* Simplify cache
* Improve test, remove unwanted log
* Add tracing logs, remove expect for block number
* Cleanup
* Add conversion method for unpin handle to Finalitynotification
* Revert unwanted changes
* Improve naming
* Make clippy happy
* Fix docs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Use `NumberFor` instead of u64 in API
* Hand over weak reference to unpin worker task
* Unwanted
* &Hash -> Hash
* Remove number from interface, rename `_unpin_handle`, LOG_TARGET
* Move RwLock one layer up
* Apply code style suggestions
* Improve comments
* Replace lru crate by schnellru
* Only insert values for pinned items + better docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Improve comments, log target and test
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Have `KeyIterator` clone the `prefix` it receives
* Stream keys in `storage_size` RPC and add a runtime limit
* Update client/rpc/Cargo.toml
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/rpc/src/state/utils.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Rename the types to signify that the cancellation is due to a timeout
* Move the test into a `mod tests`
* Add a comment regarding `biased` in `tokio::select`
* Make the `clone` explicit when calling `KeyIterator::{new, new_child}`
Co-authored-by: Bastian Köcher <git@kchr.de>
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* ed25519_verify: Support using dalek for historical blocks
The switch from `ed25519-dalek` to `ed25519-zebra` was actually a breaking change. `ed25519-zebra`
is more permissive. To support historical blocks when syncing a chain this pull request introduces
an externalities extension `UseDalekExt`. This extension is just used as a signaling mechanism to
`ed25519_verify` to use `ed25519-dalek` when it is present. Together with `ExtensionBeforeBlock` it
can be used to setup a node in way to sync historical blocks that require `ed25519-dalek`, because
they included a transaction that verified differently as when using `ed25519-zebra`.
This feature can be enabled in the following way. In the chain service file, directly after the
client is created, the following code should be added:
```
use sc_client_api::ExecutorProvider;
client.execution_extensions().set_extensions_factory(
sc_client_api::execution_extensions::ExtensionBeforeBlock::<Block, sp_io::UseDalekExt>::new(BLOCK_NUMBER_UNTIL_DALEK_SHOULD_BE_USED)
);
```
* Fix doc
* More fixes
* Update client/api/src/execution_extensions.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix merge and warning
* Fix docs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
Sebastian Kunert