* rpc/chain_head: Add backend to subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Pin blocks internally and adjust testing
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: Reference for the number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check in-memory references to pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Remove unused comment
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Place subscription handle under `Arc` and unpin blocks on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/tests: Check all pinned blocks are unpinned on drop
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* rpc/tests: Retry fetching the pinned references for CI correctness
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/service: Use 512 as maximum number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Fix merging conflicts
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust subscriptions to use pinning API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Test subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head follow to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust chain_head.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head/tests: Adjust test.rs to the new API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/builder: Use new chainHead API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/in_mem: ChainHead no longer uses `in_mem::children`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/rpc-spec-v2/src/chain_head/subscription.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* chain_head: Add block state machine
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Address feedback
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Use new_native_or_wasm_executor
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Remove 'static on Backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Add documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Lock blocks before async blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head_follower: Remove static on backend
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update client/service/src/builder.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* chain_head: Add BlockHeaderAbsent to the PartialEq impl
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client: Add better documentation around pinning constants
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head: Move subscription to dedicated module
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* subscription: Rename global pin / unpin functions
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Worker
* Reorganize and unpin onnotification drop
* Pin in state-db, pass block number
* Pin blocks in blockchain db
* Switch to reference counted LRU
* Disable pinning when we keep all blocks
* Fix pinning hint for state-db
* Remove pinning from backend layer
* Improve readability
* Add justifications to test
* Fix justification behaviour
* Remove debug prints
* Convert channels to tracing_unbounded
* Add comments to the test
* Documentation and Cleanup
* Move task start to client
* Simplify cache
* Improve test, remove unwanted log
* Add tracing logs, remove expect for block number
* Cleanup
* Add conversion method for unpin handle to Finalitynotification
* Revert unwanted changes
* Improve naming
* Make clippy happy
* Fix docs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Use `NumberFor` instead of u64 in API
* Hand over weak reference to unpin worker task
* Unwanted
* &Hash -> Hash
* Remove number from interface, rename `_unpin_handle`, LOG_TARGET
* Move RwLock one layer up
* Apply code style suggestions
* Improve comments
* Replace lru crate by schnellru
* Only insert values for pinned items + better docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Improve comments, log target and test
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Backend::append_justification
It changes the arguments of `Backend::append_justification`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of `BlockId::Number` refactoring analysis (paritytech/substrate#11292)
* Error message improved
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* single error message in beefy::finalize
* println removed
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::begin_state_operation`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block
It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* Review suggestion applied
thx to @davxy
* trigger CI job
* sc-client-db: utils::open_database(...) — return OpenDbError so that the caller could tell the `OpenDbError::DoesNotExist` clearly
* sc-client-db: utils::open_database(..) — accept the `create: bool` argument
* sc-client-db: pruning — optional argument in the DatabaseSettings
* sc-state-db: Split `Error<E>` into separate `Error<E>` and `StateDbError`
* StateDb::open: choose the pruning-mode depending on the requested and stored values
* sc-state-db: test for different combinations of stored and requested pruning-modes
* CLI-argument: mark the unsafe-pruning as deprecated
* Fix tests
* tests: do not specify --pruning when running the substrate over the existing storage
* fix types for benches
* cargo fmt
* Check whether the pruning-mode and sync-mode are compatible
* cargo fmt
* parity-db: 0.3.11 -> 0.3.12
* sc-state-db: MetaDb::set_meta — a better doc-test
* cargo fmt
* make MetaDb read-only again!
* Remove the stray newline (and run the CI once again please)
* Last nitpicks
* A more comprehensive error message
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* starting
* Updated from other branch.
* setting flag
* flag in storage struct
* fix flagging to access and insert.
* added todo to fix
* also missing serialize meta to storage proof
* extract meta.
* Isolate old trie layout.
* failing test that requires storing in meta when old hash scheme is used.
* old hash compatibility
* Db migrate.
* runing tests with both states when interesting.
* fix chain spec test with serde default.
* export state (missing trie function).
* Pending using new branch, lacking genericity on layout resolution.
* extract and set global meta
* Update to branch 4
* fix iterator with root flag (no longer insert node).
* fix trie root hashing of root
* complete basic backend.
* Remove old_hash meta from proof that do not use inner_hashing.
* fix trie test for empty (force layout on empty deltas).
* Root update fix.
* debug on meta
* Use trie key iteration that do not include value in proofs.
* switch default test ext to use inner hash.
* small integration test, and fix tx cache mgmt in ext.
test failing
* Proof scenario at state-machine level.
* trace for db upgrade
* try different param
* act more like iter_from.
* Bigger batches.
* Update trie dependency.
* drafting codec changes and refact
* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.
* alt hashing only for branch with value.
* fix trie tests
* Hash of value include the encoded size.
* removing fields(broken)
* fix trie_stream to also include value length in inner hash.
* triedbmut only using alt type if inner hashing.
* trie_stream to also only use alt hashing type when actually alt hashing.
* Refactor meta state, logic should work with change of trie treshold.
* Remove NoMeta variant.
* Remove state_hashed trigger specific functions.
* pending switching to using threshold, new storage root api does not
make much sense.
* refactoring to use state from backend (not possible payload changes).
* Applying from previous state
* Remove default from storage, genesis need a special build.
* rem empty space
* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.
* fix compilation
* Right logic to avoid switch on reencode when default layout.
* Clean up some todos
* remove trie meta from root upstream
* update upstream and fix benches.
* split some long lines.
* UPdate trie crate to work with new design.
* Finish update to refactored upstream.
* update to latest triedb changes.
* Clean up.
* fix executor test.
* rust fmt from master.
* rust format.
* rustfmt
* fix
* start host function driven versioning
* update state-machine part
* still need access to state version from runtime
* state hash in mem: wrong
* direction likely correct, but passing call to code exec for genesis
init seem awkward.
* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.
* stateversion from runtime version (core api >= 4).
* update trie, fix tests
* unused import
* clean some TODOs
* Require RuntimeVersionOf for executor
* use RuntimeVersionOf to resolve genesis state version.
* update runtime version test
* fix state-machine tests
* TODO
* Use runtime version from storage wasm with fast sync.
* rustfmt
* fmt
* fix test
* revert useless changes.
* clean some unused changes
* fmt
* removing useless trait function.
* remove remaining reference to state_hash
* fix some imports
* Follow chain state version management.
* trie update, fix and constant threshold for trie layouts.
* update deps
* Update to latest trie pr changes.
* fix benches
* Verify proof requires right layout.
* update trie_root
* Update trie deps to latest
* Update to latest trie versioning
* Removing patch
* update lock
* extrinsic for sc-service-test using layout v0.
* Adding RuntimeVersionOf to CallExecutor works.
* fmt
* error when resolving version and no wasm in storage.
* use existing utils to instantiate runtime code.
* Patch to delay runtime switch.
* Revert "Patch to delay runtime switch."
This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.
* useless closure
* remove remaining state_hash variables.
* Remove outdated comment
* useless inner hash
* fmt
* fmt and opt-in feature to apply state change.
* feature gate core version, use new test feature for node and test node
* Use a 'State' api version instead of Core one.
* fix merge of test function
* use blake macro.
* Fix state api (require declaring the api in runtime).
* Opt out feature, fix macro for io to select a given version
instead of latest.
* run test nodes on new state.
* fix
* Apply review change (docs and error).
* fmt
* use explicit runtime_interface in doc test
* fix ui test
* fix doc test
* fmt
* use default for path and specname when resolving version.
* small review related changes.
* doc value size requirement.
* rename old_state feature
* Remove macro changes
* feature rename
* state version as host function parameter
* remove flag for client api
* fix tests
* switch storage chain proof to V1
* host functions, pass by state version enum
* use WrappedRuntimeCode
* start
* state_version in runtime version
* rust fmt
* Update storage proof of max size.
* fix runtime version rpc test
* right intent of convert from compat
* fix doc test
* fix doc test
* split proof
* decode without replay, and remove some reexports.
* Decode with compatibility by default.
* switch state_version to u8. And remove RuntimeVersionBasis.
* test
* use api when reading embedded version
* fix decode with apis
* extract core version instead
* test fix
* unused import
* review changes.
Co-authored-by: kianenigma <kian@parity.io>
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Simplify some Option / Result / ? operator patterns
When those match a combinator exactly.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
* adjust after review
* adjust post-review
* primitives/runtime: initial changes on supporting multiple Justifications
* primitives/runtime: make Justifications strongly typed
* Encode/decode Justifications
* primitives/runtime: add Justification type
* backend: apply_finality and finalize_block takes a single Justification
* manual-seal: create engine id and let rpc take encoded justification
* backend: skeleton functions for appending justifications
* backend: initial implementation append_justification
Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k
* backend: guard against duplicate consensus engine id
* client/db: add check for block finality
* client/api: add append_justification to in_mem db
* client/light: add no-op append_justification
* network: fix decode call for Justification
* network: only send a single Justification in BlockData
* network: minor comment update
* protocol: update field names to distinguish single justification
* client: further field renames to plural
* client: update function names to plural justifications
* client/db: upgrade existing database for new format
* network: remove dependency on grandpa crate
* db: fix check for finalized block
* grandpa: check for multiple grandpa justifications hwne importing
* backend: update Finalizer trait to take multiple Justifications
* db: remove debugging statements in migration code
* manual-seal: update note about engine id
* db: fix check for finalized block
* client: update variable name to reflect it is now plural
* grandpa: fix incorrect empty Justications in test
* primitives: make Justifications opaque to avoid being empty
* network: fix detecting empty Justification
* runtime: doc strings for Justifications functions
* runtime: add into_justifications
* primitives: check for duplicates in when adding to Justifications
* network/test: use real grandpa engine id in test
* client: fix reviewer comments
* primitives: rename Justifications::push to append
* backend: revert changes to Finalizer trait
* backend: revert mark_finalized
* backend: revert changes to finalize_block
* backend: revert finalized_blocks
* db: add a quick early return for performance
* client: minor reviewer comments
* service/test: use local ConsensusEngineId
* network: add link to issue for sending multiple Justifications
* Apply suggestions from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Apply suggestions from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* network: tweaks to review suggestions
* network: revert change to BlockData for backwards compatibility
* Apply suggestion from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* primitives: update doc comment for Justifications
* client/db/upgrade: avoid grandpa crate dependency
* consensus: revert to single Justification for import_justification
* primitives: improve justifications docs
* style cleanups
* use and_then
* client: rename JUSTIFICATIONS db column
* network: revert to using FRNK in network-test
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Generate CHT roots on a full client
* add changes_trie_root function
* Add a test
* Line widths
* Fix sc-service-test
* Clarify comments
* Revert comments
* Reset code, almost ready for PR
* Improved build_hardcoded_spec
* Fix line widths
* Fix tests
* Fix sc-service-test
* Suggestions from code review
* Rename to LightSyncState
* It's not syncing :^(
* It syncs!
* Remove rpc call
* Convert spaces to tabs
* Moved sc-service things to export_sync_state.rs
* Fix tests
* Wait for syncing with network_status_sinks
* Remove sc-network from node-template
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Various changes, split the flag up into 2 pieces to make testing easier.
* Update client/cli/src/commands/build_spec_cmd.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Revert a lot of changes
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Allow blacklisting blocks from being finalized again after block revert
* Use BlockRules for storing unfinalized and add have_state_at in revert
* Move finalization_check in finalize_block upward
* Directly mark finalization blacklist as badblocks
* Remove obselete comment
* Simplify a few chain components creation APIs related to the service
* Fix basic-authorship doc tests
* Remove DefaultQueue
* Update client/service/src/builder.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Move ExecutionExtensions comment around
* Remove unused BlakeTwo256
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Don't clone values when calculating storage root
Instead of cloning all the keys and values of the overlay when
calculating the storage root, we pass all the values by reference. This
should probably bring some performance improvements when calculating the
storage root.
* no cow version (#6113)
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api
* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa
* drop sc-service from sc-rpc
* drop sc-service from sc-consensus-aura
* drop sc-client from manual-seal and babe
* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli
* drop sc-client from bin/node and bin/node-template
* drop sc-client
* fix tests
* remove check -p sc-client from gitlab.yml
* fix warnings
* fixes ui test
* fix light client tests
* adds associated Client type to AbstractService
* adds UsageProvider to Client
* fixed ui test, again
* tried and failed to get node-cli to compile for wasm
* thanks to tomaka for helping me get node-cli to compile for wasmm
* ui test pls pas 🙏🏾
* all tests passing 🪄
* no_run documentation code
* rm -f documentation code
* ClientProvider
* fix mega trait
* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client
* adds license to sc-consensus
Co-authored-by: Benjamin Kampmann <ben@parity.io>
Bastian Köcher