* Prune discarded blocks immediately
* state-db refactoring part 1
* Some renames
* Get rid of pending state
* Revert "Prune discarded blocks immediately"
This reverts commit b60d98c569e8af18d99087da93f0911d4f24006e.
* Cleanup
* Make clippy happy
* Minor changes
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Backend::append_justification
It changes the arguments of `Backend::append_justification`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of `BlockId::Number` refactoring analysis (paritytech/substrate#11292)
* Error message improved
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* single error message in beefy::finalize
* println removed
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::begin_state_operation`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block
It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* Review suggestion applied
thx to @davxy
* trigger CI job
* 1.Add pruning param "canonical" in sc-cli.
2.Make PruningMode's default value to ArchiveCanonical.
* Update tests in sc-state-db.
* Update tests in sc-state-db.
* 1.Add a new value `AllWithNonFinalized` in `enum BlocksPruning` which Corresponds to `blocks_pruning 0` in CLI .
2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history.
* Make some corresponding adjustments based on the content in the conversation.
* Update client/db/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review.
* 1.Change `blocks_pruning` to be like `state_pruning` .
* Fmt and add some doc.
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update doc.
* Change `new_test_with_tx_storage` to take `BlocksPruning`.
* Fmt
Co-authored-by: Bastian Köcher <git@kchr.de>
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fix leaf block removal in the backend
The fix introduced the new 'removal' method for the backend leaves set
and the improvement of the undo features.
* Update docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix docs typo
* On block block removal the new children list should be persisted.
* Align leaves set removal tests to the new interface
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Prevent duplicated leaves in the backend
* Comments...
* Use highest known heaf as a shortcut for not existing header detection
* Apply code review suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Revert non-best block
This makes `revert` also revert non-best blocks.
* Update client/db/src/lib.rs
* Do not count leaves against the maximum number to revert
* Add some explanation
* Fix bug
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update UI test output for rust 1.62.1
* switch ci to staging image to check that everything works
* fix artifacts node-bench-regression-guard
* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning
* Fix more clippy lints
* Make clippy happy by relying on auto-deref were possible
* Add tracking issue to the comments
* pin ci image
Co-authored-by: alvicsam <alvicsam@gmail.com>
* support for release as in kvdb (only if no rc).
* Start impl
* minimal implementation for paritydb rc
* Update client/db/src/parity_db.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Update client/db/src/parity_db.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Update client/db/src/parity_db.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Commit not panicking in DbAdapter
* errors from string
* update parity db version
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* sc-client-db: utils::open_database(...) — return OpenDbError so that the caller could tell the `OpenDbError::DoesNotExist` clearly
* sc-client-db: utils::open_database(..) — accept the `create: bool` argument
* sc-client-db: pruning — optional argument in the DatabaseSettings
* sc-state-db: Split `Error<E>` into separate `Error<E>` and `StateDbError`
* StateDb::open: choose the pruning-mode depending on the requested and stored values
* sc-state-db: test for different combinations of stored and requested pruning-modes
* CLI-argument: mark the unsafe-pruning as deprecated
* Fix tests
* tests: do not specify --pruning when running the substrate over the existing storage
* fix types for benches
* cargo fmt
* Check whether the pruning-mode and sync-mode are compatible
* cargo fmt
* parity-db: 0.3.11 -> 0.3.12
* sc-state-db: MetaDb::set_meta — a better doc-test
* cargo fmt
* make MetaDb read-only again!
* Remove the stray newline (and run the CI once again please)
* Last nitpicks
* A more comprehensive error message
* beefy-gadget: allow custom runtime api provider
* beefy-gadget: use mock runtime api in tests
* pallet-mmr: expose mmr root from state through runtime API
* beefy-gadget: get mmr root from runtime state
* pallet-beefy-mmr: remove MmrRoot from header digests
* frame/mmr: move mmr primitives out of frame
* frame/mmr: completely move primitives out of frame
* address review comments
* beefy-mmr: bring back mmr root from header digest
* clippy fixes for rustc 1.60
* address review comments
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Prevent possible rocksdb-corruption by running benchmark-storage
Also adds a `sanitize_key` function to strip path-prefixes from the db-keys (for databases that
use prefixed keys such as rocksdb)
fixes#10998
* Fix @cheme 's annotations.
* Update utils/frame/benchmarking-cli/src/storage/write.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make logic match the name of bool flag `invert_inserts`
* Remove unused lifetime
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Upgraded dependencies
* Adapting code to scale v3
* Empty commit to trigger CI
* Triggering CI
* Fixing UI test
* Remove superfluous dev-dep added by #9228
* Cryout for CI
cheme