mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-07-14 16:45:45 +00:00
73d9ae3284
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
165 lines
5.3 KiB
Rust
165 lines
5.3 KiB
Rust
// This file is part of Substrate.
|
|
|
|
// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd.
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! Rpc for state migration.
|
|
|
|
use jsonrpsee::{
|
|
core::{Error as JsonRpseeError, RpcResult},
|
|
proc_macros::rpc,
|
|
types::error::{CallError, ErrorCode, ErrorObject},
|
|
};
|
|
use sc_rpc_api::DenyUnsafe;
|
|
use serde::{Deserialize, Serialize};
|
|
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
|
|
use std::sync::Arc;
|
|
|
|
use sp_core::{
|
|
storage::{ChildInfo, ChildType, PrefixedStorageKey},
|
|
Hasher,
|
|
};
|
|
use sp_state_machine::backend::AsTrieBackend;
|
|
use sp_trie::{
|
|
trie_types::{TrieDB, TrieDBBuilder},
|
|
KeySpacedDB, Trie,
|
|
};
|
|
use trie_db::{
|
|
node::{NodePlan, ValuePlan},
|
|
TrieDBNodeIterator,
|
|
};
|
|
|
|
fn count_migrate<'a, H: Hasher>(
|
|
storage: &'a dyn trie_db::HashDBRef<H, Vec<u8>>,
|
|
root: &'a H::Out,
|
|
) -> std::result::Result<(u64, TrieDB<'a, 'a, H>), String> {
|
|
let mut nb = 0u64;
|
|
let trie = TrieDBBuilder::new(storage, root).build();
|
|
let iter_node =
|
|
TrieDBNodeIterator::new(&trie).map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
|
for node in iter_node {
|
|
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
|
match node.2.node_plan() {
|
|
NodePlan::Leaf { value, .. } | NodePlan::NibbledBranch { value: Some(value), .. } =>
|
|
if let ValuePlan::Inline(range) = value {
|
|
if (range.end - range.start) as u32 >=
|
|
sp_core::storage::TRIE_VALUE_NODE_THRESHOLD
|
|
{
|
|
nb += 1;
|
|
}
|
|
},
|
|
_ => (),
|
|
}
|
|
}
|
|
Ok((nb, trie))
|
|
}
|
|
|
|
/// Check trie migration status.
|
|
pub fn migration_status<H, B>(backend: &B) -> std::result::Result<(u64, u64), String>
|
|
where
|
|
H: Hasher,
|
|
H::Out: codec::Codec,
|
|
B: AsTrieBackend<H>,
|
|
{
|
|
let trie_backend = backend.as_trie_backend();
|
|
let essence = trie_backend.essence();
|
|
let (nb_to_migrate, trie) = count_migrate(essence, essence.root())?;
|
|
|
|
let mut nb_to_migrate_child = 0;
|
|
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
|
|
// get all child trie roots
|
|
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
|
|
let (key, value) = key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
|
if key[..].starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
|
|
{
|
|
let prefixed_key = PrefixedStorageKey::new(key);
|
|
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
|
|
child_roots.push((ChildInfo::new_default(unprefixed), value));
|
|
}
|
|
}
|
|
for (child_info, root) in child_roots {
|
|
let mut child_root = H::Out::default();
|
|
let storage = KeySpacedDB::new(essence, child_info.keyspace());
|
|
|
|
child_root.as_mut()[..].copy_from_slice(&root[..]);
|
|
nb_to_migrate_child += count_migrate(&storage, &child_root)?.0;
|
|
}
|
|
|
|
Ok((nb_to_migrate, nb_to_migrate_child))
|
|
}
|
|
|
|
#[derive(Serialize, Deserialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
#[serde(deny_unknown_fields)]
|
|
pub struct MigrationStatusResult {
|
|
top_remaining_to_migrate: u64,
|
|
child_remaining_to_migrate: u64,
|
|
}
|
|
|
|
/// Migration RPC methods.
|
|
#[rpc(server)]
|
|
pub trait StateMigrationApi<BlockHash> {
|
|
/// Check current migration state.
|
|
///
|
|
/// This call is performed locally without submitting any transactions. Thus executing this
|
|
/// won't change any state. Nonetheless it is a VERY costy call that should be
|
|
/// only exposed to trusted peers.
|
|
#[method(name = "state_trieMigrationStatus")]
|
|
fn call(&self, at: Option<BlockHash>) -> RpcResult<MigrationStatusResult>;
|
|
}
|
|
|
|
/// An implementation of state migration specific RPC methods.
|
|
pub struct StateMigration<C, B, BA> {
|
|
client: Arc<C>,
|
|
backend: Arc<BA>,
|
|
deny_unsafe: DenyUnsafe,
|
|
_marker: std::marker::PhantomData<(B, BA)>,
|
|
}
|
|
|
|
impl<C, B, BA> StateMigration<C, B, BA> {
|
|
/// Create new state migration rpc for the given reference to the client.
|
|
pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self {
|
|
StateMigration { client, backend, deny_unsafe, _marker: Default::default() }
|
|
}
|
|
}
|
|
|
|
impl<C, B, BA> StateMigrationApiServer<<B as BlockT>::Hash> for StateMigration<C, B, BA>
|
|
where
|
|
B: BlockT,
|
|
C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>,
|
|
BA: 'static + sc_client_api::backend::Backend<B>,
|
|
{
|
|
fn call(&self, at: Option<<B as BlockT>::Hash>) -> RpcResult<MigrationStatusResult> {
|
|
self.deny_unsafe.check_if_safe()?;
|
|
|
|
let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash));
|
|
let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?;
|
|
let (top, child) = migration_status(&state).map_err(error_into_rpc_err)?;
|
|
|
|
Ok(MigrationStatusResult {
|
|
top_remaining_to_migrate: top,
|
|
child_remaining_to_migrate: child,
|
|
})
|
|
}
|
|
}
|
|
|
|
fn error_into_rpc_err(err: impl std::fmt::Display) -> JsonRpseeError {
|
|
JsonRpseeError::Call(CallError::Custom(ErrorObject::owned(
|
|
ErrorCode::InternalError.code(),
|
|
"Error while checking migration state",
|
|
Some(err.to_string()),
|
|
)))
|
|
}
|