Files
pezkuwi-subxt/substrate/utils/frame/rpc/state-trie-migration-rpc/src/lib.rs
T
Bastian Köcher 73d9ae3284 Introduce trie level cache and remove state cache (#11407)
* trie state cache

* Also cache missing access on read.

* fix comp

* bis

* fix

* use has_lru

* remove local storage cache on size 0.

* No cache.

* local cache only

* trie cache and local cache

* storage cache (with local)

* trie cache no local cache

* Add state access benchmark

* Remove warnings etc

* Add trie cache benchmark

* No extra "clone" required

* Change benchmark to use multiple blocks

* Use patches

* Integrate shitty implementation

* More stuff

* Revert "Merge branch 'master' into trie_state_cache"

This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.

* Improve benchmark

* Adapt to latest changes

* Adapt to changes in trie

* Add a test that uses iterator

* Start fixing it

* Remove obsolete file

* Make it compile

* Start rewriting the trie node cache

* More work on the cache

* More docs and code etc

* Make data cache an optional

* Tests

* Remove debug stuff

* Recorder

* Some docs and a simple test for the recorder

* Compile fixes

* Make it compile

* More fixes

* More fixes

* Fix fix fix

* Make sure cache and recorder work together for basic stuff

* Test that data caching and recording works

* Test `TrieDBMut` with caching

* Try something

* Fixes, fixes, fixes

* Forward the recorder

* Make it compile

* Use recorder in more places

* Switch to new `with_optional_recorder` fn

* Refactor and cleanups

* Move `ProvingBackend` tests

* Simplify

* Move over all functionality to the essence

* Fix compilation

* Implement estimate encoded size for StorageProof

* Start using the `cache` everywhere

* Use the cache everywhere

* Fix compilation

* Fix tests

* Adds `TrieBackendBuilder` and enhances the tests

* Ensure that recorder drain checks that values are found as expected

* Switch over to `TrieBackendBuilder`

* Start fixing the problem with child tries and recording

* Fix recording of child tries

* Make it compile

* Overwrite `storage_hash` in `TrieBackend`

* Add `storage_cache` to  the benchmarks

* Fix `no_std` build

* Speed up cache lookup

* Extend the state access benchmark to also hash a runtime

* Fix build

* Fix compilation

* Rewrite value cache

* Add lru cache

* Ensure that the cache lru works

* Value cache should not be optional

* Add support for keeping the shared node cache in its bounds

* Make the cache configurable

* Check that the cache respects the bounds

* Adds a new test

* Fixes

* Docs and some renamings

* More docs

* Start using the new recorder

* Fix more code

* Take `self` argument

* Remove warnings

* Fix benchmark

* Fix accounting

* Rip off the state cache

* Start fixing fallout after removing the state cache

* Make it compile after trie changes

* Fix test

* Add some logging

* Some docs

* Some fixups and clean ups

* Fix benchmark

* Remove unneeded file

* Use git for patching

* Make CI happy

* Update primitives/trie/Cargo.toml

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Introduce new `AsTrieBackend` trait

* Make the LocalTrieCache not clonable

* Make it work in no_std and add docs

* Remove duplicate dependency

* Switch to ahash for better performance

* Speedup value cache merge

* Output errors on underflow

* Ensure the internal LRU map doesn't grow too much

* Use const fn to calculate the value cache element size

* Remove cache configuration

* Fix

* Clear the cache in between for more testing

* Try to come up with a failing test case

* Make the test fail

* Fix the child trie recording

* Make everything compile after the changes to trie

* Adapt to latest trie-db changes

* Fix on stable

* Update primitives/trie/src/cache.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fix wrong merge

* Docs

* Fix warnings

* Cargo.lock

* Bump pin-project

* Fix warnings

* Switch to released crate version

* More fixes

* Make clippy and rustdocs happy

* More clippy

* Print error when using deprecated `--state-cache-size`

* 🤦

* Fixes

* Fix storage_hash linkings

* Update client/rpc/src/dev/mod.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Review feedback

* encode bound

* Rework the shared value cache

Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.

* Improve of the cache by merging the old and new solution

* FMT

* Please stop coming back all the time :crying:

* Update primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Fixes

* Make clippy happy

* Ensure we don't deadlock

* Only use one lock to simplify the code

* Do not depend on `Hasher`

* Fix tests

* FMT

* Clippy 🤦

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2022-08-18 18:59:22 +00:00

165 lines
5.3 KiB
Rust

// This file is part of Substrate.
// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Rpc for state migration.
use jsonrpsee::{
core::{Error as JsonRpseeError, RpcResult},
proc_macros::rpc,
types::error::{CallError, ErrorCode, ErrorObject},
};
use sc_rpc_api::DenyUnsafe;
use serde::{Deserialize, Serialize};
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
use std::sync::Arc;
use sp_core::{
storage::{ChildInfo, ChildType, PrefixedStorageKey},
Hasher,
};
use sp_state_machine::backend::AsTrieBackend;
use sp_trie::{
trie_types::{TrieDB, TrieDBBuilder},
KeySpacedDB, Trie,
};
use trie_db::{
node::{NodePlan, ValuePlan},
TrieDBNodeIterator,
};
fn count_migrate<'a, H: Hasher>(
storage: &'a dyn trie_db::HashDBRef<H, Vec<u8>>,
root: &'a H::Out,
) -> std::result::Result<(u64, TrieDB<'a, 'a, H>), String> {
let mut nb = 0u64;
let trie = TrieDBBuilder::new(storage, root).build();
let iter_node =
TrieDBNodeIterator::new(&trie).map_err(|e| format!("TrieDB node iterator error: {}", e))?;
for node in iter_node {
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
match node.2.node_plan() {
NodePlan::Leaf { value, .. } | NodePlan::NibbledBranch { value: Some(value), .. } =>
if let ValuePlan::Inline(range) = value {
if (range.end - range.start) as u32 >=
sp_core::storage::TRIE_VALUE_NODE_THRESHOLD
{
nb += 1;
}
},
_ => (),
}
}
Ok((nb, trie))
}
/// Check trie migration status.
pub fn migration_status<H, B>(backend: &B) -> std::result::Result<(u64, u64), String>
where
H: Hasher,
H::Out: codec::Codec,
B: AsTrieBackend<H>,
{
let trie_backend = backend.as_trie_backend();
let essence = trie_backend.essence();
let (nb_to_migrate, trie) = count_migrate(essence, essence.root())?;
let mut nb_to_migrate_child = 0;
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
// get all child trie roots
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
let (key, value) = key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
if key[..].starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
{
let prefixed_key = PrefixedStorageKey::new(key);
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
child_roots.push((ChildInfo::new_default(unprefixed), value));
}
}
for (child_info, root) in child_roots {
let mut child_root = H::Out::default();
let storage = KeySpacedDB::new(essence, child_info.keyspace());
child_root.as_mut()[..].copy_from_slice(&root[..]);
nb_to_migrate_child += count_migrate(&storage, &child_root)?.0;
}
Ok((nb_to_migrate, nb_to_migrate_child))
}
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
#[serde(deny_unknown_fields)]
pub struct MigrationStatusResult {
top_remaining_to_migrate: u64,
child_remaining_to_migrate: u64,
}
/// Migration RPC methods.
#[rpc(server)]
pub trait StateMigrationApi<BlockHash> {
/// Check current migration state.
///
/// This call is performed locally without submitting any transactions. Thus executing this
/// won't change any state. Nonetheless it is a VERY costy call that should be
/// only exposed to trusted peers.
#[method(name = "state_trieMigrationStatus")]
fn call(&self, at: Option<BlockHash>) -> RpcResult<MigrationStatusResult>;
}
/// An implementation of state migration specific RPC methods.
pub struct StateMigration<C, B, BA> {
client: Arc<C>,
backend: Arc<BA>,
deny_unsafe: DenyUnsafe,
_marker: std::marker::PhantomData<(B, BA)>,
}
impl<C, B, BA> StateMigration<C, B, BA> {
/// Create new state migration rpc for the given reference to the client.
pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self {
StateMigration { client, backend, deny_unsafe, _marker: Default::default() }
}
}
impl<C, B, BA> StateMigrationApiServer<<B as BlockT>::Hash> for StateMigration<C, B, BA>
where
B: BlockT,
C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>,
BA: 'static + sc_client_api::backend::Backend<B>,
{
fn call(&self, at: Option<<B as BlockT>::Hash>) -> RpcResult<MigrationStatusResult> {
self.deny_unsafe.check_if_safe()?;
let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash));
let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?;
let (top, child) = migration_status(&state).map_err(error_into_rpc_err)?;
Ok(MigrationStatusResult {
top_remaining_to_migrate: top,
child_remaining_to_migrate: child,
})
}
}
fn error_into_rpc_err(err: impl std::fmt::Display) -> JsonRpseeError {
JsonRpseeError::Call(CallError::Custom(ErrorObject::owned(
ErrorCode::InternalError.code(),
"Error while checking migration state",
Some(err.to_string()),
)))
}