mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-05-09 00:08:00 +00:00
73d9ae3284
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
247 lines
7.2 KiB
Rust
247 lines
7.2 KiB
Rust
// This file is part of Substrate.
|
|
|
|
// Copyright (C) 2017-2022 Parity Technologies (UK) Ltd.
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! State machine in memory backend.
|
|
|
|
use crate::{
|
|
backend::Backend, trie_backend::TrieBackend, StorageCollection, StorageKey, StorageValue,
|
|
TrieBackendBuilder,
|
|
};
|
|
use codec::Codec;
|
|
use hash_db::Hasher;
|
|
use sp_core::storage::{ChildInfo, StateVersion, Storage};
|
|
use sp_trie::{empty_trie_root, GenericMemoryDB, HashKey, KeyFunction, LayoutV1, MemoryDB};
|
|
use std::collections::{BTreeMap, HashMap};
|
|
|
|
/// Create a new empty instance of in-memory backend.
|
|
///
|
|
/// It will use [`HashKey`] to store the keys internally.
|
|
pub fn new_in_mem_hash_key<H>() -> TrieBackend<MemoryDB<H>, H>
|
|
where
|
|
H: Hasher,
|
|
H::Out: Codec + Ord,
|
|
{
|
|
new_in_mem::<H, HashKey<H>>()
|
|
}
|
|
|
|
/// Create a new empty instance of in-memory backend.
|
|
pub fn new_in_mem<H, KF>() -> TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H: Hasher,
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
let db = GenericMemoryDB::default();
|
|
// V1 is same as V0 for an empty trie.
|
|
TrieBackendBuilder::new(db, empty_trie_root::<LayoutV1<H>>()).build()
|
|
}
|
|
|
|
impl<H: Hasher, KF> TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
/// Copy the state, with applied updates
|
|
pub fn update<T: IntoIterator<Item = (Option<ChildInfo>, StorageCollection)>>(
|
|
&self,
|
|
changes: T,
|
|
state_version: StateVersion,
|
|
) -> Self {
|
|
let mut clone = self.clone();
|
|
clone.insert(changes, state_version);
|
|
clone
|
|
}
|
|
|
|
/// Insert values into backend trie.
|
|
pub fn insert<T: IntoIterator<Item = (Option<ChildInfo>, StorageCollection)>>(
|
|
&mut self,
|
|
changes: T,
|
|
state_version: StateVersion,
|
|
) {
|
|
let (top, child) = changes.into_iter().partition::<Vec<_>, _>(|v| v.0.is_none());
|
|
let (root, transaction) = self.full_storage_root(
|
|
top.iter().flat_map(|(_, v)| v).map(|(k, v)| (&k[..], v.as_deref())),
|
|
child.iter().filter_map(|v| {
|
|
v.0.as_ref().map(|c| (c, v.1.iter().map(|(k, v)| (&k[..], v.as_deref()))))
|
|
}),
|
|
state_version,
|
|
);
|
|
|
|
self.apply_transaction(root, transaction);
|
|
}
|
|
|
|
/// Merge trie nodes into this backend.
|
|
pub fn update_backend(&self, root: H::Out, changes: GenericMemoryDB<H, KF>) -> Self {
|
|
let mut clone = self.backend_storage().clone();
|
|
clone.consolidate(changes);
|
|
TrieBackendBuilder::new(clone, root).build()
|
|
}
|
|
|
|
/// Apply the given transaction to this backend and set the root to the given value.
|
|
pub fn apply_transaction(&mut self, root: H::Out, transaction: GenericMemoryDB<H, KF>) {
|
|
let mut storage = sp_std::mem::take(self).into_storage();
|
|
storage.consolidate(transaction);
|
|
*self = TrieBackendBuilder::new(storage, root).build();
|
|
}
|
|
|
|
/// Compare with another in-memory backend.
|
|
pub fn eq(&self, other: &Self) -> bool {
|
|
self.root() == other.root()
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher, KF> Clone for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn clone(&self) -> Self {
|
|
TrieBackendBuilder::new(self.backend_storage().clone(), *self.root()).build()
|
|
}
|
|
}
|
|
|
|
impl<H, KF> Default for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H: Hasher,
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn default() -> Self {
|
|
new_in_mem()
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher, KF>
|
|
From<(HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>, StateVersion)>
|
|
for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn from(
|
|
(inner, state_version): (
|
|
HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>,
|
|
StateVersion,
|
|
),
|
|
) -> Self {
|
|
let mut backend = new_in_mem();
|
|
backend.insert(
|
|
inner
|
|
.into_iter()
|
|
.map(|(k, m)| (k, m.into_iter().map(|(k, v)| (k, Some(v))).collect())),
|
|
state_version,
|
|
);
|
|
backend
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher, KF> From<(Storage, StateVersion)> for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn from((inners, state_version): (Storage, StateVersion)) -> Self {
|
|
let mut inner: HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>> = inners
|
|
.children_default
|
|
.into_iter()
|
|
.map(|(_k, c)| (Some(c.child_info), c.data))
|
|
.collect();
|
|
inner.insert(None, inners.top);
|
|
(inner, state_version).into()
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher, KF> From<(BTreeMap<StorageKey, StorageValue>, StateVersion)>
|
|
for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn from((inner, state_version): (BTreeMap<StorageKey, StorageValue>, StateVersion)) -> Self {
|
|
let mut expanded = HashMap::new();
|
|
expanded.insert(None, inner);
|
|
(expanded, state_version).into()
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher, KF> From<(Vec<(Option<ChildInfo>, StorageCollection)>, StateVersion)>
|
|
for TrieBackend<GenericMemoryDB<H, KF>, H>
|
|
where
|
|
H::Out: Codec + Ord,
|
|
KF: KeyFunction<H> + Send + Sync,
|
|
{
|
|
fn from(
|
|
(inner, state_version): (Vec<(Option<ChildInfo>, StorageCollection)>, StateVersion),
|
|
) -> Self {
|
|
let mut expanded: HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>> =
|
|
HashMap::new();
|
|
for (child_info, key_values) in inner {
|
|
let entry = expanded.entry(child_info).or_default();
|
|
for (key, value) in key_values {
|
|
if let Some(value) = value {
|
|
entry.insert(key, value);
|
|
}
|
|
}
|
|
}
|
|
(expanded, state_version).into()
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use crate::backend::{AsTrieBackend, Backend};
|
|
use sp_core::storage::StateVersion;
|
|
use sp_runtime::traits::BlakeTwo256;
|
|
|
|
/// Assert in memory backend with only child trie keys works as trie backend.
|
|
#[test]
|
|
fn in_memory_with_child_trie_only() {
|
|
let state_version = StateVersion::default();
|
|
let storage = new_in_mem_hash_key::<BlakeTwo256>();
|
|
let child_info = ChildInfo::new_default(b"1");
|
|
let child_info = &child_info;
|
|
let storage = storage.update(
|
|
vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])],
|
|
state_version,
|
|
);
|
|
let trie_backend = storage.as_trie_backend();
|
|
assert_eq!(trie_backend.child_storage(child_info, b"2").unwrap(), Some(b"3".to_vec()));
|
|
let storage_key = child_info.prefixed_storage_key();
|
|
assert!(trie_backend.storage(storage_key.as_slice()).unwrap().is_some());
|
|
}
|
|
|
|
#[test]
|
|
fn insert_multiple_times_child_data_works() {
|
|
let state_version = StateVersion::default();
|
|
let mut storage = new_in_mem_hash_key::<BlakeTwo256>();
|
|
let child_info = ChildInfo::new_default(b"1");
|
|
|
|
storage.insert(
|
|
vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])],
|
|
state_version,
|
|
);
|
|
storage.insert(
|
|
vec![(Some(child_info.clone()), vec![(b"1".to_vec(), Some(b"3".to_vec()))])],
|
|
state_version,
|
|
);
|
|
|
|
assert_eq!(storage.child_storage(&child_info, &b"2"[..]), Ok(Some(b"3".to_vec())));
|
|
assert_eq!(storage.child_storage(&child_info, &b"1"[..]), Ok(Some(b"3".to_vec())));
|
|
}
|
|
}
|