mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-04-26 21:37:56 +00:00
73d9ae3284
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
357 lines
11 KiB
Rust
357 lines
11 KiB
Rust
// This file is part of Substrate.
|
|
|
|
// Copyright (C) 2017-2022 Parity Technologies (UK) Ltd.
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! State machine backends. These manage the code and storage of contracts.
|
|
|
|
#[cfg(feature = "std")]
|
|
use crate::trie_backend::TrieBackend;
|
|
use crate::{
|
|
trie_backend_essence::TrieBackendStorage, ChildStorageCollection, StorageCollection,
|
|
StorageKey, StorageValue, UsageInfo,
|
|
};
|
|
use codec::Encode;
|
|
use hash_db::Hasher;
|
|
use sp_core::storage::{ChildInfo, StateVersion, TrackedStorageKey};
|
|
#[cfg(feature = "std")]
|
|
use sp_core::traits::RuntimeCode;
|
|
use sp_std::vec::Vec;
|
|
|
|
/// A state backend is used to read state data and can have changes committed
|
|
/// to it.
|
|
///
|
|
/// The clone operation (if implemented) should be cheap.
|
|
pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
|
|
/// An error type when fetching data is not possible.
|
|
type Error: super::Error;
|
|
|
|
/// Storage changes to be applied if committing
|
|
type Transaction: Consolidate + Default + Send;
|
|
|
|
/// Type of trie backend storage.
|
|
type TrieBackendStorage: TrieBackendStorage<H, Overlay = Self::Transaction>;
|
|
|
|
/// Get keyed storage or None if there is nothing associated.
|
|
fn storage(&self, key: &[u8]) -> Result<Option<StorageValue>, Self::Error>;
|
|
|
|
/// Get keyed storage value hash or None if there is nothing associated.
|
|
fn storage_hash(&self, key: &[u8]) -> Result<Option<H::Out>, Self::Error>;
|
|
|
|
/// Get keyed child storage or None if there is nothing associated.
|
|
fn child_storage(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
key: &[u8],
|
|
) -> Result<Option<StorageValue>, Self::Error>;
|
|
|
|
/// Get child keyed storage value hash or None if there is nothing associated.
|
|
fn child_storage_hash(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
key: &[u8],
|
|
) -> Result<Option<H::Out>, Self::Error>;
|
|
|
|
/// true if a key exists in storage.
|
|
fn exists_storage(&self, key: &[u8]) -> Result<bool, Self::Error> {
|
|
Ok(self.storage_hash(key)?.is_some())
|
|
}
|
|
|
|
/// true if a key exists in child storage.
|
|
fn exists_child_storage(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
key: &[u8],
|
|
) -> Result<bool, Self::Error> {
|
|
Ok(self.child_storage_hash(child_info, key)?.is_some())
|
|
}
|
|
|
|
/// Return the next key in storage in lexicographic order or `None` if there is no value.
|
|
fn next_storage_key(&self, key: &[u8]) -> Result<Option<StorageKey>, Self::Error>;
|
|
|
|
/// Return the next key in child storage in lexicographic order or `None` if there is no value.
|
|
fn next_child_storage_key(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
key: &[u8],
|
|
) -> Result<Option<StorageKey>, Self::Error>;
|
|
|
|
/// Iterate over storage starting at key, for a given prefix and child trie.
|
|
/// Aborts as soon as `f` returns false.
|
|
/// Warning, this fails at first error when usual iteration skips errors.
|
|
/// If `allow_missing` is true, iteration stops when it reaches a missing trie node.
|
|
/// Otherwise an error is produced.
|
|
///
|
|
/// Returns `true` if trie end is reached.
|
|
fn apply_to_key_values_while<F: FnMut(Vec<u8>, Vec<u8>) -> bool>(
|
|
&self,
|
|
child_info: Option<&ChildInfo>,
|
|
prefix: Option<&[u8]>,
|
|
start_at: Option<&[u8]>,
|
|
f: F,
|
|
allow_missing: bool,
|
|
) -> Result<bool, Self::Error>;
|
|
|
|
/// Retrieve all entries keys of storage and call `f` for each of those keys.
|
|
/// Aborts as soon as `f` returns false.
|
|
fn apply_to_keys_while<F: FnMut(&[u8]) -> bool>(
|
|
&self,
|
|
child_info: Option<&ChildInfo>,
|
|
prefix: Option<&[u8]>,
|
|
start_at: Option<&[u8]>,
|
|
f: F,
|
|
);
|
|
|
|
/// Retrieve all entries keys which start with the given prefix and
|
|
/// call `f` for each of those keys.
|
|
fn for_keys_with_prefix<F: FnMut(&[u8])>(&self, prefix: &[u8], mut f: F) {
|
|
self.for_key_values_with_prefix(prefix, |k, _v| f(k))
|
|
}
|
|
|
|
/// Retrieve all entries keys and values of which start with the given prefix and
|
|
/// call `f` for each of those keys.
|
|
fn for_key_values_with_prefix<F: FnMut(&[u8], &[u8])>(&self, prefix: &[u8], f: F);
|
|
|
|
/// Retrieve all child entries keys which start with the given prefix and
|
|
/// call `f` for each of those keys.
|
|
fn for_child_keys_with_prefix<F: FnMut(&[u8])>(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
prefix: &[u8],
|
|
f: F,
|
|
);
|
|
|
|
/// Calculate the storage root, with given delta over what is already stored in
|
|
/// the backend, and produce a "transaction" that can be used to commit.
|
|
/// Does not include child storage updates.
|
|
fn storage_root<'a>(
|
|
&self,
|
|
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
|
|
state_version: StateVersion,
|
|
) -> (H::Out, Self::Transaction)
|
|
where
|
|
H::Out: Ord;
|
|
|
|
/// Calculate the child storage root, with given delta over what is already stored in
|
|
/// the backend, and produce a "transaction" that can be used to commit. The second argument
|
|
/// is true if child storage root equals default storage root.
|
|
fn child_storage_root<'a>(
|
|
&self,
|
|
child_info: &ChildInfo,
|
|
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
|
|
state_version: StateVersion,
|
|
) -> (H::Out, bool, Self::Transaction)
|
|
where
|
|
H::Out: Ord;
|
|
|
|
/// Get all key/value pairs into a Vec.
|
|
fn pairs(&self) -> Vec<(StorageKey, StorageValue)>;
|
|
|
|
/// Get all keys with given prefix
|
|
fn keys(&self, prefix: &[u8]) -> Vec<StorageKey> {
|
|
let mut all = Vec::new();
|
|
self.for_keys_with_prefix(prefix, |k| all.push(k.to_vec()));
|
|
all
|
|
}
|
|
|
|
/// Get all keys of child storage with given prefix
|
|
fn child_keys(&self, child_info: &ChildInfo, prefix: &[u8]) -> Vec<StorageKey> {
|
|
let mut all = Vec::new();
|
|
self.for_child_keys_with_prefix(child_info, prefix, |k| all.push(k.to_vec()));
|
|
all
|
|
}
|
|
|
|
/// Calculate the storage root, with given delta over what is already stored
|
|
/// in the backend, and produce a "transaction" that can be used to commit.
|
|
/// Does include child storage updates.
|
|
fn full_storage_root<'a>(
|
|
&self,
|
|
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
|
|
child_deltas: impl Iterator<
|
|
Item = (&'a ChildInfo, impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>),
|
|
>,
|
|
state_version: StateVersion,
|
|
) -> (H::Out, Self::Transaction)
|
|
where
|
|
H::Out: Ord + Encode,
|
|
{
|
|
let mut txs: Self::Transaction = Default::default();
|
|
let mut child_roots: Vec<_> = Default::default();
|
|
// child first
|
|
for (child_info, child_delta) in child_deltas {
|
|
let (child_root, empty, child_txs) =
|
|
self.child_storage_root(child_info, child_delta, state_version);
|
|
let prefixed_storage_key = child_info.prefixed_storage_key();
|
|
txs.consolidate(child_txs);
|
|
if empty {
|
|
child_roots.push((prefixed_storage_key.into_inner(), None));
|
|
} else {
|
|
child_roots.push((prefixed_storage_key.into_inner(), Some(child_root.encode())));
|
|
}
|
|
}
|
|
let (root, parent_txs) = self.storage_root(
|
|
delta
|
|
.map(|(k, v)| (k, v.as_ref().map(|v| &v[..])))
|
|
.chain(child_roots.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..])))),
|
|
state_version,
|
|
);
|
|
txs.consolidate(parent_txs);
|
|
(root, txs)
|
|
}
|
|
|
|
/// Register stats from overlay of state machine.
|
|
///
|
|
/// By default nothing is registered.
|
|
fn register_overlay_stats(&self, _stats: &crate::stats::StateMachineStats);
|
|
|
|
/// Query backend usage statistics (i/o, memory)
|
|
///
|
|
/// Not all implementations are expected to be able to do this. In the
|
|
/// case when they don't, empty statistics is returned.
|
|
fn usage_info(&self) -> UsageInfo;
|
|
|
|
/// Wipe the state database.
|
|
fn wipe(&self) -> Result<(), Self::Error> {
|
|
unimplemented!()
|
|
}
|
|
|
|
/// Commit given transaction to storage.
|
|
fn commit(
|
|
&self,
|
|
_: H::Out,
|
|
_: Self::Transaction,
|
|
_: StorageCollection,
|
|
_: ChildStorageCollection,
|
|
) -> Result<(), Self::Error> {
|
|
unimplemented!()
|
|
}
|
|
|
|
/// Get the read/write count of the db
|
|
fn read_write_count(&self) -> (u32, u32, u32, u32) {
|
|
unimplemented!()
|
|
}
|
|
|
|
/// Get the read/write count of the db
|
|
fn reset_read_write_count(&self) {
|
|
unimplemented!()
|
|
}
|
|
|
|
/// Get the whitelist for tracking db reads/writes
|
|
fn get_whitelist(&self) -> Vec<TrackedStorageKey> {
|
|
Default::default()
|
|
}
|
|
|
|
/// Update the whitelist for tracking db reads/writes
|
|
fn set_whitelist(&self, _: Vec<TrackedStorageKey>) {}
|
|
|
|
/// Estimate proof size
|
|
fn proof_size(&self) -> Option<u32> {
|
|
unimplemented!()
|
|
}
|
|
|
|
/// Extend storage info for benchmarking db
|
|
fn get_read_and_written_keys(&self) -> Vec<(Vec<u8>, u32, u32, bool)> {
|
|
unimplemented!()
|
|
}
|
|
}
|
|
|
|
/// Something that can be converted into a [`TrieBackend`].
|
|
#[cfg(feature = "std")]
|
|
pub trait AsTrieBackend<H: Hasher, C = sp_trie::cache::LocalTrieCache<H>> {
|
|
/// Type of trie backend storage.
|
|
type TrieBackendStorage: TrieBackendStorage<H>;
|
|
|
|
/// Return the type as [`TrieBackend`].
|
|
fn as_trie_backend(&self) -> &TrieBackend<Self::TrieBackendStorage, H, C>;
|
|
}
|
|
|
|
/// Trait that allows consolidate two transactions together.
|
|
pub trait Consolidate {
|
|
/// Consolidate two transactions into one.
|
|
fn consolidate(&mut self, other: Self);
|
|
}
|
|
|
|
impl Consolidate for () {
|
|
fn consolidate(&mut self, _: Self) {
|
|
()
|
|
}
|
|
}
|
|
|
|
impl Consolidate for Vec<(Option<ChildInfo>, StorageCollection)> {
|
|
fn consolidate(&mut self, mut other: Self) {
|
|
self.append(&mut other);
|
|
}
|
|
}
|
|
|
|
impl<H, KF> Consolidate for sp_trie::GenericMemoryDB<H, KF>
|
|
where
|
|
H: Hasher,
|
|
KF: sp_trie::KeyFunction<H>,
|
|
{
|
|
fn consolidate(&mut self, other: Self) {
|
|
sp_trie::GenericMemoryDB::consolidate(self, other)
|
|
}
|
|
}
|
|
|
|
/// Wrapper to create a [`RuntimeCode`] from a type that implements [`Backend`].
|
|
#[cfg(feature = "std")]
|
|
pub struct BackendRuntimeCode<'a, B, H> {
|
|
backend: &'a B,
|
|
_marker: std::marker::PhantomData<H>,
|
|
}
|
|
|
|
#[cfg(feature = "std")]
|
|
impl<'a, B: Backend<H>, H: Hasher> sp_core::traits::FetchRuntimeCode
|
|
for BackendRuntimeCode<'a, B, H>
|
|
{
|
|
fn fetch_runtime_code(&self) -> Option<std::borrow::Cow<[u8]>> {
|
|
self.backend
|
|
.storage(sp_core::storage::well_known_keys::CODE)
|
|
.ok()
|
|
.flatten()
|
|
.map(Into::into)
|
|
}
|
|
}
|
|
|
|
#[cfg(feature = "std")]
|
|
impl<'a, B: Backend<H>, H: Hasher> BackendRuntimeCode<'a, B, H>
|
|
where
|
|
H::Out: Encode,
|
|
{
|
|
/// Create a new instance.
|
|
pub fn new(backend: &'a B) -> Self {
|
|
Self { backend, _marker: std::marker::PhantomData }
|
|
}
|
|
|
|
/// Return the [`RuntimeCode`] build from the wrapped `backend`.
|
|
pub fn runtime_code(&self) -> Result<RuntimeCode, &'static str> {
|
|
let hash = self
|
|
.backend
|
|
.storage_hash(sp_core::storage::well_known_keys::CODE)
|
|
.ok()
|
|
.flatten()
|
|
.ok_or("`:code` hash not found")?
|
|
.encode();
|
|
let heap_pages = self
|
|
.backend
|
|
.storage(sp_core::storage::well_known_keys::HEAP_PAGES)
|
|
.ok()
|
|
.flatten()
|
|
.and_then(|d| codec::Decode::decode(&mut &d[..]).ok());
|
|
|
|
Ok(RuntimeCode { code_fetcher: self, hash, heap_pages })
|
|
}
|
|
}
|