mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-06-30 00:17:26 +00:00
73d9ae3284
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
192 lines
5.9 KiB
Rust
192 lines
5.9 KiB
Rust
// This file is part of Substrate.
|
|
|
|
// Copyright (C) 2020-2022 Parity Technologies (UK) Ltd.
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
use codec::{Decode, Encode};
|
|
use hash_db::{HashDB, Hasher};
|
|
use scale_info::TypeInfo;
|
|
use sp_std::{collections::btree_set::BTreeSet, iter::IntoIterator, vec::Vec};
|
|
// Note that `LayoutV1` usage here (proof compaction) is compatible
|
|
// with `LayoutV0`.
|
|
use crate::LayoutV1 as Layout;
|
|
|
|
/// A proof that some set of key-value pairs are included in the storage trie. The proof contains
|
|
/// the storage values so that the partial storage backend can be reconstructed by a verifier that
|
|
/// does not already have access to the key-value pairs.
|
|
///
|
|
/// The proof consists of the set of serialized nodes in the storage trie accessed when looking up
|
|
/// the keys covered by the proof. Verifying the proof requires constructing the partial trie from
|
|
/// the serialized nodes and performing the key lookups.
|
|
#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
|
|
pub struct StorageProof {
|
|
trie_nodes: BTreeSet<Vec<u8>>,
|
|
}
|
|
|
|
impl StorageProof {
|
|
/// Constructs a storage proof from a subset of encoded trie nodes in a storage backend.
|
|
pub fn new(trie_nodes: impl IntoIterator<Item = Vec<u8>>) -> Self {
|
|
StorageProof { trie_nodes: BTreeSet::from_iter(trie_nodes) }
|
|
}
|
|
|
|
/// Returns a new empty proof.
|
|
///
|
|
/// An empty proof is capable of only proving trivial statements (ie. that an empty set of
|
|
/// key-value pairs exist in storage).
|
|
pub fn empty() -> Self {
|
|
StorageProof { trie_nodes: BTreeSet::new() }
|
|
}
|
|
|
|
/// Returns whether this is an empty proof.
|
|
pub fn is_empty(&self) -> bool {
|
|
self.trie_nodes.is_empty()
|
|
}
|
|
|
|
/// Create an iterator over encoded trie nodes in lexicographical order constructed
|
|
/// from the proof.
|
|
pub fn iter_nodes(self) -> StorageProofNodeIterator {
|
|
StorageProofNodeIterator::new(self)
|
|
}
|
|
|
|
/// Convert into plain node vector.
|
|
pub fn into_nodes(self) -> BTreeSet<Vec<u8>> {
|
|
self.trie_nodes
|
|
}
|
|
|
|
/// Creates a [`MemoryDB`](crate::MemoryDB) from `Self`.
|
|
pub fn into_memory_db<H: Hasher>(self) -> crate::MemoryDB<H> {
|
|
self.into()
|
|
}
|
|
|
|
/// Merges multiple storage proofs covering potentially different sets of keys into one proof
|
|
/// covering all keys. The merged proof output may be smaller than the aggregate size of the
|
|
/// input proofs due to deduplication of trie nodes.
|
|
pub fn merge(proofs: impl IntoIterator<Item = Self>) -> Self {
|
|
let trie_nodes = proofs
|
|
.into_iter()
|
|
.flat_map(|proof| proof.iter_nodes())
|
|
.collect::<sp_std::collections::btree_set::BTreeSet<_>>()
|
|
.into_iter()
|
|
.collect();
|
|
|
|
Self { trie_nodes }
|
|
}
|
|
|
|
/// Encode as a compact proof with default trie layout.
|
|
pub fn into_compact_proof<H: Hasher>(
|
|
self,
|
|
root: H::Out,
|
|
) -> Result<CompactProof, crate::CompactProofError<H::Out, crate::Error<H::Out>>> {
|
|
crate::encode_compact::<Layout<H>>(self, root)
|
|
}
|
|
|
|
/// Returns the estimated encoded size of the compact proof.
|
|
///
|
|
/// Running this operation is a slow operation (build the whole compact proof) and should only
|
|
/// be in non sensitive path.
|
|
///
|
|
/// Return `None` on error.
|
|
pub fn encoded_compact_size<H: Hasher>(self, root: H::Out) -> Option<usize> {
|
|
let compact_proof = self.into_compact_proof::<H>(root);
|
|
compact_proof.ok().map(|p| p.encoded_size())
|
|
}
|
|
}
|
|
|
|
impl<H: Hasher> From<StorageProof> for crate::MemoryDB<H> {
|
|
fn from(proof: StorageProof) -> Self {
|
|
let mut db = crate::MemoryDB::default();
|
|
proof.iter_nodes().for_each(|n| {
|
|
db.insert(crate::EMPTY_PREFIX, &n);
|
|
});
|
|
db
|
|
}
|
|
}
|
|
|
|
/// Storage proof in compact form.
|
|
#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
|
|
pub struct CompactProof {
|
|
pub encoded_nodes: Vec<Vec<u8>>,
|
|
}
|
|
|
|
impl CompactProof {
|
|
/// Return an iterator on the compact encoded nodes.
|
|
pub fn iter_compact_encoded_nodes(&self) -> impl Iterator<Item = &[u8]> {
|
|
self.encoded_nodes.iter().map(Vec::as_slice)
|
|
}
|
|
|
|
/// Decode to a full storage_proof.
|
|
pub fn to_storage_proof<H: Hasher>(
|
|
&self,
|
|
expected_root: Option<&H::Out>,
|
|
) -> Result<(StorageProof, H::Out), crate::CompactProofError<H::Out, crate::Error<H::Out>>> {
|
|
let mut db = crate::MemoryDB::<H>::new(&[]);
|
|
let root = crate::decode_compact::<Layout<H>, _, _>(
|
|
&mut db,
|
|
self.iter_compact_encoded_nodes(),
|
|
expected_root,
|
|
)?;
|
|
Ok((
|
|
StorageProof::new(db.drain().into_iter().filter_map(|kv| {
|
|
if (kv.1).1 > 0 {
|
|
Some((kv.1).0)
|
|
} else {
|
|
None
|
|
}
|
|
})),
|
|
root,
|
|
))
|
|
}
|
|
|
|
/// Convert self into a [`MemoryDB`](crate::MemoryDB).
|
|
///
|
|
/// `expected_root` is the expected root of this compact proof.
|
|
///
|
|
/// Returns the memory db and the root of the trie.
|
|
pub fn to_memory_db<H: Hasher>(
|
|
&self,
|
|
expected_root: Option<&H::Out>,
|
|
) -> Result<(crate::MemoryDB<H>, H::Out), crate::CompactProofError<H::Out, crate::Error<H::Out>>>
|
|
{
|
|
let mut db = crate::MemoryDB::<H>::new(&[]);
|
|
let root = crate::decode_compact::<Layout<H>, _, _>(
|
|
&mut db,
|
|
self.iter_compact_encoded_nodes(),
|
|
expected_root,
|
|
)?;
|
|
|
|
Ok((db, root))
|
|
}
|
|
}
|
|
|
|
/// An iterator over trie nodes constructed from a storage proof. The nodes are not guaranteed to
|
|
/// be traversed in any particular order.
|
|
pub struct StorageProofNodeIterator {
|
|
inner: <BTreeSet<Vec<u8>> as IntoIterator>::IntoIter,
|
|
}
|
|
|
|
impl StorageProofNodeIterator {
|
|
fn new(proof: StorageProof) -> Self {
|
|
StorageProofNodeIterator { inner: proof.trie_nodes.into_iter() }
|
|
}
|
|
}
|
|
|
|
impl Iterator for StorageProofNodeIterator {
|
|
type Item = Vec<u8>;
|
|
|
|
fn next(&mut self) -> Option<Self::Item> {
|
|
self.inner.next()
|
|
}
|
|
}
|