Files
pezkuwi-subxt/substrate/primitives/trie/src/storage_proof.rs
T
Bastian Köcher 73d9ae3284 Introduce trie level cache and remove state cache (#11407)
* trie state cache

* Also cache missing access on read.

* fix comp

* bis

* fix

* use has_lru

* remove local storage cache on size 0.

* No cache.

* local cache only

* trie cache and local cache

* storage cache (with local)

* trie cache no local cache

* Add state access benchmark

* Remove warnings etc

* Add trie cache benchmark

* No extra "clone" required

* Change benchmark to use multiple blocks

* Use patches

* Integrate shitty implementation

* More stuff

* Revert "Merge branch 'master' into trie_state_cache"

This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.

* Improve benchmark

* Adapt to latest changes

* Adapt to changes in trie

* Add a test that uses iterator

* Start fixing it

* Remove obsolete file

* Make it compile

* Start rewriting the trie node cache

* More work on the cache

* More docs and code etc

* Make data cache an optional

* Tests

* Remove debug stuff

* Recorder

* Some docs and a simple test for the recorder

* Compile fixes

* Make it compile

* More fixes

* More fixes

* Fix fix fix

* Make sure cache and recorder work together for basic stuff

* Test that data caching and recording works

* Test `TrieDBMut` with caching

* Try something

* Fixes, fixes, fixes

* Forward the recorder

* Make it compile

* Use recorder in more places

* Switch to new `with_optional_recorder` fn

* Refactor and cleanups

* Move `ProvingBackend` tests

* Simplify

* Move over all functionality to the essence

* Fix compilation

* Implement estimate encoded size for StorageProof

* Start using the `cache` everywhere

* Use the cache everywhere

* Fix compilation

* Fix tests

* Adds `TrieBackendBuilder` and enhances the tests

* Ensure that recorder drain checks that values are found as expected

* Switch over to `TrieBackendBuilder`

* Start fixing the problem with child tries and recording

* Fix recording of child tries

* Make it compile

* Overwrite `storage_hash` in `TrieBackend`

* Add `storage_cache` to  the benchmarks

* Fix `no_std` build

* Speed up cache lookup

* Extend the state access benchmark to also hash a runtime

* Fix build

* Fix compilation

* Rewrite value cache

* Add lru cache

* Ensure that the cache lru works

* Value cache should not be optional

* Add support for keeping the shared node cache in its bounds

* Make the cache configurable

* Check that the cache respects the bounds

* Adds a new test

* Fixes

* Docs and some renamings

* More docs

* Start using the new recorder

* Fix more code

* Take `self` argument

* Remove warnings

* Fix benchmark

* Fix accounting

* Rip off the state cache

* Start fixing fallout after removing the state cache

* Make it compile after trie changes

* Fix test

* Add some logging

* Some docs

* Some fixups and clean ups

* Fix benchmark

* Remove unneeded file

* Use git for patching

* Make CI happy

* Update primitives/trie/Cargo.toml

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Introduce new `AsTrieBackend` trait

* Make the LocalTrieCache not clonable

* Make it work in no_std and add docs

* Remove duplicate dependency

* Switch to ahash for better performance

* Speedup value cache merge

* Output errors on underflow

* Ensure the internal LRU map doesn't grow too much

* Use const fn to calculate the value cache element size

* Remove cache configuration

* Fix

* Clear the cache in between for more testing

* Try to come up with a failing test case

* Make the test fail

* Fix the child trie recording

* Make everything compile after the changes to trie

* Adapt to latest trie-db changes

* Fix on stable

* Update primitives/trie/src/cache.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fix wrong merge

* Docs

* Fix warnings

* Cargo.lock

* Bump pin-project

* Fix warnings

* Switch to released crate version

* More fixes

* Make clippy and rustdocs happy

* More clippy

* Print error when using deprecated `--state-cache-size`

* 🤦

* Fixes

* Fix storage_hash linkings

* Update client/rpc/src/dev/mod.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Review feedback

* encode bound

* Rework the shared value cache

Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.

* Improve of the cache by merging the old and new solution

* FMT

* Please stop coming back all the time :crying:

* Update primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Fixes

* Make clippy happy

* Ensure we don't deadlock

* Only use one lock to simplify the code

* Do not depend on `Hasher`

* Fix tests

* FMT

* Clippy 🤦

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2022-08-18 18:59:22 +00:00

192 lines
5.9 KiB
Rust

// This file is part of Substrate.
// Copyright (C) 2020-2022 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use codec::{Decode, Encode};
use hash_db::{HashDB, Hasher};
use scale_info::TypeInfo;
use sp_std::{collections::btree_set::BTreeSet, iter::IntoIterator, vec::Vec};
// Note that `LayoutV1` usage here (proof compaction) is compatible
// with `LayoutV0`.
use crate::LayoutV1 as Layout;
/// A proof that some set of key-value pairs are included in the storage trie. The proof contains
/// the storage values so that the partial storage backend can be reconstructed by a verifier that
/// does not already have access to the key-value pairs.
///
/// The proof consists of the set of serialized nodes in the storage trie accessed when looking up
/// the keys covered by the proof. Verifying the proof requires constructing the partial trie from
/// the serialized nodes and performing the key lookups.
#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
pub struct StorageProof {
trie_nodes: BTreeSet<Vec<u8>>,
}
impl StorageProof {
/// Constructs a storage proof from a subset of encoded trie nodes in a storage backend.
pub fn new(trie_nodes: impl IntoIterator<Item = Vec<u8>>) -> Self {
StorageProof { trie_nodes: BTreeSet::from_iter(trie_nodes) }
}
/// Returns a new empty proof.
///
/// An empty proof is capable of only proving trivial statements (ie. that an empty set of
/// key-value pairs exist in storage).
pub fn empty() -> Self {
StorageProof { trie_nodes: BTreeSet::new() }
}
/// Returns whether this is an empty proof.
pub fn is_empty(&self) -> bool {
self.trie_nodes.is_empty()
}
/// Create an iterator over encoded trie nodes in lexicographical order constructed
/// from the proof.
pub fn iter_nodes(self) -> StorageProofNodeIterator {
StorageProofNodeIterator::new(self)
}
/// Convert into plain node vector.
pub fn into_nodes(self) -> BTreeSet<Vec<u8>> {
self.trie_nodes
}
/// Creates a [`MemoryDB`](crate::MemoryDB) from `Self`.
pub fn into_memory_db<H: Hasher>(self) -> crate::MemoryDB<H> {
self.into()
}
/// Merges multiple storage proofs covering potentially different sets of keys into one proof
/// covering all keys. The merged proof output may be smaller than the aggregate size of the
/// input proofs due to deduplication of trie nodes.
pub fn merge(proofs: impl IntoIterator<Item = Self>) -> Self {
let trie_nodes = proofs
.into_iter()
.flat_map(|proof| proof.iter_nodes())
.collect::<sp_std::collections::btree_set::BTreeSet<_>>()
.into_iter()
.collect();
Self { trie_nodes }
}
/// Encode as a compact proof with default trie layout.
pub fn into_compact_proof<H: Hasher>(
self,
root: H::Out,
) -> Result<CompactProof, crate::CompactProofError<H::Out, crate::Error<H::Out>>> {
crate::encode_compact::<Layout<H>>(self, root)
}
/// Returns the estimated encoded size of the compact proof.
///
/// Running this operation is a slow operation (build the whole compact proof) and should only
/// be in non sensitive path.
///
/// Return `None` on error.
pub fn encoded_compact_size<H: Hasher>(self, root: H::Out) -> Option<usize> {
let compact_proof = self.into_compact_proof::<H>(root);
compact_proof.ok().map(|p| p.encoded_size())
}
}
impl<H: Hasher> From<StorageProof> for crate::MemoryDB<H> {
fn from(proof: StorageProof) -> Self {
let mut db = crate::MemoryDB::default();
proof.iter_nodes().for_each(|n| {
db.insert(crate::EMPTY_PREFIX, &n);
});
db
}
}
/// Storage proof in compact form.
#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode, TypeInfo)]
pub struct CompactProof {
pub encoded_nodes: Vec<Vec<u8>>,
}
impl CompactProof {
/// Return an iterator on the compact encoded nodes.
pub fn iter_compact_encoded_nodes(&self) -> impl Iterator<Item = &[u8]> {
self.encoded_nodes.iter().map(Vec::as_slice)
}
/// Decode to a full storage_proof.
pub fn to_storage_proof<H: Hasher>(
&self,
expected_root: Option<&H::Out>,
) -> Result<(StorageProof, H::Out), crate::CompactProofError<H::Out, crate::Error<H::Out>>> {
let mut db = crate::MemoryDB::<H>::new(&[]);
let root = crate::decode_compact::<Layout<H>, _, _>(
&mut db,
self.iter_compact_encoded_nodes(),
expected_root,
)?;
Ok((
StorageProof::new(db.drain().into_iter().filter_map(|kv| {
if (kv.1).1 > 0 {
Some((kv.1).0)
} else {
None
}
})),
root,
))
}
/// Convert self into a [`MemoryDB`](crate::MemoryDB).
///
/// `expected_root` is the expected root of this compact proof.
///
/// Returns the memory db and the root of the trie.
pub fn to_memory_db<H: Hasher>(
&self,
expected_root: Option<&H::Out>,
) -> Result<(crate::MemoryDB<H>, H::Out), crate::CompactProofError<H::Out, crate::Error<H::Out>>>
{
let mut db = crate::MemoryDB::<H>::new(&[]);
let root = crate::decode_compact::<Layout<H>, _, _>(
&mut db,
self.iter_compact_encoded_nodes(),
expected_root,
)?;
Ok((db, root))
}
}
/// An iterator over trie nodes constructed from a storage proof. The nodes are not guaranteed to
/// be traversed in any particular order.
pub struct StorageProofNodeIterator {
inner: <BTreeSet<Vec<u8>> as IntoIterator>::IntoIter,
}
impl StorageProofNodeIterator {
fn new(proof: StorageProof) -> Self {
StorageProofNodeIterator { inner: proof.trie_nodes.into_iter() }
}
}
impl Iterator for StorageProofNodeIterator {
type Item = Vec<u8>;
fn next(&mut self) -> Option<Self::Item> {
self.inner.next()
}
}