pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-06-22 15:01:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Inner hashing of value in state trie (runtime versioning). (#9732)

Browse Source 
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
This commit is contained in:
cheme
2021-12-24 09:54:07 +01:00
committed by GitHub
parent ea30c739ea
commit 4c651637f2
78 changed files with 1814 additions and 782 deletions
Download Patch File Download Diff File Expand all files Collapse all files
substrate/primitives/api/proc-macro/src/impl_runtime_apis.rs
+7
View File
@@ -293,10 +293,17 @@ fn generate_runtime_api_base_structures() -> Result<TokenStream> {
#crate_::StorageChanges<C::StateBackend, Block>,
String
> where Self: Sized {
let at = #crate_::BlockId::Hash(parent_hash.clone());
let state_version = self.call
.runtime_version_at(&at)
.map(|v| v.state_version())
.map_err(|e| format!("Failed to get state version: {:?}", e))?;
self.changes.replace(Default::default()).into_storage_changes(
backend,
parent_hash,
self.storage_transaction_cache.replace(Default::default()),
state_version,
)
}
}
substrate/primitives/api/src/lib.rs
+4 -41
View File
@@ -84,6 +84,8 @@ pub use sp_core::NativeOrEncoded;
use sp_core::OpaqueMetadata;
#[doc(hidden)]
pub use sp_core::{offchain, ExecutionContext};
#[cfg(feature = "std")]
pub use sp_runtime::StateVersion;
#[doc(hidden)]
pub use sp_runtime::{
generic::BlockId,
@@ -269,6 +271,7 @@ pub use sp_api_proc_macro::decl_runtime_apis;
/// // Here we are exposing the runtime api versions.
/// apis: RUNTIME_API_VERSIONS,
/// transaction_version: 1,
/// state_version: 1,
/// };
///
/// # fn main() {}
@@ -654,53 +657,13 @@ pub fn deserialize_runtime_api_info(bytes: [u8; RUNTIME_API_INFO_SIZE]) -> ([u8;
(id, version)
}
#[derive(codec::Encode, codec::Decode)]
pub struct OldRuntimeVersion {
pub spec_name: RuntimeString,
pub impl_name: RuntimeString,
pub authoring_version: u32,
pub spec_version: u32,
pub impl_version: u32,
pub apis: ApisVec,
}
impl From<OldRuntimeVersion> for RuntimeVersion {
fn from(x: OldRuntimeVersion) -> Self {
Self {
spec_name: x.spec_name,
impl_name: x.impl_name,
authoring_version: x.authoring_version,
spec_version: x.spec_version,
impl_version: x.impl_version,
apis: x.apis,
transaction_version: 1,
}
}
}
impl From<RuntimeVersion> for OldRuntimeVersion {
fn from(x: RuntimeVersion) -> Self {
Self {
spec_name: x.spec_name,
impl_name: x.impl_name,
authoring_version: x.authoring_version,
spec_version: x.spec_version,
impl_version: x.impl_version,
apis: x.apis,
}
}
}
decl_runtime_apis! {
/// The `Core` runtime api that every Substrate runtime needs to implement.
#[core_trait]
#[api_version(3)]
#[api_version(4)]
pub trait Core {
/// Returns the version of the runtime.
fn version() -> RuntimeVersion;
/// Returns the version of the runtime.
#[changed_in(3)]
fn version() -> OldRuntimeVersion;
/// Execute the given block.
fn execute_block(block: Block);
/// Initialize a block with the given header.
substrate/primitives/externalities/src/lib.rs
+7 -3
View File
@@ -31,7 +31,7 @@ use sp_std::{
vec::Vec,
};
use sp_storage::{ChildInfo, TrackedStorageKey};
use sp_storage::{ChildInfo, StateVersion, TrackedStorageKey};
pub use extensions::{Extension, ExtensionStore, Extensions};
pub use scope_limited::{set_and_run_with_externalities, with_externalities};
@@ -157,7 +157,7 @@ pub trait Externalities: ExtensionStore {
/// This will also update all child storage keys in the top-level storage map.
///
/// The returned hash is defined by the `Block` and is SCALE encoded.
fn storage_root(&mut self) -> Vec<u8>;
fn storage_root(&mut self, state_version: StateVersion) -> Vec<u8>;
/// Get the trie root of a child storage map.
///
@@ -165,7 +165,11 @@ pub trait Externalities: ExtensionStore {
///
/// If the storage root equals the default hash as defined by the trie, the key in the top-level
/// storage map will be removed.
fn child_storage_root(&mut self, child_info: &ChildInfo) -> Vec<u8>;
fn child_storage_root(
&mut self,
child_info: &ChildInfo,
state_version: StateVersion,
) -> Vec<u8>;
/// Append storage item.
///
substrate/primitives/io/src/lib.rs
+134 -10
View File
@@ -51,11 +51,13 @@ use sp_core::{
offchain::{
HttpError, HttpRequestId, HttpRequestStatus, OpaqueNetworkState, StorageKind, Timestamp,
},
sr25519, LogLevel, LogLevelFilter, OpaquePeerId, H256,
sr25519,
storage::StateVersion,
LogLevel, LogLevelFilter, OpaquePeerId, H256,
};
#[cfg(feature = "std")]
use sp_trie::{trie_types::Layout, TrieConfiguration};
use sp_trie::{LayoutV0, LayoutV1, TrieConfiguration};
use sp_runtime_interface::{
pass_by::{PassBy, PassByCodec},
@@ -192,7 +194,17 @@ pub trait Storage {
///
/// Returns a `Vec<u8>` that holds the SCALE encoded hash.
fn root(&mut self) -> Vec<u8> {
self.storage_root()
self.storage_root(StateVersion::V0)
}
/// "Commit" all existing operations and compute the resulting storage root.
///
/// The hashing algorithm is defined by the `Block`.
///
/// Returns a `Vec<u8>` that holds the SCALE encoded hash.
#[version(2)]
fn root(&mut self, version: StateVersion) -> Vec<u8> {
self.storage_root(version)
}
/// Always returns `None`. This function exists for compatibility reasons.
@@ -373,7 +385,19 @@ pub trait DefaultChildStorage {
/// Returns a `Vec<u8>` that holds the SCALE encoded hash.
fn root(&mut self, storage_key: &[u8]) -> Vec<u8> {
let child_info = ChildInfo::new_default(storage_key);
self.child_storage_root(&child_info)
self.child_storage_root(&child_info, StateVersion::V0)
}
/// Default child root calculation.
///
/// "Commit" all existing operations and compute the resulting child storage root.
/// The hashing algorithm is defined by the `Block`.
///
/// Returns a `Vec<u8>` that holds the SCALE encoded hash.
#[version(2)]
fn root(&mut self, storage_key: &[u8], version: StateVersion) -> Vec<u8> {
let child_info = ChildInfo::new_default(storage_key);
self.child_storage_root(&child_info, version)
}
/// Child storage key iteration.
@@ -390,27 +414,63 @@ pub trait DefaultChildStorage {
pub trait Trie {
/// A trie root formed from the iterated items.
fn blake2_256_root(input: Vec<(Vec<u8>, Vec<u8>)>) -> H256 {
Layout::<sp_core::Blake2Hasher>::trie_root(input)
LayoutV0::<sp_core::Blake2Hasher>::trie_root(input)
}
/// A trie root formed from the iterated items.
#[version(2)]
fn blake2_256_root(input: Vec<(Vec<u8>, Vec<u8>)>, version: StateVersion) -> H256 {
match version {
StateVersion::V0 => LayoutV0::<sp_core::Blake2Hasher>::trie_root(input),
StateVersion::V1 => LayoutV1::<sp_core::Blake2Hasher>::trie_root(input),
}
}
/// A trie root formed from the enumerated items.
fn blake2_256_ordered_root(input: Vec<Vec<u8>>) -> H256 {
Layout::<sp_core::Blake2Hasher>::ordered_trie_root(input)
LayoutV0::<sp_core::Blake2Hasher>::ordered_trie_root(input)
}
/// A trie root formed from the enumerated items.
#[version(2)]
fn blake2_256_ordered_root(input: Vec<Vec<u8>>, version: StateVersion) -> H256 {
match version {
StateVersion::V0 => LayoutV0::<sp_core::Blake2Hasher>::ordered_trie_root(input),
StateVersion::V1 => LayoutV1::<sp_core::Blake2Hasher>::ordered_trie_root(input),
}
}
/// A trie root formed from the iterated items.
fn keccak_256_root(input: Vec<(Vec<u8>, Vec<u8>)>) -> H256 {
Layout::<sp_core::KeccakHasher>::trie_root(input)
LayoutV0::<sp_core::KeccakHasher>::trie_root(input)
}
/// A trie root formed from the iterated items.
#[version(2)]
fn keccak_256_root(input: Vec<(Vec<u8>, Vec<u8>)>, version: StateVersion) -> H256 {
match version {
StateVersion::V0 => LayoutV0::<sp_core::KeccakHasher>::trie_root(input),
StateVersion::V1 => LayoutV1::<sp_core::KeccakHasher>::trie_root(input),
}
}
/// A trie root formed from the enumerated items.
fn keccak_256_ordered_root(input: Vec<Vec<u8>>) -> H256 {
Layout::<sp_core::KeccakHasher>::ordered_trie_root(input)
LayoutV0::<sp_core::KeccakHasher>::ordered_trie_root(input)
}
/// A trie root formed from the enumerated items.
#[version(2)]
fn keccak_256_ordered_root(input: Vec<Vec<u8>>, version: StateVersion) -> H256 {
match version {
StateVersion::V0 => LayoutV0::<sp_core::KeccakHasher>::ordered_trie_root(input),
StateVersion::V1 => LayoutV1::<sp_core::KeccakHasher>::ordered_trie_root(input),
}
}
/// Verify trie proof
fn blake2_256_verify_proof(root: H256, proof: &[Vec<u8>], key: &[u8], value: &[u8]) -> bool {
sp_trie::verify_trie_proof::<Layout<sp_core::Blake2Hasher>, _, _, _>(
sp_trie::verify_trie_proof::<LayoutV0<sp_core::Blake2Hasher>, _, _, _>(
&root,
proof,
&[(key, Some(value))],
@@ -418,15 +478,69 @@ pub trait Trie {
.is_ok()
}
/// Verify trie proof
#[version(2)]
fn blake2_256_verify_proof(
root: H256,
proof: &[Vec<u8>],
key: &[u8],
value: &[u8],
version: StateVersion,
) -> bool {
match version {
StateVersion::V0 => sp_trie::verify_trie_proof::<
LayoutV0<sp_core::Blake2Hasher>,
_,
_,
_,
>(&root, proof, &[(key, Some(value))])
.is_ok(),
StateVersion::V1 => sp_trie::verify_trie_proof::<
LayoutV1<sp_core::Blake2Hasher>,
_,
_,
_,
>(&root, proof, &[(key, Some(value))])
.is_ok(),
}
}
/// Verify trie proof
fn keccak_256_verify_proof(root: H256, proof: &[Vec<u8>], key: &[u8], value: &[u8]) -> bool {
sp_trie::verify_trie_proof::<Layout<sp_core::KeccakHasher>, _, _, _>(
sp_trie::verify_trie_proof::<LayoutV0<sp_core::KeccakHasher>, _, _, _>(
&root,
proof,
&[(key, Some(value))],
)
.is_ok()
}
/// Verify trie proof
#[version(2)]
fn keccak_256_verify_proof(
root: H256,
proof: &[Vec<u8>],
key: &[u8],
value: &[u8],
version: StateVersion,
) -> bool {
match version {
StateVersion::V0 => sp_trie::verify_trie_proof::<
LayoutV0<sp_core::KeccakHasher>,
_,
_,
_,
>(&root, proof, &[(key, Some(value))])
.is_ok(),
StateVersion::V1 => sp_trie::verify_trie_proof::<
LayoutV1<sp_core::KeccakHasher>,
_,
_,
_,
>(&root, proof, &[(key, Some(value))])
.is_ok(),
}
}
}
/// Interface that provides miscellaneous functions for communicating between the runtime and the
@@ -1570,6 +1684,16 @@ mod tests {
assert_eq!(storage::get(b"hello"), None);
assert_eq!(storage::get(b"foo"), Some(b"bar".to_vec()));
});
let value = vec![7u8; 35];
let storage =
Storage { top: map![b"foo00".to_vec() => value.clone()], children_default: map![] };
t = BasicExternalities::new(storage);
t.execute_with(|| {
assert_eq!(storage::get(b"hello"), None);
assert_eq!(storage::get(b"foo00"), Some(value.clone()));
});
}
#[test]
substrate/primitives/runtime-interface/src/impls.rs
+4
View File
@@ -544,3 +544,7 @@ impl PassBy for sp_wasm_interface::Value {
impl PassBy for sp_storage::TrackedStorageKey {
type PassBy = Codec<Self>;
}
impl PassBy for sp_storage::StateVersion {
type PassBy = Enum<Self>;
}
substrate/primitives/runtime/src/lib.rs
+1
View File
@@ -40,6 +40,7 @@ pub use paste;
#[doc(hidden)]
pub use sp_application_crypto as app_crypto;
pub use sp_core::storage::StateVersion;
#[cfg(feature = "std")]
pub use sp_core::storage::{Storage, StorageChild};
substrate/primitives/runtime/src/traits.rs
+11 -11
View File
@@ -36,7 +36,7 @@ pub use sp_arithmetic::traits::{
CheckedShr, CheckedSub, IntegerSquareRoot, One, SaturatedConversion, Saturating,
UniqueSaturatedFrom, UniqueSaturatedInto, Zero,
};
use sp_core::{self, Hasher, RuntimeDebug, TypeId};
use sp_core::{self, storage::StateVersion, Hasher, RuntimeDebug, TypeId};
use sp_std::{
self,
convert::{TryFrom, TryInto},
@@ -459,10 +459,10 @@ pub trait Hash:
}
/// The ordered Patricia tree root of the given `input`.
fn ordered_trie_root(input: Vec<Vec<u8>>) -> Self::Output;
fn ordered_trie_root(input: Vec<Vec<u8>>, state_version: StateVersion) -> Self::Output;
/// The Patricia tree root of the given mapping.
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>) -> Self::Output;
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>, state_version: StateVersion) -> Self::Output;
}
/// Blake2-256 Hash implementation.
@@ -483,12 +483,12 @@ impl Hasher for BlakeTwo256 {
impl Hash for BlakeTwo256 {
type Output = sp_core::H256;
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>) -> Self::Output {
sp_io::trie::blake2_256_root(input)
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>, version: StateVersion) -> Self::Output {
sp_io::trie::blake2_256_root(input, version)
}
fn ordered_trie_root(input: Vec<Vec<u8>>) -> Self::Output {
sp_io::trie::blake2_256_ordered_root(input)
fn ordered_trie_root(input: Vec<Vec<u8>>, version: StateVersion) -> Self::Output {
sp_io::trie::blake2_256_ordered_root(input, version)
}
}
@@ -510,12 +510,12 @@ impl Hasher for Keccak256 {
impl Hash for Keccak256 {
type Output = sp_core::H256;
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>) -> Self::Output {
sp_io::trie::keccak_256_root(input)
fn trie_root(input: Vec<(Vec<u8>, Vec<u8>)>, version: StateVersion) -> Self::Output {
sp_io::trie::keccak_256_root(input, version)
}
fn ordered_trie_root(input: Vec<Vec<u8>>) -> Self::Output {
sp_io::trie::keccak_256_ordered_root(input)
fn ordered_trie_root(input: Vec<Vec<u8>>, version: StateVersion) -> Self::Output {
sp_io::trie::keccak_256_ordered_root(input, version)
}
}
substrate/primitives/state-machine/Cargo.toml
+2 -2
View File
@@ -18,8 +18,8 @@ log = { version = "0.4.11", optional = true }
thiserror = { version = "1.0.30", optional = true }
parking_lot = { version = "0.11.2", optional = true }
hash-db = { version = "0.15.2", default-features = false }
trie-db = { version = "0.22.6", default-features = false }
trie-root = { version = "0.16.0", default-features = false }
trie-db = { version = "0.23.0", default-features = false }
trie-root = { version = "0.17.0", default-features = false }
sp-trie = { version = "4.0.0", path = "../trie", default-features = false }
sp-core = { version = "4.1.0-dev", path = "../core", default-features = false }
sp-panic-handler = { version = "4.0.0", path = "../panic-handler", optional = true }
substrate/primitives/state-machine/src/backend.rs
+13 -5
View File
@@ -23,7 +23,7 @@ use crate::{
};
use codec::Encode;
use hash_db::Hasher;
use sp_core::storage::{ChildInfo, TrackedStorageKey};
use sp_core::storage::{ChildInfo, StateVersion, TrackedStorageKey};
#[cfg(feature = "std")]
use sp_core::traits::RuntimeCode;
use sp_std::vec::Vec;
@@ -140,6 +140,7 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (H::Out, Self::Transaction)
where
H::Out: Ord;
@@ -151,6 +152,7 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (H::Out, bool, Self::Transaction)
where
H::Out: Ord;
@@ -176,7 +178,6 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
fn as_trie_backend(&self) -> Option<&TrieBackend<Self::TrieBackendStorage, H>> {
None
}
/// Calculate the storage root, with given delta over what is already stored
/// in the backend, and produce a "transaction" that can be used to commit.
/// Does include child storage updates.
@@ -186,6 +187,7 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
child_deltas: impl Iterator<
Item = (&'a ChildInfo, impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>),
>,
state_version: StateVersion,
) -> (H::Out, Self::Transaction)
where
H::Out: Ord + Encode,
@@ -194,7 +196,8 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
let mut child_roots: Vec<_> = Default::default();
// child first
for (child_info, child_delta) in child_deltas {
let (child_root, empty, child_txs) = self.child_storage_root(&child_info, child_delta);
let (child_root, empty, child_txs) =
self.child_storage_root(&child_info, child_delta, state_version);
let prefixed_storage_key = child_info.prefixed_storage_key();
txs.consolidate(child_txs);
if empty {
@@ -205,8 +208,9 @@ pub trait Backend<H: Hasher>: sp_std::fmt::Debug {
}
let (root, parent_txs) = self.storage_root(
delta
.map(|(k, v)| (k, v.as_ref().map(|v| &v[..])))
.map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..])))
.chain(child_roots.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..])))),
state_version,
);
txs.consolidate(parent_txs);
(root, txs)
@@ -286,7 +290,11 @@ impl Consolidate for Vec<(Option<ChildInfo>, StorageCollection)> {
}
}
impl<H: Hasher, KF: sp_trie::KeyFunction<H>> Consolidate for sp_trie::GenericMemoryDB<H, KF> {
impl<H, KF> Consolidate for sp_trie::GenericMemoryDB<H, KF>
where
H: Hasher,
KF: sp_trie::KeyFunction<H>,
{
fn consolidate(&mut self, other: Self) {
sp_trie::GenericMemoryDB::consolidate(self, other)
}
substrate/primitives/state-machine/src/basic.rs
+20 -10
View File
@@ -23,13 +23,14 @@ use hash_db::Hasher;
use log::warn;
use sp_core::{
storage::{
well_known_keys::is_child_storage_key, ChildInfo, Storage, StorageChild, TrackedStorageKey,
well_known_keys::is_child_storage_key, ChildInfo, StateVersion, Storage, StorageChild,
TrackedStorageKey,
},
traits::Externalities,
Blake2Hasher,
};
use sp_externalities::{Extension, Extensions};
use sp_trie::{empty_child_trie_root, trie_types::Layout, TrieConfiguration};
use sp_trie::{empty_child_trie_root, LayoutV0, LayoutV1, TrieConfiguration};
use std::{
any::{Any, TypeId},
collections::BTreeMap,
@@ -273,7 +274,7 @@ impl Externalities for BasicExternalities {
crate::ext::StorageAppend::new(current).append(value);
}
fn storage_root(&mut self) -> Vec<u8> {
fn storage_root(&mut self, state_version: StateVersion) -> Vec<u8> {
let mut top = self.inner.top.clone();
let prefixed_keys: Vec<_> = self
.inner
@@ -284,9 +285,9 @@ impl Externalities for BasicExternalities {
// Single child trie implementation currently allows using the same child
// empty root for all child trie. Using null storage key until multiple
// type of child trie support.
let empty_hash = empty_child_trie_root::<Layout<Blake2Hasher>>();
let empty_hash = empty_child_trie_root::<LayoutV1<Blake2Hasher>>();
for (prefixed_storage_key, child_info) in prefixed_keys {
let child_root = self.child_storage_root(&child_info);
let child_root = self.child_storage_root(&child_info, state_version);
if &empty_hash[..] == &child_root[..] {
top.remove(prefixed_storage_key.as_slice());
} else {
@@ -294,17 +295,26 @@ impl Externalities for BasicExternalities {
}
}
Layout::<Blake2Hasher>::trie_root(self.inner.top.clone()).as_ref().into()
match state_version {
StateVersion::V0 =>
LayoutV0::<Blake2Hasher>::trie_root(self.inner.top.clone()).as_ref().into(),
StateVersion::V1 =>
LayoutV1::<Blake2Hasher>::trie_root(self.inner.top.clone()).as_ref().into(),
}
}
fn child_storage_root(&mut self, child_info: &ChildInfo) -> Vec<u8> {
fn child_storage_root(
&mut self,
child_info: &ChildInfo,
state_version: StateVersion,
) -> Vec<u8> {
if let Some(child) = self.inner.children_default.get(child_info.storage_key()) {
let delta = child.data.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref())));
crate::in_memory_backend::new_in_mem::<Blake2Hasher>()
.child_storage_root(&child.child_info, delta)
.child_storage_root(&child.child_info, delta, state_version)
.0
} else {
empty_child_trie_root::<Layout<Blake2Hasher>>()
empty_child_trie_root::<LayoutV1<Blake2Hasher>>()
}
.encode()
}
@@ -389,7 +399,7 @@ mod tests {
const ROOT: [u8; 32] =
hex!("39245109cef3758c2eed2ccba8d9b370a917850af3824bc8348d505df2c298fa");
assert_eq!(&ext.storage_root()[..], &ROOT);
assert_eq!(&ext.storage_root(StateVersion::default())[..], &ROOT);
}
#[test]
substrate/primitives/state-machine/src/ext.rs
+103 -66
View File
@@ -24,9 +24,11 @@ use codec::{Decode, Encode, EncodeAppend};
use hash_db::Hasher;
#[cfg(feature = "std")]
use sp_core::hexdisplay::HexDisplay;
use sp_core::storage::{well_known_keys::is_child_storage_key, ChildInfo, TrackedStorageKey};
use sp_core::storage::{
well_known_keys::is_child_storage_key, ChildInfo, StateVersion, TrackedStorageKey,
};
use sp_externalities::{Extension, ExtensionStore, Externalities};
use sp_trie::{empty_child_trie_root, trie_types::Layout};
use sp_trie::{empty_child_trie_root, LayoutV1};
use crate::{log_error, trace, warn, StorageTransactionCache};
use sp_std::{
@@ -505,7 +507,7 @@ where
StorageAppend::new(current_value).append(value);
}
fn storage_root(&mut self) -> Vec<u8> {
fn storage_root(&mut self, state_version: StateVersion) -> Vec<u8> {
let _guard = guard();
if let Some(ref root) = self.storage_transaction_cache.transaction_storage_root {
trace!(
@@ -518,7 +520,9 @@ where
return root.encode()
}
let root = self.overlay.storage_root(self.backend, self.storage_transaction_cache);
let root =
self.overlay
.storage_root(self.backend, self.storage_transaction_cache, state_version);
trace!(
target: "state",
method = "StorageRoot",
@@ -529,7 +533,11 @@ where
root.encode()
}
fn child_storage_root(&mut self, child_info: &ChildInfo) -> Vec<u8> {
fn child_storage_root(
&mut self,
child_info: &ChildInfo,
state_version: StateVersion,
) -> Vec<u8> {
let _guard = guard();
let storage_key = child_info.storage_key();
let prefixed_storage_key = child_info.prefixed_storage_key();
@@ -537,7 +545,8 @@ where
let root = self
.storage(prefixed_storage_key.as_slice())
.and_then(|k| Decode::decode(&mut &k[..]).ok())
.unwrap_or_else(|| empty_child_trie_root::<Layout<H>>());
// V1 is equivalent to V0 on empty root.
.unwrap_or_else(|| empty_child_trie_root::<LayoutV1<H>>());
trace!(
target: "state",
method = "ChildStorageRoot",
@@ -550,7 +559,7 @@ where
} else {
let root = if let Some((changes, info)) = self.overlay.child_changes(storage_key) {
let delta = changes.map(|(k, v)| (k.as_ref(), v.value().map(AsRef::as_ref)));
Some(self.backend.child_storage_root(info, delta))
Some(self.backend.child_storage_root(info, delta, state_version))
} else {
None
};
@@ -583,7 +592,8 @@ where
let root = self
.storage(prefixed_storage_key.as_slice())
.and_then(|k| Decode::decode(&mut &k[..]).ok())
.unwrap_or_else(|| empty_child_trie_root::<Layout<H>>());
// V1 is equivalent to V0 on empty root.
.unwrap_or_else(|| empty_child_trie_root::<LayoutV1<H>>());
trace!(
target: "state",
@@ -648,7 +658,12 @@ where
self.overlay.rollback_transaction().expect(BENCHMARKING_FN);
}
self.overlay
.drain_storage_changes(self.backend, Default::default(), self.storage_transaction_cache)
.drain_storage_changes(
self.backend,
Default::default(),
self.storage_transaction_cache,
Default::default(), // using any state
)
.expect(EXT_NOT_ALLOWED_TO_FAIL);
self.backend.wipe().expect(EXT_NOT_ALLOWED_TO_FAIL);
self.mark_dirty();
@@ -658,12 +673,19 @@ where
}
fn commit(&mut self) {
// Bench always use latest state.
let state_version = StateVersion::default();
for _ in 0..self.overlay.transaction_depth() {
self.overlay.commit_transaction().expect(BENCHMARKING_FN);
}
let changes = self
.overlay
.drain_storage_changes(self.backend, Default::default(), self.storage_transaction_cache)
.drain_storage_changes(
self.backend,
Default::default(),
self.storage_transaction_cache,
state_version,
)
.expect(EXT_NOT_ALLOWED_TO_FAIL);
self.backend
.commit(
@@ -878,15 +900,18 @@ mod tests {
let mut overlay = OverlayedChanges::default();
overlay.set_storage(vec![20], None);
overlay.set_storage(vec![30], Some(vec![31]));
let backend = Storage {
top: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![40] => vec![40]
],
children_default: map![],
}
.into();
let backend = (
Storage {
top: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![40] => vec![40]
],
children_default: map![],
},
StateVersion::default(),
)
.into();
let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
@@ -924,13 +949,16 @@ mod tests {
overlay.set_storage(vec![27], None);
overlay.set_storage(vec![28], None);
overlay.set_storage(vec![29], None);
let backend = Storage {
top: map![
vec![30] => vec![30]
],
children_default: map![],
}
.into();
let backend = (
Storage {
top: map![
vec![30] => vec![30]
],
children_default: map![],
},
StateVersion::default(),
)
.into();
let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
@@ -948,20 +976,23 @@ mod tests {
let mut overlay = OverlayedChanges::default();
overlay.set_child_storage(child_info, vec![20], None);
overlay.set_child_storage(child_info, vec![30], Some(vec![31]));
let backend = Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![40] => vec![40]
],
child_info: child_info.to_owned(),
}
],
}
.into();
let backend = (
Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![40] => vec![40]
],
child_info: child_info.to_owned(),
}
],
},
StateVersion::default(),
)
.into();
let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
@@ -993,20 +1024,23 @@ mod tests {
let mut overlay = OverlayedChanges::default();
overlay.set_child_storage(child_info, vec![20], None);
overlay.set_child_storage(child_info, vec![30], Some(vec![31]));
let backend = Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![30] => vec![40]
],
child_info: child_info.to_owned(),
}
],
}
.into();
let backend = (
Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![10] => vec![10],
vec![20] => vec![20],
vec![30] => vec![40]
],
child_info: child_info.to_owned(),
}
],
},
StateVersion::default(),
)
.into();
let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
@@ -1032,18 +1066,21 @@ mod tests {
let child_info = &child_info;
let mut cache = StorageTransactionCache::default();
let mut overlay = OverlayedChanges::default();
let backend = Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![30] => vec![40]
],
child_info: child_info.to_owned(),
}
],
}
.into();
let backend = (
Storage {
top: map![],
children_default: map![
child_info.storage_key().to_vec() => StorageChild {
data: map![
vec![30] => vec![40]
],
child_info: child_info.to_owned(),
}
],
},
StateVersion::default(),
)
.into();
let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
substrate/primitives/state-machine/src/in_memory_backend.rs
+44 -21
View File
@@ -22,8 +22,8 @@ use crate::{
};
use codec::Codec;
use hash_db::Hasher;
use sp_core::storage::{ChildInfo, Storage};
use sp_trie::{empty_trie_root, Layout, MemoryDB};
use sp_core::storage::{ChildInfo, StateVersion, Storage};
use sp_trie::{empty_trie_root, LayoutV1, MemoryDB};
use std::collections::{BTreeMap, HashMap};
/// Create a new empty instance of in-memory backend.
@@ -32,7 +32,8 @@ where
H::Out: Codec + Ord,
{
let db = MemoryDB::default();
TrieBackend::new(db, empty_trie_root::<Layout<H>>())
// V1 is same as V0 for an empty trie.
TrieBackend::new(db, empty_trie_root::<LayoutV1<H>>())
}
impl<H: Hasher> TrieBackend<MemoryDB<H>, H>
@@ -43,9 +44,10 @@ where
pub fn update<T: IntoIterator<Item = (Option<ChildInfo>, StorageCollection)>>(
&self,
changes: T,
state_version: StateVersion,
) -> Self {
let mut clone = self.clone();
clone.insert(changes);
clone.insert(changes, state_version);
clone
}
@@ -53,6 +55,7 @@ where
pub fn insert<T: IntoIterator<Item = (Option<ChildInfo>, StorageCollection)>>(
&mut self,
changes: T,
state_version: StateVersion,
) {
let (top, child) = changes.into_iter().partition::<Vec<_>, _>(|v| v.0.is_none());
let (root, transaction) = self.full_storage_root(
@@ -60,6 +63,7 @@ where
child.iter().filter_map(|v| {
v.0.as_ref().map(|c| (c, v.1.iter().map(|(k, v)| (&k[..], v.as_deref()))))
}),
state_version,
);
self.apply_transaction(root, transaction);
@@ -103,53 +107,63 @@ where
}
}
impl<H: Hasher> From<HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>>
impl<H: Hasher> From<(HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>, StateVersion)>
for TrieBackend<MemoryDB<H>, H>
where
H::Out: Codec + Ord,
{
fn from(inner: HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>) -> Self {
fn from(
(inner, state_version): (
HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>>,
StateVersion,
),
) -> Self {
let mut backend = new_in_mem();
backend.insert(
inner
.into_iter()
.map(|(k, m)| (k, m.into_iter().map(|(k, v)| (k, Some(v))).collect())),
state_version,
);
backend
}
}
impl<H: Hasher> From<Storage> for TrieBackend<MemoryDB<H>, H>
impl<H: Hasher> From<(Storage, StateVersion)> for TrieBackend<MemoryDB<H>, H>
where
H::Out: Codec + Ord,
{
fn from(inners: Storage) -> Self {
fn from((inners, state_version): (Storage, StateVersion)) -> Self {
let mut inner: HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>> = inners
.children_default
.into_iter()
.map(|(_k, c)| (Some(c.child_info), c.data))
.collect();
inner.insert(None, inners.top);
inner.into()
(inner, state_version).into()
}
}
impl<H: Hasher> From<BTreeMap<StorageKey, StorageValue>> for TrieBackend<MemoryDB<H>, H>
impl<H: Hasher> From<(BTreeMap<StorageKey, StorageValue>, StateVersion)>
for TrieBackend<MemoryDB<H>, H>
where
H::Out: Codec + Ord,
{
fn from(inner: BTreeMap<StorageKey, StorageValue>) -> Self {
fn from((inner, state_version): (BTreeMap<StorageKey, StorageValue>, StateVersion)) -> Self {
let mut expanded = HashMap::new();
expanded.insert(None, inner);
expanded.into()
(expanded, state_version).into()
}
}
impl<H: Hasher> From<Vec<(Option<ChildInfo>, StorageCollection)>> for TrieBackend<MemoryDB<H>, H>
impl<H: Hasher> From<(Vec<(Option<ChildInfo>, StorageCollection)>, StateVersion)>
for TrieBackend<MemoryDB<H>, H>
where
H::Out: Codec + Ord,
{
fn from(inner: Vec<(Option<ChildInfo>, StorageCollection)>) -> Self {
fn from(
(inner, state_version): (Vec<(Option<ChildInfo>, StorageCollection)>, StateVersion),
) -> Self {
let mut expanded: HashMap<Option<ChildInfo>, BTreeMap<StorageKey, StorageValue>> =
HashMap::new();
for (child_info, key_values) in inner {
@@ -160,7 +174,7 @@ where
}
}
}
expanded.into()
(expanded, state_version).into()
}
}
@@ -168,16 +182,20 @@ where
mod tests {
use super::*;
use crate::backend::Backend;
use sp_core::storage::StateVersion;
use sp_runtime::traits::BlakeTwo256;
/// Assert in memory backend with only child trie keys works as trie backend.
#[test]
fn in_memory_with_child_trie_only() {
let state_version = StateVersion::default();
let storage = new_in_mem::<BlakeTwo256>();
let child_info = ChildInfo::new_default(b"1");
let child_info = &child_info;
let storage = storage
.update(vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])]);
let storage = storage.update(
vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])],
state_version,
);
let trie_backend = storage.as_trie_backend().unwrap();
assert_eq!(trie_backend.child_storage(child_info, b"2").unwrap(), Some(b"3".to_vec()));
let storage_key = child_info.prefixed_storage_key();
@@ -186,13 +204,18 @@ mod tests {
#[test]
fn insert_multiple_times_child_data_works() {
let state_version = StateVersion::default();
let mut storage = new_in_mem::<BlakeTwo256>();
let child_info = ChildInfo::new_default(b"1");
storage
.insert(vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])]);
storage
.insert(vec![(Some(child_info.clone()), vec![(b"1".to_vec(), Some(b"3".to_vec()))])]);
storage.insert(
vec![(Some(child_info.clone()), vec![(b"2".to_vec(), Some(b"3".to_vec()))])],
state_version,
);
storage.insert(
vec![(Some(child_info.clone()), vec![(b"1".to_vec(), Some(b"3".to_vec()))])],
state_version,
);
assert_eq!(storage.child_storage(&child_info, &b"2"[..]), Ok(Some(b"3".to_vec())));
assert_eq!(storage.child_storage(&child_info, &b"1"[..]), Ok(Some(b"3".to_vec())));
substrate/primitives/state-machine/src/lib.rs
+128 -31
View File
@@ -126,6 +126,7 @@ impl sp_std::fmt::Display for DefaultError {
pub use crate::{
backend::Backend,
error::{Error, ExecutionError},
ext::Ext,
overlayed_changes::{
ChildStorageCollection, IndexOperation, OffchainChangesCollection,
@@ -136,7 +137,6 @@ pub use crate::{
trie_backend::TrieBackend,
trie_backend_essence::{Storage, TrieBackendStorage},
};
pub use error::{Error, ExecutionError};
#[cfg(feature = "std")]
mod std_reexport {
@@ -151,8 +151,8 @@ mod std_reexport {
testing::TestExternalities,
};
pub use sp_trie::{
trie_types::{Layout, TrieDBMut},
CompactProof, DBValue, MemoryDB, StorageProof, TrieMut,
trie_types::{TrieDBMutV0, TrieDBMutV1},
CompactProof, DBValue, LayoutV0, LayoutV1, MemoryDB, StorageProof, TrieMut,
};
}
@@ -1353,7 +1353,7 @@ mod tests {
use codec::{Decode, Encode};
use sp_core::{
map,
storage::ChildInfo,
storage::{ChildInfo, StateVersion},
testing::TaskExecutor,
traits::{CodeExecutor, Externalities, RuntimeCode},
NativeOrEncoded, NeverNativeValue,
@@ -1416,7 +1416,11 @@ mod tests {
#[test]
fn execute_works() {
let backend = trie_backend::tests::test_trie();
execute_works_inner(StateVersion::V0);
execute_works_inner(StateVersion::V1);
}
fn execute_works_inner(state_version: StateVersion) {
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1440,7 +1444,11 @@ mod tests {
#[test]
fn execute_works_with_native_else_wasm() {
let backend = trie_backend::tests::test_trie();
execute_works_with_native_else_wasm_inner(StateVersion::V0);
execute_works_with_native_else_wasm_inner(StateVersion::V1);
}
fn execute_works_with_native_else_wasm_inner(state_version: StateVersion) {
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1464,8 +1472,12 @@ mod tests {
#[test]
fn dual_execution_strategy_detects_consensus_failure() {
dual_execution_strategy_detects_consensus_failure_inner(StateVersion::V0);
dual_execution_strategy_detects_consensus_failure_inner(StateVersion::V1);
}
fn dual_execution_strategy_detects_consensus_failure_inner(state_version: StateVersion) {
let mut consensus_failed = false;
let backend = trie_backend::tests::test_trie();
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1498,6 +1510,10 @@ mod tests {
#[test]
fn prove_execution_and_proof_check_works() {
prove_execution_and_proof_check_works_inner(StateVersion::V0);
prove_execution_and_proof_check_works_inner(StateVersion::V1);
}
fn prove_execution_and_proof_check_works_inner(state_version: StateVersion) {
let executor = DummyCodeExecutor {
native_available: true,
native_succeeds: true,
@@ -1505,8 +1521,8 @@ mod tests {
};
// fetch execution proof from 'remote' full node
let mut remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let mut remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let (remote_result, remote_proof) = prove_execution(
&mut remote_backend,
&mut Default::default(),
@@ -1544,7 +1560,7 @@ mod tests {
b"abc".to_vec() => b"2".to_vec(),
b"bbb".to_vec() => b"3".to_vec()
];
let state = InMemoryBackend::<BlakeTwo256>::from(initial);
let state = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let backend = state.as_trie_backend().unwrap();
let mut overlay = OverlayedChanges::default();
@@ -1613,7 +1629,7 @@ mod tests {
b"d".to_vec() => b"3".to_vec()
],
];
let backend = InMemoryBackend::<BlakeTwo256>::from(initial);
let backend = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let mut overlay = OverlayedChanges::default();
overlay.set_child_storage(&child_info, b"1".to_vec(), Some(b"1312".to_vec()));
@@ -1655,7 +1671,7 @@ mod tests {
b"d".to_vec() => b"3".to_vec()
],
];
let backend = InMemoryBackend::<BlakeTwo256>::from(initial);
let backend = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let mut overlay = OverlayedChanges::default();
let mut cache = StorageTransactionCache::default();
let mut ext = Ext::new(&mut overlay, &mut cache, &backend, None);
@@ -1789,13 +1805,17 @@ mod tests {
#[test]
fn prove_read_and_proof_check_works() {
prove_read_and_proof_check_works_inner(StateVersion::V0);
prove_read_and_proof_check_works_inner(StateVersion::V1);
}
fn prove_read_and_proof_check_works_inner(state_version: StateVersion) {
let child_info = ChildInfo::new_default(b"sub1");
let missing_child_info = ChildInfo::new_default(b"sub1sub2"); // key will include other child root to proof.
let child_info = &child_info;
let missing_child_info = &missing_child_info;
// fetch read proof from 'remote' full node
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_read(remote_backend, &[b"value2"]).unwrap();
let remote_proof = test_compact(remote_proof, &remote_root);
// check proof locally
@@ -1812,8 +1832,8 @@ mod tests {
);
assert_eq!(local_result2, false);
// on child trie
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_child_read(remote_backend, child_info, &[b"value3"]).unwrap();
let remote_proof = test_compact(remote_proof, &remote_root);
let local_result1 = read_child_proof_check::<BlakeTwo256, _>(
@@ -1877,7 +1897,8 @@ mod tests {
storage.insert(Some(child_info), items);
}
let trie: InMemoryBackend<BlakeTwo256> = storage.clone().into();
let trie: InMemoryBackend<BlakeTwo256> =
(storage.clone(), StateVersion::default()).into();
let trie_root = trie.root().clone();
let backend = crate::ProvingBackend::new(&trie);
let mut queries = Vec::new();
@@ -1940,15 +1961,16 @@ mod tests {
#[test]
fn prove_read_with_size_limit_works() {
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(::std::iter::empty()).0;
let state_version = StateVersion::V0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(::std::iter::empty(), state_version).0;
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 0, None).unwrap();
// Always contains at least some nodes.
assert_eq!(proof.into_memory_db::<BlakeTwo256>().drain().len(), 3);
assert_eq!(count, 1);
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 800, Some(&[])).unwrap();
assert_eq!(proof.clone().into_memory_db::<BlakeTwo256>().drain().len(), 9);
@@ -1971,7 +1993,7 @@ mod tests {
assert_eq!(results.len() as u32, 101);
assert_eq!(completed, false);
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 50000, Some(&[])).unwrap();
assert_eq!(proof.clone().into_memory_db::<BlakeTwo256>().drain().len(), 11);
@@ -1989,10 +2011,66 @@ mod tests {
assert_eq!(completed, true);
}
#[test]
fn inner_state_versioning_switch_proofs() {
let mut state_version = StateVersion::V0;
let (mut mdb, mut root) = trie_backend::tests::test_db(state_version);
{
let mut trie = TrieDBMutV0::from_existing(&mut mdb, &mut root).unwrap();
trie.insert(b"foo", vec![1u8; 1_000].as_slice()) // big inner hash
.expect("insert failed");
trie.insert(b"foo2", vec![3u8; 16].as_slice()) // no inner hash
.expect("insert failed");
trie.insert(b"foo222", vec![5u8; 100].as_slice()) // inner hash
.expect("insert failed");
}
let check_proof = |mdb, root, state_version| -> StorageProof {
let remote_backend = TrieBackend::new(mdb, root);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_read(remote_backend, &[b"foo222"]).unwrap();
// check proof locally
let local_result1 =
read_proof_check::<BlakeTwo256, _>(remote_root, remote_proof.clone(), &[b"foo222"])
.unwrap();
// check that results are correct
assert_eq!(
local_result1.into_iter().collect::<Vec<_>>(),
vec![(b"foo222".to_vec(), Some(vec![5u8; 100]))],
);
remote_proof
};
let remote_proof = check_proof(mdb.clone(), root.clone(), state_version);
// check full values in proof
assert!(remote_proof.encode().len() > 1_100);
assert!(remote_proof.encoded_size() > 1_100);
let root1 = root.clone();
// do switch
state_version = StateVersion::V1;
{
let mut trie = TrieDBMutV1::from_existing(&mut mdb, &mut root).unwrap();
trie.insert(b"foo222", vec![5u8; 100].as_slice()) // inner hash
.expect("insert failed");
// update with same value do change
trie.insert(b"foo", vec![1u8; 1000].as_slice()) // inner hash
.expect("insert failed");
}
let root3 = root.clone();
assert!(root1 != root3);
let remote_proof = check_proof(mdb.clone(), root.clone(), state_version);
// nodes foo is replaced by its hashed value form.
assert!(remote_proof.encode().len() < 1000);
assert!(remote_proof.encoded_size() < 1000);
assert_eq!(remote_proof.encode().len(), remote_proof.encoded_size());
}
#[test]
fn prove_range_with_child_works() {
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(::std::iter::empty()).0;
let state_version = StateVersion::V0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let mut start_at = smallvec::SmallVec::<[Vec<u8>; 2]>::new();
let trie_backend = remote_backend.as_trie_backend().unwrap();
let max_iter = 1000;
@@ -2030,20 +2108,34 @@ mod tests {
#[test]
fn compact_multiple_child_trie() {
let size_no_inner_hash = compact_multiple_child_trie_inner(StateVersion::V0);
let size_inner_hash = compact_multiple_child_trie_inner(StateVersion::V1);
assert!(size_inner_hash < size_no_inner_hash);
}
fn compact_multiple_child_trie_inner(state_version: StateVersion) -> usize {
// this root will be queried
let child_info1 = ChildInfo::new_default(b"sub1");
// this root will not be include in proof
let child_info2 = ChildInfo::new_default(b"sub2");
// this root will be include in proof
let child_info3 = ChildInfo::new_default(b"sub");
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let long_vec: Vec<u8> = (0..1024usize).map(|_| 8u8).collect();
let (remote_root, transaction) = remote_backend.full_storage_root(
std::iter::empty(),
vec![
(
&child_info1,
vec![(&b"key1"[..], Some(&b"val2"[..])), (&b"key2"[..], Some(&b"val3"[..]))]
.into_iter(),
vec![
// a inner hashable node
(&b"k"[..], Some(&long_vec[..])),
// need to ensure this is not an inline node
// otherwhise we do not know what is accessed when
// storing proof.
(&b"key1"[..], Some(&vec![5u8; 32][..])),
(&b"key2"[..], Some(&b"val3"[..])),
]
.into_iter(),
),
(
&child_info2,
@@ -2057,11 +2149,13 @@ mod tests {
),
]
.into_iter(),
state_version,
);
let mut remote_storage = remote_backend.into_storage();
remote_storage.consolidate(transaction);
let remote_backend = TrieBackend::new(remote_storage, remote_root);
let remote_proof = prove_child_read(remote_backend, &child_info1, &[b"key1"]).unwrap();
let size = remote_proof.encoded_size();
let remote_proof = test_compact(remote_proof, &remote_root);
let local_result1 = read_child_proof_check::<BlakeTwo256, _>(
remote_root,
@@ -2071,11 +2165,13 @@ mod tests {
)
.unwrap();
assert_eq!(local_result1.len(), 1);
assert_eq!(local_result1.get(&b"key1"[..]), Some(&Some(b"val2".to_vec())));
assert_eq!(local_result1.get(&b"key1"[..]), Some(&Some(vec![5u8; 32])));
size
}
#[test]
fn child_storage_uuid() {
let state_version = StateVersion::V0;
let child_info_1 = ChildInfo::new_default(b"sub_test1");
let child_info_2 = ChildInfo::new_default(b"sub_test2");
@@ -2083,12 +2179,12 @@ mod tests {
let mut overlay = OverlayedChanges::default();
let mut transaction = {
let backend = test_trie();
let backend = test_trie(state_version);
let mut cache = StorageTransactionCache::default();
let mut ext = Ext::new(&mut overlay, &mut cache, &backend, None);
ext.set_child_storage(&child_info_1, b"abc".to_vec(), b"def".to_vec());
ext.set_child_storage(&child_info_2, b"abc".to_vec(), b"def".to_vec());
ext.storage_root();
ext.storage_root(state_version);
cache.transaction.unwrap()
};
let mut duplicate = false;
@@ -2108,7 +2204,7 @@ mod tests {
b"aaa".to_vec() => b"0".to_vec(),
b"bbb".to_vec() => b"".to_vec()
];
let state = InMemoryBackend::<BlakeTwo256>::from(initial);
let state = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let backend = state.as_trie_backend().unwrap();
let mut overlay = OverlayedChanges::default();
@@ -2134,12 +2230,13 @@ mod tests {
#[test]
fn runtime_registered_extensions_are_removed_after_execution() {
let state_version = StateVersion::default();
use sp_externalities::ExternalitiesExt;
sp_externalities::decl_extension! {
struct DummyExt(u32);
}
let backend = trie_backend::tests::test_trie();
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
substrate/primitives/state-machine/src/overlayed_changes/mod.rs
+10 -6
View File
@@ -27,7 +27,7 @@ use hash_db::Hasher;
pub use offchain::OffchainOverlayedChanges;
use sp_core::{
offchain::OffchainOverlayedChange,
storage::{well_known_keys::EXTRINSIC_INDEX, ChildInfo},
storage::{well_known_keys::EXTRINSIC_INDEX, ChildInfo, StateVersion},
};
#[cfg(feature = "std")]
use sp_externalities::{Extension, Extensions};
@@ -502,11 +502,12 @@ impl OverlayedChanges {
backend: &B,
parent_hash: H::Out,
mut cache: StorageTransactionCache<B::Transaction, H>,
state_version: StateVersion,
) -> Result<StorageChanges<B::Transaction, H>, DefaultError>
where
H::Out: Ord + Encode + 'static,
{
self.drain_storage_changes(backend, parent_hash, &mut cache)
self.drain_storage_changes(backend, parent_hash, &mut cache, state_version)
}
/// Drain all changes into a [`StorageChanges`] instance. Leave empty overlay in place.
@@ -515,13 +516,14 @@ impl OverlayedChanges {
backend: &B,
_parent_hash: H::Out,
mut cache: &mut StorageTransactionCache<B::Transaction, H>,
state_version: StateVersion,
) -> Result<StorageChanges<B::Transaction, H>, DefaultError>
where
H::Out: Ord + Encode + 'static,
{
// If the transaction does not exist, we generate it.
if cache.transaction.is_none() {
self.storage_root(backend, &mut cache);
self.storage_root(backend, &mut cache, state_version);
}
let (transaction, transaction_storage_root) = cache
@@ -580,6 +582,7 @@ impl OverlayedChanges {
&self,
backend: &B,
cache: &mut StorageTransactionCache<B::Transaction, H>,
state_version: StateVersion,
) -> H::Out
where
H::Out: Ord + Encode,
@@ -589,7 +592,7 @@ impl OverlayedChanges {
(info, changes.map(|(k, v)| (&k[..], v.value().map(|v| &v[..]))))
});
let (root, transaction) = backend.full_storage_root(delta, child_delta);
let (root, transaction) = backend.full_storage_root(delta, child_delta, state_version);
cache.transaction = Some(transaction);
cache.transaction_storage_root = Some(root);
@@ -830,6 +833,7 @@ mod tests {
#[test]
fn overlayed_storage_root_works() {
let state_version = StateVersion::default();
let initial: BTreeMap<_, _> = vec![
(b"doe".to_vec(), b"reindeer".to_vec()),
(b"dog".to_vec(), b"puppyXXX".to_vec()),
@@ -838,7 +842,7 @@ mod tests {
]
.into_iter()
.collect();
let backend = InMemoryBackend::<Blake2Hasher>::from(initial);
let backend = InMemoryBackend::<Blake2Hasher>::from((initial, state_version));
let mut overlay = OverlayedChanges::default();
overlay.start_transaction();
@@ -856,7 +860,7 @@ mod tests {
const ROOT: [u8; 32] =
hex!("39245109cef3758c2eed2ccba8d9b370a917850af3824bc8348d505df2c298fa");
assert_eq!(&ext.storage_root()[..], &ROOT);
assert_eq!(&ext.storage_root(state_version)[..], &ROOT);
}
#[test]
substrate/primitives/state-machine/src/proving_backend.rs
+69 -30
View File
@@ -26,14 +26,11 @@ use codec::{Codec, Decode, Encode};
use hash_db::{HashDB, Hasher, Prefix, EMPTY_PREFIX};
use log::debug;
use parking_lot::RwLock;
use sp_core::storage::ChildInfo;
use sp_core::storage::{ChildInfo, StateVersion};
pub use sp_trie::trie_types::TrieError;
use sp_trie::{
empty_child_trie_root, read_child_trie_value_with, read_trie_value_with, record_all_keys,
MemoryDB, StorageProof,
};
pub use sp_trie::{
trie_types::{Layout, TrieError},
Recorder,
LayoutV1, MemoryDB, Recorder, StorageProof,
};
use std::{
collections::{hash_map::Entry, HashMap},
@@ -59,7 +56,8 @@ where
let map_e = |e| format!("Trie lookup error: {}", e);
read_trie_value_with::<Layout<H>, _, Ephemeral<S, H>>(
// V1 is equivalent to V0 on read.
read_trie_value_with::<LayoutV1<H>, _, Ephemeral<S, H>>(
&eph,
self.backend.root(),
key,
@@ -78,14 +76,16 @@ where
let root = self
.storage(storage_key)?
.and_then(|r| Decode::decode(&mut &r[..]).ok())
.unwrap_or_else(|| empty_child_trie_root::<Layout<H>>());
// V1 is equivalent to V0 on empty trie
.unwrap_or_else(|| empty_child_trie_root::<LayoutV1<H>>());
let mut read_overlay = S::Overlay::default();
let eph = Ephemeral::new(self.backend.backend_storage(), &mut read_overlay);
let map_e = |e| format!("Trie lookup error: {}", e);
read_child_trie_value_with::<Layout<H>, _, _>(
// V1 is equivalent to V0 on read
read_child_trie_value_with::<LayoutV1<H>, _, _>(
child_info.keyspace(),
&eph,
&root.as_ref(),
@@ -102,7 +102,8 @@ where
let mut iter = move || -> Result<(), Box<TrieError<H::Out>>> {
let root = self.backend.root();
record_all_keys::<Layout<H>, _>(&eph, root, &mut *self.proof_recorder)
// V1 and V is equivalent to V0 on read and recorder is key read.
record_all_keys::<LayoutV1<H>, _>(&eph, root, &mut *self.proof_recorder)
};
if let Err(e) = iter() {
@@ -338,22 +339,24 @@ where
fn storage_root<'b>(
&self,
delta: impl Iterator<Item = (&'b [u8], Option<&'b [u8]>)>,
state_version: StateVersion,
) -> (H::Out, Self::Transaction)
where
H::Out: Ord,
{
self.0.storage_root(delta)
self.0.storage_root(delta, state_version)
}
fn child_storage_root<'b>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'b [u8], Option<&'b [u8]>)>,
state_version: StateVersion,
) -> (H::Out, bool, Self::Transaction)
where
H::Out: Ord,
{
self.0.child_storage_root(child_info, delta)
self.0.child_storage_root(child_info, delta, state_version)
}
fn register_overlay_stats(&self, _stats: &crate::stats::StateMachineStats) {}
@@ -401,13 +404,21 @@ mod tests {
#[test]
fn proof_is_empty_until_value_is_read() {
let trie_backend = test_trie();
proof_is_empty_until_value_is_read_inner(StateVersion::V0);
proof_is_empty_until_value_is_read_inner(StateVersion::V1);
}
fn proof_is_empty_until_value_is_read_inner(test_hash: StateVersion) {
let trie_backend = test_trie(test_hash);
assert!(test_proving(&trie_backend).extract_proof().is_empty());
}
#[test]
fn proof_is_non_empty_after_value_is_read() {
let trie_backend = test_trie();
proof_is_non_empty_after_value_is_read_inner(StateVersion::V0);
proof_is_non_empty_after_value_is_read_inner(StateVersion::V1);
}
fn proof_is_non_empty_after_value_is_read_inner(test_hash: StateVersion) {
let trie_backend = test_trie(test_hash);
let backend = test_proving(&trie_backend);
assert_eq!(backend.storage(b"key").unwrap(), Some(b"value".to_vec()));
assert!(!backend.extract_proof().is_empty());
@@ -425,58 +436,82 @@ mod tests {
#[test]
fn passes_through_backend_calls() {
let trie_backend = test_trie();
passes_through_backend_calls_inner(StateVersion::V0);
passes_through_backend_calls_inner(StateVersion::V1);
}
fn passes_through_backend_calls_inner(state_version: StateVersion) {
let trie_backend = test_trie(state_version);
let proving_backend = test_proving(&trie_backend);
assert_eq!(trie_backend.storage(b"key").unwrap(), proving_backend.storage(b"key").unwrap());
assert_eq!(trie_backend.pairs(), proving_backend.pairs());
let (trie_root, mut trie_mdb) = trie_backend.storage_root(std::iter::empty());
let (proving_root, mut proving_mdb) = proving_backend.storage_root(std::iter::empty());
let (trie_root, mut trie_mdb) =
trie_backend.storage_root(std::iter::empty(), state_version);
let (proving_root, mut proving_mdb) =
proving_backend.storage_root(std::iter::empty(), state_version);
assert_eq!(trie_root, proving_root);
assert_eq!(trie_mdb.drain(), proving_mdb.drain());
}
#[test]
fn proof_recorded_and_checked() {
let contents = (0..64).map(|i| (vec![i], Some(vec![i]))).collect::<Vec<_>>();
fn proof_recorded_and_checked_top() {
proof_recorded_and_checked_inner(StateVersion::V0);
proof_recorded_and_checked_inner(StateVersion::V1);
}
fn proof_recorded_and_checked_inner(state_version: StateVersion) {
let size_content = 34; // above hashable value treshold.
let value_range = 0..64;
let contents = value_range
.clone()
.map(|i| (vec![i], Some(vec![i; size_content])))
.collect::<Vec<_>>();
let in_memory = InMemoryBackend::<BlakeTwo256>::default();
let in_memory = in_memory.update(vec![(None, contents)]);
let in_memory_root = in_memory.storage_root(::std::iter::empty()).0;
(0..64).for_each(|i| assert_eq!(in_memory.storage(&[i]).unwrap().unwrap(), vec![i]));
let in_memory = in_memory.update(vec![(None, contents)], state_version);
let in_memory_root = in_memory.storage_root(std::iter::empty(), state_version).0;
value_range.clone().for_each(|i| {
assert_eq!(in_memory.storage(&[i]).unwrap().unwrap(), vec![i; size_content])
});
let trie = in_memory.as_trie_backend().unwrap();
let trie_root = trie.storage_root(::std::iter::empty()).0;
let trie_root = trie.storage_root(std::iter::empty(), state_version).0;
assert_eq!(in_memory_root, trie_root);
(0..64).for_each(|i| assert_eq!(trie.storage(&[i]).unwrap().unwrap(), vec![i]));
value_range
.clone()
.for_each(|i| assert_eq!(trie.storage(&[i]).unwrap().unwrap(), vec![i; size_content]));
let proving = ProvingBackend::new(trie);
assert_eq!(proving.storage(&[42]).unwrap().unwrap(), vec![42]);
assert_eq!(proving.storage(&[42]).unwrap().unwrap(), vec![42; size_content]);
let proof = proving.extract_proof();
let proof_check =
create_proof_check_backend::<BlakeTwo256>(in_memory_root.into(), proof).unwrap();
assert_eq!(proof_check.storage(&[42]).unwrap().unwrap(), vec![42]);
assert_eq!(proof_check.storage(&[42]).unwrap().unwrap(), vec![42; size_content]);
}
#[test]
fn proof_recorded_and_checked_with_child() {
proof_recorded_and_checked_with_child_inner(StateVersion::V0);
proof_recorded_and_checked_with_child_inner(StateVersion::V1);
}
fn proof_recorded_and_checked_with_child_inner(state_version: StateVersion) {
let child_info_1 = ChildInfo::new_default(b"sub1");
let child_info_2 = ChildInfo::new_default(b"sub2");
let child_info_1 = &child_info_1;
let child_info_2 = &child_info_2;
let contents = vec![
(None, (0..64).map(|i| (vec![i], Some(vec![i]))).collect()),
(None, (0..64).map(|i| (vec![i], Some(vec![i]))).collect::<Vec<_>>()),
(Some(child_info_1.clone()), (28..65).map(|i| (vec![i], Some(vec![i]))).collect()),
(Some(child_info_2.clone()), (10..15).map(|i| (vec![i], Some(vec![i]))).collect()),
];
let in_memory = InMemoryBackend::<BlakeTwo256>::default();
let in_memory = in_memory.update(contents);
let in_memory = in_memory.update(contents, state_version);
let child_storage_keys = vec![child_info_1.to_owned(), child_info_2.to_owned()];
let in_memory_root = in_memory
.full_storage_root(
std::iter::empty(),
child_storage_keys.iter().map(|k| (k, std::iter::empty())),
state_version,
)
.0;
(0..64).for_each(|i| assert_eq!(in_memory.storage(&[i]).unwrap().unwrap(), vec![i]));
@@ -488,7 +523,7 @@ mod tests {
});
let trie = in_memory.as_trie_backend().unwrap();
let trie_root = trie.storage_root(std::iter::empty()).0;
let trie_root = trie.storage_root(std::iter::empty(), state_version).0;
assert_eq!(in_memory_root, trie_root);
(0..64).for_each(|i| assert_eq!(trie.storage(&[i]).unwrap().unwrap(), vec![i]));
@@ -516,7 +551,11 @@ mod tests {
#[test]
fn storage_proof_encoded_size_estimation_works() {
let trie_backend = test_trie();
storage_proof_encoded_size_estimation_works_inner(StateVersion::V0);
storage_proof_encoded_size_estimation_works_inner(StateVersion::V1);
}
fn storage_proof_encoded_size_estimation_works_inner(state_version: StateVersion) {
let trie_backend = test_trie(state_version);
let backend = test_proving(&trie_backend);
let check_estimation =
substrate/primitives/state-machine/src/read_only.rs
+7 -3
View File
@@ -21,7 +21,7 @@ use crate::{Backend, StorageKey, StorageValue};
use codec::Encode;
use hash_db::Hasher;
use sp_core::{
storage::{ChildInfo, TrackedStorageKey},
storage::{ChildInfo, StateVersion, TrackedStorageKey},
traits::Externalities,
Blake2Hasher,
};
@@ -145,11 +145,15 @@ impl<'a, H: Hasher, B: 'a + Backend<H>> Externalities for ReadOnlyExternalities<
unimplemented!("storage_append is not supported in ReadOnlyExternalities")
}
fn storage_root(&mut self) -> Vec<u8> {
fn storage_root(&mut self, _state_version: StateVersion) -> Vec<u8> {
unimplemented!("storage_root is not supported in ReadOnlyExternalities")
}
fn child_storage_root(&mut self, _child_info: &ChildInfo) -> Vec<u8> {
fn child_storage_root(
&mut self,
_child_info: &ChildInfo,
_state_version: StateVersion,
) -> Vec<u8> {
unimplemented!("child_storage_root is not supported in ReadOnlyExternalities")
}
substrate/primitives/state-machine/src/testing.rs
+44 -14
View File
@@ -32,7 +32,7 @@ use sp_core::{
offchain::testing::TestPersistentOffchainDB,
storage::{
well_known_keys::{is_child_storage_key, CODE},
Storage,
StateVersion, Storage,
},
testing::TaskExecutor,
traits::TaskExecutorExt,
@@ -54,6 +54,8 @@ where
pub backend: InMemoryBackend<H>,
/// Extensions.
pub extensions: Extensions,
/// State version to use during tests.
pub state_version: StateVersion,
}
impl<H: Hasher> TestExternalities<H>
@@ -72,18 +74,31 @@ where
/// Create a new instance of `TestExternalities` with storage.
pub fn new(storage: Storage) -> Self {
Self::new_with_code(&[], storage)
Self::new_with_code_and_state(&[], storage, Default::default())
}
/// Create a new instance of `TestExternalities` with storage for a given state version.
pub fn new_with_state_version(storage: Storage, state_version: StateVersion) -> Self {
Self::new_with_code_and_state(&[], storage, state_version)
}
/// New empty test externalities.
pub fn new_empty() -> Self {
Self::new_with_code(&[], Storage::default())
Self::new_with_code_and_state(&[], Storage::default(), Default::default())
}
/// Create a new instance of `TestExternalities` with code and storage.
pub fn new_with_code(code: &[u8], mut storage: Storage) -> Self {
let overlay = OverlayedChanges::default();
pub fn new_with_code(code: &[u8], storage: Storage) -> Self {
Self::new_with_code_and_state(code, storage, Default::default())
}
/// Create a new instance of `TestExternalities` with code and storage for a given state
/// version.
pub fn new_with_code_and_state(
code: &[u8],
mut storage: Storage,
state_version: StateVersion,
) -> Self {
assert!(storage.top.keys().all(|key| !is_child_storage_key(key)));
assert!(storage.children_default.keys().all(|key| is_child_storage_key(key)));
@@ -94,12 +109,15 @@ where
let offchain_db = TestPersistentOffchainDB::new();
let backend = (storage, state_version).into();
TestExternalities {
overlay,
overlay: OverlayedChanges::default(),
offchain_db,
extensions,
backend: storage.into(),
backend,
storage_transaction_cache: Default::default(),
state_version,
}
}
@@ -120,14 +138,14 @@ where
/// Insert key/value into backend
pub fn insert(&mut self, k: StorageKey, v: StorageValue) {
self.backend.insert(vec![(None, vec![(k, Some(v))])]);
self.backend.insert(vec![(None, vec![(k, Some(v))])], self.state_version);
}
/// Insert key/value into backend.
///
/// This only supports inserting keys in child tries.
pub fn insert_child(&mut self, c: sp_core::storage::ChildInfo, k: StorageKey, v: StorageValue) {
self.backend.insert(vec![(Some(c), vec![(k, Some(v))])]);
self.backend.insert(vec![(Some(c), vec![(k, Some(v))])], self.state_version);
}
/// Registers the given extension for this instance.
@@ -151,7 +169,7 @@ where
))
}
self.backend.update(transaction)
self.backend.update(transaction, self.state_version)
}
/// Commit all pending changes to the underlying backend.
@@ -164,6 +182,7 @@ where
&self.backend,
Default::default(),
&mut Default::default(),
self.state_version,
)?;
self.backend
@@ -240,7 +259,8 @@ where
H::Out: Ord + 'static + codec::Codec,
{
fn default() -> Self {
Self::new(Default::default())
// default to default version.
Self::new_with_state_version(Storage::default(), Default::default())
}
}
@@ -249,7 +269,16 @@ where
H::Out: Ord + 'static + codec::Codec,
{
fn from(storage: Storage) -> Self {
Self::new(storage)
Self::new_with_state_version(storage, Default::default())
}
}
impl<H: Hasher> From<(Storage, StateVersion)> for TestExternalities<H>
where
H::Out: Ord + 'static + codec::Codec,
{
fn from((storage, state_version): (Storage, StateVersion)) -> Self {
Self::new_with_state_version(storage, state_version)
}
}
@@ -309,14 +338,15 @@ mod tests {
#[test]
fn commit_should_work() {
let mut ext = TestExternalities::<BlakeTwo256>::default();
let storage = Storage::default(); // avoid adding the trie threshold.
let mut ext = TestExternalities::<BlakeTwo256>::from((storage, Default::default()));
let mut ext = ext.ext();
ext.set_storage(b"doe".to_vec(), b"reindeer".to_vec());
ext.set_storage(b"dog".to_vec(), b"puppy".to_vec());
ext.set_storage(b"dogglesworth".to_vec(), b"cat".to_vec());
let root =
H256::from(hex!("ed4d8c799d996add422395a6abd7545491d40bd838d738afafa1b8a4de625489"));
assert_eq!(H256::from_slice(ext.storage_root().as_slice()), root);
assert_eq!(H256::from_slice(ext.storage_root(Default::default()).as_slice()), root);
}
#[test]
substrate/primitives/state-machine/src/trie_backend.rs
+113 -42
View File
@@ -24,12 +24,12 @@ use crate::{
};
use codec::{Codec, Decode};
use hash_db::Hasher;
use sp_core::storage::{ChildInfo, ChildType};
use sp_core::storage::{ChildInfo, ChildType, StateVersion};
use sp_std::{boxed::Box, vec::Vec};
use sp_trie::{
child_delta_trie_root, delta_trie_root, empty_child_trie_root,
trie_types::{Layout, TrieDB, TrieError},
Trie,
trie_types::{TrieDB, TrieError},
LayoutV0, LayoutV1, Trie,
};
/// Patricia trie-based backend. Transaction type is an overlay of changes to commit.
@@ -186,6 +186,7 @@ where
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (H::Out, Self::Transaction)
where
H::Out: Ord,
@@ -195,8 +196,14 @@ where
{
let mut eph = Ephemeral::new(self.essence.backend_storage(), &mut write_overlay);
let res = match state_version {
StateVersion::V0 =>
delta_trie_root::<LayoutV0<H>, _, _, _, _, _>(&mut eph, root, delta),
StateVersion::V1 =>
delta_trie_root::<LayoutV1<H>, _, _, _, _, _>(&mut eph, root, delta),
};
match delta_trie_root::<Layout<H>, _, _, _, _, _>(&mut eph, root, delta) {
match res {
Ok(ret) => root = ret,
Err(e) => warn!(target: "trie", "Failed to write to trie: {}", e),
}
@@ -209,14 +216,14 @@ where
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (H::Out, bool, Self::Transaction)
where
H::Out: Ord,
{
let default_root = match child_info.child_type() {
ChildType::ParentKeyId => empty_child_trie_root::<Layout<H>>(),
ChildType::ParentKeyId => empty_child_trie_root::<LayoutV1<H>>(),
};
let mut write_overlay = S::Overlay::default();
let prefixed_storage_key = child_info.prefixed_storage_key();
let mut root = match self.storage(prefixed_storage_key.as_slice()) {
@@ -231,13 +238,20 @@ where
{
let mut eph = Ephemeral::new(self.essence.backend_storage(), &mut write_overlay);
match child_delta_trie_root::<Layout<H>, _, _, _, _, _, _>(
child_info.keyspace(),
&mut eph,
root,
delta,
) {
match match state_version {
StateVersion::V0 => child_delta_trie_root::<LayoutV0<H>, _, _, _, _, _, _>(
child_info.keyspace(),
&mut eph,
root,
delta,
),
StateVersion::V1 => child_delta_trie_root::<LayoutV1<H>, _, _, _, _, _, _>(
child_info.keyspace(),
&mut eph,
root,
delta,
),
} {
Ok(ret) => root = ret,
Err(e) => warn!(target: "trie", "Failed to write to trie: {}", e),
}
@@ -269,52 +283,91 @@ pub mod tests {
use codec::Encode;
use sp_core::H256;
use sp_runtime::traits::BlakeTwo256;
use sp_trie::{trie_types::TrieDBMut, KeySpacedDBMut, PrefixedMemoryDB, TrieMut};
use sp_trie::{
trie_types::{TrieDBMutV0, TrieDBMutV1},
KeySpacedDBMut, PrefixedMemoryDB, TrieMut,
};
use std::{collections::HashSet, iter};
const CHILD_KEY_1: &[u8] = b"sub1";
fn test_db() -> (PrefixedMemoryDB<BlakeTwo256>, H256) {
pub(crate) fn test_db(state_version: StateVersion) -> (PrefixedMemoryDB<BlakeTwo256>, H256) {
let child_info = ChildInfo::new_default(CHILD_KEY_1);
let mut root = H256::default();
let mut mdb = PrefixedMemoryDB::<BlakeTwo256>::default();
{
let mut mdb = KeySpacedDBMut::new(&mut mdb, child_info.keyspace());
let mut trie = TrieDBMut::new(&mut mdb, &mut root);
trie.insert(b"value3", &[142; 33]).expect("insert failed");
trie.insert(b"value4", &[124; 33]).expect("insert failed");
match state_version {
StateVersion::V0 => {
let mut trie = TrieDBMutV0::new(&mut mdb, &mut root);
trie.insert(b"value3", &[142; 33]).expect("insert failed");
trie.insert(b"value4", &[124; 33]).expect("insert failed");
},
StateVersion::V1 => {
let mut trie = TrieDBMutV1::new(&mut mdb, &mut root);
trie.insert(b"value3", &[142; 33]).expect("insert failed");
trie.insert(b"value4", &[124; 33]).expect("insert failed");
},
};
};
{
let mut sub_root = Vec::new();
root.encode_to(&mut sub_root);
let mut trie = TrieDBMut::new(&mut mdb, &mut root);
trie.insert(child_info.prefixed_storage_key().as_slice(), &sub_root[..])
.expect("insert failed");
trie.insert(b"key", b"value").expect("insert failed");
trie.insert(b"value1", &[42]).expect("insert failed");
trie.insert(b"value2", &[24]).expect("insert failed");
trie.insert(b":code", b"return 42").expect("insert failed");
for i in 128u8..255u8 {
trie.insert(&[i], &[i]).unwrap();
fn build<L: sp_trie::TrieLayout>(
mut trie: sp_trie::TrieDBMut<L>,
child_info: &ChildInfo,
sub_root: &[u8],
) {
trie.insert(child_info.prefixed_storage_key().as_slice(), sub_root)
.expect("insert failed");
trie.insert(b"key", b"value").expect("insert failed");
trie.insert(b"value1", &[42]).expect("insert failed");
trie.insert(b"value2", &[24]).expect("insert failed");
trie.insert(b":code", b"return 42").expect("insert failed");
for i in 128u8..255u8 {
trie.insert(&[i], &[i]).unwrap();
}
}
match state_version {
StateVersion::V0 => {
let trie = TrieDBMutV0::new(&mut mdb, &mut root);
build(trie, &child_info, &sub_root[..])
},
StateVersion::V1 => {
let trie = TrieDBMutV1::new(&mut mdb, &mut root);
build(trie, &child_info, &sub_root[..])
},
};
}
(mdb, root)
}
pub(crate) fn test_trie() -> TrieBackend<PrefixedMemoryDB<BlakeTwo256>, BlakeTwo256> {
let (mdb, root) = test_db();
pub(crate) fn test_trie(
hashed_value: StateVersion,
) -> TrieBackend<PrefixedMemoryDB<BlakeTwo256>, BlakeTwo256> {
let (mdb, root) = test_db(hashed_value);
TrieBackend::new(mdb, root)
}
#[test]
fn read_from_storage_returns_some() {
assert_eq!(test_trie().storage(b"key").unwrap(), Some(b"value".to_vec()));
read_from_storage_returns_some_inner(StateVersion::V0);
read_from_storage_returns_some_inner(StateVersion::V1);
}
fn read_from_storage_returns_some_inner(state_version: StateVersion) {
assert_eq!(test_trie(state_version).storage(b"key").unwrap(), Some(b"value".to_vec()));
}
#[test]
fn read_from_child_storage_returns_some() {
let test_trie = test_trie();
read_from_child_storage_returns_some_inner(StateVersion::V0);
read_from_child_storage_returns_some_inner(StateVersion::V1);
}
fn read_from_child_storage_returns_some_inner(state_version: StateVersion) {
let test_trie = test_trie(state_version);
assert_eq!(
test_trie
.child_storage(&ChildInfo::new_default(CHILD_KEY_1), b"value3")
@@ -341,12 +394,20 @@ pub mod tests {
#[test]
fn read_from_storage_returns_none() {
assert_eq!(test_trie().storage(b"non-existing-key").unwrap(), None);
read_from_storage_returns_none_inner(StateVersion::V0);
read_from_storage_returns_none_inner(StateVersion::V1);
}
fn read_from_storage_returns_none_inner(state_version: StateVersion) {
assert_eq!(test_trie(state_version).storage(b"non-existing-key").unwrap(), None);
}
#[test]
fn pairs_are_not_empty_on_non_empty_storage() {
assert!(!test_trie().pairs().is_empty());
pairs_are_not_empty_on_non_empty_storage_inner(StateVersion::V0);
pairs_are_not_empty_on_non_empty_storage_inner(StateVersion::V1);
}
fn pairs_are_not_empty_on_non_empty_storage_inner(state_version: StateVersion) {
assert!(!test_trie(state_version).pairs().is_empty());
}
#[test]
@@ -361,25 +422,35 @@ pub mod tests {
#[test]
fn storage_root_is_non_default() {
assert!(test_trie().storage_root(iter::empty()).0 != H256::repeat_byte(0));
storage_root_is_non_default_inner(StateVersion::V0);
storage_root_is_non_default_inner(StateVersion::V1);
}
#[test]
fn storage_root_transaction_is_empty() {
assert!(test_trie().storage_root(iter::empty()).1.drain().is_empty());
fn storage_root_is_non_default_inner(state_version: StateVersion) {
assert!(
test_trie(state_version).storage_root(iter::empty(), state_version).0 !=
H256::repeat_byte(0)
);
}
#[test]
fn storage_root_transaction_is_non_empty() {
let (new_root, mut tx) =
test_trie().storage_root(iter::once((&b"new-key"[..], Some(&b"new-value"[..]))));
storage_root_transaction_is_non_empty_inner(StateVersion::V0);
storage_root_transaction_is_non_empty_inner(StateVersion::V1);
}
fn storage_root_transaction_is_non_empty_inner(state_version: StateVersion) {
let (new_root, mut tx) = test_trie(state_version)
.storage_root(iter::once((&b"new-key"[..], Some(&b"new-value"[..]))), state_version);
assert!(!tx.drain().is_empty());
assert!(new_root != test_trie().storage_root(iter::empty()).0);
assert!(new_root != test_trie(state_version).storage_root(iter::empty(), state_version).0);
}
#[test]
fn prefix_walking_works() {
let trie = test_trie();
prefix_walking_works_inner(StateVersion::V0);
prefix_walking_works_inner(StateVersion::V1);
}
fn prefix_walking_works_inner(state_version: StateVersion) {
let trie = test_trie(state_version);
let mut seen = HashSet::new();
trie.for_keys_with_prefix(b"value", |key| {
substrate/primitives/state-machine/src/trie_backend_essence.rs
+95 -72
View File
@@ -20,20 +20,23 @@
use crate::{backend::Consolidate, debug, warn, StorageKey, StorageValue};
use codec::Encode;
use hash_db::{self, Hasher, Prefix};
use hash_db::{self, AsHashDB, HashDB, HashDBRef, Hasher, Prefix};
#[cfg(feature = "std")]
use parking_lot::RwLock;
use sp_core::storage::ChildInfo;
use sp_std::{boxed::Box, vec::Vec};
use sp_trie::{
empty_child_trie_root, read_child_trie_value, read_trie_value,
trie_types::{Layout, TrieDB, TrieError},
DBValue, KeySpacedDB, MemoryDB, PrefixedMemoryDB, Trie, TrieDBIterator,
trie_types::{TrieDB, TrieError},
DBValue, KeySpacedDB, PrefixedMemoryDB, Trie, TrieDBIterator, TrieDBKeyIterator,
};
#[cfg(feature = "std")]
use std::collections::HashMap;
#[cfg(feature = "std")]
use std::sync::Arc;
// In this module, we only use layout for read operation and empty root,
// where V1 and V0 are equivalent.
use sp_trie::LayoutV1 as Layout;
#[cfg(not(feature = "std"))]
macro_rules! format {
@@ -178,7 +181,7 @@ where
child_info: Option<&ChildInfo>,
key: &[u8],
) -> Result<Option<StorageKey>> {
let dyn_eph: &dyn hash_db::HashDBRef<_, _>;
let dyn_eph: &dyn HashDBRef<_, _>;
let keyspace_eph;
if let Some(child_info) = child_info.as_ref() {
keyspace_eph = KeySpacedDB::new(self, child_info.keyspace());
@@ -189,7 +192,7 @@ where
let trie =
TrieDB::<H>::new(dyn_eph, root).map_err(|e| format!("TrieDB creation error: {}", e))?;
let mut iter = trie.iter().map_err(|e| format!("TrieDB iteration error: {}", e))?;
let mut iter = trie.key_iter().map_err(|e| format!("TrieDB iteration error: {}", e))?;
// The key just after the one given in input, basically `key++0`.
// Note: We are sure this is the next key if:
@@ -205,7 +208,7 @@ where
let next_element = iter.next();
let next_key = if let Some(next_element) = next_element {
let (next_key, _) =
let next_key =
next_element.map_err(|e| format!("TrieDB iterator next error: {}", e))?;
Some(next_key)
} else {
@@ -291,17 +294,7 @@ where
&self.root
};
let _ = self.trie_iter_inner(
root,
prefix,
|k, _v| {
f(&k);
true
},
child_info,
None,
false,
);
self.trie_iter_key_inner(root, prefix, |k| f(k), child_info)
}
/// Execute given closure for all keys starting with prefix.
@@ -320,32 +313,70 @@ where
};
let mut root = H::Out::default();
root.as_mut().copy_from_slice(&root_vec);
let _ = self.trie_iter_inner(
self.trie_iter_key_inner(
&root,
Some(prefix),
|k, _v| {
f(&k);
|k| {
f(k);
true
},
Some(child_info),
None,
false,
);
)
}
/// Execute given closure for all keys starting with prefix.
pub fn for_keys_with_prefix<F: FnMut(&[u8])>(&self, prefix: &[u8], mut f: F) {
let _ = self.trie_iter_inner(
self.trie_iter_key_inner(
&self.root,
Some(prefix),
|k, _v| {
f(&k);
|k| {
f(k);
true
},
None,
None,
false,
);
)
}
fn trie_iter_key_inner<F: FnMut(&[u8]) -> bool>(
&self,
root: &H::Out,
prefix: Option<&[u8]>,
mut f: F,
child_info: Option<&ChildInfo>,
) {
let mut iter = move |db| -> sp_std::result::Result<(), Box<TrieError<H::Out>>> {
let trie = TrieDB::<H>::new(db, root)?;
let iter = if let Some(prefix) = prefix.as_ref() {
TrieDBKeyIterator::new_prefixed(&trie, prefix)?
} else {
TrieDBKeyIterator::new(&trie)?
};
for x in iter {
let key = x?;
debug_assert!(prefix
.as_ref()
.map(|prefix| key.starts_with(prefix))
.unwrap_or(true));
if !f(&key) {
break
}
}
Ok(())
};
let result = if let Some(child_info) = child_info {
let db = KeySpacedDB::new(self, child_info.keyspace());
iter(&db)
} else {
iter(self)
};
if let Err(e) = result {
debug!(target: "trie", "Error while iterating by prefix: {}", e);
}
}
fn trie_iter_inner<F: FnMut(Vec<u8>, Vec<u8>) -> bool>(
@@ -414,13 +445,13 @@ pub(crate) struct Ephemeral<'a, S: 'a + TrieBackendStorage<H>, H: 'a + Hasher> {
overlay: &'a mut S::Overlay,
}
impl<'a, S: 'a + TrieBackendStorage<H>, H: 'a + Hasher> hash_db::AsHashDB<H, DBValue>
impl<'a, S: 'a + TrieBackendStorage<H>, H: 'a + Hasher> AsHashDB<H, DBValue>
for Ephemeral<'a, S, H>
{
fn as_hash_db<'b>(&'b self) -> &'b (dyn hash_db::HashDB<H, DBValue> + 'b) {
fn as_hash_db<'b>(&'b self) -> &'b (dyn HashDB<H, DBValue> + 'b) {
self
}
fn as_hash_db_mut<'b>(&'b mut self) -> &'b mut (dyn hash_db::HashDB<H, DBValue> + 'b) {
fn as_hash_db_mut<'b>(&'b mut self) -> &'b mut (dyn HashDB<H, DBValue> + 'b) {
self
}
}
@@ -435,7 +466,7 @@ impl<'a, S: 'a + TrieBackendStorage<H>, H: Hasher> hash_db::HashDB<H, DBValue>
for Ephemeral<'a, S, H>
{
fn get(&self, key: &H::Out, prefix: Prefix) -> Option<DBValue> {
if let Some(val) = hash_db::HashDB::get(self.overlay, key, prefix) {
if let Some(val) = HashDB::get(self.overlay, key, prefix) {
Some(val)
} else {
match self.storage.get(&key, prefix) {
@@ -449,38 +480,37 @@ impl<'a, S: 'a + TrieBackendStorage<H>, H: Hasher> hash_db::HashDB<H, DBValue>
}
fn contains(&self, key: &H::Out, prefix: Prefix) -> bool {
hash_db::HashDB::get(self, key, prefix).is_some()
HashDB::get(self, key, prefix).is_some()
}
fn insert(&mut self, prefix: Prefix, value: &[u8]) -> H::Out {
hash_db::HashDB::insert(self.overlay, prefix, value)
HashDB::insert(self.overlay, prefix, value)
}
fn emplace(&mut self, key: H::Out, prefix: Prefix, value: DBValue) {
hash_db::HashDB::emplace(self.overlay, key, prefix, value)
HashDB::emplace(self.overlay, key, prefix, value)
}
fn remove(&mut self, key: &H::Out, prefix: Prefix) {
hash_db::HashDB::remove(self.overlay, key, prefix)
HashDB::remove(self.overlay, key, prefix)
}
}
impl<'a, S: 'a + TrieBackendStorage<H>, H: Hasher> hash_db::HashDBRef<H, DBValue>
for Ephemeral<'a, S, H>
{
impl<'a, S: 'a + TrieBackendStorage<H>, H: Hasher> HashDBRef<H, DBValue> for Ephemeral<'a, S, H> {
fn get(&self, key: &H::Out, prefix: Prefix) -> Option<DBValue> {
hash_db::HashDB::get(self, key, prefix)
HashDB::get(self, key, prefix)
}
fn contains(&self, key: &H::Out, prefix: Prefix) -> bool {
hash_db::HashDB::contains(self, key, prefix)
HashDB::contains(self, key, prefix)
}
}
/// Key-value pairs storage that is used by trie backend essence.
pub trait TrieBackendStorage<H: Hasher>: Send + Sync {
/// Type of in-memory overlay.
type Overlay: hash_db::HashDB<H, DBValue> + Default + Consolidate;
type Overlay: HashDB<H, DBValue> + Default + Consolidate;
/// Get the value stored at key.
fn get(&self, key: &H::Out, prefix: Prefix) -> Result<Option<DBValue>>;
}
@@ -495,35 +525,28 @@ impl<H: Hasher> TrieBackendStorage<H> for Arc<dyn Storage<H>> {
}
}
// This implementation is used by test storage trie clients.
impl<H: Hasher> TrieBackendStorage<H> for PrefixedMemoryDB<H> {
type Overlay = PrefixedMemoryDB<H>;
fn get(&self, key: &H::Out, prefix: Prefix) -> Result<Option<DBValue>> {
Ok(hash_db::HashDB::get(self, key, prefix))
}
}
impl<H: Hasher> TrieBackendStorage<H> for MemoryDB<H> {
type Overlay = MemoryDB<H>;
fn get(&self, key: &H::Out, prefix: Prefix) -> Result<Option<DBValue>> {
Ok(hash_db::HashDB::get(self, key, prefix))
}
}
impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::AsHashDB<H, DBValue>
for TrieBackendEssence<S, H>
impl<H, KF> TrieBackendStorage<H> for sp_trie::GenericMemoryDB<H, KF>
where
H: Hasher,
KF: sp_trie::KeyFunction<H> + Send + Sync,
{
fn as_hash_db<'b>(&'b self) -> &'b (dyn hash_db::HashDB<H, DBValue> + 'b) {
type Overlay = Self;
fn get(&self, key: &H::Out, prefix: Prefix) -> Result<Option<DBValue>> {
Ok(hash_db::HashDB::get(self, key, prefix))
}
}
impl<S: TrieBackendStorage<H>, H: Hasher> AsHashDB<H, DBValue> for TrieBackendEssence<S, H> {
fn as_hash_db<'b>(&'b self) -> &'b (dyn HashDB<H, DBValue> + 'b) {
self
}
fn as_hash_db_mut<'b>(&'b mut self) -> &'b mut (dyn hash_db::HashDB<H, DBValue> + 'b) {
fn as_hash_db_mut<'b>(&'b mut self) -> &'b mut (dyn HashDB<H, DBValue> + 'b) {
self
}
}
impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::HashDB<H, DBValue> for TrieBackendEssence<S, H> {
impl<S: TrieBackendStorage<H>, H: Hasher> HashDB<H, DBValue> for TrieBackendEssence<S, H> {
fn get(&self, key: &H::Out, prefix: Prefix) -> Option<DBValue> {
if *key == self.empty {
return Some([0u8].to_vec())
@@ -538,7 +561,7 @@ impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::HashDB<H, DBValue> for TrieBa
}
fn contains(&self, key: &H::Out, prefix: Prefix) -> bool {
hash_db::HashDB::get(self, key, prefix).is_some()
HashDB::get(self, key, prefix).is_some()
}
fn insert(&mut self, _prefix: Prefix, _value: &[u8]) -> H::Out {
@@ -554,15 +577,13 @@ impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::HashDB<H, DBValue> for TrieBa
}
}
impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::HashDBRef<H, DBValue>
for TrieBackendEssence<S, H>
{
impl<S: TrieBackendStorage<H>, H: Hasher> HashDBRef<H, DBValue> for TrieBackendEssence<S, H> {
fn get(&self, key: &H::Out, prefix: Prefix) -> Option<DBValue> {
hash_db::HashDB::get(self, key, prefix)
HashDB::get(self, key, prefix)
}
fn contains(&self, key: &H::Out, prefix: Prefix) -> bool {
hash_db::HashDB::contains(self, key, prefix)
HashDB::contains(self, key, prefix)
}
}
@@ -570,7 +591,9 @@ impl<S: TrieBackendStorage<H>, H: Hasher> hash_db::HashDBRef<H, DBValue>
mod test {
use super::*;
use sp_core::{Blake2Hasher, H256};
use sp_trie::{trie_types::TrieDBMut, KeySpacedDBMut, PrefixedMemoryDB, TrieMut};
use sp_trie::{
trie_types::TrieDBMutV1 as TrieDBMut, KeySpacedDBMut, PrefixedMemoryDB, TrieMut,
};
#[test]
fn next_storage_key_and_next_child_storage_key_work() {
substrate/primitives/storage/src/lib.rs
+51
View File
@@ -237,6 +237,9 @@ pub mod well_known_keys {
}
}
/// Threshold size to start using trie value nodes in state.
pub const TRIE_VALUE_NODE_THRESHOLD: u32 = 33;
/// Information related to a child state.
#[derive(Debug, Clone)]
#[cfg_attr(feature = "std", derive(PartialEq, Eq, Hash, PartialOrd, Ord, Encode, Decode))]
@@ -401,6 +404,54 @@ impl ChildTrieParentKeyId {
}
}
/// Different possible state version.
///
/// V0 and V1 uses a same trie implementation, but V1 will write external value node in the trie for
/// value with size at least `TRIE_VALUE_NODE_THRESHOLD`.
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
pub enum StateVersion {
/// Old state version, no value nodes.
V0 = 0,
/// New state version can use value nodes.
V1 = 1,
}
impl Default for StateVersion {
fn default() -> Self {
StateVersion::V1
}
}
impl From<StateVersion> for u8 {
fn from(version: StateVersion) -> u8 {
version as u8
}
}
impl sp_std::convert::TryFrom<u8> for StateVersion {
type Error = ();
fn try_from(val: u8) -> sp_std::result::Result<StateVersion, ()> {
match val {
0 => Ok(StateVersion::V0),
1 => Ok(StateVersion::V1),
_ => Err(()),
}
}
}
impl StateVersion {
/// If defined, values in state of size bigger or equal
/// to this threshold will use a separate trie node.
/// Otherwhise, value will be inlined in branch or leaf
/// node.
pub fn state_value_threshold(&self) -> Option<u32> {
match self {
StateVersion::V0 => None,
StateVersion::V1 => Some(TRIE_VALUE_NODE_THRESHOLD),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
substrate/primitives/tasks/src/async_externalities.rs
+7 -3
View File
@@ -19,7 +19,7 @@
//! Async externalities.
use sp_core::{
storage::{ChildInfo, TrackedStorageKey},
storage::{ChildInfo, StateVersion, TrackedStorageKey},
traits::{Externalities, RuntimeSpawn, RuntimeSpawnExt, SpawnNamed, TaskExecutorExt},
};
use sp_externalities::{Extensions, ExternalitiesExt as _};
@@ -126,11 +126,15 @@ impl Externalities for AsyncExternalities {
panic!("`storage_append`: should not be used in async externalities!")
}
fn storage_root(&mut self) -> Vec<u8> {
fn storage_root(&mut self, _state_version: StateVersion) -> Vec<u8> {
panic!("`storage_root`: should not be used in async externalities!")
}
fn child_storage_root(&mut self, _child_info: &ChildInfo) -> Vec<u8> {
fn child_storage_root(
&mut self,
_child_info: &ChildInfo,
_state_version: StateVersion,
) -> Vec<u8> {
panic!("`child_storage_root`: should not be used in async externalities!")
}
substrate/primitives/transaction-storage-proof/src/lib.rs
+1 -1
View File
@@ -143,7 +143,7 @@ pub mod registration {
use sp_trie::TrieMut;
type Hasher = sp_core::Blake2Hasher;
type TrieLayout = sp_trie::Layout<Hasher>;
type TrieLayout = sp_trie::LayoutV1<Hasher>;
/// Create a new inherent data provider instance for a given parent block hash.
pub fn new_data_provider<B, C>(
substrate/primitives/trie/Cargo.toml
+4 -4
View File
@@ -22,13 +22,13 @@ codec = { package = "parity-scale-codec", version = "2.0.0", default-features =
scale-info = { version = "1.0", default-features = false, features = ["derive"] }
sp-std = { version = "4.0.0", default-features = false, path = "../std" }
hash-db = { version = "0.15.2", default-features = false }
trie-db = { version = "0.22.6", default-features = false }
trie-root = { version = "0.16.0", default-features = false }
memory-db = { version = "0.27.0", default-features = false }
trie-db = { version = "0.23.0", default-features = false }
trie-root = { version = "0.17.0", default-features = false }
memory-db = { version = "0.28.0", default-features = false }
sp-core = { version = "4.1.0-dev", default-features = false, path = "../core" }
[dev-dependencies]
trie-bench = "0.28.0"
trie-bench = "0.29.0"
trie-standardmap = "0.15.2"
criterion = "0.3.3"
hex-literal = "0.3.4"
substrate/primitives/trie/benches/bench.rs
+2 -2
View File
@@ -21,11 +21,11 @@ criterion_main!(benches);
fn benchmark(c: &mut Criterion) {
trie_bench::standard_benchmark::<
sp_trie::Layout<sp_runtime::traits::BlakeTwo256>,
sp_trie::LayoutV1<sp_runtime::traits::BlakeTwo256>,
sp_trie::TrieStream,
>(c, "substrate-blake2");
trie_bench::standard_benchmark::<
sp_trie::Layout<sp_runtime::traits::BlakeTwo256>,
sp_trie::LayoutV1<sp_runtime::traits::BlakeTwo256>,
sp_trie::TrieStream,
>(c, "substrate-keccak");
}
substrate/primitives/trie/src/lib.rs
+166 -76
View File
@@ -45,31 +45,43 @@ pub use trie_db::proof::VerifyError;
use trie_db::proof::{generate_proof, verify_proof};
/// Various re-exports from the `trie-db` crate.
pub use trie_db::{
nibble_ops, CError, DBValue, Query, Recorder, Trie, TrieConfiguration, TrieDBIterator,
nibble_ops,
node::{NodePlan, ValuePlan},
CError, DBValue, Query, Recorder, Trie, TrieConfiguration, TrieDBIterator, TrieDBKeyIterator,
TrieLayout, TrieMut,
};
/// The Substrate format implementation of `TrieStream`.
pub use trie_stream::TrieStream;
#[derive(Default)]
/// substrate trie layout
pub struct Layout<H>(sp_std::marker::PhantomData<H>);
pub struct LayoutV0<H>(sp_std::marker::PhantomData<H>);
impl<H: Hasher> TrieLayout for Layout<H> {
/// substrate trie layout, with external value nodes.
pub struct LayoutV1<H>(sp_std::marker::PhantomData<H>);
impl<H> TrieLayout for LayoutV0<H>
where
H: Hasher,
{
const USE_EXTENSION: bool = false;
const ALLOW_EMPTY: bool = true;
const MAX_INLINE_VALUE: Option<u32> = None;
type Hash = H;
type Codec = NodeCodec<Self::Hash>;
}
impl<H: Hasher> TrieConfiguration for Layout<H> {
impl<H> TrieConfiguration for LayoutV0<H>
where
H: Hasher,
{
fn trie_root<I, A, B>(input: I) -> <Self::Hash as Hasher>::Out
where
I: IntoIterator<Item = (A, B)>,
A: AsRef<[u8]> + Ord,
B: AsRef<[u8]>,
{
trie_root::trie_root_no_extension::<H, TrieStream, _, _, _>(input)
trie_root::trie_root_no_extension::<H, TrieStream, _, _, _>(input, Self::MAX_INLINE_VALUE)
}
fn trie_root_unhashed<I, A, B>(input: I) -> Vec<u8>
@@ -78,7 +90,52 @@ impl<H: Hasher> TrieConfiguration for Layout<H> {
A: AsRef<[u8]> + Ord,
B: AsRef<[u8]>,
{
trie_root::unhashed_trie_no_extension::<H, TrieStream, _, _, _>(input)
trie_root::unhashed_trie_no_extension::<H, TrieStream, _, _, _>(
input,
Self::MAX_INLINE_VALUE,
)
}
fn encode_index(input: u32) -> Vec<u8> {
codec::Encode::encode(&codec::Compact(input))
}
}
impl<H> TrieLayout for LayoutV1<H>
where
H: Hasher,
{
const USE_EXTENSION: bool = false;
const ALLOW_EMPTY: bool = true;
const MAX_INLINE_VALUE: Option<u32> = Some(sp_core::storage::TRIE_VALUE_NODE_THRESHOLD);
type Hash = H;
type Codec = NodeCodec<Self::Hash>;
}
impl<H> TrieConfiguration for LayoutV1<H>
where
H: Hasher,
{
fn trie_root<I, A, B>(input: I) -> <Self::Hash as Hasher>::Out
where
I: IntoIterator<Item = (A, B)>,
A: AsRef<[u8]> + Ord,
B: AsRef<[u8]>,
{
trie_root::trie_root_no_extension::<H, TrieStream, _, _, _>(input, Self::MAX_INLINE_VALUE)
}
fn trie_root_unhashed<I, A, B>(input: I) -> Vec<u8>
where
I: IntoIterator<Item = (A, B)>,
A: AsRef<[u8]> + Ord,
B: AsRef<[u8]>,
{
trie_root::unhashed_trie_no_extension::<H, TrieStream, _, _, _>(
input,
Self::MAX_INLINE_VALUE,
)
}
fn encode_index(input: u32) -> Vec<u8> {
@@ -118,17 +175,22 @@ pub type TrieDBMut<'a, L> = trie_db::TrieDBMut<'a, L>;
pub type Lookup<'a, L, Q> = trie_db::Lookup<'a, L, Q>;
/// Hash type for a trie layout.
pub type TrieHash<L> = <<L as TrieLayout>::Hash as Hasher>::Out;
/// This module is for non generic definition of trie type.
/// Only the `Hasher` trait is generic in this case.
pub mod trie_types {
pub type Layout<H> = super::Layout<H>;
use super::*;
/// Persistent trie database read-access interface for the a given hasher.
pub type TrieDB<'a, H> = super::TrieDB<'a, Layout<H>>;
/// Read only V1 and V0 are compatible, thus we always use V1.
pub type TrieDB<'a, H> = super::TrieDB<'a, LayoutV1<H>>;
/// Persistent trie database write-access interface for the a given hasher.
pub type TrieDBMut<'a, H> = super::TrieDBMut<'a, Layout<H>>;
pub type TrieDBMutV0<'a, H> = super::TrieDBMut<'a, LayoutV0<H>>;
/// Persistent trie database write-access interface for the a given hasher.
pub type TrieDBMutV1<'a, H> = super::TrieDBMut<'a, LayoutV1<H>>;
/// Querying interface, as in `trie_db` but less generic.
pub type Lookup<'a, H, Q> = trie_db::Lookup<'a, Layout<H>, Q>;
pub type LookupV0<'a, H, Q> = trie_db::Lookup<'a, LayoutV0<H>, Q>;
/// Querying interface, as in `trie_db` but less generic.
pub type LookupV1<'a, H, Q> = trie_db::Lookup<'a, LayoutV1<H>, Q>;
/// As in `trie_db`, but less generic, error type for the crate.
pub type TrieError<H> = trie_db::TrieError<H, super::Error>;
}
@@ -141,16 +203,18 @@ pub mod trie_types {
/// For a key `K` that is included in the `db` a proof of inclusion is generated.
/// For a key `K` that is not included in the `db` a proof of non-inclusion is generated.
/// These can be later checked in `verify_trie_proof`.
pub fn generate_trie_proof<'a, L: TrieConfiguration, I, K, DB>(
pub fn generate_trie_proof<'a, L, I, K, DB>(
db: &DB,
root: TrieHash<L>,
keys: I,
) -> Result<Vec<Vec<u8>>, Box<TrieError<L>>>
where
L: TrieConfiguration,
I: IntoIterator<Item = &'a K>,
K: 'a + AsRef<[u8]>,
DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>,
{
// Can use default layout (read only).
let trie = TrieDB::<L>::new(db, &root)?;
generate_proof(&trie, keys)
}
@@ -163,17 +227,18 @@ where
/// checked for inclusion in the proof.
/// If the value is omitted (`(key, None)`), this key will be checked for non-inclusion in the
/// proof.
pub fn verify_trie_proof<'a, L: TrieConfiguration, I, K, V>(
pub fn verify_trie_proof<'a, L, I, K, V>(
root: &TrieHash<L>,
proof: &[Vec<u8>],
items: I,
) -> Result<(), VerifyError<TrieHash<L>, error::Error>>
) -> Result<(), VerifyError<TrieHash<L>, CError<L>>>
where
L: TrieConfiguration,
I: IntoIterator<Item = &'a (K, Option<V>)>,
K: 'a + AsRef<[u8]>,
V: 'a + AsRef<[u8]>,
{
verify_proof::<Layout<L::Hash>, _, _, _>(root, proof, items)
verify_proof::<L, _, _, _>(root, proof, items)
}
/// Determine a trie root given a hash DB and delta values.
@@ -207,28 +272,33 @@ where
}
/// Read a value from the trie.
pub fn read_trie_value<L: TrieConfiguration, DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>>(
pub fn read_trie_value<L, DB>(
db: &DB,
root: &TrieHash<L>,
key: &[u8],
) -> Result<Option<Vec<u8>>, Box<TrieError<L>>> {
TrieDB::<L>::new(&*db, root)?.get(key).map(|x| x.map(|val| val.to_vec()))
) -> Result<Option<Vec<u8>>, Box<TrieError<L>>>
where
L: TrieConfiguration,
DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>,
{
Ok(TrieDB::<L>::new(&*db, root)?.get(key).map(|x| x.map(|val| val.to_vec()))?)
}
/// Read a value from the trie with given Query.
pub fn read_trie_value_with<
L: TrieConfiguration,
Q: Query<L::Hash, Item = DBValue>,
DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>,
>(
pub fn read_trie_value_with<L, Q, DB>(
db: &DB,
root: &TrieHash<L>,
key: &[u8],
query: Q,
) -> Result<Option<Vec<u8>>, Box<TrieError<L>>> {
TrieDB::<L>::new(&*db, root)?
) -> Result<Option<Vec<u8>>, Box<TrieError<L>>>
where
L: TrieConfiguration,
Q: Query<L::Hash, Item = DBValue>,
DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>,
{
Ok(TrieDB::<L>::new(&*db, root)?
.get_with(key, query)
.map(|x| x.map(|val| val.to_vec()))
.map(|x| x.map(|val| val.to_vec()))?)
}
/// Determine the empty trie root.
@@ -319,7 +389,7 @@ where
}
/// Read a value from the child trie with given query.
pub fn read_child_trie_value_with<L: TrieConfiguration, Q: Query<L::Hash, Item = DBValue>, DB>(
pub fn read_child_trie_value_with<L, Q, DB>(
keyspace: &[u8],
db: &DB,
root_slice: &[u8],
@@ -327,6 +397,8 @@ pub fn read_child_trie_value_with<L: TrieConfiguration, Q: Query<L::Hash, Item =
query: Q,
) -> Result<Option<Vec<u8>>, Box<TrieError<L>>>
where
L: TrieConfiguration,
Q: Query<L::Hash, Item = DBValue>,
DB: hash_db::HashDBRef<L::Hash, trie_db::DBValue>,
{
let mut root = TrieHash::<L>::default();
@@ -444,11 +516,15 @@ where
/// Constants used into trie simplification codec.
mod trie_constants {
pub const EMPTY_TRIE: u8 = 0;
pub const NIBBLE_SIZE_BOUND: usize = u16::MAX as usize;
const FIRST_PREFIX: u8 = 0b_00 << 6;
pub const NIBBLE_SIZE_BOUND: usize = u16::max_value() as usize;
pub const LEAF_PREFIX_MASK: u8 = 0b_01 << 6;
pub const BRANCH_WITHOUT_MASK: u8 = 0b_10 << 6;
pub const BRANCH_WITH_MASK: u8 = 0b_11 << 6;
pub const EMPTY_TRIE: u8 = FIRST_PREFIX | (0b_00 << 4);
pub const ALT_HASHING_LEAF_PREFIX_MASK: u8 = FIRST_PREFIX | (0b_1 << 5);
pub const ALT_HASHING_BRANCH_WITH_MASK: u8 = FIRST_PREFIX | (0b_01 << 4);
pub const ESCAPE_COMPACT_HEADER: u8 = EMPTY_TRIE | 0b_00_01;
}
#[cfg(test)]
@@ -461,7 +537,11 @@ mod tests {
use trie_db::{DBValue, NodeCodec as NodeCodecT, Trie, TrieMut};
use trie_standardmap::{Alphabet, StandardMap, ValueMode};
type Layout = super::Layout<Blake2Hasher>;
type LayoutV0 = super::LayoutV0<Blake2Hasher>;
type LayoutV1 = super::LayoutV1<Blake2Hasher>;
type MemoryDBMeta<H> =
memory_db::MemoryDB<H, memory_db::HashKey<H>, trie_db::DBValue, MemTracker>;
fn hashed_null_node<T: TrieConfiguration>() -> TrieHash<T> {
<T::Codec as NodeCodecT>::hashed_null_node()
@@ -473,7 +553,7 @@ mod tests {
let d = T::trie_root_unhashed(input.clone());
println!("Data: {:#x?}, {:#x?}", d, Blake2Hasher::hash(&d[..]));
let persistent = {
let mut memdb = MemoryDB::default();
let mut memdb = MemoryDBMeta::default();
let mut root = Default::default();
let mut t = TrieDBMut::<T>::new(&mut memdb, &mut root);
for (x, y) in input.iter().rev() {
@@ -486,7 +566,7 @@ mod tests {
}
fn check_iteration<T: TrieConfiguration>(input: &Vec<(&[u8], &[u8])>) {
let mut memdb = MemoryDB::default();
let mut memdb = MemoryDBMeta::default();
let mut root = Default::default();
{
let mut t = TrieDBMut::<T>::new(&mut memdb, &mut root);
@@ -506,14 +586,21 @@ mod tests {
}
}
fn check_input(input: &Vec<(&[u8], &[u8])>) {
check_equivalent::<LayoutV0>(input);
check_iteration::<LayoutV0>(input);
check_equivalent::<LayoutV1>(input);
check_iteration::<LayoutV1>(input);
}
#[test]
fn default_trie_root() {
let mut db = MemoryDB::default();
let mut root = TrieHash::<Layout>::default();
let mut empty = TrieDBMut::<Layout>::new(&mut db, &mut root);
let mut root = TrieHash::<LayoutV1>::default();
let mut empty = TrieDBMut::<LayoutV1>::new(&mut db, &mut root);
empty.commit();
let root1 = empty.root().as_ref().to_vec();
let root2: Vec<u8> = Layout::trie_root::<_, Vec<u8>, Vec<u8>>(std::iter::empty())
let root2: Vec<u8> = LayoutV1::trie_root::<_, Vec<u8>, Vec<u8>>(std::iter::empty())
.as_ref()
.iter()
.cloned()
@@ -525,31 +612,27 @@ mod tests {
#[test]
fn empty_is_equivalent() {
let input: Vec<(&[u8], &[u8])> = vec![];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
fn leaf_is_equivalent() {
let input: Vec<(&[u8], &[u8])> = vec![(&[0xaa][..], &[0xbb][..])];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
fn branch_is_equivalent() {
let input: Vec<(&[u8], &[u8])> =
vec![(&[0xaa][..], &[0x10][..]), (&[0xba][..], &[0x11][..])];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
fn extension_and_branch_is_equivalent() {
let input: Vec<(&[u8], &[u8])> =
vec![(&[0xaa][..], &[0x10][..]), (&[0xab][..], &[0x11][..])];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
@@ -564,8 +647,7 @@ mod tests {
let mut d = st.make();
d.sort_by(|&(ref a, _), &(ref b, _)| a.cmp(b));
let dr = d.iter().map(|v| (&v.0[..], &v.1[..])).collect();
check_equivalent::<Layout>(&dr);
check_iteration::<Layout>(&dr);
check_input(&dr);
}
#[test]
@@ -575,8 +657,7 @@ mod tests {
(&[0xaa, 0xaa][..], &[0xaa][..]),
(&[0xaa, 0xbb][..], &[0xab][..]),
];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
@@ -589,8 +670,7 @@ mod tests {
(&[0xbb, 0xbb][..], &[0xbb][..]),
(&[0xbb, 0xcc][..], &[0xbc][..]),
];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
@@ -602,8 +682,7 @@ mod tests {
),
(&[0xba][..], &[0x11][..]),
];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
#[test]
@@ -618,15 +697,17 @@ mod tests {
&b"ABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABCABC"[..],
),
];
check_equivalent::<Layout>(&input);
check_iteration::<Layout>(&input);
check_input(&input);
}
fn populate_trie<'db, T: TrieConfiguration>(
fn populate_trie<'db, T>(
db: &'db mut dyn HashDB<T::Hash, DBValue>,
root: &'db mut TrieHash<T>,
v: &[(Vec<u8>, Vec<u8>)],
) -> TrieDBMut<'db, T> {
) -> TrieDBMut<'db, T>
where
T: TrieConfiguration,
{
let mut t = TrieDBMut::<T>::new(db, root);
for i in 0..v.len() {
let key: &[u8] = &v[i].0;
@@ -648,8 +729,12 @@ mod tests {
#[test]
fn random_should_work() {
random_should_work_inner::<LayoutV1>();
random_should_work_inner::<LayoutV0>();
}
fn random_should_work_inner<L: TrieConfiguration>() {
let mut seed = <Blake2Hasher as Hasher>::Out::zero();
for test_i in 0..10000 {
for test_i in 0..10_000 {
if test_i % 50 == 0 {
println!("{:?} of 10000 stress tests done", test_i);
}
@@ -662,10 +747,11 @@ mod tests {
}
.make_with(seed.as_fixed_bytes_mut());
let real = Layout::trie_root(x.clone());
let real = L::trie_root(x.clone());
let mut memdb = MemoryDB::default();
let mut root = Default::default();
let mut memtrie = populate_trie::<Layout>(&mut memdb, &mut root, &x);
let mut memtrie = populate_trie::<L>(&mut memdb, &mut root, &x);
memtrie.commit();
if *memtrie.root() != real {
@@ -677,9 +763,9 @@ mod tests {
}
}
assert_eq!(*memtrie.root(), real);
unpopulate_trie::<Layout>(&mut memtrie, &x);
unpopulate_trie::<L>(&mut memtrie, &x);
memtrie.commit();
let hashed_null_node = hashed_null_node::<Layout>();
let hashed_null_node = hashed_null_node::<L>();
if *memtrie.root() != hashed_null_node {
println!("- TRIE MISMATCH");
println!("");
@@ -699,7 +785,7 @@ mod tests {
#[test]
fn codec_trie_empty() {
let input: Vec<(&[u8], &[u8])> = vec![];
let trie = Layout::trie_root_unhashed::<_, _, _>(input);
let trie = LayoutV1::trie_root_unhashed(input);
println!("trie: {:#x?}", trie);
assert_eq!(trie, vec![0x0]);
}
@@ -707,7 +793,7 @@ mod tests {
#[test]
fn codec_trie_single_tuple() {
let input = vec![(vec![0xaa], vec![0xbb])];
let trie = Layout::trie_root_unhashed::<_, _, _>(input);
let trie = LayoutV1::trie_root_unhashed(input);
println!("trie: {:#x?}", trie);
assert_eq!(
trie,
@@ -723,7 +809,7 @@ mod tests {
#[test]
fn codec_trie_two_tuples_disjoint_keys() {
let input = vec![(&[0x48, 0x19], &[0xfe]), (&[0x13, 0x14], &[0xff])];
let trie = Layout::trie_root_unhashed::<_, _, _>(input);
let trie = LayoutV1::trie_root_unhashed(input);
println!("trie: {:#x?}", trie);
let mut ex = Vec::<u8>::new();
ex.push(0x80); // branch, no value (0b_10..) no nibble
@@ -747,6 +833,10 @@ mod tests {
#[test]
fn iterator_works() {
iterator_works_inner::<LayoutV1>();
iterator_works_inner::<LayoutV0>();
}
fn iterator_works_inner<Layout: TrieConfiguration>() {
let pairs = vec![
(hex!("0103000000000000000464").to_vec(), hex!("0400000000").to_vec()),
(hex!("0103000000000000000469").to_vec(), hex!("0401000000").to_vec()),
@@ -777,15 +867,15 @@ mod tests {
let mut memdb = MemoryDB::default();
let mut root = Default::default();
populate_trie::<Layout>(&mut memdb, &mut root, &pairs);
populate_trie::<LayoutV1>(&mut memdb, &mut root, &pairs);
let non_included_key: Vec<u8> = hex!("0909").to_vec();
let proof =
generate_trie_proof::<Layout, _, _, _>(&memdb, root, &[non_included_key.clone()])
generate_trie_proof::<LayoutV1, _, _, _>(&memdb, root, &[non_included_key.clone()])
.unwrap();
// Verifying that the K was not included into the trie should work.
assert!(verify_trie_proof::<Layout, _, _, Vec<u8>>(
assert!(verify_trie_proof::<LayoutV1, _, _, Vec<u8>>(
&root,
&proof,
&[(non_included_key.clone(), None)],
@@ -793,7 +883,7 @@ mod tests {
.is_ok());
// Verifying that the K was included into the trie should fail.
assert!(verify_trie_proof::<Layout, _, _, Vec<u8>>(
assert!(verify_trie_proof::<LayoutV1, _, _, Vec<u8>>(
&root,
&proof,
&[(non_included_key, Some(hex!("1010").to_vec()))],
@@ -810,13 +900,13 @@ mod tests {
let mut memdb = MemoryDB::default();
let mut root = Default::default();
populate_trie::<Layout>(&mut memdb, &mut root, &pairs);
populate_trie::<LayoutV1>(&mut memdb, &mut root, &pairs);
let proof =
generate_trie_proof::<Layout, _, _, _>(&memdb, root, &[pairs[0].0.clone()]).unwrap();
generate_trie_proof::<LayoutV1, _, _, _>(&memdb, root, &[pairs[0].0.clone()]).unwrap();
// Check that a K, V included into the proof are verified.
assert!(verify_trie_proof::<Layout, _, _, _>(
assert!(verify_trie_proof::<LayoutV1, _, _, _>(
&root,
&proof,
&[(pairs[0].0.clone(), Some(pairs[0].1.clone()))]
@@ -824,7 +914,7 @@ mod tests {
.is_ok());
// Absence of the V is not verified with the proof that has K, V included.
assert!(verify_trie_proof::<Layout, _, _, Vec<u8>>(
assert!(verify_trie_proof::<LayoutV1, _, _, Vec<u8>>(
&root,
&proof,
&[(pairs[0].0.clone(), None)]
@@ -832,7 +922,7 @@ mod tests {
.is_err());
// K not included into the trie is not verified.
assert!(verify_trie_proof::<Layout, _, _, _>(
assert!(verify_trie_proof::<LayoutV1, _, _, _>(
&root,
&proof,
&[(hex!("4242").to_vec(), Some(pairs[0].1.clone()))]
@@ -840,7 +930,7 @@ mod tests {
.is_err());
// K included into the trie but not included into the proof is not verified.
assert!(verify_trie_proof::<Layout, _, _, _>(
assert!(verify_trie_proof::<LayoutV1, _, _, _>(
&root,
&proof,
&[(pairs[1].0.clone(), Some(pairs[1].1.clone()))]
@@ -865,13 +955,13 @@ mod tests {
.unwrap();
let proof_db = proof.into_memory_db::<Blake2Hasher>();
let first_storage_root = delta_trie_root::<Layout, _, _, _, _, _>(
let first_storage_root = delta_trie_root::<LayoutV0, _, _, _, _, _>(
&mut proof_db.clone(),
storage_root,
valid_delta,
)
.unwrap();
let second_storage_root = delta_trie_root::<Layout, _, _, _, _, _>(
let second_storage_root = delta_trie_root::<LayoutV0, _, _, _, _, _>(
&mut proof_db.clone(),
storage_root,
invalid_delta,
substrate/primitives/trie/src/node_codec.rs
+87 -27
View File
@@ -24,7 +24,7 @@ use hash_db::Hasher;
use sp_std::{borrow::Borrow, marker::PhantomData, ops::Range, vec::Vec};
use trie_db::{
self, nibble_ops,
node::{NibbleSlicePlan, NodeHandlePlan, NodePlan},
node::{NibbleSlicePlan, NodeHandlePlan, NodePlan, Value, ValuePlan},
ChildReference, NodeCodec as NodeCodecT, Partial,
};
@@ -80,7 +80,11 @@ impl<'a> Input for ByteSliceInput<'a> {
#[derive(Default, Clone)]
pub struct NodeCodec<H>(PhantomData<H>);
impl<H: Hasher> NodeCodecT for NodeCodec<H> {
impl<H> NodeCodecT for NodeCodec<H>
where
H: Hasher,
{
const ESCAPE_HEADER: Option<u8> = Some(trie_constants::ESCAPE_COMPACT_HEADER);
type Error = Error;
type HashOut = H::Out;
@@ -88,11 +92,22 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
H::hash(<Self as NodeCodecT>::empty_node())
}
fn decode_plan(data: &[u8]) -> sp_std::result::Result<NodePlan, Self::Error> {
fn decode_plan(data: &[u8]) -> Result<NodePlan, Self::Error> {
let mut input = ByteSliceInput::new(data);
match NodeHeader::decode(&mut input)? {
let header = NodeHeader::decode(&mut input)?;
let contains_hash = header.contains_hash_of_value();
let branch_has_value = if let NodeHeader::Branch(has_value, _) = &header {
*has_value
} else {
// hashed_value_branch
true
};
match header {
NodeHeader::Null => Ok(NodePlan::Empty),
NodeHeader::Branch(has_value, nibble_count) => {
NodeHeader::HashedValueBranch(nibble_count) | NodeHeader::Branch(_, nibble_count) => {
let padding = nibble_count % nibble_ops::NIBBLE_PER_BYTE != 0;
// check that the padding is valid (if any)
if padding && nibble_ops::pad_left(data[input.offset]) != 0 {
@@ -105,9 +120,13 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
let partial_padding = nibble_ops::number_padding(nibble_count);
let bitmap_range = input.take(BITMAP_LENGTH)?;
let bitmap = Bitmap::decode(&data[bitmap_range])?;
let value = if has_value {
let count = <Compact<u32>>::decode(&mut input)?.0 as usize;
Some(input.take(count)?)
let value = if branch_has_value {
Some(if contains_hash {
ValuePlan::Node(input.take(H::LENGTH)?)
} else {
let count = <Compact<u32>>::decode(&mut input)?.0 as usize;
ValuePlan::Inline(input.take(count)?)
})
} else {
None
};
@@ -132,7 +151,7 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
children,
})
},
NodeHeader::Leaf(nibble_count) => {
NodeHeader::HashedValueLeaf(nibble_count) | NodeHeader::Leaf(nibble_count) => {
let padding = nibble_count % nibble_ops::NIBBLE_PER_BYTE != 0;
// check that the padding is valid (if any)
if padding && nibble_ops::pad_left(data[input.offset]) != 0 {
@@ -143,10 +162,16 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
nibble_ops::NIBBLE_PER_BYTE,
)?;
let partial_padding = nibble_ops::number_padding(nibble_count);
let count = <Compact<u32>>::decode(&mut input)?.0 as usize;
let value = if contains_hash {
ValuePlan::Node(input.take(H::LENGTH)?)
} else {
let count = <Compact<u32>>::decode(&mut input)?.0 as usize;
ValuePlan::Inline(input.take(count)?)
};
Ok(NodePlan::Leaf {
partial: NibbleSlicePlan::new(partial, partial_padding),
value: input.take(count)?,
value,
})
},
}
@@ -160,9 +185,23 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
&[trie_constants::EMPTY_TRIE]
}
fn leaf_node(partial: Partial, value: &[u8]) -> Vec<u8> {
let mut output = partial_encode(partial, NodeKind::Leaf);
value.encode_to(&mut output);
fn leaf_node(partial: Partial, value: Value) -> Vec<u8> {
let contains_hash = matches!(&value, Value::Node(..));
let mut output = if contains_hash {
partial_encode(partial, NodeKind::HashedValueLeaf)
} else {
partial_encode(partial, NodeKind::Leaf)
};
match value {
Value::Inline(value) => {
Compact(value.len() as u32).encode_to(&mut output);
output.extend_from_slice(value);
},
Value::Node(hash, _) => {
debug_assert!(hash.len() == H::LENGTH);
output.extend_from_slice(hash);
},
}
output
}
@@ -171,33 +210,46 @@ impl<H: Hasher> NodeCodecT for NodeCodec<H> {
_nbnibble: usize,
_child: ChildReference<<H as Hasher>::Out>,
) -> Vec<u8> {
unreachable!()
unreachable!("No extension codec.")
}
fn branch_node(
_children: impl Iterator<Item = impl Borrow<Option<ChildReference<<H as Hasher>::Out>>>>,
_maybe_value: Option<&[u8]>,
_maybe_value: Option<Value>,
) -> Vec<u8> {
unreachable!()
unreachable!("No extension codec.")
}
fn branch_node_nibbled(
partial: impl Iterator<Item = u8>,
number_nibble: usize,
children: impl Iterator<Item = impl Borrow<Option<ChildReference<<H as Hasher>::Out>>>>,
maybe_value: Option<&[u8]>,
value: Option<Value>,
) -> Vec<u8> {
let mut output = if maybe_value.is_some() {
partial_from_iterator_encode(partial, number_nibble, NodeKind::BranchWithValue)
} else {
partial_from_iterator_encode(partial, number_nibble, NodeKind::BranchNoValue)
let contains_hash = matches!(&value, Some(Value::Node(..)));
let mut output = match (&value, contains_hash) {
(&None, _) =>
partial_from_iterator_encode(partial, number_nibble, NodeKind::BranchNoValue),
(_, false) =>
partial_from_iterator_encode(partial, number_nibble, NodeKind::BranchWithValue),
(_, true) =>
partial_from_iterator_encode(partial, number_nibble, NodeKind::HashedValueBranch),
};
let bitmap_index = output.len();
let mut bitmap: [u8; BITMAP_LENGTH] = [0; BITMAP_LENGTH];
(0..BITMAP_LENGTH).for_each(|_| output.push(0));
if let Some(value) = maybe_value {
value.encode_to(&mut output);
};
match value {
Some(Value::Inline(value)) => {
Compact(value.len() as u32).encode_to(&mut output);
output.extend_from_slice(value);
},
Some(Value::Node(hash, _)) => {
debug_assert!(hash.len() == H::LENGTH);
output.extend_from_slice(hash);
},
None => (),
}
Bitmap::encode(
children.map(|maybe_child| match maybe_child.borrow() {
Some(ChildReference::Hash(h)) => {
@@ -229,11 +281,15 @@ fn partial_from_iterator_encode<I: Iterator<Item = u8>>(
) -> Vec<u8> {
let nibble_count = sp_std::cmp::min(trie_constants::NIBBLE_SIZE_BOUND, nibble_count);
let mut output = Vec::with_capacity(3 + (nibble_count / nibble_ops::NIBBLE_PER_BYTE));
let mut output = Vec::with_capacity(4 + (nibble_count / nibble_ops::NIBBLE_PER_BYTE));
match node_kind {
NodeKind::Leaf => NodeHeader::Leaf(nibble_count).encode_to(&mut output),
NodeKind::BranchWithValue => NodeHeader::Branch(true, nibble_count).encode_to(&mut output),
NodeKind::BranchNoValue => NodeHeader::Branch(false, nibble_count).encode_to(&mut output),
NodeKind::HashedValueLeaf =>
NodeHeader::HashedValueLeaf(nibble_count).encode_to(&mut output),
NodeKind::HashedValueBranch =>
NodeHeader::HashedValueBranch(nibble_count).encode_to(&mut output),
};
output.extend(partial);
output
@@ -247,11 +303,15 @@ fn partial_encode(partial: Partial, node_kind: NodeKind) -> Vec<u8> {
let nibble_count = sp_std::cmp::min(trie_constants::NIBBLE_SIZE_BOUND, nibble_count);
let mut output = Vec::with_capacity(3 + partial.1.len());
let mut output = Vec::with_capacity(4 + partial.1.len());
match node_kind {
NodeKind::Leaf => NodeHeader::Leaf(nibble_count).encode_to(&mut output),
NodeKind::BranchWithValue => NodeHeader::Branch(true, nibble_count).encode_to(&mut output),
NodeKind::BranchNoValue => NodeHeader::Branch(false, nibble_count).encode_to(&mut output),
NodeKind::HashedValueLeaf =>
NodeHeader::HashedValueLeaf(nibble_count).encode_to(&mut output),
NodeKind::HashedValueBranch =>
NodeHeader::HashedValueBranch(nibble_count).encode_to(&mut output),
};
if number_nibble_encoded > 0 {
output.push(nibble_ops::pad_right((partial.0).1));
substrate/primitives/trie/src/node_header.rs
+74 -19
View File
@@ -25,8 +25,20 @@ use sp_std::iter::once;
#[derive(Copy, Clone, PartialEq, Eq, sp_core::RuntimeDebug)]
pub(crate) enum NodeHeader {
Null,
// contains wether there is a value and nibble count
Branch(bool, usize),
// contains nibble count
Leaf(usize),
// contains nibble count.
HashedValueBranch(usize),
// contains nibble count.
HashedValueLeaf(usize),
}
impl NodeHeader {
pub(crate) fn contains_hash_of_value(&self) -> bool {
matches!(self, NodeHeader::HashedValueBranch(_) | NodeHeader::HashedValueLeaf(_))
}
}
/// NodeHeader without content
@@ -34,6 +46,8 @@ pub(crate) enum NodeKind {
Leaf,
BranchNoValue,
BranchWithValue,
HashedValueLeaf,
HashedValueBranch,
}
impl Encode for NodeHeader {
@@ -41,11 +55,27 @@ impl Encode for NodeHeader {
match self {
NodeHeader::Null => output.push_byte(trie_constants::EMPTY_TRIE),
NodeHeader::Branch(true, nibble_count) =>
encode_size_and_prefix(*nibble_count, trie_constants::BRANCH_WITH_MASK, output),
NodeHeader::Branch(false, nibble_count) =>
encode_size_and_prefix(*nibble_count, trie_constants::BRANCH_WITHOUT_MASK, output),
encode_size_and_prefix(*nibble_count, trie_constants::BRANCH_WITH_MASK, 2, output),
NodeHeader::Branch(false, nibble_count) => encode_size_and_prefix(
*nibble_count,
trie_constants::BRANCH_WITHOUT_MASK,
2,
output,
),
NodeHeader::Leaf(nibble_count) =>
encode_size_and_prefix(*nibble_count, trie_constants::LEAF_PREFIX_MASK, output),
encode_size_and_prefix(*nibble_count, trie_constants::LEAF_PREFIX_MASK, 2, output),
NodeHeader::HashedValueBranch(nibble_count) => encode_size_and_prefix(
*nibble_count,
trie_constants::ALT_HASHING_BRANCH_WITH_MASK,
4,
output,
),
NodeHeader::HashedValueLeaf(nibble_count) => encode_size_and_prefix(
*nibble_count,
trie_constants::ALT_HASHING_LEAF_PREFIX_MASK,
3,
output,
),
}
}
}
@@ -59,13 +89,22 @@ impl Decode for NodeHeader {
return Ok(NodeHeader::Null)
}
match i & (0b11 << 6) {
trie_constants::LEAF_PREFIX_MASK => Ok(NodeHeader::Leaf(decode_size(i, input)?)),
trie_constants::BRANCH_WITHOUT_MASK =>
Ok(NodeHeader::Branch(false, decode_size(i, input)?)),
trie_constants::LEAF_PREFIX_MASK => Ok(NodeHeader::Leaf(decode_size(i, input, 2)?)),
trie_constants::BRANCH_WITH_MASK =>
Ok(NodeHeader::Branch(true, decode_size(i, input)?)),
// do not allow any special encoding
_ => Err("Unallowed encoding".into()),
Ok(NodeHeader::Branch(true, decode_size(i, input, 2)?)),
trie_constants::BRANCH_WITHOUT_MASK =>
Ok(NodeHeader::Branch(false, decode_size(i, input, 2)?)),
trie_constants::EMPTY_TRIE => {
if i & (0b111 << 5) == trie_constants::ALT_HASHING_LEAF_PREFIX_MASK {
Ok(NodeHeader::HashedValueLeaf(decode_size(i, input, 3)?))
} else if i & (0b1111 << 4) == trie_constants::ALT_HASHING_BRANCH_WITH_MASK {
Ok(NodeHeader::HashedValueBranch(decode_size(i, input, 4)?))
} else {
// do not allow any special encoding
Err("Unallowed encoding".into())
}
},
_ => unreachable!(),
}
}
}
@@ -73,12 +112,20 @@ impl Decode for NodeHeader {
/// Returns an iterator over encoded bytes for node header and size.
/// Size encoding allows unlimited, length inefficient, representation, but
/// is bounded to 16 bit maximum value to avoid possible DOS.
pub(crate) fn size_and_prefix_iterator(size: usize, prefix: u8) -> impl Iterator<Item = u8> {
pub(crate) fn size_and_prefix_iterator(
size: usize,
prefix: u8,
prefix_mask: usize,
) -> impl Iterator<Item = u8> {
let size = sp_std::cmp::min(trie_constants::NIBBLE_SIZE_BOUND, size);
let l1 = sp_std::cmp::min(62, size);
let (first_byte, mut rem) =
if size == l1 { (once(prefix + l1 as u8), 0) } else { (once(prefix + 63), size - l1) };
let max_value = 255u8 >> prefix_mask;
let l1 = sp_std::cmp::min((max_value as usize).saturating_sub(1), size);
let (first_byte, mut rem) = if size == l1 {
(once(prefix + l1 as u8), 0)
} else {
(once(prefix + max_value as u8), size - l1)
};
let next_bytes = move || {
if rem > 0 {
if rem < 256 {
@@ -97,16 +144,24 @@ pub(crate) fn size_and_prefix_iterator(size: usize, prefix: u8) -> impl Iterator
}
/// Encodes size and prefix to a stream output.
fn encode_size_and_prefix<W: Output + ?Sized>(size: usize, prefix: u8, out: &mut W) {
for b in size_and_prefix_iterator(size, prefix) {
fn encode_size_and_prefix<W>(size: usize, prefix: u8, prefix_mask: usize, out: &mut W)
where
W: Output + ?Sized,
{
for b in size_and_prefix_iterator(size, prefix, prefix_mask) {
out.push_byte(b)
}
}
/// Decode size only from stream input and header byte.
fn decode_size(first: u8, input: &mut impl Input) -> Result<usize, codec::Error> {
let mut result = (first & 255u8 >> 2) as usize;
if result < 63 {
fn decode_size(
first: u8,
input: &mut impl Input,
prefix_mask: usize,
) -> Result<usize, codec::Error> {
let max_value = 255u8 >> prefix_mask;
let mut result = (first & max_value) as usize;
if result < max_value as usize {
return Ok(result)
}
result -= 1;
substrate/primitives/trie/src/storage_proof.rs
+7 -4
View File
@@ -19,6 +19,9 @@ use codec::{Decode, Encode};
use hash_db::{HashDB, Hasher};
use scale_info::TypeInfo;
use sp_std::vec::Vec;
// Note that `LayoutV1` usage here (proof compaction) is compatible
// with `LayoutV0`.
use crate::LayoutV1 as Layout;
/// A proof that some set of key-value pairs are included in the storage trie. The proof contains
/// the storage values so that the partial storage backend can be reconstructed by a verifier that
@@ -95,8 +98,8 @@ impl StorageProof {
pub fn into_compact_proof<H: Hasher>(
self,
root: H::Out,
) -> Result<CompactProof, crate::CompactProofError<crate::Layout<H>>> {
crate::encode_compact::<crate::Layout<H>>(self, root)
) -> Result<CompactProof, crate::CompactProofError<Layout<H>>> {
crate::encode_compact::<Layout<H>>(self, root)
}
/// Returns the estimated encoded size of the compact proof.
@@ -124,9 +127,9 @@ impl CompactProof {
pub fn to_storage_proof<H: Hasher>(
&self,
expected_root: Option<&H::Out>,
) -> Result<(StorageProof, H::Out), crate::CompactProofError<crate::Layout<H>>> {
) -> Result<(StorageProof, H::Out), crate::CompactProofError<Layout<H>>> {
let mut db = crate::MemoryDB::<H>::new(&[]);
let root = crate::decode_compact::<crate::Layout<H>, _, _>(
let root = crate::decode_compact::<Layout<H>, _, _>(
&mut db,
self.iter_compact_encoded_nodes(),
expected_root,
substrate/primitives/trie/src/trie_codec.rs
+2 -4
View File
@@ -112,8 +112,7 @@ where
I: IntoIterator<Item = &'a [u8]>,
{
let mut nodes_iter = encoded.into_iter();
let (top_root, _nb_used) =
trie_db::decode_compact_from_iter::<L, _, _, _>(db, &mut nodes_iter)?;
let (top_root, _nb_used) = trie_db::decode_compact_from_iter::<L, _, _>(db, &mut nodes_iter)?;
// Only check root if expected root is passed as argument.
if let Some(expected_root) = expected_root {
@@ -164,8 +163,7 @@ where
let mut nodes_iter = nodes_iter.peekable();
for child_root in child_tries.into_iter() {
if previous_extracted_child_trie.is_none() && nodes_iter.peek().is_some() {
let (top_root, _) =
trie_db::decode_compact_from_iter::<L, _, _, _>(db, &mut nodes_iter)?;
let (top_root, _) = trie_db::decode_compact_from_iter::<L, _, _>(db, &mut nodes_iter)?;
previous_extracted_child_trie = Some(top_root);
}
substrate/primitives/trie/src/trie_stream.rs
+44 -37
View File
@@ -18,21 +18,18 @@
//! `TrieStream` implementation for Substrate's trie format.
use crate::{
node_codec::Bitmap,
node_header::{size_and_prefix_iterator, NodeKind},
trie_constants,
};
use codec::Encode;
use codec::{Compact, Encode};
use hash_db::Hasher;
use sp_std::vec::Vec;
use trie_root;
const BRANCH_NODE_NO_VALUE: u8 = 254;
const BRANCH_NODE_WITH_VALUE: u8 = 255;
#[derive(Default, Clone)]
/// Codec-flavored TrieStream.
pub struct TrieStream {
/// Current node buffer.
buffer: Vec<u8>,
}
@@ -60,51 +57,76 @@ fn fuse_nibbles_node<'a>(nibbles: &'a [u8], kind: NodeKind) -> impl Iterator<Ite
let size = sp_std::cmp::min(trie_constants::NIBBLE_SIZE_BOUND, nibbles.len());
let iter_start = match kind {
NodeKind::Leaf => size_and_prefix_iterator(size, trie_constants::LEAF_PREFIX_MASK),
NodeKind::Leaf => size_and_prefix_iterator(size, trie_constants::LEAF_PREFIX_MASK, 2),
NodeKind::BranchNoValue =>
size_and_prefix_iterator(size, trie_constants::BRANCH_WITHOUT_MASK),
size_and_prefix_iterator(size, trie_constants::BRANCH_WITHOUT_MASK, 2),
NodeKind::BranchWithValue =>
size_and_prefix_iterator(size, trie_constants::BRANCH_WITH_MASK),
size_and_prefix_iterator(size, trie_constants::BRANCH_WITH_MASK, 2),
NodeKind::HashedValueLeaf =>
size_and_prefix_iterator(size, trie_constants::ALT_HASHING_LEAF_PREFIX_MASK, 3),
NodeKind::HashedValueBranch =>
size_and_prefix_iterator(size, trie_constants::ALT_HASHING_BRANCH_WITH_MASK, 4),
};
iter_start
.chain(if nibbles.len() % 2 == 1 { Some(nibbles[0]) } else { None })
.chain(nibbles[nibbles.len() % 2..].chunks(2).map(|ch| ch[0] << 4 | ch[1]))
}
use trie_root::Value as TrieStreamValue;
impl trie_root::TrieStream for TrieStream {
fn new() -> Self {
TrieStream { buffer: Vec::new() }
Self { buffer: Vec::new() }
}
fn append_empty_data(&mut self) {
self.buffer.push(trie_constants::EMPTY_TRIE);
}
fn append_leaf(&mut self, key: &[u8], value: &[u8]) {
self.buffer.extend(fuse_nibbles_node(key, NodeKind::Leaf));
value.encode_to(&mut self.buffer);
fn append_leaf(&mut self, key: &[u8], value: TrieStreamValue) {
let kind = match &value {
TrieStreamValue::Inline(..) => NodeKind::Leaf,
TrieStreamValue::Node(..) => NodeKind::HashedValueLeaf,
};
self.buffer.extend(fuse_nibbles_node(key, kind));
match &value {
TrieStreamValue::Inline(value) => {
Compact(value.len() as u32).encode_to(&mut self.buffer);
self.buffer.extend_from_slice(value);
},
TrieStreamValue::Node(hash) => {
self.buffer.extend_from_slice(hash.as_slice());
},
};
}
fn begin_branch(
&mut self,
maybe_partial: Option<&[u8]>,
maybe_value: Option<&[u8]>,
maybe_value: Option<TrieStreamValue>,
has_children: impl Iterator<Item = bool>,
) {
if let Some(partial) = maybe_partial {
if maybe_value.is_some() {
self.buffer.extend(fuse_nibbles_node(partial, NodeKind::BranchWithValue));
} else {
self.buffer.extend(fuse_nibbles_node(partial, NodeKind::BranchNoValue));
}
let kind = match &maybe_value {
None => NodeKind::BranchNoValue,
Some(TrieStreamValue::Inline(..)) => NodeKind::BranchWithValue,
Some(TrieStreamValue::Node(..)) => NodeKind::HashedValueBranch,
};
self.buffer.extend(fuse_nibbles_node(partial, kind));
let bm = branch_node_bit_mask(has_children);
self.buffer.extend([bm.0, bm.1].iter());
} else {
debug_assert!(false, "trie stream codec only for no extension trie");
self.buffer.extend(&branch_node(maybe_value.is_some(), has_children));
unreachable!("trie stream codec only for no extension trie");
}
if let Some(value) = maybe_value {
value.encode_to(&mut self.buffer);
match maybe_value {
None => (),
Some(TrieStreamValue::Inline(value)) => {
Compact(value.len() as u32).encode_to(&mut self.buffer);
self.buffer.extend_from_slice(value);
},
Some(TrieStreamValue::Node(hash)) => {
self.buffer.extend_from_slice(hash.as_slice());
},
}
}
@@ -124,18 +146,3 @@ impl trie_root::TrieStream for TrieStream {
self.buffer
}
}
fn branch_node(has_value: bool, has_children: impl Iterator<Item = bool>) -> [u8; 3] {
let mut result = [0, 0, 0];
branch_node_buffered(has_value, has_children, &mut result[..]);
result
}
fn branch_node_buffered<I>(has_value: bool, has_children: I, output: &mut [u8])
where
I: Iterator<Item = bool>,
{
let first = if has_value { BRANCH_NODE_WITH_VALUE } else { BRANCH_NODE_NO_VALUE };
output[0] = first;
Bitmap::encode(has_children, &mut output[1..]);
}
substrate/primitives/version/Cargo.toml
+1
View File
@@ -22,6 +22,7 @@ sp-std = { version = "4.0.0", default-features = false, path = "../std" }
sp-runtime = { version = "4.0.0", default-features = false, path = "../runtime" }
sp-version-proc-macro = { version = "4.0.0-dev", default-features = false, path = "proc-macro" }
parity-wasm = { version = "0.42.2", optional = true }
sp-core-hashing-proc-macro = { version = "4.0.0-dev", path = "../core/hashing/proc-macro" }
thiserror = { version = "1.0.30", optional = true }
[features]
substrate/primitives/version/proc-macro/src/decl_runtime_version.rs
+22 -2
View File
@@ -63,6 +63,7 @@ struct RuntimeVersion {
impl_version: u32,
apis: u8,
transaction_version: u32,
state_version: u8,
}
#[derive(Default, Debug)]
@@ -73,6 +74,7 @@ struct ParseRuntimeVersion {
spec_version: Option<u32>,
impl_version: Option<u32>,
transaction_version: Option<u32>,
state_version: Option<u8>,
}
impl ParseRuntimeVersion {
@@ -122,6 +124,8 @@ impl ParseRuntimeVersion {
parse_once(&mut self.impl_version, field_value, Self::parse_num_literal)?;
} else if field_name == "transaction_version" {
parse_once(&mut self.transaction_version, field_value, Self::parse_num_literal)?;
} else if field_name == "state_version" {
parse_once(&mut self.state_version, field_value, Self::parse_num_literal_u8)?;
} else if field_name == "apis" {
// Intentionally ignored
//
@@ -147,6 +151,18 @@ impl ParseRuntimeVersion {
lit.base10_parse::<u32>()
}
fn parse_num_literal_u8(expr: &Expr) -> Result<u8> {
let lit = match *expr {
Expr::Lit(ExprLit { lit: Lit::Int(ref lit), .. }) => lit,
_ =>
return Err(Error::new(
expr.span(),
"only numeric literals (e.g. `10`) are supported here",
)),
};
lit.base10_parse::<u8>()
}
fn parse_str_literal(expr: &Expr) -> Result<String> {
let mac = match *expr {
Expr::Macro(syn::ExprMacro { ref mac, .. }) => mac,
@@ -182,6 +198,7 @@ impl ParseRuntimeVersion {
spec_version,
impl_version,
transaction_version,
state_version,
} = self;
Ok(RuntimeVersion {
@@ -191,6 +208,7 @@ impl ParseRuntimeVersion {
spec_version: required!(spec_version),
impl_version: required!(impl_version),
transaction_version: required!(transaction_version),
state_version: required!(state_version),
apis: 0,
})
}
@@ -210,7 +228,6 @@ fn generate_emit_link_section_decl(contents: &[u8], section_name: &str) -> Token
#[cfg(test)]
mod tests {
use super::*;
use codec::DecodeAll;
use std::borrow::Cow;
#[test]
@@ -223,11 +240,13 @@ mod tests {
impl_version: 1,
apis: 0,
transaction_version: 2,
state_version: 1,
}
.encode();
assert_eq!(
sp_version::RuntimeVersion::decode_all(&mut &version_bytes[..]).unwrap(),
sp_version::RuntimeVersion::decode_with_version_hint(&mut &version_bytes[..], Some(4))
.unwrap(),
sp_version::RuntimeVersion {
spec_name: "hello".into(),
impl_name: "world".into(),
@@ -236,6 +255,7 @@ mod tests {
impl_version: 1,
apis: Cow::Owned(vec![]),
transaction_version: 2,
state_version: 1,
},
);
}
substrate/primitives/version/src/lib.rs
+74 -4
View File
@@ -40,10 +40,10 @@ use std::collections::HashSet;
#[cfg(feature = "std")]
use std::fmt;
use codec::{Decode, Encode};
use codec::{Decode, Encode, Input};
use scale_info::TypeInfo;
pub use sp_runtime::create_runtime_str;
use sp_runtime::RuntimeString;
pub use sp_runtime::{create_runtime_str, StateVersion};
#[doc(hidden)]
pub use sp_std;
@@ -79,6 +79,7 @@ pub mod embed;
/// impl_version: 1,
/// apis: RUNTIME_API_VERSIONS,
/// transaction_version: 2,
/// state_version: 1,
/// };
///
/// # const RUNTIME_API_VERSIONS: sp_version::ApisVec = sp_version::create_apis_vec!([]);
@@ -154,7 +155,7 @@ macro_rules! create_apis_vec {
/// In particular: bug fixes should result in an increment of `spec_version` and possibly
/// `authoring_version`, absolutely not `impl_version` since they change the semantics of the
/// runtime.
#[derive(Clone, PartialEq, Eq, Encode, Decode, Default, sp_runtime::RuntimeDebug, TypeInfo)]
#[derive(Clone, PartialEq, Eq, Encode, Default, sp_runtime::RuntimeDebug, TypeInfo)]
#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "std", serde(rename_all = "camelCase"))]
pub struct RuntimeVersion {
@@ -207,6 +208,53 @@ pub struct RuntimeVersion {
///
/// It need *not* change when a new module is added or when a dispatchable is added.
pub transaction_version: u32,
/// Version of the state implementation used by this runtime.
/// Use of an incorrect version is consensus breaking.
pub state_version: u8,
}
impl RuntimeVersion {
/// `Decode` while giving a "version hint"
///
/// There exists multiple versions of [`RuntimeVersion`] and they are versioned using the `Core`
/// runtime api:
/// - `Core` version < 3 is a runtime version without a transaction version and state version.
/// - `Core` version 3 is a runtime version without a state version.
/// - `Core` version 4 is the latest runtime version.
pub fn decode_with_version_hint<I: Input>(
input: &mut I,
core_version: Option<u32>,
) -> Result<RuntimeVersion, codec::Error> {
let spec_name = Decode::decode(input)?;
let impl_name = Decode::decode(input)?;
let authoring_version = Decode::decode(input)?;
let spec_version = Decode::decode(input)?;
let impl_version = Decode::decode(input)?;
let apis = Decode::decode(input)?;
let core_version =
if core_version.is_some() { core_version } else { core_version_from_apis(&apis) };
let transaction_version =
if core_version.map(|v| v >= 3).unwrap_or(false) { Decode::decode(input)? } else { 1 };
let state_version =
if core_version.map(|v| v >= 4).unwrap_or(false) { Decode::decode(input)? } else { 0 };
Ok(RuntimeVersion {
spec_name,
impl_name,
authoring_version,
spec_version,
impl_version,
apis,
transaction_version,
state_version,
})
}
}
impl Decode for RuntimeVersion {
fn decode<I: Input>(input: &mut I) -> Result<Self, codec::Error> {
Self::decode_with_version_hint(input, None)
}
}
#[cfg(feature = "std")]
@@ -225,6 +273,16 @@ impl fmt::Display for RuntimeVersion {
}
}
fn has_api_with<P: Fn(u32) -> bool>(apis: &ApisVec, id: &ApiId, predicate: P) -> bool {
apis.iter().any(|(s, v)| s == id && predicate(*v))
}
/// Returns the version of the `Core` runtime api.
pub fn core_version_from_apis(apis: &ApisVec) -> Option<u32> {
let id = sp_core_hashing_proc_macro::blake2b_64!(b"Core");
apis.iter().find(|(s, _v)| s == &id).map(|(_s, v)| *v)
}
#[cfg(feature = "std")]
impl RuntimeVersion {
/// Check if this version matches other version for calling into runtime.
@@ -237,7 +295,7 @@ impl RuntimeVersion {
/// Check if the given api with `api_id` is implemented and the version passes the given
/// `predicate`.
pub fn has_api_with<P: Fn(u32) -> bool>(&self, id: &ApiId, predicate: P) -> bool {
self.apis.iter().any(|(s, v)| s == id && predicate(*v))
has_api_with(&self.apis, id, predicate)
}
/// Returns the api version found for api with `id`.
@@ -246,6 +304,18 @@ impl RuntimeVersion {
}
}
impl RuntimeVersion {
/// Returns state version to use for update.
///
/// For runtime with core api version less than 4,
/// V0 trie version will be applied to state.
/// Otherwhise, V1 trie version will be use.
pub fn state_version(&self) -> StateVersion {
// If version > than 1, keep using latest version.
self.state_version.try_into().unwrap_or(StateVersion::V1)
}
}
/// The version of the native runtime.
///
/// In contrast to the bare [`RuntimeVersion`] this also carries a list of `spec_version`s of