pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-06-14 16:51:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Inner hashing of value in state trie (runtime versioning). (#9732)

Browse Source 
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
This commit is contained in:
cheme
2021-12-24 09:54:07 +01:00
committed by GitHub
parent ea30c739ea
commit 4c651637f2
78 changed files with 1814 additions and 782 deletions
Download Patch File Download Diff File Expand all files Collapse all files
substrate/primitives/state-machine/src/lib.rs
+128 -31
View File
@@ -126,6 +126,7 @@ impl sp_std::fmt::Display for DefaultError {
pub use crate::{
backend::Backend,
error::{Error, ExecutionError},
ext::Ext,
overlayed_changes::{
ChildStorageCollection, IndexOperation, OffchainChangesCollection,
@@ -136,7 +137,6 @@ pub use crate::{
trie_backend::TrieBackend,
trie_backend_essence::{Storage, TrieBackendStorage},
};
pub use error::{Error, ExecutionError};
#[cfg(feature = "std")]
mod std_reexport {
@@ -151,8 +151,8 @@ mod std_reexport {
testing::TestExternalities,
};
pub use sp_trie::{
trie_types::{Layout, TrieDBMut},
CompactProof, DBValue, MemoryDB, StorageProof, TrieMut,
trie_types::{TrieDBMutV0, TrieDBMutV1},
CompactProof, DBValue, LayoutV0, LayoutV1, MemoryDB, StorageProof, TrieMut,
};
}
@@ -1353,7 +1353,7 @@ mod tests {
use codec::{Decode, Encode};
use sp_core::{
map,
storage::ChildInfo,
storage::{ChildInfo, StateVersion},
testing::TaskExecutor,
traits::{CodeExecutor, Externalities, RuntimeCode},
NativeOrEncoded, NeverNativeValue,
@@ -1416,7 +1416,11 @@ mod tests {
#[test]
fn execute_works() {
let backend = trie_backend::tests::test_trie();
execute_works_inner(StateVersion::V0);
execute_works_inner(StateVersion::V1);
}
fn execute_works_inner(state_version: StateVersion) {
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1440,7 +1444,11 @@ mod tests {
#[test]
fn execute_works_with_native_else_wasm() {
let backend = trie_backend::tests::test_trie();
execute_works_with_native_else_wasm_inner(StateVersion::V0);
execute_works_with_native_else_wasm_inner(StateVersion::V1);
}
fn execute_works_with_native_else_wasm_inner(state_version: StateVersion) {
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1464,8 +1472,12 @@ mod tests {
#[test]
fn dual_execution_strategy_detects_consensus_failure() {
dual_execution_strategy_detects_consensus_failure_inner(StateVersion::V0);
dual_execution_strategy_detects_consensus_failure_inner(StateVersion::V1);
}
fn dual_execution_strategy_detects_consensus_failure_inner(state_version: StateVersion) {
let mut consensus_failed = false;
let backend = trie_backend::tests::test_trie();
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();
@@ -1498,6 +1510,10 @@ mod tests {
#[test]
fn prove_execution_and_proof_check_works() {
prove_execution_and_proof_check_works_inner(StateVersion::V0);
prove_execution_and_proof_check_works_inner(StateVersion::V1);
}
fn prove_execution_and_proof_check_works_inner(state_version: StateVersion) {
let executor = DummyCodeExecutor {
native_available: true,
native_succeeds: true,
@@ -1505,8 +1521,8 @@ mod tests {
};
// fetch execution proof from 'remote' full node
let mut remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let mut remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let (remote_result, remote_proof) = prove_execution(
&mut remote_backend,
&mut Default::default(),
@@ -1544,7 +1560,7 @@ mod tests {
b"abc".to_vec() => b"2".to_vec(),
b"bbb".to_vec() => b"3".to_vec()
];
let state = InMemoryBackend::<BlakeTwo256>::from(initial);
let state = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let backend = state.as_trie_backend().unwrap();
let mut overlay = OverlayedChanges::default();
@@ -1613,7 +1629,7 @@ mod tests {
b"d".to_vec() => b"3".to_vec()
],
];
let backend = InMemoryBackend::<BlakeTwo256>::from(initial);
let backend = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let mut overlay = OverlayedChanges::default();
overlay.set_child_storage(&child_info, b"1".to_vec(), Some(b"1312".to_vec()));
@@ -1655,7 +1671,7 @@ mod tests {
b"d".to_vec() => b"3".to_vec()
],
];
let backend = InMemoryBackend::<BlakeTwo256>::from(initial);
let backend = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let mut overlay = OverlayedChanges::default();
let mut cache = StorageTransactionCache::default();
let mut ext = Ext::new(&mut overlay, &mut cache, &backend, None);
@@ -1789,13 +1805,17 @@ mod tests {
#[test]
fn prove_read_and_proof_check_works() {
prove_read_and_proof_check_works_inner(StateVersion::V0);
prove_read_and_proof_check_works_inner(StateVersion::V1);
}
fn prove_read_and_proof_check_works_inner(state_version: StateVersion) {
let child_info = ChildInfo::new_default(b"sub1");
let missing_child_info = ChildInfo::new_default(b"sub1sub2"); // key will include other child root to proof.
let child_info = &child_info;
let missing_child_info = &missing_child_info;
// fetch read proof from 'remote' full node
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_read(remote_backend, &[b"value2"]).unwrap();
let remote_proof = test_compact(remote_proof, &remote_root);
// check proof locally
@@ -1812,8 +1832,8 @@ mod tests {
);
assert_eq!(local_result2, false);
// on child trie
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(std::iter::empty()).0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_child_read(remote_backend, child_info, &[b"value3"]).unwrap();
let remote_proof = test_compact(remote_proof, &remote_root);
let local_result1 = read_child_proof_check::<BlakeTwo256, _>(
@@ -1877,7 +1897,8 @@ mod tests {
storage.insert(Some(child_info), items);
}
let trie: InMemoryBackend<BlakeTwo256> = storage.clone().into();
let trie: InMemoryBackend<BlakeTwo256> =
(storage.clone(), StateVersion::default()).into();
let trie_root = trie.root().clone();
let backend = crate::ProvingBackend::new(&trie);
let mut queries = Vec::new();
@@ -1940,15 +1961,16 @@ mod tests {
#[test]
fn prove_read_with_size_limit_works() {
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(::std::iter::empty()).0;
let state_version = StateVersion::V0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(::std::iter::empty(), state_version).0;
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 0, None).unwrap();
// Always contains at least some nodes.
assert_eq!(proof.into_memory_db::<BlakeTwo256>().drain().len(), 3);
assert_eq!(count, 1);
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 800, Some(&[])).unwrap();
assert_eq!(proof.clone().into_memory_db::<BlakeTwo256>().drain().len(), 9);
@@ -1971,7 +1993,7 @@ mod tests {
assert_eq!(results.len() as u32, 101);
assert_eq!(completed, false);
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let (proof, count) =
prove_range_read_with_size(remote_backend, None, None, 50000, Some(&[])).unwrap();
assert_eq!(proof.clone().into_memory_db::<BlakeTwo256>().drain().len(), 11);
@@ -1989,10 +2011,66 @@ mod tests {
assert_eq!(completed, true);
}
#[test]
fn inner_state_versioning_switch_proofs() {
let mut state_version = StateVersion::V0;
let (mut mdb, mut root) = trie_backend::tests::test_db(state_version);
{
let mut trie = TrieDBMutV0::from_existing(&mut mdb, &mut root).unwrap();
trie.insert(b"foo", vec![1u8; 1_000].as_slice()) // big inner hash
.expect("insert failed");
trie.insert(b"foo2", vec![3u8; 16].as_slice()) // no inner hash
.expect("insert failed");
trie.insert(b"foo222", vec![5u8; 100].as_slice()) // inner hash
.expect("insert failed");
}
let check_proof = |mdb, root, state_version| -> StorageProof {
let remote_backend = TrieBackend::new(mdb, root);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let remote_proof = prove_read(remote_backend, &[b"foo222"]).unwrap();
// check proof locally
let local_result1 =
read_proof_check::<BlakeTwo256, _>(remote_root, remote_proof.clone(), &[b"foo222"])
.unwrap();
// check that results are correct
assert_eq!(
local_result1.into_iter().collect::<Vec<_>>(),
vec![(b"foo222".to_vec(), Some(vec![5u8; 100]))],
);
remote_proof
};
let remote_proof = check_proof(mdb.clone(), root.clone(), state_version);
// check full values in proof
assert!(remote_proof.encode().len() > 1_100);
assert!(remote_proof.encoded_size() > 1_100);
let root1 = root.clone();
// do switch
state_version = StateVersion::V1;
{
let mut trie = TrieDBMutV1::from_existing(&mut mdb, &mut root).unwrap();
trie.insert(b"foo222", vec![5u8; 100].as_slice()) // inner hash
.expect("insert failed");
// update with same value do change
trie.insert(b"foo", vec![1u8; 1000].as_slice()) // inner hash
.expect("insert failed");
}
let root3 = root.clone();
assert!(root1 != root3);
let remote_proof = check_proof(mdb.clone(), root.clone(), state_version);
// nodes foo is replaced by its hashed value form.
assert!(remote_proof.encode().len() < 1000);
assert!(remote_proof.encoded_size() < 1000);
assert_eq!(remote_proof.encode().len(), remote_proof.encoded_size());
}
#[test]
fn prove_range_with_child_works() {
let remote_backend = trie_backend::tests::test_trie();
let remote_root = remote_backend.storage_root(::std::iter::empty()).0;
let state_version = StateVersion::V0;
let remote_backend = trie_backend::tests::test_trie(state_version);
let remote_root = remote_backend.storage_root(std::iter::empty(), state_version).0;
let mut start_at = smallvec::SmallVec::<[Vec<u8>; 2]>::new();
let trie_backend = remote_backend.as_trie_backend().unwrap();
let max_iter = 1000;
@@ -2030,20 +2108,34 @@ mod tests {
#[test]
fn compact_multiple_child_trie() {
let size_no_inner_hash = compact_multiple_child_trie_inner(StateVersion::V0);
let size_inner_hash = compact_multiple_child_trie_inner(StateVersion::V1);
assert!(size_inner_hash < size_no_inner_hash);
}
fn compact_multiple_child_trie_inner(state_version: StateVersion) -> usize {
// this root will be queried
let child_info1 = ChildInfo::new_default(b"sub1");
// this root will not be include in proof
let child_info2 = ChildInfo::new_default(b"sub2");
// this root will be include in proof
let child_info3 = ChildInfo::new_default(b"sub");
let remote_backend = trie_backend::tests::test_trie();
let remote_backend = trie_backend::tests::test_trie(state_version);
let long_vec: Vec<u8> = (0..1024usize).map(|_| 8u8).collect();
let (remote_root, transaction) = remote_backend.full_storage_root(
std::iter::empty(),
vec![
(
&child_info1,
vec![(&b"key1"[..], Some(&b"val2"[..])), (&b"key2"[..], Some(&b"val3"[..]))]
.into_iter(),
vec![
// a inner hashable node
(&b"k"[..], Some(&long_vec[..])),
// need to ensure this is not an inline node
// otherwhise we do not know what is accessed when
// storing proof.
(&b"key1"[..], Some(&vec![5u8; 32][..])),
(&b"key2"[..], Some(&b"val3"[..])),
]
.into_iter(),
),
(
&child_info2,
@@ -2057,11 +2149,13 @@ mod tests {
),
]
.into_iter(),
state_version,
);
let mut remote_storage = remote_backend.into_storage();
remote_storage.consolidate(transaction);
let remote_backend = TrieBackend::new(remote_storage, remote_root);
let remote_proof = prove_child_read(remote_backend, &child_info1, &[b"key1"]).unwrap();
let size = remote_proof.encoded_size();
let remote_proof = test_compact(remote_proof, &remote_root);
let local_result1 = read_child_proof_check::<BlakeTwo256, _>(
remote_root,
@@ -2071,11 +2165,13 @@ mod tests {
)
.unwrap();
assert_eq!(local_result1.len(), 1);
assert_eq!(local_result1.get(&b"key1"[..]), Some(&Some(b"val2".to_vec())));
assert_eq!(local_result1.get(&b"key1"[..]), Some(&Some(vec![5u8; 32])));
size
}
#[test]
fn child_storage_uuid() {
let state_version = StateVersion::V0;
let child_info_1 = ChildInfo::new_default(b"sub_test1");
let child_info_2 = ChildInfo::new_default(b"sub_test2");
@@ -2083,12 +2179,12 @@ mod tests {
let mut overlay = OverlayedChanges::default();
let mut transaction = {
let backend = test_trie();
let backend = test_trie(state_version);
let mut cache = StorageTransactionCache::default();
let mut ext = Ext::new(&mut overlay, &mut cache, &backend, None);
ext.set_child_storage(&child_info_1, b"abc".to_vec(), b"def".to_vec());
ext.set_child_storage(&child_info_2, b"abc".to_vec(), b"def".to_vec());
ext.storage_root();
ext.storage_root(state_version);
cache.transaction.unwrap()
};
let mut duplicate = false;
@@ -2108,7 +2204,7 @@ mod tests {
b"aaa".to_vec() => b"0".to_vec(),
b"bbb".to_vec() => b"".to_vec()
];
let state = InMemoryBackend::<BlakeTwo256>::from(initial);
let state = InMemoryBackend::<BlakeTwo256>::from((initial, StateVersion::default()));
let backend = state.as_trie_backend().unwrap();
let mut overlay = OverlayedChanges::default();
@@ -2134,12 +2230,13 @@ mod tests {
#[test]
fn runtime_registered_extensions_are_removed_after_execution() {
let state_version = StateVersion::default();
use sp_externalities::ExternalitiesExt;
sp_externalities::decl_extension! {
struct DummyExt(u32);
}
let backend = trie_backend::tests::test_trie();
let backend = trie_backend::tests::test_trie(state_version);
let mut overlayed_changes = Default::default();
let wasm_code = RuntimeCode::empty();