Inner hashing of value in state trie (runtime versioning). (#9732)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
This commit is contained in:
cheme
2021-12-24 09:54:07 +01:00
committed by GitHub
parent ea30c739ea
commit 4c651637f2
78 changed files with 1814 additions and 782 deletions
+7 -2
View File
@@ -29,7 +29,7 @@ use sp_core::offchain::OffchainStorage;
use sp_runtime::{
generic::BlockId,
traits::{Block as BlockT, HashFor, NumberFor},
Justification, Justifications, Storage,
Justification, Justifications, StateVersion, Storage,
};
use sp_state_machine::{
ChildStorageCollection, IndexOperation, OffchainChangesCollection, StorageCollection,
@@ -166,10 +166,15 @@ pub trait BlockImportOperation<Block: BlockT> {
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash>;
/// Inject storage data into the database replacing any existing data.
fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash>;
fn reset_storage(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash>;
/// Set storage changes.
fn update_storage(
+2 -2
View File
@@ -19,7 +19,7 @@
//! A method call executor interface.
use codec::{Decode, Encode};
use sc_executor::RuntimeVersion;
use sc_executor::{RuntimeVersion, RuntimeVersionOf};
use sp_core::NativeOrEncoded;
use sp_externalities::Extensions;
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
@@ -42,7 +42,7 @@ pub trait ExecutorProvider<Block: BlockT> {
}
/// Method call executor.
pub trait CallExecutor<B: BlockT> {
pub trait CallExecutor<B: BlockT>: RuntimeVersionOf {
/// Externalities error type.
type Error: sp_state_machine::Error;
+11 -4
View File
@@ -26,7 +26,7 @@ use sp_core::{
use sp_runtime::{
generic::BlockId,
traits::{Block as BlockT, HashFor, Header as HeaderT, NumberFor, Zero},
Justification, Justifications, Storage,
Justification, Justifications, StateVersion, Storage,
};
use sp_state_machine::{
Backend as StateBackend, ChildStorageCollection, InMemoryBackend, IndexOperation,
@@ -506,6 +506,7 @@ where
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
check_genesis_storage(&storage)?;
@@ -519,6 +520,7 @@ where
let (root, transaction) = self.old_state.full_storage_root(
storage.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
child_delta,
state_version,
);
if commit {
@@ -566,12 +568,17 @@ where
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, commit)
self.apply_storage(storage, commit, state_version)
}
fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, true)
fn reset_storage(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, true, state_version)
}
fn insert_aux<I>(&mut self, ops: I) -> sp_blockchain::Result<()>
@@ -232,6 +232,7 @@ where
header.extrinsics_root().clone(),
HashFor::<Block>::ordered_trie_root(
self.extrinsics.iter().map(Encode::encode).collect(),
sp_runtime::StateVersion::V0,
),
);
+9 -4
View File
@@ -34,7 +34,7 @@ use sp_core::{
};
use sp_runtime::{
traits::{Block as BlockT, HashFor},
Storage,
StateVersion, Storage,
};
use sp_state_machine::{
backend::Backend as StateBackend, ChildStorageCollection, DBValue, ProofRecorder,
@@ -73,6 +73,7 @@ impl<Block: BlockT> sp_state_machine::Storage<HashFor<Block>> for StorageDb<Bloc
}
}
}
/// State that manages the backend database reference. Allows runtime to control the database.
pub struct BenchmarkingState<B: BlockT> {
root: Cell<B::Hash>,
@@ -105,9 +106,10 @@ impl<B: BlockT> BenchmarkingState<B> {
record_proof: bool,
enable_tracking: bool,
) -> Result<Self, String> {
let state_version = sp_runtime::StateVersion::default();
let mut root = B::Hash::default();
let mut mdb = MemoryDB::<HashFor<B>>::default();
sp_state_machine::TrieDBMut::<HashFor<B>>::new(&mut mdb, &mut root);
sp_state_machine::TrieDBMutV1::<HashFor<B>>::new(&mut mdb, &mut root);
let mut state = BenchmarkingState {
state: RefCell::new(None),
@@ -138,6 +140,7 @@ impl<B: BlockT> BenchmarkingState<B> {
state.state.borrow_mut().as_mut().unwrap().full_storage_root(
genesis.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
child_delta,
state_version,
);
state.genesis = transaction.clone().drain();
state.genesis_root = root.clone();
@@ -415,6 +418,7 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, Self::Transaction)
where
B::Hash: Ord,
@@ -422,13 +426,14 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
self.state
.borrow()
.as_ref()
.map_or(Default::default(), |s| s.storage_root(delta))
.map_or(Default::default(), |s| s.storage_root(delta, state_version))
}
fn child_storage_root<'a>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, bool, Self::Transaction)
where
B::Hash: Ord,
@@ -436,7 +441,7 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
self.state
.borrow()
.as_ref()
.map_or(Default::default(), |s| s.child_storage_root(child_info, delta))
.map_or(Default::default(), |s| s.child_storage_root(child_info, delta, state_version))
}
fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {
+70 -38
View File
@@ -82,7 +82,7 @@ use sp_runtime::{
Block as BlockT, Hash, HashFor, Header as HeaderT, NumberFor, One, SaturatedConversion,
Zero,
},
Justification, Justifications, Storage,
Justification, Justifications, StateVersion, Storage,
};
use sp_state_machine::{
backend::Backend as StateBackend, ChildStorageCollection, DBValue, IndexOperation,
@@ -235,22 +235,24 @@ impl<B: BlockT> StateBackend<HashFor<B>> for RefTrackingState<B> {
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, Self::Transaction)
where
B::Hash: Ord,
{
self.state.storage_root(delta)
self.state.storage_root(delta, state_version)
}
fn child_storage_root<'a>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, bool, Self::Transaction)
where
B::Hash: Ord,
{
self.state.child_storage_root(child_info, delta)
self.state.child_storage_root(child_info, delta, state_version)
}
fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {
@@ -760,7 +762,11 @@ impl<Block: BlockT> BlockImportOperation<Block> {
}
}
fn apply_new_state(&mut self, storage: Storage) -> ClientResult<Block::Hash> {
fn apply_new_state(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> ClientResult<Block::Hash> {
if storage.top.keys().any(|k| well_known_keys::is_child_storage_key(&k)) {
return Err(sp_blockchain::Error::InvalidState.into())
}
@@ -775,6 +781,7 @@ impl<Block: BlockT> BlockImportOperation<Block> {
let (root, transaction) = self.old_state.full_storage_root(
storage.top.iter().map(|(k, v)| (&k[..], Some(&v[..]))),
child_delta,
state_version,
);
self.db_updates = transaction;
@@ -814,14 +821,23 @@ impl<Block: BlockT> sc_client_api::backend::BlockImportOperation<Block>
Ok(())
}
fn reset_storage(&mut self, storage: Storage) -> ClientResult<Block::Hash> {
let root = self.apply_new_state(storage)?;
fn reset_storage(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> ClientResult<Block::Hash> {
let root = self.apply_new_state(storage, state_version)?;
self.commit_state = true;
Ok(root)
}
fn set_genesis_state(&mut self, storage: Storage, commit: bool) -> ClientResult<Block::Hash> {
let root = self.apply_new_state(storage)?;
fn set_genesis_state(
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> ClientResult<Block::Hash> {
let root = self.apply_new_state(storage, state_version)?;
self.commit_state = commit;
Ok(root)
}
@@ -924,7 +940,8 @@ impl<Block: BlockT> EmptyStorage<Block> {
pub fn new() -> Self {
let mut root = Block::Hash::default();
let mut mdb = MemoryDB::<HashFor<Block>>::default();
sp_state_machine::TrieDBMut::<HashFor<Block>>::new(&mut mdb, &mut root);
// both triedbmut are the same on empty storage.
sp_state_machine::TrieDBMutV1::<HashFor<Block>>::new(&mut mdb, &mut root);
EmptyStorage(root)
}
}
@@ -2262,7 +2279,7 @@ pub(crate) mod tests {
use sp_runtime::{
testing::{Block as RawBlock, ExtrinsicWrapper, Header},
traits::{BlakeTwo256, Hash},
ConsensusEngineId,
ConsensusEngineId, StateVersion,
};
const CONS0_ENGINE_ID: ConsensusEngineId = *b"CON0";
@@ -2295,7 +2312,7 @@ pub(crate) mod tests {
let header = Header {
number,
parent_hash,
state_root: BlakeTwo256::trie_root(Vec::new()),
state_root: BlakeTwo256::trie_root(Vec::new(), StateVersion::V1),
digest,
extrinsics_root,
};
@@ -2375,6 +2392,10 @@ pub(crate) mod tests {
#[test]
fn set_state_data() {
set_state_data_inner(StateVersion::V0);
set_state_data_inner(StateVersion::V1);
}
fn set_state_data_inner(state_version: StateVersion) {
let db = Backend::<Block>::new_test(2, 0);
let hash = {
let mut op = db.begin_operation().unwrap();
@@ -2390,15 +2411,18 @@ pub(crate) mod tests {
header.state_root = op
.old_state
.storage_root(storage.iter().map(|(x, y)| (&x[..], Some(&y[..]))))
.storage_root(storage.iter().map(|(x, y)| (&x[..], Some(&y[..]))), state_version)
.0
.into();
let hash = header.hash();
op.reset_storage(Storage {
top: storage.into_iter().collect(),
children_default: Default::default(),
})
op.reset_storage(
Storage {
top: storage.into_iter().collect(),
children_default: Default::default(),
},
state_version,
)
.unwrap();
op.set_block_data(header.clone(), Some(vec![]), None, None, NewBlockState::Best)
.unwrap();
@@ -2427,9 +2451,10 @@ pub(crate) mod tests {
let storage = vec![(vec![1, 3, 5], None), (vec![5, 5, 5], Some(vec![4, 5, 6]))];
let (root, overlay) = op
.old_state
.storage_root(storage.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..]))));
let (root, overlay) = op.old_state.storage_root(
storage.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..]))),
state_version,
);
op.update_db_storage(overlay).unwrap();
header.state_root = root.into();
@@ -2450,6 +2475,7 @@ pub(crate) mod tests {
#[test]
fn delete_only_when_negative_rc() {
sp_tracing::try_init_simple();
let state_version = StateVersion::default();
let key;
let backend = Backend::<Block>::new_test(1, 0);
@@ -2466,13 +2492,14 @@ pub(crate) mod tests {
extrinsics_root: Default::default(),
};
header.state_root = op.old_state.storage_root(std::iter::empty()).0.into();
header.state_root =
op.old_state.storage_root(std::iter::empty(), state_version).0.into();
let hash = header.hash();
op.reset_storage(Storage {
top: Default::default(),
children_default: Default::default(),
})
op.reset_storage(
Storage { top: Default::default(), children_default: Default::default() },
state_version,
)
.unwrap();
key = op.db_updates.insert(EMPTY_PREFIX, b"hello");
@@ -2506,7 +2533,7 @@ pub(crate) mod tests {
header.state_root = op
.old_state
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))))
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))), state_version)
.0
.into();
let hash = header.hash();
@@ -2543,7 +2570,7 @@ pub(crate) mod tests {
header.state_root = op
.old_state
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))))
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))), state_version)
.0
.into();
let hash = header.hash();
@@ -2577,7 +2604,7 @@ pub(crate) mod tests {
header.state_root = op
.old_state
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))))
.storage_root(storage.iter().cloned().map(|(x, y)| (x, Some(y))), state_version)
.0
.into();
@@ -2912,6 +2939,7 @@ pub(crate) mod tests {
#[test]
fn storage_hash_is_cached_correctly() {
let state_version = StateVersion::default();
let backend = Backend::<Block>::new_test(10, 10);
let hash0 = {
@@ -2931,15 +2959,18 @@ pub(crate) mod tests {
header.state_root = op
.old_state
.storage_root(storage.iter().map(|(x, y)| (&x[..], Some(&y[..]))))
.storage_root(storage.iter().map(|(x, y)| (&x[..], Some(&y[..]))), state_version)
.0
.into();
let hash = header.hash();
op.reset_storage(Storage {
top: storage.into_iter().collect(),
children_default: Default::default(),
})
op.reset_storage(
Storage {
top: storage.into_iter().collect(),
children_default: Default::default(),
},
state_version,
)
.unwrap();
op.set_block_data(header.clone(), Some(vec![]), None, None, NewBlockState::Best)
.unwrap();
@@ -2968,9 +2999,10 @@ pub(crate) mod tests {
let storage = vec![(b"test".to_vec(), Some(b"test2".to_vec()))];
let (root, overlay) = op
.old_state
.storage_root(storage.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..]))));
let (root, overlay) = op.old_state.storage_root(
storage.iter().map(|(k, v)| (&k[..], v.as_ref().map(|v| &v[..]))),
state_version,
);
op.update_db_storage(overlay).unwrap();
header.state_root = root.into();
let hash = header.hash();
@@ -3212,7 +3244,7 @@ pub(crate) mod tests {
let header = Header {
number: 1,
parent_hash: block0,
state_root: BlakeTwo256::trie_root(Vec::new()),
state_root: BlakeTwo256::trie_root(Vec::new(), StateVersion::V1),
digest: Default::default(),
extrinsics_root: Default::default(),
};
@@ -3224,7 +3256,7 @@ pub(crate) mod tests {
let header = Header {
number: 2,
parent_hash: block1,
state_root: BlakeTwo256::trie_root(Vec::new()),
state_root: BlakeTwo256::trie_root(Vec::new(), StateVersion::V1),
digest: Default::default(),
extrinsics_root: Default::default(),
};
@@ -3247,7 +3279,7 @@ pub(crate) mod tests {
let header = Header {
number: 1,
parent_hash: block0,
state_root: BlakeTwo256::trie_root(Vec::new()),
state_root: BlakeTwo256::trie_root(Vec::new(), StateVersion::V1),
digest: Default::default(),
extrinsics_root: Default::default(),
};
+16 -6
View File
@@ -26,7 +26,10 @@ use linked_hash_map::{Entry, LinkedHashMap};
use log::trace;
use parking_lot::{RwLock, RwLockUpgradableReadGuard};
use sp_core::{hexdisplay::HexDisplay, storage::ChildInfo};
use sp_runtime::traits::{Block as BlockT, HashFor, Header, NumberFor};
use sp_runtime::{
traits::{Block as BlockT, HashFor, Header, NumberFor},
StateVersion,
};
use sp_state_machine::{
backend::Backend as StateBackend, ChildStorageCollection, StorageCollection, StorageKey,
StorageValue, TrieBackend,
@@ -673,22 +676,24 @@ impl<S: StateBackend<HashFor<B>>, B: BlockT> StateBackend<HashFor<B>> for Cachin
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, Self::Transaction)
where
B::Hash: Ord,
{
self.state.storage_root(delta)
self.state.storage_root(delta, state_version)
}
fn child_storage_root<'a>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, bool, Self::Transaction)
where
B::Hash: Ord,
{
self.state.child_storage_root(child_info, delta)
self.state.child_storage_root(child_info, delta, state_version)
}
fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {
@@ -871,22 +876,24 @@ impl<S: StateBackend<HashFor<B>>, B: BlockT> StateBackend<HashFor<B>>
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, Self::Transaction)
where
B::Hash: Ord,
{
self.caching_state().storage_root(delta)
self.caching_state().storage_root(delta, state_version)
}
fn child_storage_root<'a>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, bool, Self::Transaction)
where
B::Hash: Ord,
{
self.caching_state().child_storage_root(child_info, delta)
self.caching_state().child_storage_root(child_info, delta, state_version)
}
fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {
@@ -1182,7 +1189,10 @@ mod tests {
let shared = new_shared_cache::<Block>(256 * 1024, (0, 1));
let mut backend = InMemoryBackend::<BlakeTwo256>::default();
backend.insert(std::iter::once((None, vec![(key.clone(), Some(vec![1]))])));
backend.insert(
std::iter::once((None, vec![(key.clone(), Some(vec![1]))])),
Default::default(),
);
let mut s = CachingState::new(backend.clone(), shared.clone(), Some(root_parent));
s.cache.sync_cache(
@@ -180,6 +180,7 @@ sp_core::wasm_export_functions! {
b"one"[..].into(),
b"two"[..].into(),
],
sp_core::storage::StateVersion::V1,
).as_ref().to_vec()
}
@@ -33,7 +33,7 @@ use sp_core::{
};
use sp_runtime::traits::BlakeTwo256;
use sp_state_machine::TestExternalities as CoreTestExternalities;
use sp_trie::{trie_types::Layout, TrieConfiguration};
use sp_trie::{LayoutV1 as Layout, TrieConfiguration};
use std::sync::Arc;
use tracing_subscriber::layer::SubscriberExt;
@@ -215,21 +215,22 @@ fn panicking_should_work(wasm_method: WasmExecutionMethod) {
test_wasm_execution!(storage_should_work);
fn storage_should_work(wasm_method: WasmExecutionMethod) {
let mut ext = TestExternalities::default();
// Test value must be bigger than 32 bytes
// to test the trie versioning.
let value = vec![7u8; 60];
{
let mut ext = ext.ext();
ext.set_storage(b"foo".to_vec(), b"bar".to_vec());
let output =
call_in_wasm("test_data_in", &b"Hello world".to_vec().encode(), wasm_method, &mut ext)
.unwrap();
let output = call_in_wasm("test_data_in", &value.encode(), wasm_method, &mut ext).unwrap();
assert_eq!(output, b"all ok!".to_vec().encode());
}
let expected = TestExternalities::new(sp_core::storage::Storage {
top: map![
b"input".to_vec() => b"Hello world".to_vec(),
b"input".to_vec() => value,
b"foo".to_vec() => b"bar".to_vec(),
b"baz".to_vec() => b"bar".to_vec()
],
+56 -30
View File
@@ -334,22 +334,11 @@ where
}
fn decode_version(mut version: &[u8]) -> Result<RuntimeVersion, WasmError> {
let v: RuntimeVersion = sp_api::OldRuntimeVersion::decode(&mut &version[..])
.map_err(|_| {
WasmError::Instantiation(
"failed to decode \"Core_version\" result using old runtime version".into(),
)
})?
.into();
let core_api_id = sp_core_hashing_proc_macro::blake2b_64!(b"Core");
if v.has_api_with(&core_api_id, |v| v >= 3) {
sp_api::RuntimeVersion::decode(&mut version).map_err(|_| {
WasmError::Instantiation("failed to decode \"Core_version\" result".into())
})
} else {
Ok(v)
}
Decode::decode(&mut version).map_err(|_| {
WasmError::Instantiation(
"failed to decode \"Core_version\" result using old runtime version".into(),
)
})
}
fn decode_runtime_apis(apis: &[u8]) -> Result<Vec<([u8; 8], u32)>, WasmError> {
@@ -373,17 +362,25 @@ fn decode_runtime_apis(apis: &[u8]) -> Result<Vec<([u8; 8], u32)>, WasmError> {
/// sections, `Err` will be returned.
pub fn read_embedded_version(blob: &RuntimeBlob) -> Result<Option<RuntimeVersion>, WasmError> {
if let Some(mut version_section) = blob.custom_section_contents("runtime_version") {
// We do not use `decode_version` here because the runtime_version section is not supposed
// to ever contain a legacy version. Apart from that `decode_version` relies on presence
// of a special API in the `apis` field to treat the input as a non-legacy version. However
// the structure found in the `runtime_version` always contain an empty `apis` field.
// Therefore the version read will be mistakenly treated as an legacy one.
let mut decoded_version = sp_api::RuntimeVersion::decode(&mut version_section)
.map_err(|_| WasmError::Instantiation("failed to decode version section".into()))?;
let apis = blob
.custom_section_contents("runtime_apis")
.map(decode_runtime_apis)
.transpose()?
.map(Into::into);
// Don't stop on this and check if there is a special section that encodes all runtime APIs.
if let Some(apis_section) = blob.custom_section_contents("runtime_apis") {
decoded_version.apis = decode_runtime_apis(apis_section)?.into();
let core_version = apis.as_ref().and_then(|apis| sp_version::core_version_from_apis(apis));
// We do not use `RuntimeVersion::decode` here because that `decode_version` relies on
// presence of a special API in the `apis` field to treat the input as a non-legacy version.
// However the structure found in the `runtime_version` always contain an empty `apis`
// field. Therefore the version read will be mistakenly treated as an legacy one.
let mut decoded_version = sp_version::RuntimeVersion::decode_with_version_hint(
&mut version_section,
core_version,
)
.map_err(|_| WasmError::Instantiation("failed to decode version section".into()))?;
if let Some(apis) = apis {
decoded_version.apis = apis;
}
Ok(Some(decoded_version))
@@ -455,9 +452,20 @@ mod tests {
use super::*;
use codec::Encode;
use sp_api::{Core, RuntimeApiInfo};
use sp_runtime::RuntimeString;
use sp_wasm_interface::HostFunctions;
use substrate_test_runtime::Block;
#[derive(Encode)]
pub struct OldRuntimeVersion {
pub spec_name: RuntimeString,
pub impl_name: RuntimeString,
pub authoring_version: u32,
pub spec_version: u32,
pub impl_version: u32,
pub apis: sp_version::ApisVec,
}
#[test]
fn host_functions_are_equal() {
let host_functions = sp_io::SubstrateHostFunctions::host_functions();
@@ -468,7 +476,7 @@ mod tests {
#[test]
fn old_runtime_version_decodes() {
let old_runtime_version = sp_api::OldRuntimeVersion {
let old_runtime_version = OldRuntimeVersion {
spec_name: "test".into(),
impl_name: "test".into(),
authoring_version: 1,
@@ -479,11 +487,12 @@ mod tests {
let version = decode_version(&old_runtime_version.encode()).unwrap();
assert_eq!(1, version.transaction_version);
assert_eq!(0, version.state_version);
}
#[test]
fn old_runtime_version_decodes_fails_with_version_3() {
let old_runtime_version = sp_api::OldRuntimeVersion {
let old_runtime_version = OldRuntimeVersion {
spec_name: "test".into(),
impl_name: "test".into(),
authoring_version: 1,
@@ -505,10 +514,27 @@ mod tests {
impl_version: 1,
apis: sp_api::create_apis_vec!([(<dyn Core::<Block>>::ID, 3)]),
transaction_version: 3,
state_version: 4,
};
let version = decode_version(&old_runtime_version.encode()).unwrap();
assert_eq!(3, version.transaction_version);
assert_eq!(0, version.state_version);
let old_runtime_version = sp_api::RuntimeVersion {
spec_name: "test".into(),
impl_name: "test".into(),
authoring_version: 1,
spec_version: 1,
impl_version: 1,
apis: sp_api::create_apis_vec!([(<dyn Core::<Block>>::ID, 4)]),
transaction_version: 3,
state_version: 4,
};
let version = decode_version(&old_runtime_version.encode()).unwrap();
assert_eq!(3, version.transaction_version);
assert_eq!(4, version.state_version);
}
#[test]
@@ -518,15 +544,15 @@ mod tests {
sp_maybe_compressed_blob::CODE_BLOB_BOMB_LIMIT,
)
.expect("Decompressing works");
let runtime_version = RuntimeVersion {
spec_name: "test_replace".into(),
impl_name: "test_replace".into(),
authoring_version: 100,
spec_version: 100,
impl_version: 100,
apis: sp_api::create_apis_vec!([(<dyn Core::<Block>>::ID, 3)]),
apis: sp_api::create_apis_vec!([(<dyn Core::<Block>>::ID, 4)]),
transaction_version: 100,
state_version: 1,
};
let embedded = sp_version::embed::embed_runtime_version(&wasm, runtime_version.clone())
@@ -2539,8 +2539,10 @@ fn validate_blocks<Block: BlockT>(
}
if let (Some(header), Some(body)) = (&b.header, &b.body) {
let expected = *header.extrinsics_root();
let got =
HashFor::<Block>::ordered_trie_root(body.iter().map(Encode::encode).collect());
let got = HashFor::<Block>::ordered_trie_root(
body.iter().map(Encode::encode).collect(),
sp_runtime::StateVersion::V0,
);
if expected != got {
debug!(
target:"sync",
+2 -2
View File
@@ -526,11 +526,11 @@ fn should_return_runtime_version() {
);
let result = "{\"specName\":\"test\",\"implName\":\"parity-test\",\"authoringVersion\":1,\
\"specVersion\":2,\"implVersion\":2,\"apis\":[[\"0xdf6acb689907609b\",3],\
\"specVersion\":2,\"implVersion\":2,\"apis\":[[\"0xdf6acb689907609b\",4],\
[\"0x37e397fc7c91f5e4\",1],[\"0xd2bc9897eed08f15\",3],[\"0x40fe3ad401f8959a\",5],\
[\"0xc6e9a76309f39b09\",1],[\"0xdd718d5cc53262d4\",1],[\"0xcbca25e39f142387\",2],\
[\"0xf78b278be53f454c\",2],[\"0xab3c0572291feb8b\",1],[\"0xbc9d89904f5b923f\",1]],\
\"transactionVersion\":1}";
\"transactionVersion\":1,\"stateVersion\":1}";
let runtime_version = executor::block_on(api.runtime_version(None.into())).unwrap();
let serialized = serde_json::to_string(&runtime_version).unwrap();
@@ -90,7 +90,7 @@ where
Block: BlockT,
B: backend::Backend<Block>,
{
let spec = self.runtime_version(id)?;
let spec = CallExecutor::runtime_version(self, id)?;
let code = if let Some(d) = self
.wasm_override
.as_ref()
@@ -321,6 +321,20 @@ where
}
}
impl<B, E, Block> RuntimeVersionOf for LocalCallExecutor<Block, B, E>
where
E: RuntimeVersionOf,
Block: BlockT,
{
fn runtime_version(
&self,
ext: &mut dyn sp_externalities::Externalities,
runtime_code: &sp_core::traits::RuntimeCode,
) -> Result<sp_version::RuntimeVersion, sc_executor::error::Error> {
RuntimeVersionOf::runtime_version(&self.executor, ext, runtime_code)
}
}
impl<Block, B, E> sp_version::GetRuntimeVersionAt<Block> for LocalCallExecutor<Block, B, E>
where
B: backend::Backend<Block>,
+51 -13
View File
@@ -45,7 +45,7 @@ use sc_client_api::{
use sc_consensus::{
BlockCheckParams, BlockImportParams, ForkChoiceStrategy, ImportResult, StateAction,
};
use sc_executor::RuntimeVersion;
use sc_executor::{RuntimeVersion, RuntimeVersionOf};
use sc_telemetry::{telemetry, TelemetryHandle, SUBSTRATE_INFO};
use sp_api::{
ApiExt, ApiRef, CallApiAt, CallApiAtParams, ConstructRuntimeApi, Core as CoreApi,
@@ -60,8 +60,8 @@ use sp_consensus::{BlockOrigin, BlockStatus, Error as ConsensusError};
use sc_utils::mpsc::{tracing_unbounded, TracingUnboundedSender};
use sp_core::{
storage::{
well_known_keys, ChildInfo, ChildType, PrefixedStorageKey, StorageChild, StorageData,
StorageKey,
well_known_keys, ChildInfo, ChildType, PrefixedStorageKey, Storage, StorageChild,
StorageData, StorageKey,
},
NativeOrEncoded,
};
@@ -72,7 +72,7 @@ use sp_runtime::{
traits::{
Block as BlockT, HashFor, Header as HeaderT, NumberFor, One, SaturatedConversion, Zero,
},
BuildStorage, Digest, Justification, Justifications,
BuildStorage, Digest, Justification, Justifications, StateVersion,
};
use sp_state_machine::{
prove_child_read, prove_range_read_with_child_with_size, prove_read,
@@ -81,7 +81,7 @@ use sp_state_machine::{
};
use sp_trie::{CompactProof, StorageProof};
use std::{
collections::{HashMap, HashSet},
collections::{hash_map::DefaultHasher, HashMap, HashSet},
marker::PhantomData,
panic::UnwindSafe,
path::PathBuf,
@@ -93,7 +93,6 @@ use std::{
use {
super::call_executor::LocalCallExecutor,
sc_client_api::in_mem,
sc_executor::RuntimeVersionOf,
sp_core::traits::{CodeExecutor, SpawnNamed},
};
@@ -334,8 +333,11 @@ where
if info.finalized_state.is_none() {
let genesis_storage =
build_genesis_storage.build_storage().map_err(sp_blockchain::Error::Storage)?;
let genesis_state_version =
Self::resolve_state_version_from_wasm(&genesis_storage, &executor)?;
let mut op = backend.begin_operation()?;
let state_root = op.set_genesis_state(genesis_storage, !config.no_genesis)?;
let state_root =
op.set_genesis_state(genesis_storage, !config.no_genesis, genesis_state_version)?;
let genesis_block = genesis::construct_genesis_block::<Block>(state_root.into());
info!(
"🔨 Initializing Genesis block/state (state: {}, header-hash: {})",
@@ -403,7 +405,7 @@ where
/// Get the RuntimeVersion at a given block.
pub fn runtime_version_at(&self, id: &BlockId<Block>) -> sp_blockchain::Result<RuntimeVersion> {
self.executor.runtime_version(id)
CallExecutor::runtime_version(&self.executor, id)
}
/// Apply a checked and validated block to an operation. If a justification is provided
@@ -606,7 +608,11 @@ where
}
}
let state_root = operation.op.reset_storage(storage)?;
// This is use by fast sync for runtime version to be resolvable from
// changes.
let state_version =
Self::resolve_state_version_from_wasm(&storage, &self.executor)?;
let state_root = operation.op.reset_storage(storage, state_version)?;
if state_root != *import_headers.post().state_root() {
// State root mismatch when importing state. This should not happen in
// safe fast sync mode, but may happen in unsafe mode.
@@ -1041,6 +1047,35 @@ where
trace!("Collected {} uncles", uncles.len());
Ok(uncles)
}
fn resolve_state_version_from_wasm(
storage: &Storage,
executor: &E,
) -> sp_blockchain::Result<StateVersion> {
if let Some(wasm) = storage.top.get(well_known_keys::CODE) {
let mut ext = sp_state_machine::BasicExternalities::new_empty(); // just to read runtime version.
let code_fetcher = sp_core::traits::WrappedRuntimeCode(wasm.as_slice().into());
let runtime_code = sp_core::traits::RuntimeCode {
code_fetcher: &code_fetcher,
heap_pages: None,
hash: {
use std::hash::{Hash, Hasher};
let mut state = DefaultHasher::new();
wasm.hash(&mut state);
state.finish().to_le_bytes().to_vec()
},
};
let runtime_version =
RuntimeVersionOf::runtime_version(executor, &mut ext, &runtime_code)
.map_err(|e| sp_blockchain::Error::VersionInvalid(format!("{:?}", e)))?;
Ok(runtime_version.state_version())
} else {
Err(sp_blockchain::Error::VersionInvalid(
"Runtime missing from initial storage, could not read state version.".to_string(),
))
}
}
}
impl<B, E, Block, RA> UsageProvider<Block> for Client<B, E, Block, RA>
@@ -1095,12 +1130,14 @@ where
size_limit: usize,
) -> sp_blockchain::Result<(CompactProof, u32)> {
let state = self.state_at(id)?;
let root = state.storage_root(std::iter::empty()).0;
// this is a read proof, using version V0 or V1 is equivalent.
let root = state.storage_root(std::iter::empty(), StateVersion::V0).0;
let (proof, count) = prove_range_read_with_child_with_size::<_, HashFor<Block>>(
state, size_limit, start_key,
)?;
let proof = sp_trie::encode_compact::<sp_trie::Layout<HashFor<Block>>>(proof, root)
// This is read proof only, we can use either LayoutV0 or LayoutV1.
let proof = sp_trie::encode_compact::<sp_trie::LayoutV0<HashFor<Block>>>(proof, root)
.map_err(|e| sp_blockchain::Error::from_state(Box::new(e)))?;
Ok((proof, count))
}
@@ -1225,7 +1262,8 @@ where
start_key: &[Vec<u8>],
) -> sp_blockchain::Result<(KeyValueStates, usize)> {
let mut db = sp_state_machine::MemoryDB::<HashFor<Block>>::new(&[]);
let _ = sp_trie::decode_compact::<sp_state_machine::Layout<HashFor<Block>>, _, _>(
// Compact encoding
let _ = sp_trie::decode_compact::<sp_state_machine::LayoutV0<HashFor<Block>>, _, _>(
&mut db,
proof.iter_compact_encoded_nodes(),
Some(&root),
@@ -1594,7 +1632,7 @@ where
}
fn runtime_version_at(&self, at: &BlockId<Block>) -> Result<RuntimeVersion, sp_api::ApiError> {
self.runtime_version_at(at).map_err(Into::into)
CallExecutor::runtime_version(&self.executor, at).map_err(Into::into)
}
}
@@ -22,8 +22,10 @@ use sp_runtime::traits::{Block as BlockT, Hash as HashT, Header as HeaderT, Zero
/// Create a genesis block, given the initial storage.
pub fn construct_genesis_block<Block: BlockT>(state_root: Block::Hash) -> Block {
let extrinsics_root =
<<<Block as BlockT>::Header as HeaderT>::Hashing as HashT>::trie_root(Vec::new());
let extrinsics_root = <<<Block as BlockT>::Header as HeaderT>::Hashing as HashT>::trie_root(
Vec::new(),
sp_runtime::StateVersion::V0,
);
Block::new(
<<Block as BlockT>::Header as HeaderT>::new(
@@ -34,13 +34,13 @@ use sp_core::{testing::TaskExecutor, H256};
use sp_runtime::{
generic::BlockId,
traits::{BlakeTwo256, Block as BlockT, Header as HeaderT},
ConsensusEngineId, Justifications,
ConsensusEngineId, Justifications, StateVersion,
};
use sp_state_machine::{
backend::Backend as _, ExecutionStrategy, InMemoryBackend, OverlayedChanges, StateMachine,
};
use sp_storage::{ChildInfo, StorageKey};
use sp_trie::{trie_types::Layout, TrieConfiguration};
use sp_trie::{LayoutV0, TrieConfiguration};
use std::{collections::HashSet, sync::Arc};
use substrate_test_runtime::TestAPI;
use substrate_test_runtime_client::{
@@ -90,7 +90,7 @@ fn construct_block(
let transactions = txs.into_iter().map(|tx| tx.into_signed_tx()).collect::<Vec<_>>();
let iter = transactions.iter().map(Encode::encode);
let extrinsics_root = Layout::<BlakeTwo256>::ordered_trie_root(iter).into();
let extrinsics_root = LayoutV0::<BlakeTwo256>::ordered_trie_root(iter).into();
let mut header = Header {
parent_hash,
@@ -177,7 +177,7 @@ fn construct_genesis_should_work_with_native() {
.genesis_map();
let genesis_hash = insert_genesis_block(&mut storage);
let backend = InMemoryBackend::from(storage);
let backend = InMemoryBackend::from((storage, StateVersion::default()));
let (b1data, _b1hash) = block1(genesis_hash, &backend);
let backend_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&backend);
let runtime_code = backend_runtime_code.runtime_code().expect("Code is part of the backend");
@@ -210,7 +210,7 @@ fn construct_genesis_should_work_with_wasm() {
.genesis_map();
let genesis_hash = insert_genesis_block(&mut storage);
let backend = InMemoryBackend::from(storage);
let backend = InMemoryBackend::from((storage, StateVersion::default()));
let (b1data, _b1hash) = block1(genesis_hash, &backend);
let backend_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&backend);
let runtime_code = backend_runtime_code.runtime_code().expect("Code is part of the backend");
@@ -243,7 +243,7 @@ fn construct_genesis_with_bad_transaction_should_panic() {
.genesis_map();
let genesis_hash = insert_genesis_block(&mut storage);
let backend = InMemoryBackend::from(storage);
let backend = InMemoryBackend::from((storage, StateVersion::default()));
let (b1data, _b1hash) = block1(genesis_hash, &backend);
let backend_runtime_code = sp_state_machine::backend::BackendRuntimeCode::new(&backend);
let runtime_code = backend_runtime_code.runtime_code().expect("Code is part of the backend");
@@ -418,8 +418,8 @@ fn uncles_with_multiple_forks() {
// block tree:
// G -> A1 -> A2 -> A3 -> A4 -> A5
// A1 -> B2 -> B3 -> B4
// B2 -> C3
// A1 -> D2
// B2 -> C3
// A1 -> D2
let mut client = substrate_test_runtime_client::new();
// G -> A1