Inner hashing of value in state trie (runtime versioning). (#9732)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>

substrate/primitives/state-machine/src/ext.rs

@@ -24,9 +24,11 @@ use codec::{Decode, Encode, EncodeAppend};
 use hash_db::Hasher;
 #[cfg(feature = "std")]
 use sp_core::hexdisplay::HexDisplay;
-use sp_core::storage::{well_known_keys::is_child_storage_key, ChildInfo, TrackedStorageKey};
+use sp_core::storage::{
+	well_known_keys::is_child_storage_key, ChildInfo, StateVersion, TrackedStorageKey,
+};
 use sp_externalities::{Extension, ExtensionStore, Externalities};
-use sp_trie::{empty_child_trie_root, trie_types::Layout};
+use sp_trie::{empty_child_trie_root, LayoutV1};
 
 use crate::{log_error, trace, warn, StorageTransactionCache};
 use sp_std::{
@@ -505,7 +507,7 @@ where
 		StorageAppend::new(current_value).append(value);
 	}
 
-	fn storage_root(&mut self) -> Vec<u8> {
+	fn storage_root(&mut self, state_version: StateVersion) -> Vec<u8> {
 		let _guard = guard();
 		if let Some(ref root) = self.storage_transaction_cache.transaction_storage_root {
 			trace!(
@@ -518,7 +520,9 @@ where
 			return root.encode()
 		}
 
-		let root = self.overlay.storage_root(self.backend, self.storage_transaction_cache);
+		let root =
+			self.overlay
+				.storage_root(self.backend, self.storage_transaction_cache, state_version);
 		trace!(
 			target: "state",
 			method = "StorageRoot",
@@ -529,7 +533,11 @@ where
 		root.encode()
 	}
 
-	fn child_storage_root(&mut self, child_info: &ChildInfo) -> Vec<u8> {
+	fn child_storage_root(
+		&mut self,
+		child_info: &ChildInfo,
+		state_version: StateVersion,
+	) -> Vec<u8> {
 		let _guard = guard();
 		let storage_key = child_info.storage_key();
 		let prefixed_storage_key = child_info.prefixed_storage_key();
@@ -537,7 +545,8 @@ where
 			let root = self
 				.storage(prefixed_storage_key.as_slice())
 				.and_then(|k| Decode::decode(&mut &k[..]).ok())
-				.unwrap_or_else(|| empty_child_trie_root::<Layout<H>>());
+				// V1 is equivalent to V0 on empty root.
+				.unwrap_or_else(|| empty_child_trie_root::<LayoutV1<H>>());
 			trace!(
 				target: "state",
 				method = "ChildStorageRoot",
@@ -550,7 +559,7 @@ where
 		} else {
 			let root = if let Some((changes, info)) = self.overlay.child_changes(storage_key) {
 				let delta = changes.map(|(k, v)| (k.as_ref(), v.value().map(AsRef::as_ref)));
-				Some(self.backend.child_storage_root(info, delta))
+				Some(self.backend.child_storage_root(info, delta, state_version))
 			} else {
 				None
 			};
@@ -583,7 +592,8 @@ where
 				let root = self
 					.storage(prefixed_storage_key.as_slice())
 					.and_then(|k| Decode::decode(&mut &k[..]).ok())
-					.unwrap_or_else(|| empty_child_trie_root::<Layout<H>>());
+					// V1 is equivalent to V0 on empty root.
+					.unwrap_or_else(|| empty_child_trie_root::<LayoutV1<H>>());
 
 				trace!(
 					target: "state",
@@ -648,7 +658,12 @@ where
 			self.overlay.rollback_transaction().expect(BENCHMARKING_FN);
 		}
 		self.overlay
-			.drain_storage_changes(self.backend, Default::default(), self.storage_transaction_cache)
+			.drain_storage_changes(
+				self.backend,
+				Default::default(),
+				self.storage_transaction_cache,
+				Default::default(), // using any state
+			)
 			.expect(EXT_NOT_ALLOWED_TO_FAIL);
 		self.backend.wipe().expect(EXT_NOT_ALLOWED_TO_FAIL);
 		self.mark_dirty();
@@ -658,12 +673,19 @@ where
 	}
 
 	fn commit(&mut self) {
+		// Bench always use latest state.
+		let state_version = StateVersion::default();
 		for _ in 0..self.overlay.transaction_depth() {
 			self.overlay.commit_transaction().expect(BENCHMARKING_FN);
 		}
 		let changes = self
 			.overlay
-			.drain_storage_changes(self.backend, Default::default(), self.storage_transaction_cache)
+			.drain_storage_changes(
+				self.backend,
+				Default::default(),
+				self.storage_transaction_cache,
+				state_version,
+			)
 			.expect(EXT_NOT_ALLOWED_TO_FAIL);
 		self.backend
 			.commit(
@@ -878,15 +900,18 @@ mod tests {
 		let mut overlay = OverlayedChanges::default();
 		overlay.set_storage(vec![20], None);
 		overlay.set_storage(vec![30], Some(vec![31]));
-		let backend = Storage {
-			top: map![
-				vec![10] => vec![10],
-				vec![20] => vec![20],
-				vec![40] => vec![40]
-			],
-			children_default: map![],
-		}
-		.into();
+		let backend = (
+			Storage {
+				top: map![
+					vec![10] => vec![10],
+					vec![20] => vec![20],
+					vec![40] => vec![40]
+				],
+				children_default: map![],
+			},
+			StateVersion::default(),
+		)
+			.into();
 
 		let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
 
@@ -924,13 +949,16 @@ mod tests {
 		overlay.set_storage(vec![27], None);
 		overlay.set_storage(vec![28], None);
 		overlay.set_storage(vec![29], None);
-		let backend = Storage {
-			top: map![
-				vec![30] => vec![30]
-			],
-			children_default: map![],
-		}
-		.into();
+		let backend = (
+			Storage {
+				top: map![
+					vec![30] => vec![30]
+				],
+				children_default: map![],
+			},
+			StateVersion::default(),
+		)
+			.into();
 
 		let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
 
@@ -948,20 +976,23 @@ mod tests {
 		let mut overlay = OverlayedChanges::default();
 		overlay.set_child_storage(child_info, vec![20], None);
 		overlay.set_child_storage(child_info, vec![30], Some(vec![31]));
-		let backend = Storage {
-			top: map![],
-			children_default: map![
-				child_info.storage_key().to_vec() => StorageChild {
-					data: map![
-						vec![10] => vec![10],
-						vec![20] => vec![20],
-						vec![40] => vec![40]
-					],
-					child_info: child_info.to_owned(),
-				}
-			],
-		}
-		.into();
+		let backend = (
+			Storage {
+				top: map![],
+				children_default: map![
+					child_info.storage_key().to_vec() => StorageChild {
+						data: map![
+							vec![10] => vec![10],
+							vec![20] => vec![20],
+							vec![40] => vec![40]
+						],
+						child_info: child_info.to_owned(),
+					}
+				],
+			},
+			StateVersion::default(),
+		)
+			.into();
 
 		let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
 
@@ -993,20 +1024,23 @@ mod tests {
 		let mut overlay = OverlayedChanges::default();
 		overlay.set_child_storage(child_info, vec![20], None);
 		overlay.set_child_storage(child_info, vec![30], Some(vec![31]));
-		let backend = Storage {
-			top: map![],
-			children_default: map![
-				child_info.storage_key().to_vec() => StorageChild {
-					data: map![
-						vec![10] => vec![10],
-						vec![20] => vec![20],
-						vec![30] => vec![40]
-					],
-					child_info: child_info.to_owned(),
-				}
-			],
-		}
-		.into();
+		let backend = (
+			Storage {
+				top: map![],
+				children_default: map![
+					child_info.storage_key().to_vec() => StorageChild {
+						data: map![
+							vec![10] => vec![10],
+							vec![20] => vec![20],
+							vec![30] => vec![40]
+						],
+						child_info: child_info.to_owned(),
+					}
+				],
+			},
+			StateVersion::default(),
+		)
+			.into();
 
 		let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);
 
@@ -1032,18 +1066,21 @@ mod tests {
 		let child_info = &child_info;
 		let mut cache = StorageTransactionCache::default();
 		let mut overlay = OverlayedChanges::default();
-		let backend = Storage {
-			top: map![],
-			children_default: map![
-				child_info.storage_key().to_vec() => StorageChild {
-					data: map![
-						vec![30] => vec![40]
-					],
-					child_info: child_info.to_owned(),
-				}
-			],
-		}
-		.into();
+		let backend = (
+			Storage {
+				top: map![],
+				children_default: map![
+					child_info.storage_key().to_vec() => StorageChild {
+						data: map![
+							vec![30] => vec![40]
+						],
+						child_info: child_info.to_owned(),
+					}
+				],
+			},
+			StateVersion::default(),
+		)
+			.into();
 
 		let ext = TestExt::new(&mut overlay, &mut cache, &backend, None);