Inner hashing of value in state trie (runtime versioning). (#9732)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>

substrate/primitives/version/src/lib.rs

@@ -40,10 +40,10 @@ use std::collections::HashSet;
 #[cfg(feature = "std")]
 use std::fmt;
 
-use codec::{Decode, Encode};
+use codec::{Decode, Encode, Input};
 use scale_info::TypeInfo;
-pub use sp_runtime::create_runtime_str;
 use sp_runtime::RuntimeString;
+pub use sp_runtime::{create_runtime_str, StateVersion};
 #[doc(hidden)]
 pub use sp_std;
 
@@ -79,6 +79,7 @@ pub mod embed;
 /// 	impl_version: 1,
 /// 	apis: RUNTIME_API_VERSIONS,
 /// 	transaction_version: 2,
+/// 	state_version: 1,
 /// };
 ///
 /// # const RUNTIME_API_VERSIONS: sp_version::ApisVec = sp_version::create_apis_vec!([]);
@@ -154,7 +155,7 @@ macro_rules! create_apis_vec {
 /// In particular: bug fixes should result in an increment of `spec_version` and possibly
 /// `authoring_version`, absolutely not `impl_version` since they change the semantics of the
 /// runtime.
-#[derive(Clone, PartialEq, Eq, Encode, Decode, Default, sp_runtime::RuntimeDebug, TypeInfo)]
+#[derive(Clone, PartialEq, Eq, Encode, Default, sp_runtime::RuntimeDebug, TypeInfo)]
 #[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
 #[cfg_attr(feature = "std", serde(rename_all = "camelCase"))]
 pub struct RuntimeVersion {
@@ -207,6 +208,53 @@ pub struct RuntimeVersion {
 	///
 	/// It need *not* change when a new module is added or when a dispatchable is added.
 	pub transaction_version: u32,
+
+	/// Version of the state implementation used by this runtime.
+	/// Use of an incorrect version is consensus breaking.
+	pub state_version: u8,
+}
+
+impl RuntimeVersion {
+	/// `Decode` while giving a "version hint"
+	///
+	/// There exists multiple versions of [`RuntimeVersion`] and they are versioned using the `Core`
+	/// runtime api:
+	/// - `Core` version < 3 is a runtime version without a transaction version and state version.
+	/// - `Core` version 3 is a runtime version without a state version.
+	/// - `Core` version 4 is the latest runtime version.
+	pub fn decode_with_version_hint<I: Input>(
+		input: &mut I,
+		core_version: Option<u32>,
+	) -> Result<RuntimeVersion, codec::Error> {
+		let spec_name = Decode::decode(input)?;
+		let impl_name = Decode::decode(input)?;
+		let authoring_version = Decode::decode(input)?;
+		let spec_version = Decode::decode(input)?;
+		let impl_version = Decode::decode(input)?;
+		let apis = Decode::decode(input)?;
+		let core_version =
+			if core_version.is_some() { core_version } else { core_version_from_apis(&apis) };
+		let transaction_version =
+			if core_version.map(|v| v >= 3).unwrap_or(false) { Decode::decode(input)? } else { 1 };
+		let state_version =
+			if core_version.map(|v| v >= 4).unwrap_or(false) { Decode::decode(input)? } else { 0 };
+		Ok(RuntimeVersion {
+			spec_name,
+			impl_name,
+			authoring_version,
+			spec_version,
+			impl_version,
+			apis,
+			transaction_version,
+			state_version,
+		})
+	}
+}
+
+impl Decode for RuntimeVersion {
+	fn decode<I: Input>(input: &mut I) -> Result<Self, codec::Error> {
+		Self::decode_with_version_hint(input, None)
+	}
 }
 
 #[cfg(feature = "std")]
@@ -225,6 +273,16 @@ impl fmt::Display for RuntimeVersion {
 	}
 }
 
+fn has_api_with<P: Fn(u32) -> bool>(apis: &ApisVec, id: &ApiId, predicate: P) -> bool {
+	apis.iter().any(|(s, v)| s == id && predicate(*v))
+}
+
+/// Returns the version of the `Core` runtime api.
+pub fn core_version_from_apis(apis: &ApisVec) -> Option<u32> {
+	let id = sp_core_hashing_proc_macro::blake2b_64!(b"Core");
+	apis.iter().find(|(s, _v)| s == &id).map(|(_s, v)| *v)
+}
+
 #[cfg(feature = "std")]
 impl RuntimeVersion {
 	/// Check if this version matches other version for calling into runtime.
@@ -237,7 +295,7 @@ impl RuntimeVersion {
 	/// Check if the given api with `api_id` is implemented and the version passes the given
 	/// `predicate`.
 	pub fn has_api_with<P: Fn(u32) -> bool>(&self, id: &ApiId, predicate: P) -> bool {
-		self.apis.iter().any(|(s, v)| s == id && predicate(*v))
+		has_api_with(&self.apis, id, predicate)
 	}
 
 	/// Returns the api version found for api with `id`.
@@ -246,6 +304,18 @@ impl RuntimeVersion {
 	}
 }
 
+impl RuntimeVersion {
+	/// Returns state version to use for update.
+	///
+	/// For runtime with core api version less than 4,
+	/// V0 trie version will be applied to state.
+	/// Otherwhise, V1 trie version will be use.
+	pub fn state_version(&self) -> StateVersion {
+		// If version > than 1, keep using latest version.
+		self.state_version.try_into().unwrap_or(StateVersion::V1)
+	}
+}
+
 /// The version of the native runtime.
 ///
 /// In contrast to the bare [`RuntimeVersion`] this also carries a list of `spec_version`s of