Trie version migration pallet (#10073)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* migration pallet

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* fix test

* fix child migration calls.

* useless closure

* remove remaining state_hash variables.

* Fix and add more tests

* Remove outdated comment

* useless inner hash

* fmt

* remote tests

* finally ksm works

* batches are broken

* clean the benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Update frame/state-trie-migration/src/lib.rs

* brand new version

* fix build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* new test structure

* new testing stuff from emeric

* Add commit_all, still not working

* Fix all tests

* add comment

* we have PoV tracking baby

* document stuff, but proof size is still wrong

* FUCK YEAH

* a big batch of review comments

* add more tests

* tweak test

* update config

* some remote-ext stuff

* delete some of the old stuff

* sync more files with master to minimize the diff

* Fix all tests

* make signed migration a bit more relaxed

* add witness check to signed submissions

* allow custom migration to also go above limit

* Fix these pesky tests

* ==== removal of the unsigned stuff ====

* Make all tests work again

* separate the tests from the logic so it can be reused easier

* fix overall build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Slightly better termination

* some final tweaks

* Fix tests

* Restrict access to signed migrations

* address most of the review comments

* fix defensive

* New simplified code

* Fix weights

* fmt

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make the tests correctly fail

* Fix build

* Fix build

* try and fix the benchmarks

* fix build

* Fix cargo file

* Fix runtime deposit

* make rustdoc happy

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>

substrate/utils/frame/remote-externalities/src/lib.rs

@@ -40,7 +40,7 @@ use sp_core::{
 	},
 };
 pub use sp_io::TestExternalities;
-use sp_runtime::traits::Block as BlockT;
+use sp_runtime::{traits::Block as BlockT, StateVersion};
 use std::{
 	fs,
 	path::{Path, PathBuf},
@@ -56,6 +56,7 @@ type ChildKeyValues = Vec<(ChildInfo, Vec<KeyValue>)>;
 const LOG_TARGET: &str = "remote-ext";
 const DEFAULT_TARGET: &str = "wss://rpc.polkadot.io:443";
 const BATCH_SIZE: usize = 1000;
+const PAGE: u32 = 512;
 
 #[rpc(client)]
 pub trait RpcApi<Hash> {
@@ -117,12 +118,6 @@ pub struct OfflineConfig {
 	pub state_snapshot: SnapshotConfig,
 }
 
-impl<P: Into<PathBuf>> From<P> for SnapshotConfig {
-	fn from(p: P) -> Self {
-		Self { path: p.into() }
-	}
-}
-
 /// Description of the transport protocol (for online execution).
 #[derive(Debug, Clone)]
 pub enum Transport {
@@ -187,6 +182,8 @@ pub struct OnlineConfig<B: BlockT> {
 	pub pallets: Vec<String>,
 	/// Transport config.
 	pub transport: Transport,
+	/// Lookout for child-keys, and scrape them as well if set to true.
+	pub scrape_children: bool,
 }
 
 impl<B: BlockT> OnlineConfig<B> {
@@ -205,10 +202,17 @@ impl<B: BlockT> Default for OnlineConfig<B> {
 			at: None,
 			state_snapshot: None,
 			pallets: vec![],
+			scrape_children: true,
 		}
 	}
 }
 
+impl<B: BlockT> From<String> for OnlineConfig<B> {
+	fn from(s: String) -> Self {
+		Self { transport: s.into(), ..Default::default() }
+	}
+}
+
 /// Configuration of the state snapshot.
 #[derive(Clone)]
 pub struct SnapshotConfig {
@@ -222,6 +226,12 @@ impl SnapshotConfig {
 	}
 }
 
+impl From<String> for SnapshotConfig {
+	fn from(s: String) -> Self {
+		Self::new(s)
+	}
+}
+
 impl Default for SnapshotConfig {
 	fn default() -> Self {
 		Self { path: Path::new("SNAPSHOT").into() }
@@ -242,6 +252,8 @@ pub struct Builder<B: BlockT> {
 	hashed_blacklist: Vec<Vec<u8>>,
 	/// connectivity mode, online or offline.
 	mode: Mode<B>,
+	/// The state version being used.
+	state_version: StateVersion,
 }
 
 // NOTE: ideally we would use `DefaultNoBound` here, but not worth bringing in frame-support for
@@ -254,6 +266,7 @@ impl<B: BlockT + DeserializeOwned> Default for Builder<B> {
 			hashed_prefixes: Default::default(),
 			hashed_keys: Default::default(),
 			hashed_blacklist: Default::default(),
+			state_version: StateVersion::V1,
 		}
 	}
 }
@@ -306,7 +319,6 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 		prefix: StorageKey,
 		at: B::Hash,
 	) -> Result<Vec<StorageKey>, &'static str> {
-		const PAGE: u32 = 512;
 		let mut last_key: Option<StorageKey> = None;
 		let mut all_keys: Vec<StorageKey> = vec![];
 		let keys = loop {
@@ -320,6 +332,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 					"rpc get_keys failed"
 				})?;
 			let page_len = page.len();
+
 			all_keys.extend(page);
 
 			if page_len < PAGE as usize {
@@ -362,11 +375,12 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 				.cloned()
 				.map(|key| ("state_getStorage", rpc_params![key, at]))
 				.collect::<Vec<_>>();
+
 			let values = client.batch_request::<Option<StorageData>>(batch).await.map_err(|e| {
 				log::error!(
 					target: LOG_TARGET,
 					"failed to execute batch: {:?}. Error: {:?}",
-					chunk_keys,
+					chunk_keys.iter().map(|k| HexDisplay::from(k)).collect::<Vec<_>>(),
 					e
 				);
 				"batch failed."
@@ -693,7 +707,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 
 		// inject manual key values.
 		if !self.hashed_key_values.is_empty() {
-			log::debug!(
+			log::info!(
 				target: LOG_TARGET,
 				"extending externalities with {} manually injected key-values",
 				self.hashed_key_values.len()
@@ -703,7 +717,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 
 		// exclude manual key values.
 		if !self.hashed_blacklist.is_empty() {
-			log::debug!(
+			log::info!(
 				target: LOG_TARGET,
 				"excluding externalities from {} keys",
 				self.hashed_blacklist.len()
@@ -795,6 +809,12 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 		self
 	}
 
+	/// The state version to use.
+	pub fn state_version(mut self, version: StateVersion) -> Self {
+		self.state_version = version;
+		self
+	}
+
 	/// overwrite the `at` value, if `mode` is set to [`Mode::Online`].
 	///
 	/// noop if `mode` is [`Mode::Offline`]
@@ -808,8 +828,13 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 
 	/// Build the test externalities.
 	pub async fn build(self) -> Result<TestExternalities, &'static str> {
+		let state_version = self.state_version;
 		let (top_kv, child_kv) = self.pre_build().await?;
-		let mut ext = TestExternalities::new_with_code(Default::default(), Default::default());
+		let mut ext = TestExternalities::new_with_code_and_state(
+			Default::default(),
+			Default::default(),
+			state_version,
+		);
 
 		info!(target: LOG_TARGET, "injecting a total of {} top keys", top_kv.len());
 		for (k, v) in top_kv {
@@ -1165,4 +1190,21 @@ mod remote_tests {
 			std::fs::remove_file(d.path()).unwrap();
 		}
 	}
+
+	#[tokio::test]
+	async fn can_build_child_tree() {
+		init_logger();
+		Builder::<Block>::new()
+			.mode(Mode::Online(OnlineConfig {
+				// transport: "wss://kusama-rpc.polkadot.io".to_owned().into(),
+				transport: "ws://kianenigma-archive:9924".to_owned().into(),
+				// transport: "ws://localhost:9999".to_owned().into(),
+				pallets: vec!["Crowdloan".to_owned()],
+				..Default::default()
+			}))
+			.build()
+			.await
+			.expect(REMOTE_INACCESSIBLE)
+			.execute_with(|| {});
+	}
 }