Trie version migration pallet (#10073)

* starting * Updated from other branch. * setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right logic to avoid switch on reencode when default layout. * Clean up some todos * remove trie meta from root upstream * update upstream and fix benches. * split some long lines. * UPdate trie crate to work with new design. * Finish update to refactored upstream. * update to latest triedb changes. * Clean up. * fix executor test. * rust fmt from master. * rust format. * rustfmt * fix * start host function driven versioning * update state-machine part * still need access to state version from runtime * state hash in mem: wrong * direction likely correct, but passing call to code exec for genesis init seem awkward. * state version serialize in runtime, wrong approach, just initialize it with no threshold for core api < 4 seems more proper. * stateversion from runtime version (core api >= 4). * update trie, fix tests * unused import * clean some TODOs * Require RuntimeVersionOf for executor * use RuntimeVersionOf to resolve genesis state version. * update runtime version test * fix state-machine tests * TODO * Use runtime version from storage wasm with fast sync. * rustfmt * fmt * fix test * revert useless changes. * clean some unused changes * fmt * removing useless trait function. * remove remaining reference to state_hash * fix some imports * Follow chain state version management. * trie update, fix and constant threshold for trie layouts. * update deps * Update to latest trie pr changes. * fix benches * Verify proof requires right layout. * update trie_root * Update trie deps to latest * Update to latest trie versioning * Removing patch * update lock * extrinsic for sc-service-test using layout v0. * Adding RuntimeVersionOf to CallExecutor works. * fmt * error when resolving version and no wasm in storage. * use existing utils to instantiate runtime code. * migration pallet * Patch to delay runtime switch. * Revert "Patch to delay runtime switch." This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da. * fix test * fix child migration calls. * useless closure * remove remaining state_hash variables. * Fix and add more tests * Remove outdated comment * useless inner hash * fmt * remote tests * finally ksm works * batches are broken * clean the benchmarks * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> * Update frame/state-trie-migration/src/lib.rs * brand new version * fix build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update primitives/storage/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * fmt and opt-in feature to apply state change. * feature gate core version, use new test feature for node and test node * Use a 'State' api version instead of Core one. * fix merge of test function * use blake macro. * Fix state api (require declaring the api in runtime). * Opt out feature, fix macro for io to select a given version instead of latest. * run test nodes on new state. * fix * new test structure * new testing stuff from emeric * Add commit_all, still not working * Fix all tests * add comment * we have PoV tracking baby * document stuff, but proof size is still wrong * FUCK YEAH * a big batch of review comments * add more tests * tweak test * update config * some remote-ext stuff * delete some of the old stuff * sync more files with master to minimize the diff * Fix all tests * make signed migration a bit more relaxed * add witness check to signed submissions * allow custom migration to also go above limit * Fix these pesky tests * ==== removal of the unsigned stuff ==== * Make all tests work again * separate the tests from the logic so it can be reused easier * fix overall build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Slightly better termination * some final tweaks * Fix tests * Restrict access to signed migrations * address most of the review comments * fix defensive * New simplified code * Fix weights * fmt * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * make the tests correctly fail * Fix build * Fix build * try and fix the benchmarks * fix build * Fix cargo file * Fix runtime deposit * make rustdoc happy * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io>
2026-06-12 04:11:07 +00:00 · 2022-03-04 15:54:23 +00:00
parent ded1f7706d
commit 6cfdbe5caa
16 changed files with 1954 additions and 41 deletions
@@ -5010,6 +5010,7 @@ dependencies = [
 "pallet-society",
 "pallet-staking",
 "pallet-staking-reward-curve",
+ "pallet-state-trie-migration",
 "pallet-sudo",
 "pallet-timestamp",
 "pallet-tips",
@@ -6419,6 +6420,30 @@ dependencies = [
 "sp-arithmetic",
 ]

+[[package]]
+name = "pallet-state-trie-migration"
+version = "4.0.0-dev"
+dependencies = [
+ "frame-benchmarking",
+ "frame-support",
+ "frame-system",
+ "log 0.4.14",
+ "pallet-balances",
+ "parity-scale-codec",
+ "parking_lot 0.12.0",
+ "remote-externalities",
+ "scale-info",
+ "serde",
+ "sp-core",
+ "sp-io",
+ "sp-runtime",
+ "sp-std",
+ "sp-tracing",
+ "thousands",
+ "tokio",
+ "zstd",
+]
+
 [[package]]
 name = "pallet-sudo"
 version = "4.0.0-dev"
@@ -10838,6 +10863,12 @@ dependencies = [
 "syn",
 ]

+[[package]]
+name = "thousands"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3bf63baf9f5039dadc247375c29eb13706706cfde997d0330d05aa63a77d8820"
+
 [[package]]
 name = "thread_local"
 version = "1.1.3"
@@ -87,6 +87,7 @@ pallet-session = { version = "4.0.0-dev", features = [ "historical" ], path = ".
 pallet-session-benchmarking = { version = "4.0.0-dev", path = "../../../frame/session/benchmarking", default-features = false, optional = true }
 pallet-staking = { version = "4.0.0-dev", default-features = false, path = "../../../frame/staking" }
 pallet-staking-reward-curve = { version = "4.0.0-dev", default-features = false, path = "../../../frame/staking/reward-curve" }
+pallet-state-trie-migration = { version = "4.0.0-dev", default-features = false, path = "../../../frame/state-trie-migration" }
 pallet-scheduler = { version = "4.0.0-dev", default-features = false, path = "../../../frame/scheduler" }
 pallet-society = { version = "4.0.0-dev", default-features = false, path = "../../../frame/society" }
 pallet-sudo = { version = "4.0.0-dev", default-features = false, path = "../../../frame/sudo" }
@@ -153,6 +154,7 @@ std = [
 	"sp-runtime/std",
 	"sp-staking/std",
 	"pallet-staking/std",
+	"pallet-state-trie-migration/std",
 	"sp-session/std",
 	"pallet-sudo/std",
 	"frame-support/std",
@@ -214,6 +216,7 @@ runtime-benchmarks = [
 	"pallet-session-benchmarking",
 	"pallet-society/runtime-benchmarks",
 	"pallet-staking/runtime-benchmarks",
+	"pallet-state-trie-migration/runtime-benchmarks",
 	"pallet-timestamp/runtime-benchmarks",
 	"pallet-tips/runtime-benchmarks",
 	"pallet-transaction-storage/runtime-benchmarks",
@@ -261,6 +264,7 @@ try-runtime = [
 	"pallet-session/try-runtime",
 	"pallet-society/try-runtime",
 	"pallet-staking/try-runtime",
+	"pallet-state-trie-migration/try-runtime",
 	"pallet-sudo/try-runtime",
 	"pallet-timestamp/try-runtime",
 	"pallet-tips/try-runtime",
@@ -39,7 +39,7 @@ use frame_support::{
 };
 use frame_system::{
 	limits::{BlockLength, BlockWeights},
-	EnsureRoot,
+	EnsureRoot, EnsureSigned,
 };
 pub use node_primitives::{AccountId, Signature};
 use node_primitives::{AccountIndex, Balance, BlockNumber, Hash, Index, Moment};
@@ -1367,6 +1367,28 @@ impl pallet_whitelist::Config for Runtime {
 	type WeightInfo = pallet_whitelist::weights::SubstrateWeight<Runtime>;
 }

+parameter_types! {
+	pub const SignedMigrationMaxLimits: pallet_state_trie_migration::MigrationLimits =
+		pallet_state_trie_migration::MigrationLimits { size: 1024 * 1024 / 2, item: 512 };
+	pub const MigrationSignedDepositPerItem: Balance = 1 * CENTS;
+	pub const MigrationSignedDepositBase: Balance = 20 * DOLLARS;
+}
+
+impl pallet_state_trie_migration::Config for Runtime {
+	type Event = Event;
+	type ControlOrigin = EnsureRoot<AccountId>;
+	type Currency = Balances;
+	type SignedDepositPerItem = MigrationSignedDepositPerItem;
+	type SignedDepositBase = MigrationSignedDepositBase;
+	type SignedMigrationMaxLimits = SignedMigrationMaxLimits;
+	// Warning: this is not advised, as it might allow the chain to be temporarily DOS-ed.
+	// Preferably, if the chain's governance/maintenance team is planning on using a specific
+	// account for the migration, put it here to make sure only that account can trigger the signed
+	// migrations.
+	type SignedFilter = EnsureSigned<Self::AccountId>;
+	type WeightInfo = ();
+}
+
 construct_runtime!(
 	pub enum Runtime where
 		Block = Block,
@@ -1418,6 +1440,7 @@ construct_runtime!(
 		Uniques: pallet_uniques,
 		TransactionStorage: pallet_transaction_storage,
 		BagsList: pallet_bags_list,
+		StateTrieMigration: pallet_state_trie_migration,
 		ChildBounties: pallet_child_bounties,
 		Referenda: pallet_referenda,
 		ConvictionVoting: pallet_conviction_voting,
@@ -1512,6 +1535,7 @@ mod benches {
 		[pallet_scheduler, Scheduler]
 		[pallet_session, SessionBench::<Runtime>]
 		[pallet_staking, Staking]
+		[pallet_state_trie_migration, StateTrieMigration]
 		[frame_system, SystemBench::<Runtime>]
 		[pallet_timestamp, Timestamp]
 		[pallet_tips, Tips]
@@ -34,8 +34,7 @@ pub async fn execute<Runtime: RuntimeT, Block: BlockT + DeserializeOwned>(
 		.mode(Mode::Online(OnlineConfig {
 			transport: ws_url.to_string().into(),
 			pallets: vec![pallet_staking::Pallet::<Runtime>::name().to_string()],
-			at: None,
-			state_snapshot: None,
+			..Default::default()
 		}))
 		.build()
 		.await
@@ -35,8 +35,7 @@ pub async fn execute<Runtime: crate::RuntimeT, Block: BlockT + DeserializeOwned>
 		.mode(Mode::Online(OnlineConfig {
 			transport: ws_url.to_string().into(),
 			pallets: vec![pallet_bags_list::Pallet::<Runtime>::name().to_string()],
-			at: None,
-			state_snapshot: None,
+			..Default::default()
 		}))
 		.inject_hashed_prefix(&<pallet_staking::Bonded<Runtime>>::prefix_hash())
 		.inject_hashed_prefix(&<pallet_staking::Ledger<Runtime>>::prefix_hash())
@@ -35,7 +35,7 @@ pub async fn execute<Runtime: crate::RuntimeT, Block: BlockT + DeserializeOwned>
 			// is bags-list.
 			pallets: vec![pallet_bags_list::Pallet::<Runtime>::name().to_string()],
 			at: None,
-			state_snapshot: None,
+			..Default::default()
 		}))
 		.inject_hashed_prefix(&<pallet_staking::Bonded<Runtime>>::prefix_hash())
 		.inject_hashed_prefix(&<pallet_staking::Ledger<Runtime>>::prefix_hash())
@@ -0,0 +1,57 @@
+[package]
+name = "pallet-state-trie-migration"
+version = "4.0.0-dev"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2021"
+license = "Apache-2.0"
+homepage = "https://substrate.dev"
+repository = "https://github.com/paritytech/substrate/"
+description = "FRAME pallet migration of trie"
+readme = "README.md"
+
+[package.metadata.docs.rs]
+targets = ["x86_64-unknown-linux-gnu"]
+
+[dependencies]
+scale-info = { version = "2.0.1", default-features = false, features = ["derive"] }
+codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false }
+log = { version = "0.4.14", default-features = false }
+
+sp-std = { default-features = false, path = "../../primitives/std" }
+sp-io = { default-features = false, path = "../../primitives/io" }
+sp-core = { default-features = false, path = "../../primitives/core" }
+sp-runtime = { default-features = false, path = "../../primitives/runtime" }
+
+frame-support = { default-features = false, path = "../support" }
+frame-system = { default-features = false, path = "../system" }
+frame-benchmarking = { default-features = false, path = "../benchmarking", optional = true }
+
+serde = { version = "1.0.133", optional = true }
+thousands = { version = "0.2.0", optional = true }
+remote-externalities = { path = "../../utils/frame/remote-externalities", optional = true }
+zstd = { version = "0.9.0", optional = true }
+
+[dev-dependencies]
+pallet-balances = { path = "../balances" }
+parking_lot = "0.12.0"
+sp-tracing = { path = "../../primitives/tracing" }
+tokio = { version = "1.10", features = ["macros"] }
+
+[features]
+default = ["std"]
+std = [
+	"log/std",
+	"scale-info/std",
+	"codec/std",
+	"frame-benchmarking/std",
+	"frame-support/std",
+	"frame-system/std",
+	"sp-core/std",
+	"sp-io/std",
+	"sp-runtime/std",
+	"sp-std/std"
+]
+runtime-benchmarks = ["frame-benchmarking"]
+try-runtime = ["frame-support/try-runtime"]
+
+remote-test = [ "std", "zstd", "serde", "thousands", "remote-externalities" ]
@@ -0,0 +1,137 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2022 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Autogenerated weights for pallet_state_trie_migration
+//!
+//! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 4.0.0-dev
+//! DATE: 2022-03-04, STEPS: `50`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("dev"), DB CACHE: 1024
+
+// Executed Command:
+// target/production/substrate
+// benchmark
+// --chain=dev
+// --steps=50
+// --repeat=20
+// --pallet=pallet_state_trie_migration
+// --extrinsic=*
+// --execution=wasm
+// --wasm-execution=compiled
+// --heap-pages=4096
+// --output=./frame/state-trie-migration/src/weights.rs
+// --template=./.maintain/frame-weight-template.hbs
+
+#![cfg_attr(rustfmt, rustfmt_skip)]
+#![allow(unused_parens)]
+#![allow(unused_imports)]
+
+use frame_support::{traits::Get, weights::{Weight, constants::RocksDbWeight}};
+use sp_std::marker::PhantomData;
+
+/// Weight functions needed for pallet_state_trie_migration.
+pub trait WeightInfo {
+	fn continue_migrate() -> Weight;
+	fn continue_migrate_wrong_witness() -> Weight;
+	fn migrate_custom_top_success() -> Weight;
+	fn migrate_custom_top_fail() -> Weight;
+	fn migrate_custom_child_success() -> Weight;
+	fn migrate_custom_child_fail() -> Weight;
+	fn process_top_key(v: u32, ) -> Weight;
+}
+
+/// Weights for pallet_state_trie_migration using the Substrate node and recommended hardware.
+pub struct SubstrateWeight<T>(PhantomData<T>);
+impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
+	// Storage: StateTrieMigration MigrationProcess (r:1 w:1)
+	fn continue_migrate() -> Weight {
+		(13_385_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(1 as Weight))
+			.saturating_add(T::DbWeight::get().writes(1 as Weight))
+	}
+	// Storage: StateTrieMigration MigrationProcess (r:1 w:0)
+	fn continue_migrate_wrong_witness() -> Weight {
+		(1_757_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(1 as Weight))
+	}
+	fn migrate_custom_top_success() -> Weight {
+		(12_813_000 as Weight)
+	}
+	// Storage: unknown [0x666f6f] (r:1 w:1)
+	fn migrate_custom_top_fail() -> Weight {
+		(24_961_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(1 as Weight))
+			.saturating_add(T::DbWeight::get().writes(1 as Weight))
+	}
+	fn migrate_custom_child_success() -> Weight {
+		(13_132_000 as Weight)
+	}
+	// Storage: unknown [0x666f6f] (r:1 w:1)
+	fn migrate_custom_child_fail() -> Weight {
+		(29_215_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(1 as Weight))
+			.saturating_add(T::DbWeight::get().writes(1 as Weight))
+	}
+	// Storage: unknown [0x6b6579] (r:1 w:1)
+	fn process_top_key(v: u32, ) -> Weight {
+		(0 as Weight)
+			// Standard Error: 0
+			.saturating_add((2_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add(T::DbWeight::get().reads(1 as Weight))
+			.saturating_add(T::DbWeight::get().writes(1 as Weight))
+	}
+}
+
+// For backwards compatibility and tests
+impl WeightInfo for () {
+	// Storage: StateTrieMigration MigrationProcess (r:1 w:1)
+	fn continue_migrate() -> Weight {
+		(13_385_000 as Weight)
+			.saturating_add(RocksDbWeight::get().reads(1 as Weight))
+			.saturating_add(RocksDbWeight::get().writes(1 as Weight))
+	}
+	// Storage: StateTrieMigration MigrationProcess (r:1 w:0)
+	fn continue_migrate_wrong_witness() -> Weight {
+		(1_757_000 as Weight)
+			.saturating_add(RocksDbWeight::get().reads(1 as Weight))
+	}
+	fn migrate_custom_top_success() -> Weight {
+		(12_813_000 as Weight)
+	}
+	// Storage: unknown [0x666f6f] (r:1 w:1)
+	fn migrate_custom_top_fail() -> Weight {
+		(24_961_000 as Weight)
+			.saturating_add(RocksDbWeight::get().reads(1 as Weight))
+			.saturating_add(RocksDbWeight::get().writes(1 as Weight))
+	}
+	fn migrate_custom_child_success() -> Weight {
+		(13_132_000 as Weight)
+	}
+	// Storage: unknown [0x666f6f] (r:1 w:1)
+	fn migrate_custom_child_fail() -> Weight {
+		(29_215_000 as Weight)
+			.saturating_add(RocksDbWeight::get().reads(1 as Weight))
+			.saturating_add(RocksDbWeight::get().writes(1 as Weight))
+	}
+	// Storage: unknown [0x6b6579] (r:1 w:1)
+	fn process_top_key(v: u32, ) -> Weight {
+		(0 as Weight)
+			// Standard Error: 0
+			.saturating_add((2_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add(RocksDbWeight::get().reads(1 as Weight))
+			.saturating_add(RocksDbWeight::get().writes(1 as Weight))
+	}
+}
@@ -38,18 +38,18 @@ macro_rules! defensive {
 		frame_support::log::error!(
 			target: "runtime",
 			"{}",
-			$crate::traits::misc::DEFENSIVE_OP_PUBLIC_ERROR
+			$crate::traits::DEFENSIVE_OP_PUBLIC_ERROR
 		);
-		debug_assert!(false, "{}", $crate::traits::misc::DEFENSIVE_OP_INTERNAL_ERROR);
+		debug_assert!(false, "{}", $crate::traits::DEFENSIVE_OP_INTERNAL_ERROR);
 	};
 	($error:tt) => {
 		frame_support::log::error!(
 			target: "runtime",
 			"{}: {:?}",
-			$crate::traits::misc::DEFENSIVE_OP_PUBLIC_ERROR,
+			$crate::traits::DEFENSIVE_OP_PUBLIC_ERROR,
 			$error
 		);
-		debug_assert!(false, "{}: {:?}", $crate::traits::misc::DEFENSIVE_OP_INTERNAL_ERROR, $error);
+		debug_assert!(false, "{}: {:?}", $crate::traits::DEFENSIVE_OP_INTERNAL_ERROR, $error);
 	}
 }

@@ -96,7 +96,7 @@ pub enum ExecutionContext {
 	/// We distinguish between major sync and import so that validators who are running
 	/// their initial sync (or catching up after some time offline) can use the faster
 	/// native runtime (since we can reasonably assume the network as a whole has already
-	/// come to a broad conensus on the block and it probably hasn't been crafted
+	/// come to a broad consensus on the block and it probably hasn't been crafted
 	/// specifically to attack this node), but when importing blocks at the head of the
 	/// chain in normal operation they can use the safer Wasm version.
 	Syncing,
@@ -23,8 +23,8 @@ use std::{
 };

 use crate::{
-	backend::Backend, ext::Ext, InMemoryBackend, InMemoryProvingBackend, OverlayedChanges,
-	StorageKey, StorageTransactionCache, StorageValue,
+	backend::Backend, ext::Ext, InMemoryBackend, OverlayedChanges, StorageKey,
+	StorageTransactionCache, StorageValue,
 };

 use hash_db::Hasher;
@@ -100,7 +100,6 @@ where
 		state_version: StateVersion,
 	) -> Self {
 		assert!(storage.top.keys().all(|key| !is_child_storage_key(key)));
-		assert!(storage.children_default.keys().all(|key| is_child_storage_key(key)));

 		storage.top.insert(CODE.to_vec(), code.to_vec());

@@ -204,7 +203,7 @@ where
 	/// This implementation will wipe the proof recorded in between calls. Consecutive calls will
 	/// get their own proof from scratch.
 	pub fn execute_and_prove<R>(&mut self, execute: impl FnOnce() -> R) -> (R, StorageProof) {
-		let proving_backend = InMemoryProvingBackend::new(&self.backend);
+		let proving_backend = crate::InMemoryProvingBackend::new(&self.backend);
 		let mut proving_ext = Ext::new(
 			&mut self.overlay,
 			&mut self.storage_transaction_cache,
@@ -23,21 +23,23 @@ use codec::Encode;
 use hash_db::{self, AsHashDB, HashDB, HashDBRef, Hasher, Prefix};
 #[cfg(feature = "std")]
 use parking_lot::RwLock;
-use sp_core::storage::{ChildInfo, ChildType, StateVersion};
+use sp_core::storage::{ChildInfo, ChildType, PrefixedStorageKey, StateVersion};
 use sp_std::{boxed::Box, vec::Vec};
 use sp_trie::{
 	child_delta_trie_root, delta_trie_root, empty_child_trie_root, read_child_trie_value,
 	read_trie_value,
 	trie_types::{TrieDB, TrieError},
-	DBValue, KeySpacedDB, PrefixedMemoryDB, Trie, TrieDBIterator, TrieDBKeyIterator,
+	DBValue, KeySpacedDB, LayoutV1 as Layout, PrefixedMemoryDB, Trie, TrieDBIterator,
+	TrieDBKeyIterator,
 };
 #[cfg(feature = "std")]
 use std::collections::HashMap;
 #[cfg(feature = "std")]
 use std::sync::Arc;
-// In this module, we only use layout for read operation and empty root,
-// where V1 and V0 are equivalent.
-use sp_trie::LayoutV1 as Layout;
+use trie_db::{
+	node::{NodePlan, ValuePlan},
+	TrieDBNodeIterator,
+};

 #[cfg(not(feature = "std"))]
 macro_rules! format {
@@ -431,6 +433,72 @@ where
 		);
 	}

+	/// Check remaining state item to migrate. Note this function should be remove when all state
+	/// migration did finished as it is only an utility.
+	// original author: @cheme
+	pub fn check_migration_state(&self) -> Result<(u64, u64)> {
+		let threshold: u32 = sp_core::storage::TRIE_VALUE_NODE_THRESHOLD;
+		let mut nb_to_migrate = 0;
+		let mut nb_to_migrate_child = 0;
+
+		let trie = sp_trie::trie_types::TrieDB::new(self, &self.root)
+			.map_err(|e| format!("TrieDB creation error: {}", e))?;
+		let iter_node = TrieDBNodeIterator::new(&trie)
+			.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+		for node in iter_node {
+			let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+			match node.2.node_plan() {
+				NodePlan::Leaf { value, .. } |
+				NodePlan::NibbledBranch { value: Some(value), .. } =>
+					if let ValuePlan::Inline(range) = value {
+						if (range.end - range.start) as u32 >= threshold {
+							nb_to_migrate += 1;
+						}
+					},
+				_ => (),
+			}
+		}
+
+		let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
+		// get all child trie roots
+		for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
+			let (key, value) =
+				key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+			if key[..]
+				.starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
+			{
+				let prefixed_key = PrefixedStorageKey::new(key);
+				let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
+				child_roots.push((ChildInfo::new_default(unprefixed), value));
+			}
+		}
+		for (child_info, root) in child_roots {
+			let mut child_root = H::Out::default();
+			let storage = KeySpacedDB::new(self, child_info.keyspace());
+
+			child_root.as_mut()[..].copy_from_slice(&root[..]);
+			let trie = sp_trie::trie_types::TrieDB::new(&storage, &child_root)
+				.map_err(|e| format!("New child TrieDB error: {}", e))?;
+			let iter_node = TrieDBNodeIterator::new(&trie)
+				.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+			for node in iter_node {
+				let node = node.map_err(|e| format!("Child TrieDB node iterator error: {}", e))?;
+				match node.2.node_plan() {
+					NodePlan::Leaf { value, .. } |
+					NodePlan::NibbledBranch { value: Some(value), .. } =>
+						if let ValuePlan::Inline(range) = value {
+							if (range.end - range.start) as u32 >= threshold {
+								nb_to_migrate_child += 1;
+							}
+						},
+					_ => (),
+				}
+			}
+		}
+
+		Ok((nb_to_migrate, nb_to_migrate_child))
+	}
+
 	/// Returns all `(key, value)` pairs in the trie.
 	pub fn pairs(&self) -> Vec<(StorageKey, StorageValue)> {
 		let collect_all = || -> sp_std::result::Result<_, Box<TrieError<H::Out>>> {
@@ -119,8 +119,7 @@ impl DerefMut for PrefixedStorageKey {
 }

 impl PrefixedStorageKey {
-	/// Create a prefixed storage key from its byte array
-	/// representation.
+	/// Create a prefixed storage key from its byte array representation.
 	pub fn new(inner: Vec<u8>) -> Self {
 		PrefixedStorageKey(inner)
 	}
@@ -130,9 +129,7 @@ impl PrefixedStorageKey {
 		PrefixedStorageKey::ref_cast(inner)
 	}

-	/// Get inner key, this should
-	/// only be needed when writing
-	/// into parent trie to avoid an
+	/// Get inner key, this should only be needed when writing into parent trie to avoid an
 	/// allocation.
 	pub fn into_inner(self) -> Vec<u8> {
 		self.0
@@ -171,10 +168,8 @@ pub struct StorageChild {
 pub struct Storage {
 	/// Top trie storage data.
 	pub top: StorageMap,
-	/// Children trie storage data.
-	/// The key does not including prefix, for the `default`
-	/// trie kind, so this is exclusively for the `ChildType::ParentKeyId`
-	/// tries.
+	/// Children trie storage data. Key does not include prefix, only for the `default` trie kind,
+	/// of `ChildType::ParentKeyId` type.
 	pub children_default: std::collections::HashMap<Vec<u8>, StorageChild>,
 }

@@ -40,7 +40,7 @@ use sp_core::{
 	},
 };
 pub use sp_io::TestExternalities;
-use sp_runtime::traits::Block as BlockT;
+use sp_runtime::{traits::Block as BlockT, StateVersion};
 use std::{
 	fs,
 	path::{Path, PathBuf},
@@ -56,6 +56,7 @@ type ChildKeyValues = Vec<(ChildInfo, Vec<KeyValue>)>;
 const LOG_TARGET: &str = "remote-ext";
 const DEFAULT_TARGET: &str = "wss://rpc.polkadot.io:443";
 const BATCH_SIZE: usize = 1000;
+const PAGE: u32 = 512;

 #[rpc(client)]
 pub trait RpcApi<Hash> {
@@ -117,12 +118,6 @@ pub struct OfflineConfig {
 	pub state_snapshot: SnapshotConfig,
 }

-impl<P: Into<PathBuf>> From<P> for SnapshotConfig {
-	fn from(p: P) -> Self {
-		Self { path: p.into() }
-	}
-}
-
 /// Description of the transport protocol (for online execution).
 #[derive(Debug, Clone)]
 pub enum Transport {
@@ -187,6 +182,8 @@ pub struct OnlineConfig<B: BlockT> {
 	pub pallets: Vec<String>,
 	/// Transport config.
 	pub transport: Transport,
+	/// Lookout for child-keys, and scrape them as well if set to true.
+	pub scrape_children: bool,
 }

 impl<B: BlockT> OnlineConfig<B> {
@@ -205,10 +202,17 @@ impl<B: BlockT> Default for OnlineConfig<B> {
 			at: None,
 			state_snapshot: None,
 			pallets: vec![],
+			scrape_children: true,
 		}
 	}
 }

+impl<B: BlockT> From<String> for OnlineConfig<B> {
+	fn from(s: String) -> Self {
+		Self { transport: s.into(), ..Default::default() }
+	}
+}
+
 /// Configuration of the state snapshot.
 #[derive(Clone)]
 pub struct SnapshotConfig {
@@ -222,6 +226,12 @@ impl SnapshotConfig {
 	}
 }

+impl From<String> for SnapshotConfig {
+	fn from(s: String) -> Self {
+		Self::new(s)
+	}
+}
+
 impl Default for SnapshotConfig {
 	fn default() -> Self {
 		Self { path: Path::new("SNAPSHOT").into() }
@@ -242,6 +252,8 @@ pub struct Builder<B: BlockT> {
 	hashed_blacklist: Vec<Vec<u8>>,
 	/// connectivity mode, online or offline.
 	mode: Mode<B>,
+	/// The state version being used.
+	state_version: StateVersion,
 }

 // NOTE: ideally we would use `DefaultNoBound` here, but not worth bringing in frame-support for
@@ -254,6 +266,7 @@ impl<B: BlockT + DeserializeOwned> Default for Builder<B> {
 			hashed_prefixes: Default::default(),
 			hashed_keys: Default::default(),
 			hashed_blacklist: Default::default(),
+			state_version: StateVersion::V1,
 		}
 	}
 }
@@ -306,7 +319,6 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 		prefix: StorageKey,
 		at: B::Hash,
 	) -> Result<Vec<StorageKey>, &'static str> {
-		const PAGE: u32 = 512;
 		let mut last_key: Option<StorageKey> = None;
 		let mut all_keys: Vec<StorageKey> = vec![];
 		let keys = loop {
@@ -320,6 +332,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 					"rpc get_keys failed"
 				})?;
 			let page_len = page.len();
+
 			all_keys.extend(page);

 			if page_len < PAGE as usize {
@@ -362,11 +375,12 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 				.cloned()
 				.map(|key| ("state_getStorage", rpc_params![key, at]))
 				.collect::<Vec<_>>();
+
 			let values = client.batch_request::<Option<StorageData>>(batch).await.map_err(|e| {
 				log::error!(
 					target: LOG_TARGET,
 					"failed to execute batch: {:?}. Error: {:?}",
-					chunk_keys,
+					chunk_keys.iter().map(|k| HexDisplay::from(k)).collect::<Vec<_>>(),
 					e
 				);
 				"batch failed."
@@ -693,7 +707,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {

 		// inject manual key values.
 		if !self.hashed_key_values.is_empty() {
-			log::debug!(
+			log::info!(
 				target: LOG_TARGET,
 				"extending externalities with {} manually injected key-values",
 				self.hashed_key_values.len()
@@ -703,7 +717,7 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {

 		// exclude manual key values.
 		if !self.hashed_blacklist.is_empty() {
-			log::debug!(
+			log::info!(
 				target: LOG_TARGET,
 				"excluding externalities from {} keys",
 				self.hashed_blacklist.len()
@@ -795,6 +809,12 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {
 		self
 	}

+	/// The state version to use.
+	pub fn state_version(mut self, version: StateVersion) -> Self {
+		self.state_version = version;
+		self
+	}
+
 	/// overwrite the `at` value, if `mode` is set to [`Mode::Online`].
 	///
 	/// noop if `mode` is [`Mode::Offline`]
@@ -808,8 +828,13 @@ impl<B: BlockT + DeserializeOwned> Builder<B> {

 	/// Build the test externalities.
 	pub async fn build(self) -> Result<TestExternalities, &'static str> {
+		let state_version = self.state_version;
 		let (top_kv, child_kv) = self.pre_build().await?;
-		let mut ext = TestExternalities::new_with_code(Default::default(), Default::default());
+		let mut ext = TestExternalities::new_with_code_and_state(
+			Default::default(),
+			Default::default(),
+			state_version,
+		);

 		info!(target: LOG_TARGET, "injecting a total of {} top keys", top_kv.len());
 		for (k, v) in top_kv {
@@ -1165,4 +1190,21 @@ mod remote_tests {
 			std::fs::remove_file(d.path()).unwrap();
 		}
 	}
+
+	#[tokio::test]
+	async fn can_build_child_tree() {
+		init_logger();
+		Builder::<Block>::new()
+			.mode(Mode::Online(OnlineConfig {
+				// transport: "wss://kusama-rpc.polkadot.io".to_owned().into(),
+				transport: "ws://kianenigma-archive:9924".to_owned().into(),
+				// transport: "ws://localhost:9999".to_owned().into(),
+				pallets: vec!["Crowdloan".to_owned()],
+				..Default::default()
+			}))
+			.build()
+			.await
+			.expect(REMOTE_INACCESSIBLE)
+			.execute_with(|| {});
+	}
 }
@@ -493,6 +493,7 @@ impl State {
 						transport: uri.to_owned().into(),
 						state_snapshot: snapshot_path.as_ref().map(SnapshotConfig::new),
 						pallets: pallets.clone().unwrap_or_default(),
+						scrape_children: true,
 						at,
 					}))
 					.inject_hashed_key(