State migration rpc (#10981)

* setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right logic to avoid switch on reencode when default layout. * Clean up some todos * remove trie meta from root upstream * update upstream and fix benches. * split some long lines. * UPdate trie crate to work with new design. * Finish update to refactored upstream. * update to latest triedb changes. * Clean up. * fix executor test. * rust fmt from master. * rust format. * rustfmt * fix * start host function driven versioning * update state-machine part * still need access to state version from runtime * state hash in mem: wrong * direction likely correct, but passing call to code exec for genesis init seem awkward. * state version serialize in runtime, wrong approach, just initialize it with no threshold for core api < 4 seems more proper. * stateversion from runtime version (core api >= 4). * update trie, fix tests * unused import * clean some TODOs * Require RuntimeVersionOf for executor * use RuntimeVersionOf to resolve genesis state version. * update runtime version test * fix state-machine tests * TODO * Use runtime version from storage wasm with fast sync. * rustfmt * fmt * fix test * revert useless changes. * clean some unused changes * fmt * removing useless trait function. * remove remaining reference to state_hash * fix some imports * Follow chain state version management. * trie update, fix and constant threshold for trie layouts. * update deps * Update to latest trie pr changes. * fix benches * Verify proof requires right layout. * update trie_root * Update trie deps to latest * Update to latest trie versioning * Removing patch * update lock * extrinsic for sc-service-test using layout v0. * Adding RuntimeVersionOf to CallExecutor works. * fmt * error when resolving version and no wasm in storage. * use existing utils to instantiate runtime code. * migration pallet * Patch to delay runtime switch. * Revert "Patch to delay runtime switch." This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da. * fix test * fix child migration calls. * useless closure * remove remaining state_hash variables. * Fix and add more tests * Remove outdated comment * useless inner hash * fmt * remote tests * finally ksm works * batches are broken * clean the benchmarks * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> * Update frame/state-trie-migration/src/lib.rs * brand new version * fix build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update primitives/storage/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * fmt and opt-in feature to apply state change. * feature gate core version, use new test feature for node and test node * Use a 'State' api version instead of Core one. * fix merge of test function * use blake macro. * Fix state api (require declaring the api in runtime). * Opt out feature, fix macro for io to select a given version instead of latest. * run test nodes on new state. * fix * new test structure * new testing stuff from emeric * Add commit_all, still not working * Fix all tests * add comment * we have PoV tracking baby * document stuff, but proof size is still wrong * FUCK YEAH * a big batch of review comments * add more tests * tweak test * update config * some remote-ext stuff * delete some of the old stuff * sync more files with master to minimize the diff * Fix all tests * make signed migration a bit more relaxed * add witness check to signed submissions * allow custom migration to also go above limit * Fix these pesky tests * ==== removal of the unsigned stuff ==== * Make all tests work again * separate the tests from the logic so it can be reused easier * fix overall build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Slightly better termination * some final tweaks * Fix tests * Restrict access to signed migrations * mig rpc * fix * better rpc name * Make rpc unsafe * address most of the review comments * fix defensive * New simplified code * Fix weights * fmt * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * make the tests correctly fail * Fix build * Fix build * try and fix the benchmarks * fix build * Fix cargo file * Fix runtime deposit * make rustdoc happy * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs * update rpc deps, try to process empty keys. * move rpc crate * move check backend out of state machine * Add primitive crate. * module code * fix runtime test * StateMigrationStatusProvider * Pass backend to rpc. * fmt * review changes * move rpc crate * try remove primitive crate * Update utils/frame/rpc/state-trie-migration-rpc/Cargo.toml Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * review changes. Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io>
2026-04-26 22:47:56 +00:00 · 2022-03-16 12:43:24 +01:00
parent 6acf5b963f
commit 800cc1d4be
12 changed files with 332 additions and 165 deletions
@@ -0,0 +1,38 @@
+[package]
+name = "substrate-state-trie-migration-rpc"
+version = "4.0.0-dev"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2021"
+license = "Apache-2.0"
+homepage = "https://substrate.io"
+repository = "https://github.com/paritytech/substrate/"
+description = "Node-specific RPC methods for interaction with state trie migration."
+readme = "README.md"
+
+[package.metadata.docs.rs]
+targets = ["x86_64-unknown-linux-gnu"]
+
+[dependencies]
+scale-info = { version = "2.0.1", default-features = false, features = ["derive"] }
+codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false }
+serde = { version = "1", features = ["derive"] }
+log = { version = "0.4.14", default-features = false }
+
+sp-std = { path = "../../../../primitives/std" }
+sp-io = { path = "../../../../primitives/io" }
+sp-core = { path = "../../../../primitives/core" }
+sp-state-machine = { path = "../../../../primitives/state-machine" }
+sp-trie = { path = "../../../../primitives/trie" }
+trie-db = { version = "0.23.1" }
+
+jsonrpc-core = "18.0.0"
+jsonrpc-core-client = "18.0.0"
+jsonrpc-derive = "18.0.0"
+
+# Substrate Dependencies
+sc-client-api = { version = "4.0.0-dev", path = "../../../../client/api" }
+sc-rpc-api = { version = "0.10.0-dev", path = "../../../../client/rpc-api" }
+sp-runtime = { version = "6.0.0", path = "../../../../primitives/runtime" }
+
+[dev-dependencies]
+serde_json = "1"
@@ -0,0 +1,3 @@
+Node-specific RPC methods for interaction with trie migration.
+
+License: Apache-2.0
@@ -0,0 +1,164 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 	http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Rpc for state migration.
+
+use jsonrpc_core::{Error, ErrorCode, Result};
+use jsonrpc_derive::rpc;
+use sc_rpc_api::DenyUnsafe;
+use serde::{Deserialize, Serialize};
+use sp_runtime::{generic::BlockId, traits::Block as BlockT};
+use std::sync::Arc;
+
+use sp_core::{
+	storage::{ChildInfo, ChildType, PrefixedStorageKey},
+	Hasher,
+};
+use sp_state_machine::Backend;
+use sp_trie::{trie_types::TrieDB, KeySpacedDB, Trie};
+use trie_db::{
+	node::{NodePlan, ValuePlan},
+	TrieDBNodeIterator,
+};
+
+fn count_migrate<'a, H: Hasher>(
+	storage: &'a dyn trie_db::HashDBRef<H, Vec<u8>>,
+	root: &'a H::Out,
+) -> std::result::Result<(u64, TrieDB<'a, H>), String> {
+	let mut nb = 0u64;
+	let trie = TrieDB::new(storage, root).map_err(|e| format!("TrieDB creation error: {}", e))?;
+	let iter_node =
+		TrieDBNodeIterator::new(&trie).map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+	for node in iter_node {
+		let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+		match node.2.node_plan() {
+			NodePlan::Leaf { value, .. } | NodePlan::NibbledBranch { value: Some(value), .. } =>
+				if let ValuePlan::Inline(range) = value {
+					if (range.end - range.start) as u32 >=
+						sp_core::storage::TRIE_VALUE_NODE_THRESHOLD
+					{
+						nb += 1;
+					}
+				},
+			_ => (),
+		}
+	}
+	Ok((nb, trie))
+}
+
+/// Check trie migration status.
+pub fn migration_status<H, B>(backend: &B) -> std::result::Result<(u64, u64), String>
+where
+	H: Hasher,
+	H::Out: codec::Codec,
+	B: Backend<H>,
+{
+	let trie_backend = if let Some(backend) = backend.as_trie_backend() {
+		backend
+	} else {
+		return Err("No access to trie from backend.".to_string())
+	};
+	let essence = trie_backend.essence();
+	let (nb_to_migrate, trie) = count_migrate(essence, &essence.root())?;
+
+	let mut nb_to_migrate_child = 0;
+	let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
+	// get all child trie roots
+	for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
+		let (key, value) = key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
+		if key[..].starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
+		{
+			let prefixed_key = PrefixedStorageKey::new(key);
+			let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
+			child_roots.push((ChildInfo::new_default(unprefixed), value));
+		}
+	}
+	for (child_info, root) in child_roots {
+		let mut child_root = H::Out::default();
+		let storage = KeySpacedDB::new(essence, child_info.keyspace());
+
+		child_root.as_mut()[..].copy_from_slice(&root[..]);
+		nb_to_migrate_child += count_migrate(&storage, &child_root)?.0;
+	}
+
+	Ok((nb_to_migrate, nb_to_migrate_child))
+}
+
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+#[serde(deny_unknown_fields)]
+pub struct MigrationStatusResult {
+	top_remaining_to_migrate: u64,
+	child_remaining_to_migrate: u64,
+}
+
+/// Migration RPC methods.
+#[rpc]
+pub trait StateMigrationApi<BlockHash> {
+	/// Check current migration state.
+	///
+	/// This call is performed locally without submitting any transactions. Thus executing this
+	/// won't change any state. Nonetheless it is a VERY costy call that should be
+	/// only exposed to trusted peers.
+	#[rpc(name = "state_trieMigrationStatus")]
+	fn call(&self, at: Option<BlockHash>) -> Result<MigrationStatusResult>;
+}
+
+/// An implementation of state migration specific RPC methods.
+pub struct MigrationRpc<C, B, BA> {
+	client: Arc<C>,
+	backend: Arc<BA>,
+	deny_unsafe: DenyUnsafe,
+	_marker: std::marker::PhantomData<(B, BA)>,
+}
+
+impl<C, B, BA> MigrationRpc<C, B, BA> {
+	/// Create new state migration rpc for the given reference to the client.
+	pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self {
+		MigrationRpc { client, backend, deny_unsafe, _marker: Default::default() }
+	}
+}
+
+impl<C, B, BA> StateMigrationApi<<B as BlockT>::Hash> for MigrationRpc<C, B, BA>
+where
+	B: BlockT,
+	C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>,
+	BA: 'static + sc_client_api::backend::Backend<B>,
+{
+	fn call(&self, at: Option<<B as BlockT>::Hash>) -> Result<MigrationStatusResult> {
+		if let Err(err) = self.deny_unsafe.check_if_safe() {
+			return Err(err.into())
+		}
+
+		let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash));
+		let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?;
+		let (top, child) = migration_status(&state).map_err(error_into_rpc_err)?;
+
+		Ok(MigrationStatusResult {
+			top_remaining_to_migrate: top,
+			child_remaining_to_migrate: child,
+		})
+	}
+}
+
+fn error_into_rpc_err(err: impl std::fmt::Display) -> Error {
+	Error {
+		code: ErrorCode::InternalError,
+		message: "Error while checking migration state".into(),
+		data: Some(err.to_string().into()),
+	}
+}