mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-05-07 22:28:02 +00:00
State migration rpc (#10981)
* setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right logic to avoid switch on reencode when default layout. * Clean up some todos * remove trie meta from root upstream * update upstream and fix benches. * split some long lines. * UPdate trie crate to work with new design. * Finish update to refactored upstream. * update to latest triedb changes. * Clean up. * fix executor test. * rust fmt from master. * rust format. * rustfmt * fix * start host function driven versioning * update state-machine part * still need access to state version from runtime * state hash in mem: wrong * direction likely correct, but passing call to code exec for genesis init seem awkward. * state version serialize in runtime, wrong approach, just initialize it with no threshold for core api < 4 seems more proper. * stateversion from runtime version (core api >= 4). * update trie, fix tests * unused import * clean some TODOs * Require RuntimeVersionOf for executor * use RuntimeVersionOf to resolve genesis state version. * update runtime version test * fix state-machine tests * TODO * Use runtime version from storage wasm with fast sync. * rustfmt * fmt * fix test * revert useless changes. * clean some unused changes * fmt * removing useless trait function. * remove remaining reference to state_hash * fix some imports * Follow chain state version management. * trie update, fix and constant threshold for trie layouts. * update deps * Update to latest trie pr changes. * fix benches * Verify proof requires right layout. * update trie_root * Update trie deps to latest * Update to latest trie versioning * Removing patch * update lock * extrinsic for sc-service-test using layout v0. * Adding RuntimeVersionOf to CallExecutor works. * fmt * error when resolving version and no wasm in storage. * use existing utils to instantiate runtime code. * migration pallet * Patch to delay runtime switch. * Revert "Patch to delay runtime switch." This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da. * fix test * fix child migration calls. * useless closure * remove remaining state_hash variables. * Fix and add more tests * Remove outdated comment * useless inner hash * fmt * remote tests * finally ksm works * batches are broken * clean the benchmarks * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> * Update frame/state-trie-migration/src/lib.rs * brand new version * fix build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update primitives/storage/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * fmt and opt-in feature to apply state change. * feature gate core version, use new test feature for node and test node * Use a 'State' api version instead of Core one. * fix merge of test function * use blake macro. * Fix state api (require declaring the api in runtime). * Opt out feature, fix macro for io to select a given version instead of latest. * run test nodes on new state. * fix * new test structure * new testing stuff from emeric * Add commit_all, still not working * Fix all tests * add comment * we have PoV tracking baby * document stuff, but proof size is still wrong * FUCK YEAH * a big batch of review comments * add more tests * tweak test * update config * some remote-ext stuff * delete some of the old stuff * sync more files with master to minimize the diff * Fix all tests * make signed migration a bit more relaxed * add witness check to signed submissions * allow custom migration to also go above limit * Fix these pesky tests * ==== removal of the unsigned stuff ==== * Make all tests work again * separate the tests from the logic so it can be reused easier * fix overall build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Slightly better termination * some final tweaks * Fix tests * Restrict access to signed migrations * mig rpc * fix * better rpc name * Make rpc unsafe * address most of the review comments * fix defensive * New simplified code * Fix weights * fmt * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * make the tests correctly fail * Fix build * Fix build * try and fix the benchmarks * fix build * Fix cargo file * Fix runtime deposit * make rustdoc happy * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs * update rpc deps, try to process empty keys. * move rpc crate * move check backend out of state machine * Add primitive crate. * module code * fix runtime test * StateMigrationStatusProvider * Pass backend to rpc. * fmt * review changes * move rpc crate * try remove primitive crate * Update utils/frame/rpc/state-trie-migration-rpc/Cargo.toml Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * review changes. Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io>
This commit is contained in:
Generated
+25
-1
@@ -4987,6 +4987,7 @@ dependencies = [
|
||||
"sp-keystore",
|
||||
"sp-runtime",
|
||||
"substrate-frame-rpc-system",
|
||||
"substrate-state-trie-migration-rpc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -6472,6 +6473,7 @@ dependencies = [
|
||||
"sp-runtime",
|
||||
"sp-std",
|
||||
"sp-tracing",
|
||||
"substrate-state-trie-migration-rpc",
|
||||
"thousands",
|
||||
"tokio",
|
||||
"zstd",
|
||||
@@ -10300,7 +10302,6 @@ dependencies = [
|
||||
"sp-trie",
|
||||
"thiserror",
|
||||
"tracing",
|
||||
"trie-db",
|
||||
"trie-root",
|
||||
]
|
||||
|
||||
@@ -10628,6 +10629,29 @@ dependencies = [
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "substrate-state-trie-migration-rpc"
|
||||
version = "4.0.0-dev"
|
||||
dependencies = [
|
||||
"jsonrpc-core",
|
||||
"jsonrpc-core-client",
|
||||
"jsonrpc-derive",
|
||||
"log 0.4.14",
|
||||
"parity-scale-codec",
|
||||
"sc-client-api",
|
||||
"sc-rpc-api",
|
||||
"scale-info",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sp-core",
|
||||
"sp-io",
|
||||
"sp-runtime",
|
||||
"sp-state-machine",
|
||||
"sp-std",
|
||||
"sp-trie",
|
||||
"trie-db",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "substrate-test-client"
|
||||
version = "2.0.1"
|
||||
|
||||
@@ -119,6 +119,7 @@ members = [
|
||||
"frame/staking",
|
||||
"frame/staking/reward-curve",
|
||||
"frame/staking/reward-fn",
|
||||
"frame/state-trie-migration",
|
||||
"frame/sudo",
|
||||
"frame/support",
|
||||
"frame/support/procedural",
|
||||
@@ -210,6 +211,7 @@ members = [
|
||||
"utils/frame/remote-externalities",
|
||||
"utils/frame/frame-utilities-cli",
|
||||
"utils/frame/try-runtime/cli",
|
||||
"utils/frame/rpc/state-trie-migration-rpc",
|
||||
"utils/frame/rpc/support",
|
||||
"utils/frame/rpc/system",
|
||||
"utils/frame/generate-bags",
|
||||
|
||||
@@ -252,6 +252,7 @@ pub fn new_partial(
|
||||
let keystore = keystore_container.sync_keystore();
|
||||
let chain_spec = config.chain_spec.cloned_box();
|
||||
|
||||
let rpc_backend = backend.clone();
|
||||
let rpc_extensions_builder = move |deny_unsafe, subscription_executor| {
|
||||
let deps = node_rpc::FullDeps {
|
||||
client: client.clone(),
|
||||
@@ -273,7 +274,7 @@ pub fn new_partial(
|
||||
},
|
||||
};
|
||||
|
||||
node_rpc::create_full(deps).map_err(Into::into)
|
||||
node_rpc::create_full(deps, rpc_backend.clone()).map_err(Into::into)
|
||||
};
|
||||
|
||||
(rpc_extensions_builder, rpc_setup)
|
||||
|
||||
@@ -35,3 +35,4 @@ sp-consensus-babe = { version = "0.10.0-dev", path = "../../../primitives/consen
|
||||
sp-runtime = { version = "6.0.0", path = "../../../primitives/runtime" }
|
||||
sc-transaction-pool-api = { version = "4.0.0-dev", path = "../../../client/transaction-pool/api" }
|
||||
substrate-frame-rpc-system = { version = "4.0.0-dev", path = "../../../utils/frame/rpc/system" }
|
||||
substrate-state-trie-migration-rpc = { version = "4.0.0-dev", path = "../../../utils/frame/rpc/state-trie-migration-rpc/" }
|
||||
|
||||
@@ -99,6 +99,7 @@ pub type IoHandler = jsonrpc_core::IoHandler<sc_rpc::Metadata>;
|
||||
/// Instantiate all Full RPC extensions.
|
||||
pub fn create_full<C, P, SC, B>(
|
||||
deps: FullDeps<C, P, SC, B>,
|
||||
backend: Arc<B>,
|
||||
) -> Result<jsonrpc_core::IoHandler<sc_rpc_api::Metadata>, Box<dyn std::error::Error + Send + Sync>>
|
||||
where
|
||||
C: ProvideRuntimeApi<Block>
|
||||
@@ -159,6 +160,10 @@ where
|
||||
finality_provider,
|
||||
)));
|
||||
|
||||
io.extend_with(substrate_state_trie_migration_rpc::StateMigrationApi::to_delegate(
|
||||
substrate_state_trie_migration_rpc::MigrationRpc::new(client.clone(), backend, deny_unsafe),
|
||||
));
|
||||
|
||||
io.extend_with(sc_sync_state_rpc::SyncStateRpcApi::to_delegate(
|
||||
sc_sync_state_rpc::SyncStateRpcHandler::new(
|
||||
chain_spec,
|
||||
|
||||
@@ -21,6 +21,7 @@ sp-std = { default-features = false, path = "../../primitives/std" }
|
||||
sp-io = { default-features = false, path = "../../primitives/io" }
|
||||
sp-core = { default-features = false, path = "../../primitives/core" }
|
||||
sp-runtime = { default-features = false, path = "../../primitives/runtime" }
|
||||
substrate-state-trie-migration-rpc = { optional = true, path = "../../utils/frame/rpc/state-trie-migration-rpc" }
|
||||
|
||||
frame-support = { default-features = false, path = "../support" }
|
||||
frame-system = { default-features = false, path = "../system" }
|
||||
@@ -49,9 +50,9 @@ std = [
|
||||
"sp-core/std",
|
||||
"sp-io/std",
|
||||
"sp-runtime/std",
|
||||
"sp-std/std"
|
||||
"sp-std/std",
|
||||
]
|
||||
runtime-benchmarks = ["frame-benchmarking"]
|
||||
try-runtime = ["frame-support/try-runtime"]
|
||||
|
||||
remote-test = [ "std", "zstd", "serde", "thousands", "remote-externalities" ]
|
||||
remote-test = [ "std", "zstd", "serde", "thousands", "remote-externalities", "substrate-state-trie-migration-rpc" ]
|
||||
|
||||
@@ -123,6 +123,13 @@ pub mod pallet {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Encode, Decode, scale_info::TypeInfo, PartialEq, Eq)]
|
||||
pub(crate) enum Progress {
|
||||
ToStart,
|
||||
LastKey(Vec<u8>),
|
||||
Complete,
|
||||
}
|
||||
|
||||
/// A migration task stored in state.
|
||||
///
|
||||
/// It tracks the last top and child keys read.
|
||||
@@ -130,20 +137,13 @@ pub mod pallet {
|
||||
#[codec(mel_bound(T: Config))]
|
||||
#[scale_info(skip_type_params(T))]
|
||||
pub struct MigrationTask<T: Config> {
|
||||
/// The last top key that we migrated.
|
||||
/// The current top trie migration progress.
|
||||
pub(crate) progress_top: Progress,
|
||||
/// The current child trie migration progress.
|
||||
///
|
||||
/// If it does not exist, it means that the migration is done and no further keys exist.
|
||||
pub(crate) last_top: Option<Vec<u8>>,
|
||||
/// The last child key that we have processed.
|
||||
///
|
||||
/// This is a child key under the current `self.last_top`.
|
||||
///
|
||||
/// If this is set, no further top keys are processed until the child key migration is
|
||||
/// complete.
|
||||
pub(crate) last_child: Option<Vec<u8>>,
|
||||
|
||||
/// A marker to indicate if the previous tick was a child tree migration or not.
|
||||
pub(crate) prev_tick_child: bool,
|
||||
/// If `ToStart`, no further top keys are processed until the child key migration is
|
||||
/// `Complete`.
|
||||
pub(crate) progress_child: Progress,
|
||||
|
||||
/// Dynamic counter for the number of items that we have processed in this execution from
|
||||
/// the top trie.
|
||||
@@ -182,18 +182,22 @@ pub mod pallet {
|
||||
pub(crate) _ph: sp_std::marker::PhantomData<T>,
|
||||
}
|
||||
|
||||
impl sp_std::fmt::Debug for Progress {
|
||||
fn fmt(&self, f: &mut sp_std::fmt::Formatter<'_>) -> sp_std::fmt::Result {
|
||||
match self {
|
||||
Progress::ToStart => f.write_str("To start"),
|
||||
Progress::LastKey(key) =>
|
||||
write!(f, "Last: {:?}", sp_core::hexdisplay::HexDisplay::from(key)),
|
||||
Progress::Complete => f.write_str("Complete"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<T: Config> sp_std::fmt::Debug for MigrationTask<T> {
|
||||
fn fmt(&self, f: &mut sp_std::fmt::Formatter<'_>) -> sp_std::fmt::Result {
|
||||
f.debug_struct("MigrationTask")
|
||||
.field(
|
||||
"top",
|
||||
&self.last_top.as_ref().map(|d| sp_core::hexdisplay::HexDisplay::from(d)),
|
||||
)
|
||||
.field(
|
||||
"child",
|
||||
&self.last_child.as_ref().map(|d| sp_core::hexdisplay::HexDisplay::from(d)),
|
||||
)
|
||||
.field("prev_tick_child", &self.prev_tick_child)
|
||||
.field("top", &self.progress_top)
|
||||
.field("child", &self.progress_child)
|
||||
.field("dyn_top_items", &self.dyn_top_items)
|
||||
.field("dyn_child_items", &self.dyn_child_items)
|
||||
.field("dyn_size", &self.dyn_size)
|
||||
@@ -207,12 +211,11 @@ pub mod pallet {
|
||||
impl<T: Config> Default for MigrationTask<T> {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
last_top: Some(Default::default()),
|
||||
last_child: Default::default(),
|
||||
progress_top: Progress::ToStart,
|
||||
progress_child: Progress::ToStart,
|
||||
dyn_child_items: Default::default(),
|
||||
dyn_top_items: Default::default(),
|
||||
dyn_size: Default::default(),
|
||||
prev_tick_child: Default::default(),
|
||||
_ph: Default::default(),
|
||||
size: Default::default(),
|
||||
top_items: Default::default(),
|
||||
@@ -224,7 +227,7 @@ pub mod pallet {
|
||||
impl<T: Config> MigrationTask<T> {
|
||||
/// Return true if the task is finished.
|
||||
pub(crate) fn finished(&self) -> bool {
|
||||
self.last_top.is_none() && self.last_child.is_none()
|
||||
matches!(self.progress_top, Progress::Complete)
|
||||
}
|
||||
|
||||
/// Check if there's any work left, or if we have exhausted the limits already.
|
||||
@@ -269,51 +272,36 @@ pub mod pallet {
|
||||
///
|
||||
/// This function is *the* core of this entire pallet.
|
||||
fn migrate_tick(&mut self) {
|
||||
match (self.last_top.as_ref(), self.last_child.as_ref()) {
|
||||
(Some(_), Some(_)) => {
|
||||
match (&self.progress_top, &self.progress_child) {
|
||||
(Progress::ToStart, _) => {
|
||||
self.migrate_top();
|
||||
},
|
||||
(Progress::LastKey(_), Progress::LastKey(_)) => {
|
||||
// we're in the middle of doing work on a child tree.
|
||||
self.migrate_child();
|
||||
},
|
||||
(Some(ref top_key), None) => {
|
||||
// we have a top key and no child key. 3 possibilities exist:
|
||||
// 1. we continue the top key migrations.
|
||||
// 2. this is the root of a child key, and we start processing child keys (and
|
||||
// should call `migrate_child`).
|
||||
(Progress::LastKey(top_key), Progress::ToStart) => {
|
||||
// 3. this is the root of a child key, and we are finishing all child-keys (and
|
||||
// should call `migrate_top`).
|
||||
|
||||
// NOTE: this block is written intentionally to verbosely for easy of
|
||||
// verification.
|
||||
match (
|
||||
top_key.starts_with(DEFAULT_CHILD_STORAGE_KEY_PREFIX),
|
||||
self.prev_tick_child,
|
||||
) {
|
||||
(false, false) => {
|
||||
// continue the top key migration
|
||||
self.migrate_top();
|
||||
},
|
||||
(true, false) => {
|
||||
self.last_child = Some(Default::default());
|
||||
self.migrate_child();
|
||||
self.prev_tick_child = true;
|
||||
},
|
||||
(true, true) => {
|
||||
// we're done with migrating a child-root.
|
||||
self.prev_tick_child = false;
|
||||
self.migrate_top();
|
||||
},
|
||||
(false, true) => {
|
||||
// should never happen.
|
||||
log!(error, "LOGIC ERROR: unreachable code [0].");
|
||||
Pallet::<T>::halt();
|
||||
},
|
||||
};
|
||||
if !top_key.starts_with(DEFAULT_CHILD_STORAGE_KEY_PREFIX) {
|
||||
// we continue the top key migrations.
|
||||
// continue the top key migration
|
||||
self.migrate_top();
|
||||
} else {
|
||||
// this is the root of a child key, and we start processing child keys (and
|
||||
// should call `migrate_child`).
|
||||
self.migrate_child();
|
||||
}
|
||||
},
|
||||
(None, Some(_)) => {
|
||||
log!(error, "LOGIC ERROR: unreachable code [1].");
|
||||
Pallet::<T>::halt()
|
||||
(Progress::LastKey(_), Progress::Complete) => {
|
||||
// we're done with migrating a child-root.
|
||||
self.migrate_top();
|
||||
self.progress_child = Progress::ToStart;
|
||||
},
|
||||
(None, None) => {
|
||||
(Progress::Complete, _) => {
|
||||
// nada
|
||||
},
|
||||
}
|
||||
@@ -324,19 +312,26 @@ pub mod pallet {
|
||||
/// It updates the dynamic counters.
|
||||
fn migrate_child(&mut self) {
|
||||
use sp_io::default_child_storage as child_io;
|
||||
let (last_child, last_top) = match (&self.last_child, &self.last_top) {
|
||||
(Some(last_child), Some(last_top)) => (last_child, last_top),
|
||||
let (maybe_current_child, child_root) = match (&self.progress_child, &self.progress_top)
|
||||
{
|
||||
(Progress::LastKey(last_child), Progress::LastKey(last_top)) => {
|
||||
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
|
||||
let maybe_current_child = child_io::next_key(child_root, &last_child);
|
||||
(maybe_current_child, child_root)
|
||||
},
|
||||
(Progress::ToStart, Progress::LastKey(last_top)) => {
|
||||
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
|
||||
// Start with the empty key as first key.
|
||||
(Some(Vec::new()), child_root)
|
||||
},
|
||||
_ => {
|
||||
// defensive: this function is only called when both of these values exist.
|
||||
// much that we can do otherwise..
|
||||
// defensive: there must be an ongoing top migration.
|
||||
frame_support::defensive!("cannot migrate child key.");
|
||||
return
|
||||
},
|
||||
};
|
||||
|
||||
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
|
||||
let maybe_current_child = child_io::next_key(child_root, &last_child);
|
||||
if let Some(ref current_child) = maybe_current_child {
|
||||
if let Some(current_child) = maybe_current_child.as_ref() {
|
||||
let added_size = if let Some(data) = child_io::get(child_root, ¤t_child) {
|
||||
child_io::set(child_root, current_child, &data);
|
||||
data.len() as u32
|
||||
@@ -348,25 +343,28 @@ pub mod pallet {
|
||||
}
|
||||
|
||||
log!(trace, "migrated a child key, next_child_key: {:?}", maybe_current_child);
|
||||
self.last_child = maybe_current_child;
|
||||
self.progress_child = match maybe_current_child {
|
||||
Some(last_child) => Progress::LastKey(last_child),
|
||||
None => Progress::Complete,
|
||||
}
|
||||
}
|
||||
|
||||
/// Migrate the current top key, setting it to its new value, if one exists.
|
||||
///
|
||||
/// It updates the dynamic counters.
|
||||
fn migrate_top(&mut self) {
|
||||
let last_top = match &self.last_top {
|
||||
Some(last_top) => last_top,
|
||||
None => {
|
||||
// defensive: this function is only called when this value exist.
|
||||
// much that we can do otherwise..
|
||||
let maybe_current_top = match &self.progress_top {
|
||||
Progress::LastKey(last_top) => sp_io::storage::next_key(last_top),
|
||||
// Start with the empty key as first key.
|
||||
Progress::ToStart => Some(Vec::new()),
|
||||
Progress::Complete => {
|
||||
// defensive: there must be an ongoing top migration.
|
||||
frame_support::defensive!("cannot migrate top key.");
|
||||
return
|
||||
},
|
||||
};
|
||||
|
||||
let maybe_current_top = sp_io::storage::next_key(last_top);
|
||||
if let Some(ref current_top) = maybe_current_top {
|
||||
if let Some(current_top) = maybe_current_top.as_ref() {
|
||||
let added_size = if let Some(data) = sp_io::storage::get(¤t_top) {
|
||||
sp_io::storage::set(¤t_top, &data);
|
||||
data.len() as u32
|
||||
@@ -378,7 +376,10 @@ pub mod pallet {
|
||||
}
|
||||
|
||||
log!(trace, "migrated a top key, next_top_key = {:?}", maybe_current_top);
|
||||
self.last_top = maybe_current_top;
|
||||
self.progress_top = match maybe_current_top {
|
||||
Some(last_top) => Progress::LastKey(last_top),
|
||||
None => Progress::Complete,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -811,7 +812,7 @@ mod benchmarks {
|
||||
continue_migrate_wrong_witness {
|
||||
let null = MigrationLimits::default();
|
||||
let caller = frame_benchmarking::whitelisted_caller();
|
||||
let bad_witness = MigrationTask { last_top: Some(vec![1u8]), ..Default::default() };
|
||||
let bad_witness = MigrationTask { progress_top: Progress::LastKey(vec![1u8]), ..Default::default() };
|
||||
}: {
|
||||
assert!(
|
||||
StateTrieMigration::<T>::continue_migrate(
|
||||
@@ -1141,15 +1142,12 @@ mod test {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn detects_value_in_empty_top_key() {
|
||||
let limit = MigrationLimits { item: 1, size: 1000 };
|
||||
let initial_keys = Some(vec![(vec![], vec![66u8; 77])]);
|
||||
let mut ext = new_test_ext(StateVersion::V0, false, initial_keys.clone(), None);
|
||||
|
||||
let root_upgraded = ext.execute_with(|| {
|
||||
sp_io::storage::set(&[], &vec![66u8; 77]);
|
||||
|
||||
AutoLimits::<Test>::put(Some(limit));
|
||||
let root = run_to_block(30).0;
|
||||
|
||||
@@ -1168,9 +1166,7 @@ mod test {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn detects_value_in_first_child_key() {
|
||||
use frame_support::storage::child;
|
||||
let limit = MigrationLimits { item: 1, size: 1000 };
|
||||
let initial_child = Some(vec![(b"chk1".to_vec(), vec![], vec![66u8; 77])]);
|
||||
let mut ext = new_test_ext(StateVersion::V0, false, None, initial_child.clone());
|
||||
@@ -1186,7 +1182,6 @@ mod test {
|
||||
|
||||
let mut ext2 = new_test_ext(StateVersion::V1, false, None, initial_child);
|
||||
let root = ext2.execute_with(|| {
|
||||
child::put(&child::ChildInfo::new_default(b"chk1"), &[], &vec![66u8; 77]);
|
||||
AutoLimits::<Test>::put(Some(limit));
|
||||
run_to_block(30).0
|
||||
});
|
||||
@@ -1214,7 +1209,7 @@ mod test {
|
||||
// eventually everything is over.
|
||||
assert!(matches!(
|
||||
StateTrieMigration::migration_process(),
|
||||
MigrationTask { last_child: None, last_top: None, .. }
|
||||
MigrationTask { progress_top: Progress::Complete, .. }
|
||||
));
|
||||
root
|
||||
});
|
||||
@@ -1276,7 +1271,10 @@ mod test {
|
||||
Origin::signed(1),
|
||||
MigrationLimits { item: 5, size: 100 },
|
||||
100,
|
||||
MigrationTask { last_top: Some(vec![1u8]), ..Default::default() }
|
||||
MigrationTask {
|
||||
progress_top: Progress::LastKey(vec![1u8]),
|
||||
..Default::default()
|
||||
}
|
||||
),
|
||||
Error::<Test>::BadWitness
|
||||
);
|
||||
@@ -1451,7 +1449,8 @@ pub(crate) mod remote_tests {
|
||||
// set the version to 1, as if the upgrade happened.
|
||||
ext.state_version = sp_core::storage::StateVersion::V1;
|
||||
|
||||
let (top_left, child_left) = ext.as_backend().essence().check_migration_state().unwrap();
|
||||
let (top_left, child_left) =
|
||||
substrate_state_trie_migration_rpc::migration_status(&ext.as_backend()).unwrap();
|
||||
assert!(
|
||||
top_left > 0,
|
||||
"no node needs migrating, this probably means that state was initialized with `StateVersion::V1`",
|
||||
@@ -1509,7 +1508,8 @@ pub(crate) mod remote_tests {
|
||||
)
|
||||
});
|
||||
|
||||
let (top_left, child_left) = ext.as_backend().essence().check_migration_state().unwrap();
|
||||
let (top_left, child_left) =
|
||||
substrate_state_trie_migration_rpc::migration_status(&ext.as_backend()).unwrap();
|
||||
assert_eq!(top_left, 0);
|
||||
assert_eq!(child_left, 0);
|
||||
}
|
||||
|
||||
@@ -18,7 +18,6 @@ log = { version = "0.4.11", optional = true }
|
||||
thiserror = { version = "1.0.30", optional = true }
|
||||
parking_lot = { version = "0.12.0", optional = true }
|
||||
hash-db = { version = "0.15.2", default-features = false }
|
||||
trie-db = { version = "0.23.1", default-features = false }
|
||||
trie-root = { version = "0.17.0", default-features = false }
|
||||
sp-trie = { version = "6.0.0", path = "../trie", default-features = false }
|
||||
sp-core = { version = "6.0.0", path = "../core", default-features = false }
|
||||
@@ -47,7 +46,6 @@ std = [
|
||||
"sp-externalities/std",
|
||||
"sp-std/std",
|
||||
"sp-trie/std",
|
||||
"trie-db/std",
|
||||
"trie-root/std",
|
||||
"log",
|
||||
"thiserror",
|
||||
|
||||
@@ -23,7 +23,7 @@ use codec::Encode;
|
||||
use hash_db::{self, AsHashDB, HashDB, HashDBRef, Hasher, Prefix};
|
||||
#[cfg(feature = "std")]
|
||||
use parking_lot::RwLock;
|
||||
use sp_core::storage::{ChildInfo, ChildType, PrefixedStorageKey, StateVersion};
|
||||
use sp_core::storage::{ChildInfo, ChildType, StateVersion};
|
||||
use sp_std::{boxed::Box, vec::Vec};
|
||||
use sp_trie::{
|
||||
child_delta_trie_root, delta_trie_root, empty_child_trie_root, read_child_trie_value,
|
||||
@@ -36,10 +36,6 @@ use sp_trie::{
|
||||
use std::collections::HashMap;
|
||||
#[cfg(feature = "std")]
|
||||
use std::sync::Arc;
|
||||
use trie_db::{
|
||||
node::{NodePlan, ValuePlan},
|
||||
TrieDBNodeIterator,
|
||||
};
|
||||
|
||||
#[cfg(not(feature = "std"))]
|
||||
macro_rules! format {
|
||||
@@ -433,72 +429,6 @@ where
|
||||
);
|
||||
}
|
||||
|
||||
/// Check remaining state item to migrate. Note this function should be remove when all state
|
||||
/// migration did finished as it is only an utility.
|
||||
// original author: @cheme
|
||||
pub fn check_migration_state(&self) -> Result<(u64, u64)> {
|
||||
let threshold: u32 = sp_core::storage::TRIE_VALUE_NODE_THRESHOLD;
|
||||
let mut nb_to_migrate = 0;
|
||||
let mut nb_to_migrate_child = 0;
|
||||
|
||||
let trie = sp_trie::trie_types::TrieDB::new(self, &self.root)
|
||||
.map_err(|e| format!("TrieDB creation error: {}", e))?;
|
||||
let iter_node = TrieDBNodeIterator::new(&trie)
|
||||
.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
for node in iter_node {
|
||||
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
match node.2.node_plan() {
|
||||
NodePlan::Leaf { value, .. } |
|
||||
NodePlan::NibbledBranch { value: Some(value), .. } =>
|
||||
if let ValuePlan::Inline(range) = value {
|
||||
if (range.end - range.start) as u32 >= threshold {
|
||||
nb_to_migrate += 1;
|
||||
}
|
||||
},
|
||||
_ => (),
|
||||
}
|
||||
}
|
||||
|
||||
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
|
||||
// get all child trie roots
|
||||
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
|
||||
let (key, value) =
|
||||
key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
if key[..]
|
||||
.starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
|
||||
{
|
||||
let prefixed_key = PrefixedStorageKey::new(key);
|
||||
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
|
||||
child_roots.push((ChildInfo::new_default(unprefixed), value));
|
||||
}
|
||||
}
|
||||
for (child_info, root) in child_roots {
|
||||
let mut child_root = H::Out::default();
|
||||
let storage = KeySpacedDB::new(self, child_info.keyspace());
|
||||
|
||||
child_root.as_mut()[..].copy_from_slice(&root[..]);
|
||||
let trie = sp_trie::trie_types::TrieDB::new(&storage, &child_root)
|
||||
.map_err(|e| format!("New child TrieDB error: {}", e))?;
|
||||
let iter_node = TrieDBNodeIterator::new(&trie)
|
||||
.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
for node in iter_node {
|
||||
let node = node.map_err(|e| format!("Child TrieDB node iterator error: {}", e))?;
|
||||
match node.2.node_plan() {
|
||||
NodePlan::Leaf { value, .. } |
|
||||
NodePlan::NibbledBranch { value: Some(value), .. } =>
|
||||
if let ValuePlan::Inline(range) = value {
|
||||
if (range.end - range.start) as u32 >= threshold {
|
||||
nb_to_migrate_child += 1;
|
||||
}
|
||||
},
|
||||
_ => (),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok((nb_to_migrate, nb_to_migrate_child))
|
||||
}
|
||||
|
||||
/// Returns all `(key, value)` pairs in the trie.
|
||||
pub fn pairs(&self) -> Vec<(StorageKey, StorageValue)> {
|
||||
let collect_all = || -> sp_std::result::Result<_, Box<TrieError<H::Out>>> {
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
[package]
|
||||
name = "substrate-state-trie-migration-rpc"
|
||||
version = "4.0.0-dev"
|
||||
authors = ["Parity Technologies <admin@parity.io>"]
|
||||
edition = "2021"
|
||||
license = "Apache-2.0"
|
||||
homepage = "https://substrate.io"
|
||||
repository = "https://github.com/paritytech/substrate/"
|
||||
description = "Node-specific RPC methods for interaction with state trie migration."
|
||||
readme = "README.md"
|
||||
|
||||
[package.metadata.docs.rs]
|
||||
targets = ["x86_64-unknown-linux-gnu"]
|
||||
|
||||
[dependencies]
|
||||
scale-info = { version = "2.0.1", default-features = false, features = ["derive"] }
|
||||
codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
log = { version = "0.4.14", default-features = false }
|
||||
|
||||
sp-std = { path = "../../../../primitives/std" }
|
||||
sp-io = { path = "../../../../primitives/io" }
|
||||
sp-core = { path = "../../../../primitives/core" }
|
||||
sp-state-machine = { path = "../../../../primitives/state-machine" }
|
||||
sp-trie = { path = "../../../../primitives/trie" }
|
||||
trie-db = { version = "0.23.1" }
|
||||
|
||||
jsonrpc-core = "18.0.0"
|
||||
jsonrpc-core-client = "18.0.0"
|
||||
jsonrpc-derive = "18.0.0"
|
||||
|
||||
# Substrate Dependencies
|
||||
sc-client-api = { version = "4.0.0-dev", path = "../../../../client/api" }
|
||||
sc-rpc-api = { version = "0.10.0-dev", path = "../../../../client/rpc-api" }
|
||||
sp-runtime = { version = "6.0.0", path = "../../../../primitives/runtime" }
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = "1"
|
||||
@@ -0,0 +1,3 @@
|
||||
Node-specific RPC methods for interaction with trie migration.
|
||||
|
||||
License: Apache-2.0
|
||||
@@ -0,0 +1,164 @@
|
||||
// This file is part of Substrate.
|
||||
|
||||
// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Rpc for state migration.
|
||||
|
||||
use jsonrpc_core::{Error, ErrorCode, Result};
|
||||
use jsonrpc_derive::rpc;
|
||||
use sc_rpc_api::DenyUnsafe;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
|
||||
use std::sync::Arc;
|
||||
|
||||
use sp_core::{
|
||||
storage::{ChildInfo, ChildType, PrefixedStorageKey},
|
||||
Hasher,
|
||||
};
|
||||
use sp_state_machine::Backend;
|
||||
use sp_trie::{trie_types::TrieDB, KeySpacedDB, Trie};
|
||||
use trie_db::{
|
||||
node::{NodePlan, ValuePlan},
|
||||
TrieDBNodeIterator,
|
||||
};
|
||||
|
||||
fn count_migrate<'a, H: Hasher>(
|
||||
storage: &'a dyn trie_db::HashDBRef<H, Vec<u8>>,
|
||||
root: &'a H::Out,
|
||||
) -> std::result::Result<(u64, TrieDB<'a, H>), String> {
|
||||
let mut nb = 0u64;
|
||||
let trie = TrieDB::new(storage, root).map_err(|e| format!("TrieDB creation error: {}", e))?;
|
||||
let iter_node =
|
||||
TrieDBNodeIterator::new(&trie).map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
for node in iter_node {
|
||||
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
match node.2.node_plan() {
|
||||
NodePlan::Leaf { value, .. } | NodePlan::NibbledBranch { value: Some(value), .. } =>
|
||||
if let ValuePlan::Inline(range) = value {
|
||||
if (range.end - range.start) as u32 >=
|
||||
sp_core::storage::TRIE_VALUE_NODE_THRESHOLD
|
||||
{
|
||||
nb += 1;
|
||||
}
|
||||
},
|
||||
_ => (),
|
||||
}
|
||||
}
|
||||
Ok((nb, trie))
|
||||
}
|
||||
|
||||
/// Check trie migration status.
|
||||
pub fn migration_status<H, B>(backend: &B) -> std::result::Result<(u64, u64), String>
|
||||
where
|
||||
H: Hasher,
|
||||
H::Out: codec::Codec,
|
||||
B: Backend<H>,
|
||||
{
|
||||
let trie_backend = if let Some(backend) = backend.as_trie_backend() {
|
||||
backend
|
||||
} else {
|
||||
return Err("No access to trie from backend.".to_string())
|
||||
};
|
||||
let essence = trie_backend.essence();
|
||||
let (nb_to_migrate, trie) = count_migrate(essence, &essence.root())?;
|
||||
|
||||
let mut nb_to_migrate_child = 0;
|
||||
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
|
||||
// get all child trie roots
|
||||
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
|
||||
let (key, value) = key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
|
||||
if key[..].starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
|
||||
{
|
||||
let prefixed_key = PrefixedStorageKey::new(key);
|
||||
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
|
||||
child_roots.push((ChildInfo::new_default(unprefixed), value));
|
||||
}
|
||||
}
|
||||
for (child_info, root) in child_roots {
|
||||
let mut child_root = H::Out::default();
|
||||
let storage = KeySpacedDB::new(essence, child_info.keyspace());
|
||||
|
||||
child_root.as_mut()[..].copy_from_slice(&root[..]);
|
||||
nb_to_migrate_child += count_migrate(&storage, &child_root)?.0;
|
||||
}
|
||||
|
||||
Ok((nb_to_migrate, nb_to_migrate_child))
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct MigrationStatusResult {
|
||||
top_remaining_to_migrate: u64,
|
||||
child_remaining_to_migrate: u64,
|
||||
}
|
||||
|
||||
/// Migration RPC methods.
|
||||
#[rpc]
|
||||
pub trait StateMigrationApi<BlockHash> {
|
||||
/// Check current migration state.
|
||||
///
|
||||
/// This call is performed locally without submitting any transactions. Thus executing this
|
||||
/// won't change any state. Nonetheless it is a VERY costy call that should be
|
||||
/// only exposed to trusted peers.
|
||||
#[rpc(name = "state_trieMigrationStatus")]
|
||||
fn call(&self, at: Option<BlockHash>) -> Result<MigrationStatusResult>;
|
||||
}
|
||||
|
||||
/// An implementation of state migration specific RPC methods.
|
||||
pub struct MigrationRpc<C, B, BA> {
|
||||
client: Arc<C>,
|
||||
backend: Arc<BA>,
|
||||
deny_unsafe: DenyUnsafe,
|
||||
_marker: std::marker::PhantomData<(B, BA)>,
|
||||
}
|
||||
|
||||
impl<C, B, BA> MigrationRpc<C, B, BA> {
|
||||
/// Create new state migration rpc for the given reference to the client.
|
||||
pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self {
|
||||
MigrationRpc { client, backend, deny_unsafe, _marker: Default::default() }
|
||||
}
|
||||
}
|
||||
|
||||
impl<C, B, BA> StateMigrationApi<<B as BlockT>::Hash> for MigrationRpc<C, B, BA>
|
||||
where
|
||||
B: BlockT,
|
||||
C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>,
|
||||
BA: 'static + sc_client_api::backend::Backend<B>,
|
||||
{
|
||||
fn call(&self, at: Option<<B as BlockT>::Hash>) -> Result<MigrationStatusResult> {
|
||||
if let Err(err) = self.deny_unsafe.check_if_safe() {
|
||||
return Err(err.into())
|
||||
}
|
||||
|
||||
let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash));
|
||||
let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?;
|
||||
let (top, child) = migration_status(&state).map_err(error_into_rpc_err)?;
|
||||
|
||||
Ok(MigrationStatusResult {
|
||||
top_remaining_to_migrate: top,
|
||||
child_remaining_to_migrate: child,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn error_into_rpc_err(err: impl std::fmt::Display) -> Error {
|
||||
Error {
|
||||
code: ErrorCode::InternalError,
|
||||
message: "Error while checking migration state".into(),
|
||||
data: Some(err.to_string().into()),
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user