State migration rpc (#10981)

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* migration pallet

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* fix test

* fix child migration calls.

* useless closure

* remove remaining state_hash variables.

* Fix and add more tests

* Remove outdated comment

* useless inner hash

* fmt

* remote tests

* finally ksm works

* batches are broken

* clean the benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Update frame/state-trie-migration/src/lib.rs

* brand new version

* fix build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* new test structure

* new testing stuff from emeric

* Add commit_all, still not working

* Fix all tests

* add comment

* we have PoV tracking baby

* document stuff, but proof size is still wrong

* FUCK YEAH

* a big batch of review comments

* add more tests

* tweak test

* update config

* some remote-ext stuff

* delete some of the old stuff

* sync more files with master to minimize the diff

* Fix all tests

* make signed migration a bit more relaxed

* add witness check to signed submissions

* allow custom migration to also go above limit

* Fix these pesky tests

* ==== removal of the unsigned stuff ====

* Make all tests work again

* separate the tests from the logic so it can be reused easier

* fix overall build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Slightly better termination

* some final tweaks

* Fix tests

* Restrict access to signed migrations

* mig rpc

* fix

* better rpc name

* Make rpc unsafe

* address most of the review comments

* fix defensive

* New simplified code

* Fix weights

* fmt

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make the tests correctly fail

* Fix build

* Fix build

* try and fix the benchmarks

* fix build

* Fix cargo file

* Fix runtime deposit

* make rustdoc happy

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* update rpc deps, try to process empty keys.

* move rpc crate

* move check backend out of state machine

* Add primitive crate.

* module code

* fix runtime test

* StateMigrationStatusProvider

* Pass backend to rpc.

* fmt

* review changes

* move rpc crate

* try remove primitive crate

* Update utils/frame/rpc/state-trie-migration-rpc/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
This commit is contained in:
cheme
2022-03-16 12:43:24 +01:00
committed by GitHub
parent 6acf5b963f
commit 800cc1d4be
12 changed files with 332 additions and 165 deletions
+25 -1
View File
@@ -4987,6 +4987,7 @@ dependencies = [
"sp-keystore",
"sp-runtime",
"substrate-frame-rpc-system",
"substrate-state-trie-migration-rpc",
]
[[package]]
@@ -6472,6 +6473,7 @@ dependencies = [
"sp-runtime",
"sp-std",
"sp-tracing",
"substrate-state-trie-migration-rpc",
"thousands",
"tokio",
"zstd",
@@ -10300,7 +10302,6 @@ dependencies = [
"sp-trie",
"thiserror",
"tracing",
"trie-db",
"trie-root",
]
@@ -10628,6 +10629,29 @@ dependencies = [
"tokio",
]
[[package]]
name = "substrate-state-trie-migration-rpc"
version = "4.0.0-dev"
dependencies = [
"jsonrpc-core",
"jsonrpc-core-client",
"jsonrpc-derive",
"log 0.4.14",
"parity-scale-codec",
"sc-client-api",
"sc-rpc-api",
"scale-info",
"serde",
"serde_json",
"sp-core",
"sp-io",
"sp-runtime",
"sp-state-machine",
"sp-std",
"sp-trie",
"trie-db",
]
[[package]]
name = "substrate-test-client"
version = "2.0.1"
+2
View File
@@ -119,6 +119,7 @@ members = [
"frame/staking",
"frame/staking/reward-curve",
"frame/staking/reward-fn",
"frame/state-trie-migration",
"frame/sudo",
"frame/support",
"frame/support/procedural",
@@ -210,6 +211,7 @@ members = [
"utils/frame/remote-externalities",
"utils/frame/frame-utilities-cli",
"utils/frame/try-runtime/cli",
"utils/frame/rpc/state-trie-migration-rpc",
"utils/frame/rpc/support",
"utils/frame/rpc/system",
"utils/frame/generate-bags",
+2 -1
View File
@@ -252,6 +252,7 @@ pub fn new_partial(
let keystore = keystore_container.sync_keystore();
let chain_spec = config.chain_spec.cloned_box();
let rpc_backend = backend.clone();
let rpc_extensions_builder = move |deny_unsafe, subscription_executor| {
let deps = node_rpc::FullDeps {
client: client.clone(),
@@ -273,7 +274,7 @@ pub fn new_partial(
},
};
node_rpc::create_full(deps).map_err(Into::into)
node_rpc::create_full(deps, rpc_backend.clone()).map_err(Into::into)
};
(rpc_extensions_builder, rpc_setup)
+1
View File
@@ -35,3 +35,4 @@ sp-consensus-babe = { version = "0.10.0-dev", path = "../../../primitives/consen
sp-runtime = { version = "6.0.0", path = "../../../primitives/runtime" }
sc-transaction-pool-api = { version = "4.0.0-dev", path = "../../../client/transaction-pool/api" }
substrate-frame-rpc-system = { version = "4.0.0-dev", path = "../../../utils/frame/rpc/system" }
substrate-state-trie-migration-rpc = { version = "4.0.0-dev", path = "../../../utils/frame/rpc/state-trie-migration-rpc/" }
+5
View File
@@ -99,6 +99,7 @@ pub type IoHandler = jsonrpc_core::IoHandler<sc_rpc::Metadata>;
/// Instantiate all Full RPC extensions.
pub fn create_full<C, P, SC, B>(
deps: FullDeps<C, P, SC, B>,
backend: Arc<B>,
) -> Result<jsonrpc_core::IoHandler<sc_rpc_api::Metadata>, Box<dyn std::error::Error + Send + Sync>>
where
C: ProvideRuntimeApi<Block>
@@ -159,6 +160,10 @@ where
finality_provider,
)));
io.extend_with(substrate_state_trie_migration_rpc::StateMigrationApi::to_delegate(
substrate_state_trie_migration_rpc::MigrationRpc::new(client.clone(), backend, deny_unsafe),
));
io.extend_with(sc_sync_state_rpc::SyncStateRpcApi::to_delegate(
sc_sync_state_rpc::SyncStateRpcHandler::new(
chain_spec,
@@ -21,6 +21,7 @@ sp-std = { default-features = false, path = "../../primitives/std" }
sp-io = { default-features = false, path = "../../primitives/io" }
sp-core = { default-features = false, path = "../../primitives/core" }
sp-runtime = { default-features = false, path = "../../primitives/runtime" }
substrate-state-trie-migration-rpc = { optional = true, path = "../../utils/frame/rpc/state-trie-migration-rpc" }
frame-support = { default-features = false, path = "../support" }
frame-system = { default-features = false, path = "../system" }
@@ -49,9 +50,9 @@ std = [
"sp-core/std",
"sp-io/std",
"sp-runtime/std",
"sp-std/std"
"sp-std/std",
]
runtime-benchmarks = ["frame-benchmarking"]
try-runtime = ["frame-support/try-runtime"]
remote-test = [ "std", "zstd", "serde", "thousands", "remote-externalities" ]
remote-test = [ "std", "zstd", "serde", "thousands", "remote-externalities", "substrate-state-trie-migration-rpc" ]
+88 -88
View File
@@ -123,6 +123,13 @@ pub mod pallet {
}
}
#[derive(Clone, Encode, Decode, scale_info::TypeInfo, PartialEq, Eq)]
pub(crate) enum Progress {
ToStart,
LastKey(Vec<u8>),
Complete,
}
/// A migration task stored in state.
///
/// It tracks the last top and child keys read.
@@ -130,20 +137,13 @@ pub mod pallet {
#[codec(mel_bound(T: Config))]
#[scale_info(skip_type_params(T))]
pub struct MigrationTask<T: Config> {
/// The last top key that we migrated.
/// The current top trie migration progress.
pub(crate) progress_top: Progress,
/// The current child trie migration progress.
///
/// If it does not exist, it means that the migration is done and no further keys exist.
pub(crate) last_top: Option<Vec<u8>>,
/// The last child key that we have processed.
///
/// This is a child key under the current `self.last_top`.
///
/// If this is set, no further top keys are processed until the child key migration is
/// complete.
pub(crate) last_child: Option<Vec<u8>>,
/// A marker to indicate if the previous tick was a child tree migration or not.
pub(crate) prev_tick_child: bool,
/// If `ToStart`, no further top keys are processed until the child key migration is
/// `Complete`.
pub(crate) progress_child: Progress,
/// Dynamic counter for the number of items that we have processed in this execution from
/// the top trie.
@@ -182,18 +182,22 @@ pub mod pallet {
pub(crate) _ph: sp_std::marker::PhantomData<T>,
}
impl sp_std::fmt::Debug for Progress {
fn fmt(&self, f: &mut sp_std::fmt::Formatter<'_>) -> sp_std::fmt::Result {
match self {
Progress::ToStart => f.write_str("To start"),
Progress::LastKey(key) =>
write!(f, "Last: {:?}", sp_core::hexdisplay::HexDisplay::from(key)),
Progress::Complete => f.write_str("Complete"),
}
}
}
impl<T: Config> sp_std::fmt::Debug for MigrationTask<T> {
fn fmt(&self, f: &mut sp_std::fmt::Formatter<'_>) -> sp_std::fmt::Result {
f.debug_struct("MigrationTask")
.field(
"top",
&self.last_top.as_ref().map(|d| sp_core::hexdisplay::HexDisplay::from(d)),
)
.field(
"child",
&self.last_child.as_ref().map(|d| sp_core::hexdisplay::HexDisplay::from(d)),
)
.field("prev_tick_child", &self.prev_tick_child)
.field("top", &self.progress_top)
.field("child", &self.progress_child)
.field("dyn_top_items", &self.dyn_top_items)
.field("dyn_child_items", &self.dyn_child_items)
.field("dyn_size", &self.dyn_size)
@@ -207,12 +211,11 @@ pub mod pallet {
impl<T: Config> Default for MigrationTask<T> {
fn default() -> Self {
Self {
last_top: Some(Default::default()),
last_child: Default::default(),
progress_top: Progress::ToStart,
progress_child: Progress::ToStart,
dyn_child_items: Default::default(),
dyn_top_items: Default::default(),
dyn_size: Default::default(),
prev_tick_child: Default::default(),
_ph: Default::default(),
size: Default::default(),
top_items: Default::default(),
@@ -224,7 +227,7 @@ pub mod pallet {
impl<T: Config> MigrationTask<T> {
/// Return true if the task is finished.
pub(crate) fn finished(&self) -> bool {
self.last_top.is_none() && self.last_child.is_none()
matches!(self.progress_top, Progress::Complete)
}
/// Check if there's any work left, or if we have exhausted the limits already.
@@ -269,51 +272,36 @@ pub mod pallet {
///
/// This function is *the* core of this entire pallet.
fn migrate_tick(&mut self) {
match (self.last_top.as_ref(), self.last_child.as_ref()) {
(Some(_), Some(_)) => {
match (&self.progress_top, &self.progress_child) {
(Progress::ToStart, _) => {
self.migrate_top();
},
(Progress::LastKey(_), Progress::LastKey(_)) => {
// we're in the middle of doing work on a child tree.
self.migrate_child();
},
(Some(ref top_key), None) => {
// we have a top key and no child key. 3 possibilities exist:
// 1. we continue the top key migrations.
// 2. this is the root of a child key, and we start processing child keys (and
// should call `migrate_child`).
(Progress::LastKey(top_key), Progress::ToStart) => {
// 3. this is the root of a child key, and we are finishing all child-keys (and
// should call `migrate_top`).
// NOTE: this block is written intentionally to verbosely for easy of
// verification.
match (
top_key.starts_with(DEFAULT_CHILD_STORAGE_KEY_PREFIX),
self.prev_tick_child,
) {
(false, false) => {
// continue the top key migration
self.migrate_top();
},
(true, false) => {
self.last_child = Some(Default::default());
self.migrate_child();
self.prev_tick_child = true;
},
(true, true) => {
// we're done with migrating a child-root.
self.prev_tick_child = false;
self.migrate_top();
},
(false, true) => {
// should never happen.
log!(error, "LOGIC ERROR: unreachable code [0].");
Pallet::<T>::halt();
},
};
if !top_key.starts_with(DEFAULT_CHILD_STORAGE_KEY_PREFIX) {
// we continue the top key migrations.
// continue the top key migration
self.migrate_top();
} else {
// this is the root of a child key, and we start processing child keys (and
// should call `migrate_child`).
self.migrate_child();
}
},
(None, Some(_)) => {
log!(error, "LOGIC ERROR: unreachable code [1].");
Pallet::<T>::halt()
(Progress::LastKey(_), Progress::Complete) => {
// we're done with migrating a child-root.
self.migrate_top();
self.progress_child = Progress::ToStart;
},
(None, None) => {
(Progress::Complete, _) => {
// nada
},
}
@@ -324,19 +312,26 @@ pub mod pallet {
/// It updates the dynamic counters.
fn migrate_child(&mut self) {
use sp_io::default_child_storage as child_io;
let (last_child, last_top) = match (&self.last_child, &self.last_top) {
(Some(last_child), Some(last_top)) => (last_child, last_top),
let (maybe_current_child, child_root) = match (&self.progress_child, &self.progress_top)
{
(Progress::LastKey(last_child), Progress::LastKey(last_top)) => {
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
let maybe_current_child = child_io::next_key(child_root, &last_child);
(maybe_current_child, child_root)
},
(Progress::ToStart, Progress::LastKey(last_top)) => {
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
// Start with the empty key as first key.
(Some(Vec::new()), child_root)
},
_ => {
// defensive: this function is only called when both of these values exist.
// much that we can do otherwise..
// defensive: there must be an ongoing top migration.
frame_support::defensive!("cannot migrate child key.");
return
},
};
let child_root = Pallet::<T>::transform_child_key_or_halt(&last_top);
let maybe_current_child = child_io::next_key(child_root, &last_child);
if let Some(ref current_child) = maybe_current_child {
if let Some(current_child) = maybe_current_child.as_ref() {
let added_size = if let Some(data) = child_io::get(child_root, &current_child) {
child_io::set(child_root, current_child, &data);
data.len() as u32
@@ -348,25 +343,28 @@ pub mod pallet {
}
log!(trace, "migrated a child key, next_child_key: {:?}", maybe_current_child);
self.last_child = maybe_current_child;
self.progress_child = match maybe_current_child {
Some(last_child) => Progress::LastKey(last_child),
None => Progress::Complete,
}
}
/// Migrate the current top key, setting it to its new value, if one exists.
///
/// It updates the dynamic counters.
fn migrate_top(&mut self) {
let last_top = match &self.last_top {
Some(last_top) => last_top,
None => {
// defensive: this function is only called when this value exist.
// much that we can do otherwise..
let maybe_current_top = match &self.progress_top {
Progress::LastKey(last_top) => sp_io::storage::next_key(last_top),
// Start with the empty key as first key.
Progress::ToStart => Some(Vec::new()),
Progress::Complete => {
// defensive: there must be an ongoing top migration.
frame_support::defensive!("cannot migrate top key.");
return
},
};
let maybe_current_top = sp_io::storage::next_key(last_top);
if let Some(ref current_top) = maybe_current_top {
if let Some(current_top) = maybe_current_top.as_ref() {
let added_size = if let Some(data) = sp_io::storage::get(&current_top) {
sp_io::storage::set(&current_top, &data);
data.len() as u32
@@ -378,7 +376,10 @@ pub mod pallet {
}
log!(trace, "migrated a top key, next_top_key = {:?}", maybe_current_top);
self.last_top = maybe_current_top;
self.progress_top = match maybe_current_top {
Some(last_top) => Progress::LastKey(last_top),
None => Progress::Complete,
}
}
}
@@ -811,7 +812,7 @@ mod benchmarks {
continue_migrate_wrong_witness {
let null = MigrationLimits::default();
let caller = frame_benchmarking::whitelisted_caller();
let bad_witness = MigrationTask { last_top: Some(vec![1u8]), ..Default::default() };
let bad_witness = MigrationTask { progress_top: Progress::LastKey(vec![1u8]), ..Default::default() };
}: {
assert!(
StateTrieMigration::<T>::continue_migrate(
@@ -1141,15 +1142,12 @@ mod test {
}
#[test]
#[ignore]
fn detects_value_in_empty_top_key() {
let limit = MigrationLimits { item: 1, size: 1000 };
let initial_keys = Some(vec![(vec![], vec![66u8; 77])]);
let mut ext = new_test_ext(StateVersion::V0, false, initial_keys.clone(), None);
let root_upgraded = ext.execute_with(|| {
sp_io::storage::set(&[], &vec![66u8; 77]);
AutoLimits::<Test>::put(Some(limit));
let root = run_to_block(30).0;
@@ -1168,9 +1166,7 @@ mod test {
}
#[test]
#[ignore]
fn detects_value_in_first_child_key() {
use frame_support::storage::child;
let limit = MigrationLimits { item: 1, size: 1000 };
let initial_child = Some(vec![(b"chk1".to_vec(), vec![], vec![66u8; 77])]);
let mut ext = new_test_ext(StateVersion::V0, false, None, initial_child.clone());
@@ -1186,7 +1182,6 @@ mod test {
let mut ext2 = new_test_ext(StateVersion::V1, false, None, initial_child);
let root = ext2.execute_with(|| {
child::put(&child::ChildInfo::new_default(b"chk1"), &[], &vec![66u8; 77]);
AutoLimits::<Test>::put(Some(limit));
run_to_block(30).0
});
@@ -1214,7 +1209,7 @@ mod test {
// eventually everything is over.
assert!(matches!(
StateTrieMigration::migration_process(),
MigrationTask { last_child: None, last_top: None, .. }
MigrationTask { progress_top: Progress::Complete, .. }
));
root
});
@@ -1276,7 +1271,10 @@ mod test {
Origin::signed(1),
MigrationLimits { item: 5, size: 100 },
100,
MigrationTask { last_top: Some(vec![1u8]), ..Default::default() }
MigrationTask {
progress_top: Progress::LastKey(vec![1u8]),
..Default::default()
}
),
Error::<Test>::BadWitness
);
@@ -1451,7 +1449,8 @@ pub(crate) mod remote_tests {
// set the version to 1, as if the upgrade happened.
ext.state_version = sp_core::storage::StateVersion::V1;
let (top_left, child_left) = ext.as_backend().essence().check_migration_state().unwrap();
let (top_left, child_left) =
substrate_state_trie_migration_rpc::migration_status(&ext.as_backend()).unwrap();
assert!(
top_left > 0,
"no node needs migrating, this probably means that state was initialized with `StateVersion::V1`",
@@ -1509,7 +1508,8 @@ pub(crate) mod remote_tests {
)
});
let (top_left, child_left) = ext.as_backend().essence().check_migration_state().unwrap();
let (top_left, child_left) =
substrate_state_trie_migration_rpc::migration_status(&ext.as_backend()).unwrap();
assert_eq!(top_left, 0);
assert_eq!(child_left, 0);
}
@@ -18,7 +18,6 @@ log = { version = "0.4.11", optional = true }
thiserror = { version = "1.0.30", optional = true }
parking_lot = { version = "0.12.0", optional = true }
hash-db = { version = "0.15.2", default-features = false }
trie-db = { version = "0.23.1", default-features = false }
trie-root = { version = "0.17.0", default-features = false }
sp-trie = { version = "6.0.0", path = "../trie", default-features = false }
sp-core = { version = "6.0.0", path = "../core", default-features = false }
@@ -47,7 +46,6 @@ std = [
"sp-externalities/std",
"sp-std/std",
"sp-trie/std",
"trie-db/std",
"trie-root/std",
"log",
"thiserror",
@@ -23,7 +23,7 @@ use codec::Encode;
use hash_db::{self, AsHashDB, HashDB, HashDBRef, Hasher, Prefix};
#[cfg(feature = "std")]
use parking_lot::RwLock;
use sp_core::storage::{ChildInfo, ChildType, PrefixedStorageKey, StateVersion};
use sp_core::storage::{ChildInfo, ChildType, StateVersion};
use sp_std::{boxed::Box, vec::Vec};
use sp_trie::{
child_delta_trie_root, delta_trie_root, empty_child_trie_root, read_child_trie_value,
@@ -36,10 +36,6 @@ use sp_trie::{
use std::collections::HashMap;
#[cfg(feature = "std")]
use std::sync::Arc;
use trie_db::{
node::{NodePlan, ValuePlan},
TrieDBNodeIterator,
};
#[cfg(not(feature = "std"))]
macro_rules! format {
@@ -433,72 +429,6 @@ where
);
}
/// Check remaining state item to migrate. Note this function should be remove when all state
/// migration did finished as it is only an utility.
// original author: @cheme
pub fn check_migration_state(&self) -> Result<(u64, u64)> {
let threshold: u32 = sp_core::storage::TRIE_VALUE_NODE_THRESHOLD;
let mut nb_to_migrate = 0;
let mut nb_to_migrate_child = 0;
let trie = sp_trie::trie_types::TrieDB::new(self, &self.root)
.map_err(|e| format!("TrieDB creation error: {}", e))?;
let iter_node = TrieDBNodeIterator::new(&trie)
.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
for node in iter_node {
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
match node.2.node_plan() {
NodePlan::Leaf { value, .. } |
NodePlan::NibbledBranch { value: Some(value), .. } =>
if let ValuePlan::Inline(range) = value {
if (range.end - range.start) as u32 >= threshold {
nb_to_migrate += 1;
}
},
_ => (),
}
}
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
// get all child trie roots
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
let (key, value) =
key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
if key[..]
.starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
{
let prefixed_key = PrefixedStorageKey::new(key);
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
child_roots.push((ChildInfo::new_default(unprefixed), value));
}
}
for (child_info, root) in child_roots {
let mut child_root = H::Out::default();
let storage = KeySpacedDB::new(self, child_info.keyspace());
child_root.as_mut()[..].copy_from_slice(&root[..]);
let trie = sp_trie::trie_types::TrieDB::new(&storage, &child_root)
.map_err(|e| format!("New child TrieDB error: {}", e))?;
let iter_node = TrieDBNodeIterator::new(&trie)
.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
for node in iter_node {
let node = node.map_err(|e| format!("Child TrieDB node iterator error: {}", e))?;
match node.2.node_plan() {
NodePlan::Leaf { value, .. } |
NodePlan::NibbledBranch { value: Some(value), .. } =>
if let ValuePlan::Inline(range) = value {
if (range.end - range.start) as u32 >= threshold {
nb_to_migrate_child += 1;
}
},
_ => (),
}
}
}
Ok((nb_to_migrate, nb_to_migrate_child))
}
/// Returns all `(key, value)` pairs in the trie.
pub fn pairs(&self) -> Vec<(StorageKey, StorageValue)> {
let collect_all = || -> sp_std::result::Result<_, Box<TrieError<H::Out>>> {
@@ -0,0 +1,38 @@
[package]
name = "substrate-state-trie-migration-rpc"
version = "4.0.0-dev"
authors = ["Parity Technologies <admin@parity.io>"]
edition = "2021"
license = "Apache-2.0"
homepage = "https://substrate.io"
repository = "https://github.com/paritytech/substrate/"
description = "Node-specific RPC methods for interaction with state trie migration."
readme = "README.md"
[package.metadata.docs.rs]
targets = ["x86_64-unknown-linux-gnu"]
[dependencies]
scale-info = { version = "2.0.1", default-features = false, features = ["derive"] }
codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false }
serde = { version = "1", features = ["derive"] }
log = { version = "0.4.14", default-features = false }
sp-std = { path = "../../../../primitives/std" }
sp-io = { path = "../../../../primitives/io" }
sp-core = { path = "../../../../primitives/core" }
sp-state-machine = { path = "../../../../primitives/state-machine" }
sp-trie = { path = "../../../../primitives/trie" }
trie-db = { version = "0.23.1" }
jsonrpc-core = "18.0.0"
jsonrpc-core-client = "18.0.0"
jsonrpc-derive = "18.0.0"
# Substrate Dependencies
sc-client-api = { version = "4.0.0-dev", path = "../../../../client/api" }
sc-rpc-api = { version = "0.10.0-dev", path = "../../../../client/rpc-api" }
sp-runtime = { version = "6.0.0", path = "../../../../primitives/runtime" }
[dev-dependencies]
serde_json = "1"
@@ -0,0 +1,3 @@
Node-specific RPC methods for interaction with trie migration.
License: Apache-2.0
@@ -0,0 +1,164 @@
// This file is part of Substrate.
// Copyright (C) 2019-2022 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Rpc for state migration.
use jsonrpc_core::{Error, ErrorCode, Result};
use jsonrpc_derive::rpc;
use sc_rpc_api::DenyUnsafe;
use serde::{Deserialize, Serialize};
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
use std::sync::Arc;
use sp_core::{
storage::{ChildInfo, ChildType, PrefixedStorageKey},
Hasher,
};
use sp_state_machine::Backend;
use sp_trie::{trie_types::TrieDB, KeySpacedDB, Trie};
use trie_db::{
node::{NodePlan, ValuePlan},
TrieDBNodeIterator,
};
fn count_migrate<'a, H: Hasher>(
storage: &'a dyn trie_db::HashDBRef<H, Vec<u8>>,
root: &'a H::Out,
) -> std::result::Result<(u64, TrieDB<'a, H>), String> {
let mut nb = 0u64;
let trie = TrieDB::new(storage, root).map_err(|e| format!("TrieDB creation error: {}", e))?;
let iter_node =
TrieDBNodeIterator::new(&trie).map_err(|e| format!("TrieDB node iterator error: {}", e))?;
for node in iter_node {
let node = node.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
match node.2.node_plan() {
NodePlan::Leaf { value, .. } | NodePlan::NibbledBranch { value: Some(value), .. } =>
if let ValuePlan::Inline(range) = value {
if (range.end - range.start) as u32 >=
sp_core::storage::TRIE_VALUE_NODE_THRESHOLD
{
nb += 1;
}
},
_ => (),
}
}
Ok((nb, trie))
}
/// Check trie migration status.
pub fn migration_status<H, B>(backend: &B) -> std::result::Result<(u64, u64), String>
where
H: Hasher,
H::Out: codec::Codec,
B: Backend<H>,
{
let trie_backend = if let Some(backend) = backend.as_trie_backend() {
backend
} else {
return Err("No access to trie from backend.".to_string())
};
let essence = trie_backend.essence();
let (nb_to_migrate, trie) = count_migrate(essence, &essence.root())?;
let mut nb_to_migrate_child = 0;
let mut child_roots: Vec<(ChildInfo, Vec<u8>)> = Vec::new();
// get all child trie roots
for key_value in trie.iter().map_err(|e| format!("TrieDB node iterator error: {}", e))? {
let (key, value) = key_value.map_err(|e| format!("TrieDB node iterator error: {}", e))?;
if key[..].starts_with(sp_core::storage::well_known_keys::DEFAULT_CHILD_STORAGE_KEY_PREFIX)
{
let prefixed_key = PrefixedStorageKey::new(key);
let (_type, unprefixed) = ChildType::from_prefixed_key(&prefixed_key).unwrap();
child_roots.push((ChildInfo::new_default(unprefixed), value));
}
}
for (child_info, root) in child_roots {
let mut child_root = H::Out::default();
let storage = KeySpacedDB::new(essence, child_info.keyspace());
child_root.as_mut()[..].copy_from_slice(&root[..]);
nb_to_migrate_child += count_migrate(&storage, &child_root)?.0;
}
Ok((nb_to_migrate, nb_to_migrate_child))
}
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
#[serde(deny_unknown_fields)]
pub struct MigrationStatusResult {
top_remaining_to_migrate: u64,
child_remaining_to_migrate: u64,
}
/// Migration RPC methods.
#[rpc]
pub trait StateMigrationApi<BlockHash> {
/// Check current migration state.
///
/// This call is performed locally without submitting any transactions. Thus executing this
/// won't change any state. Nonetheless it is a VERY costy call that should be
/// only exposed to trusted peers.
#[rpc(name = "state_trieMigrationStatus")]
fn call(&self, at: Option<BlockHash>) -> Result<MigrationStatusResult>;
}
/// An implementation of state migration specific RPC methods.
pub struct MigrationRpc<C, B, BA> {
client: Arc<C>,
backend: Arc<BA>,
deny_unsafe: DenyUnsafe,
_marker: std::marker::PhantomData<(B, BA)>,
}
impl<C, B, BA> MigrationRpc<C, B, BA> {
/// Create new state migration rpc for the given reference to the client.
pub fn new(client: Arc<C>, backend: Arc<BA>, deny_unsafe: DenyUnsafe) -> Self {
MigrationRpc { client, backend, deny_unsafe, _marker: Default::default() }
}
}
impl<C, B, BA> StateMigrationApi<<B as BlockT>::Hash> for MigrationRpc<C, B, BA>
where
B: BlockT,
C: Send + Sync + 'static + sc_client_api::HeaderBackend<B>,
BA: 'static + sc_client_api::backend::Backend<B>,
{
fn call(&self, at: Option<<B as BlockT>::Hash>) -> Result<MigrationStatusResult> {
if let Err(err) = self.deny_unsafe.check_if_safe() {
return Err(err.into())
}
let block_id = BlockId::hash(at.unwrap_or_else(|| self.client.info().best_hash));
let state = self.backend.state_at(block_id).map_err(error_into_rpc_err)?;
let (top, child) = migration_status(&state).map_err(error_into_rpc_err)?;
Ok(MigrationStatusResult {
top_remaining_to_migrate: top,
child_remaining_to_migrate: child,
})
}
}
fn error_into_rpc_err(err: impl std::fmt::Display) -> Error {
Error {
code: ErrorCode::InternalError,
message: "Error while checking migration state".into(),
data: Some(err.to_string().into()),
}
}