Commit Graph

42 Commits

Author SHA1 Message Date
Piotr Mikołajczyk 1fabf067d0 try-runtime::follow-chain - execute all blocks (#12048)
* extract subscription

* FinalizedHeaders

* Fool of a Took

* testability

* tests

* review comments

* clippy
2022-09-05 07:26:45 +00:00
Kian Paimani f67c06ce22 Runtime State Test + Integration with try-runtime (#10174)
* add missing version to dependencies

* Huh

* add features more

* more fixing

* last touches

* it all finally works

* remove some feature gates

* remove unused

* fix old macro

* make it work again

* fmt

* remove unused import

* ".git/.scripts/fmt.sh" 1

* Cleanup more

* fix and rename everything

* a few clippy fixes

* Add try-runtime feature

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* small fixes

* fmt

* Update bin/node-template/runtime/src/lib.rs

* fix build

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: David <dvdplm@gmail.com>

* Update utils/frame/try-runtime/cli/src/commands/execute_block.rs

Co-authored-by: David <dvdplm@gmail.com>

* address all review comments

* fix typos

* revert spec change

* last touches

* update docs

* fmt

* remove some debug_assertions

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: David <dvdplm@gmail.com>
2022-09-01 10:33:22 +00:00
Piotr Mikołajczyk 9e5b211828 try-runtime - support all state versions (#12089)
* Add argument

* Apply

* More verbose parsing

* fmt

* fmt again

* Invalid state version

* reuse parsing

* type mismatch
2022-08-27 04:56:24 +00:00
Bastian Köcher 73d9ae3284 Introduce trie level cache and remove state cache (#11407)
* trie state cache

* Also cache missing access on read.

* fix comp

* bis

* fix

* use has_lru

* remove local storage cache on size 0.

* No cache.

* local cache only

* trie cache and local cache

* storage cache (with local)

* trie cache no local cache

* Add state access benchmark

* Remove warnings etc

* Add trie cache benchmark

* No extra "clone" required

* Change benchmark to use multiple blocks

* Use patches

* Integrate shitty implementation

* More stuff

* Revert "Merge branch 'master' into trie_state_cache"

This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.

* Improve benchmark

* Adapt to latest changes

* Adapt to changes in trie

* Add a test that uses iterator

* Start fixing it

* Remove obsolete file

* Make it compile

* Start rewriting the trie node cache

* More work on the cache

* More docs and code etc

* Make data cache an optional

* Tests

* Remove debug stuff

* Recorder

* Some docs and a simple test for the recorder

* Compile fixes

* Make it compile

* More fixes

* More fixes

* Fix fix fix

* Make sure cache and recorder work together for basic stuff

* Test that data caching and recording works

* Test `TrieDBMut` with caching

* Try something

* Fixes, fixes, fixes

* Forward the recorder

* Make it compile

* Use recorder in more places

* Switch to new `with_optional_recorder` fn

* Refactor and cleanups

* Move `ProvingBackend` tests

* Simplify

* Move over all functionality to the essence

* Fix compilation

* Implement estimate encoded size for StorageProof

* Start using the `cache` everywhere

* Use the cache everywhere

* Fix compilation

* Fix tests

* Adds `TrieBackendBuilder` and enhances the tests

* Ensure that recorder drain checks that values are found as expected

* Switch over to `TrieBackendBuilder`

* Start fixing the problem with child tries and recording

* Fix recording of child tries

* Make it compile

* Overwrite `storage_hash` in `TrieBackend`

* Add `storage_cache` to  the benchmarks

* Fix `no_std` build

* Speed up cache lookup

* Extend the state access benchmark to also hash a runtime

* Fix build

* Fix compilation

* Rewrite value cache

* Add lru cache

* Ensure that the cache lru works

* Value cache should not be optional

* Add support for keeping the shared node cache in its bounds

* Make the cache configurable

* Check that the cache respects the bounds

* Adds a new test

* Fixes

* Docs and some renamings

* More docs

* Start using the new recorder

* Fix more code

* Take `self` argument

* Remove warnings

* Fix benchmark

* Fix accounting

* Rip off the state cache

* Start fixing fallout after removing the state cache

* Make it compile after trie changes

* Fix test

* Add some logging

* Some docs

* Some fixups and clean ups

* Fix benchmark

* Remove unneeded file

* Use git for patching

* Make CI happy

* Update primitives/trie/Cargo.toml

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Introduce new `AsTrieBackend` trait

* Make the LocalTrieCache not clonable

* Make it work in no_std and add docs

* Remove duplicate dependency

* Switch to ahash for better performance

* Speedup value cache merge

* Output errors on underflow

* Ensure the internal LRU map doesn't grow too much

* Use const fn to calculate the value cache element size

* Remove cache configuration

* Fix

* Clear the cache in between for more testing

* Try to come up with a failing test case

* Make the test fail

* Fix the child trie recording

* Make everything compile after the changes to trie

* Adapt to latest trie-db changes

* Fix on stable

* Update primitives/trie/src/cache.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fix wrong merge

* Docs

* Fix warnings

* Cargo.lock

* Bump pin-project

* Fix warnings

* Switch to released crate version

* More fixes

* Make clippy and rustdocs happy

* More clippy

* Print error when using deprecated `--state-cache-size`

* 🤦

* Fixes

* Fix storage_hash linkings

* Update client/rpc/src/dev/mod.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Review feedback

* encode bound

* Rework the shared value cache

Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.

* Improve of the cache by merging the old and new solution

* FMT

* Please stop coming back all the time :crying:

* Update primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Fixes

* Make clippy happy

* Ensure we don't deadlock

* Only use one lock to simplify the code

* Do not depend on `Hasher`

* Fix tests

* FMT

* Clippy 🤦

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2022-08-18 18:59:22 +00:00
Kian Paimani f8d4b99917 Revamp nomination pool reward scheme (#11669)
* make pool roles optional

* undo lock file changes?

* add migration

* add the ability for pools to chill themselves

* boilerplate of tests

* somewhat stable, but I think I found another bug as well

* Fix it all

* Add more more sophisticated test + capture one more bug.

* Update frame/staking/src/lib.rs

* reduce the diff a little bit

* add some test for the slashing bug

* cleanup

* fix lock file?

* Fix

* fmt

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/mock.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix build

* fix some fishy tests..

* add one last integrity check for MinCreateBond

* remove bad assertion -- needs to be dealt with later

* nits

* fix tests and add benchmarks for chill

* remove stuff

* fix benchmarks

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_nomination_pools --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/nomination-pools/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* remove defensive

* first working version

* bring back all tests

* ALL new tests work now

* cleanup

* make sure benchmarks and all work

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_nomination_pools --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/nomination-pools/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* round of self-review, make arithmetic safe

* fix warn

* add migration code

* Fix doc

* add precision notes

* make arithmetic fallible

* fix node runtime

* a lot of precision tests and notes and stuff

* document MaxPOintsToBalance better

* :round of self-review

* fmt

* fix some comments

* Fix proportional slashing logic

* Update frame/nomination-pools/src/tests.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/tests.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* track poinst in migration

* fix

* fmt

* fix migration

* remove event read

* Apply suggestions from code review

* Update frame/staking/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/nomination-pools/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* update

* fmt

* fmt

* add one last test

* fmt

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Bot <admin@parity.io>
2022-07-13 12:49:20 +00:00
Kian Paimani 494167a05b Optimize offchain worker memory usage a bit. (#11454)
* add missing events to elections fallback

* Merged

* add some logs and stuff

* undo a bunch of things

* undo lock file

* remove unused err

* fix build
2022-05-24 06:23:05 +00:00
Koute dd854c16e2 Switch to pooling copy-on-write instantiation strategy for WASM (#11232)
* Switch to pooling copy-on-write instantiation strategy for WASM

* Fix benchmark compilation

* Fix `cargo fmt`

* Fix compilation of another benchmark I've missed

* Cleanups according to review comments

* Move `max_memory_size` to `Semantics`

* Set `memory_guaranteed_dense_image_size` to `max_memory_size`

* Rename `wasm_instantiation_strategy` to `wasmtime_instantiation_strategy`

* Update the doc-comments regarding the instantiation strategy

* Extend the integration tests to test every instantiation strategy

* Don't drop the temporary directory until the runtime is dropped in benchmarks

* Don't drop the temporary directory until the runtime is dropped in tests
2022-05-19 07:32:53 +00:00
Falco Hirschenberger b581604aa7 Apply some clippy lints (#11154)
* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2022-04-30 21:28:27 +00:00
Davide Galassi 641a43a05d Normalization of CLI options format (CamelCase => kebab-case) (#11194)
* Convert cli options from 'CamelCase' to 'kebab-case'
* Remove not required 'as_str' for 'ExecutionStrategy' option
2022-04-11 09:45:45 +02:00
Niklas Adolfsson 1df943cd6d chore: update jsonrpsee to v0.10.1 (#11173)
* fix(bin/node): remove unsed dep jsonrpsee

* chore(remote ext): update jsonrpsee v0.10.1

* chore(try runtime): update jsonrpsee v0.10.1

* Update utils/frame/try-runtime/cli/src/commands/follow_chain.rs

* cargo fmt
2022-04-07 06:47:36 +00:00
AurevoirXavier 8781574dc4 Use Lowercase (#11059) 2022-03-18 12:35:38 +00:00
Bastian Köcher 42484508d7 Update clap to the latest version (#11017)
* Update clap to the latest version

Besides that it also removes some `structopt` leftovers from some docs.

* Fix compile errors

* More fixes
2022-03-12 14:31:37 +00:00
Liu-Cheng Xu 47f96b6584 Remove unused parent_hash in OverlayedChanges::into_storage_changes (#11011)
Ref https://github.com/paritytech/substrate/pull/10922#issuecomment-1064258443

CC @cheme
2022-03-10 19:35:11 +01:00
Kian Paimani 6cfdbe5caa Trie version migration pallet (#10073)
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* migration pallet

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* fix test

* fix child migration calls.

* useless closure

* remove remaining state_hash variables.

* Fix and add more tests

* Remove outdated comment

* useless inner hash

* fmt

* remote tests

* finally ksm works

* batches are broken

* clean the benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Update frame/state-trie-migration/src/lib.rs

* brand new version

* fix build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* new test structure

* new testing stuff from emeric

* Add commit_all, still not working

* Fix all tests

* add comment

* we have PoV tracking baby

* document stuff, but proof size is still wrong

* FUCK YEAH

* a big batch of review comments

* add more tests

* tweak test

* update config

* some remote-ext stuff

* delete some of the old stuff

* sync more files with master to minimize the diff

* Fix all tests

* make signed migration a bit more relaxed

* add witness check to signed submissions

* allow custom migration to also go above limit

* Fix these pesky tests

* ==== removal of the unsigned stuff ====

* Make all tests work again

* separate the tests from the logic so it can be reused easier

* fix overall build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Slightly better termination

* some final tweaks

* Fix tests

* Restrict access to signed migrations

* address most of the review comments

* fix defensive

* New simplified code

* Fix weights

* fmt

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make the tests correctly fail

* Fix build

* Fix build

* try and fix the benchmarks

* fix build

* Fix cargo file

* Fix runtime deposit

* make rustdoc happy

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
2022-03-04 15:54:23 +00:00
Koute 9a31b2c341 Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)
* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments
2022-02-09 09:12:55 +00:00
Zeke Mostov b9947f4661 try-runtime-cli: Add to docs (#10822)
* try-runtime-cli: Add to docs of pre/post hooks

* Add note about --ws-max-out-buffer-capacity 1000
2022-02-08 19:05:34 +00:00
Qinxuan Chen e327b734bc use clap3 instead of structopt (#10632)
* use clap3 instead of structopt

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* format

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* update ss58-registry and revert some nits

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Fix clippy and doc

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* update clap to 3.0.7

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Apply review suggestions

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* remove useless option long name

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* cargo fmt

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2022-01-24 16:28:46 +00:00
Squirrel 3dd32d5255 Make clippy _a little_ more annoying (#10570)
* Clippy: +complexity

* Update client/cli/src/arg_enums.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/inspect/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/keystore/src/testing.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update primitives/npos-elections/fuzzer/src/reduce.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Incorporating feedback

* No need for Ok

* Additional

* Needed slice

* Wigy's suggestions on less derefs

* fix count

* reverting changes brought in by option_map_unit_fn

* add --all-targets

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-01-05 14:35:30 +00:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
cheme 4c651637f2 Inner hashing of value in state trie (runtime versioning). (#9732)
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
2021-12-24 08:54:07 +00:00
Éloïs 3acd335a8d Make wasm runtime cache size configurable (#10177)
* Make wasm runtime cache size configurable

* apply review comments

* remove VersionedRuntimeValue

* fix compilation

* VersionedRuntime: replace clone by Arc

* fmt

* fix warnings

* fix tests compilation

* fmt
2021-12-09 16:10:16 +01:00
Kian Paimani 4775f11edc allow try-runtime and TestExternalities to report PoV size (#10372)
* allow try-runtime and test-externalities to report proof size

* self review

* fix test

* Fix humanized dispaly of bytes

* Fix some test

* Fix some review grumbles

* last of the review comments

* fmt

* remove unused import

* move test

* fix import

* Update primitives/state-machine/src/testing.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* last touches

* fix

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-12-04 06:11:25 +00:00
Guillaume Thiolliere d91125389a ignoring errors and some fix (#10344) 2021-11-29 08:16:23 +00:00
Arkadiy Paronyan 4cbbf0cf43 Kill the light client, CHTs and change tries. (#10080)
* Remove light client, change tries and CHTs

* Update tests

* fmt

* Restore changes_root

* Fixed benches

* Cargo fmt

* fmt

* fmt
2021-11-12 14:15:01 +01:00
David eeb80f9e0b Upgrade jsonrpsee to v0.4.1 (#10022)
* Upgrade jsonrpsee to v0.4.1

* remove needless BlockT trait bound

* use default wss port in URL

* Fix try_runtime build

* Partially fix for "remote-tests" feature

* Review feedback

* fmt

* Sort out trait bounds for benches

* Fmt

* fmt again?

* fmt with nightly-2021-09-13

* Upgrade try-runtime as well

* fmt

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
2021-11-11 11:29:28 +00:00
Dan Shields 8db7ec9cc9 Migrate to docs.substrate.io throughout (#10007)
* all reverences of substrate.dev replaced with *correct* docs.substrate.io links

* fix links in non-cargo files, errors in replace

* homepage https://substrate.io

* cargo +nightly fmt

* FMT

Co-authored-by: Bastian Köcher <info@kchr.de>
2021-10-13 12:56:56 -06:00
Kian Paimani aae41d777a allow remote-ext to exclude some keys (#10020)
* allow remote-ext to exclude some keys

* don't use deprecated, just remove.

* rename
2021-10-13 11:12:26 +00:00
Kian Paimani ea0117af5d Add some docs about runtime migration best practices (#9837)
* Add some docs about runtime migration best practices as well.

* Update docs/CONTRIBUTING.adoc

* Update docs/CONTRIBUTING.adoc

Co-authored-by: Chevdor <chevdor@users.noreply.github.com>

Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
2021-09-23 08:20:37 +00:00
Kian Paimani 8f5f89324d follow-chain testing mode for try-runtime (and revamp CLI configs). (#9788)
* deadlock, need to ask someone to help now

* Finally it seems to be working.. at least for a few blocks

* self-review

* major mega revamp

* some small fixes

* another mega refactor

* add license

* Apply suggestions from code review

* hack around signature verification

* Some fixes

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* final tweaks, hopefully.

* a little self-review

* Add the ext root check

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
2021-09-21 15:04:28 +00:00
Kian Paimani 7f53a7f9df Display warning on try-runtime spec_name mismatch (#9593)
* Display warning on try-runtime version mismatch

* detect spec-name mismatch in try-runtime

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/remote-externalities/src/rpc_api.rs

Co-authored-by: Chevdor <chevdor@users.noreply.github.com>

* remove unused import

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
2021-08-20 18:41:21 +00:00
Ashley bad4544507 Make choosing an executor (native/wasm) an explicit part of service construction (#9525)
* Split native executor stuff from wasm executor stuff

* Remove `native_runtime_version` in places

* Fix warning

* Fix test warning

* Remove redundant NativeRuntimeInfo trait

* Add a warning for use_native

* Run cargo fmt

* Revert "Add a warning for use_native"

This reverts commit 9494f765a06037e991dd60524f2ed1b14649bfd6.

* Make choosing an executor (native/wasm) an explicit part of service construction

* Add Cargo.lock

* Rename Executor to ExecutorDispatch

* Update bin/node/executor/src/lib.rs

Co-authored-by: Squirrel <gilescope@gmail.com>

* Fix tests

* Fix minor node-executor error

* Fix node cli command thing

Co-authored-by: Squirrel <gilescope@gmail.com>
2021-08-18 14:26:41 +02:00
Kian Paimani abd08e29ce Rewrap all comments to 100 line width (#9490)
* reformat everything again

* manual formatting

* last manual fix

* Fix build
2021-08-11 14:56:55 +00:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Zeke Mostov b9d03f25da try-runtime-cli: Add execute-block subcommand (#9077)
* Refactor remote_externalities::rpc_api

* try-runtime-cli: Adde `execute-block` subcommand

* Trivial

* Address some comments

* Use required_if & remove header-at usage

* Improve doc

* Update comment

* small tweaks

* add overwrite-code to shared params

* Update utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* make url a shared param

* add helper for block_at (#9153)

* add helper for block_at

* remove redundant bound

* doc for fn block_at

* Update error message

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2021-06-22 20:36:12 +00:00
Zeke Mostov f0120d3b62 [try-runtime-cli] Offchain worker support (#8966)
* make remote-ext work with ws and safe RPCs

* Update docs.

* Update utils/frame/remote-externalities/Cargo.toml

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Fix test

* Update lock file

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix build again.

* checkpoint, merging the paged rpc now

* revert lifetime stuff

* WIP: remote client init not working

* Small cleanups

* use jsonrpsee alpha.7

* WIP

* Executiing without errors

* Reorg & cleanup

* Trivial cleaning

* Add txpool & keystore extension

* Small cleaning

* More :cleaning

* Flags: page-size, override-code

* WIP

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Remove heap_pages

* Dry code extraction from state

* Formatting

* More formatting

* Add issue todo

* Use jsonrpsee 0.2.0

* Try trigger gitlab

* Fix "block_import_works" test

* fix native_big_block_import_fails_on_fallback test

* fix commit should work

* Rewrite UI tests

* Revert "Rewrite UI tests"

This reverts commit ada7f670f701c21fb399946a3f6918453f537bcb.

* try again with UI

* Use const for legacy heap pages val

* Move parse module to its own file

* Move rpc_api module to its own file

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* trait names: Block, not B

* Corect HEAP_PAGES_TEST_LEGACY export

* Update utils/frame/remote-externalities/src/rpc_api.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Revert test_ext heap_page insert; adjust storage root instead

* Doc comments for try_runtime::cli::Command

* TryRuntime stub

* trailing comma

* Remove unused dev dep in frame-executive

* Improve parse::hash variable name & error index

* Use Result for rpc_api fns

* Richer err messagges

* Remove HEAP_PAGE_TEST_LEGACY

* Update bin/node/executor/tests/basic.rs

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2021-06-09 22:05:28 +00:00
Kian Paimani ec409615a3 make remote-ext work with ws and safe RPCs (#8889)
* make remote-ext work with ws and safe RPCs

* Update docs.

* Update utils/frame/remote-externalities/Cargo.toml

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Fix test

* Update lock file

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix build again.

* revert lifetime stuff

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-05-25 18:00:50 +00:00
Bastian Köcher 6a6f330412 Make wasmtime the default when the feature is enabled (#8855)
* Make wasmtime the default when the feature is enabled

* Update client/cli/src/arg_enums.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-05-19 08:29:32 +00:00
François Garillot 052be8bbef Simplify some Option / Result / ? operator patterns (#8653)
* Simplify some Option / Result / ? operator patterns

When those match a combinator exactly.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).

* adjust after review

* adjust post-review
2021-04-23 13:36:10 +00:00
Steve Biedermann db74a34877 Use pathbuf for remote externalities (#8480)
* Combine SnapshotConfig string fields name and directory into single PathBuf field named path

* Update Cargo.lock

* fix test build failure
2021-03-29 10:47:34 +00:00
Steve Biedermann f9b6c869a3 Improve remote-externalities (#8397)
* make builder generic to allow using different hash types

* expose "cache", "block_number" and "modules" as cli options for live state

* Change Builder to be generic over Block instead of Hash
add rpc method to get hash from block number
allow passing of block numbers and hashes

* fix live tests

* fix formatting in utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* change cli to only accept block hashes
break up lines that were too long
use starts_with instead of match s.get
use unwrap_or_default instead of unwrap_or(Vec::new())

* improve error message

* fix indentation

* replace Block with sp_runtime::testing::Block

* Move cache test out of remote-test feature tests
Add cache file (contains only "Proxy" module) for local test

* simplify match expression to and_then

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Combine the two cfg attributes into one

Co-authored-by: David <dvdplm@gmail.com>

* Restrict visibility of test_prelude use statements to crate level

* Fix usage of and_then

* Rename cache to snapshot

* Remove fully qualified path for Debug

* Refine naming. snapshot to state_snapshot

* Remove unnecessary comment

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
2021-03-23 13:23:07 +00:00
Kian Paimani 33b435894a Better identifier and logging for runtime upgrades (#8123)
* A clean new attempt

* Checkpoint to move remote.

* A lot of dependency wiring to make it feature gated.

* bad macro, bad macro.

* Undo the DB mess.

* Update frame/support/src/traits.rs

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Apply suggestions from code review

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* unbreak the build

* Better logging and ids for migrations

* Fix doc.

* Test

* Update frame/try-runtime/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/try-runtime/cli/Cargo.toml

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/try-runtime/Cargo.toml

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Address most review grumbles.

* Fix build

* Add some comments

* Remove allowing one pallet at a time.

* Rework the PR

* nit

* Slightly better error handling.

* Remove files

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/support/src/dispatch.rs

* Update frame/support/src/dispatch.rs

* Fix test

* Make extension trait.

* Bring back try-runtime/std

* remove bincode

* Remove warning

* Change test features

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-02-26 15:41:23 +00:00
Kian Paimani 16a27c28a9 Migration testing runtime API/Bot (#8038)
* A clean new attempt

* Checkpoint to move remote.

* A lot of dependency wiring to make it feature gated.

* bad macro, bad macro.

* Undo the DB mess.

* Update frame/support/src/traits.rs

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Apply suggestions from code review

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* unbreak the build

* Update frame/try-runtime/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/try-runtime/cli/Cargo.toml

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/try-runtime/Cargo.toml

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Address most review grumbles.

* Fix build

* Add some comments

* Remove allowing one pallet at a time.

* More grumbles.

* relocate remote-ext

* Fix build

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-02-19 14:52:09 +00:00