Commit Graph
80 Commits
Author SHA1 Message Date
cheme 34c991e2cf Remove hashbrown from trie cache. (#2632)
Using hashmap instead (hashset do not expose entry), to get the default
random hasher her.
2023-12-08 14:28:04 +09:00
cheme f4c4c0fe29 Switch trie cache random seed (#1935)
Use a more secure seed for hashsets of cache.
2023-10-20 12:13:19 +09:00
cheme 714981776c Kusama state version switch and migration. (#7015)
* State version switch and migration init

* fix

* migration declaration (last position, westend was first)

* revert unrelated cargo fmt

* move to unrelease

* fmt?

* fmt

* Change size of processing per blocks

* Dispatch error
2023-07-04 16:17:21 +00:00
cheme ef165cec3e update trie deps (#13601) 2023-03-14 21:03:50 +00:00
cheme 94d24de1a0 Remove state migration from westend runtime. (#6737)
* clean state_trie_migration from westend

* update cargo.lock

* Revert "clean state_trie_migration from westend"

This reverts commit b1d55c688e9a8a2fbd155f0a9ddaba9766ad101c.

* cleanup.

* use storage_alias

* Revert "use storage_alias"

This reverts commit 5289745d1592a14db0063d9ee111137473993f41.

* Revert "Revert "use storage_alias""

This reverts commit d4c688acdddf29a09c79a4eef52ab4095128e5b3.

* Force alias.

* magic
2023-02-22 14:21:44 +00:00
chemeandparity-processbot 403320269e Westend state trie to version 1 (#6336)
* Switch to state V1 and add state-trie-migration pallet with dummy manual
account.

* Initialize limit on runtime upgrade.

* add prelude

* sp_std prelude only for no_std

* Disable filter for signed migration

* revert hex dep

* use NeverEnsureOrigin

* fix

* correct fix

* check init state in try-runtime

Co-authored-by: parity-processbot <>
2023-01-11 12:13:33 +00:00
chemeandparity-processbot 5ca7dc1b87 Add total nb to trie migration rpc (#12770)
* Add total nb to trie migration rpc

* fix and format

* Use struct instead of tuple

* fixes

Co-authored-by: parity-processbot <>
2022-11-25 15:06:04 +00:00
chemeandparity-processbot 79d9ea4f62 State trie migration rococo runtime changes. (#6127)
* add state-trie-migration (warn key need to be changed)

* rococo root

* restore master benchs (weights from substrate are used).

* use ord_parameter macro.

* do not upgrade runtime version yet

* apply review changes

* to test ci

* Revert "to test ci"

This reverts commit 5df6c5c74c5d172f254579cbb76caeb44af80ec2.

* test ci

* Revert "test ci"

This reverts commit 0747761644ba566f9756e195e5304758370dc52c.

Co-authored-by: parity-processbot <>
2022-11-11 11:06:44 +00:00
chemeBastian Köcherparity-processbot
91ca3a223b update paritydb and remove dev deps on rocksdb (#12641)
* update paritydb and remove dev deps on rocksdb

* feature rocksdb for node testing

* feature decl in node-bench

* revert change to rocksdb inclusion logic

* Update bin/node/bench/Cargo.toml

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: parity-processbot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
2022-11-09 09:23:44 +01:00
cheme 8b2ca711b3 Remove partial key size limit from trie codec (#12566)
* remove size limit from trie codec

* test previous upper limit is not enforced anymore

* fmt

* restore test
2022-11-08 21:24:54 +01:00
cheme e7c6a83502 Statemine state migration runtime changes. (#1743)
* init, using polkadot master

* version update

* weights copy pasted from substrate

* feature gate migration

* changes from reviews

* Change controller addresses.

* fix

* rename feature to state-trie-version-1
2022-11-07 11:09:32 +00:00
cheme b7b75ff054 Add trie migration rpc to polkadot-parachain (#1424)
* add trie migration rpc to polkadot-parachain

* pass backend

* fix
2022-11-07 10:41:45 +00:00
chemeandparity-processbot d5677cde48 Guard some invalid node for proof decoding. (#12417)
* Guard some invalid node for proof decoding.

* only forbid bitmap with no children.

* change format

* scale error.

* small test

Co-authored-by: parity-processbot <>
2022-11-05 22:26:12 +00:00
chemeandBastian Köcher 20abd5dc24 Expose and link trie migration weights (#11775)
* expose and link trie migration weights

* actually limit to one trait

* Run test on dummy weights

* fmt

Co-authored-by: Bastian Köcher <info@kchr.de>
2022-07-17 22:57:13 +00:00
cheme fc568486fa update parity-db version (#5472) 2022-05-06 15:24:16 +02:00
chemeandArkadiy Paronyan 9d57a674d7 Parity-db Change missing implementation. (#11049)
* support for release as in kvdb (only if no rc).

* Start impl

* minimal implementation for paritydb rc

* Update client/db/src/parity_db.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Update client/db/src/parity_db.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Update client/db/src/parity_db.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Commit not panicking in DbAdapter

* errors from string

* update parity db version

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2022-05-06 15:22:39 +02:00
cheme 8f784d592a Trie State migration rpc (#5180)
* update

* unsafe

* switch remote

* update rpc usage

* update cargo.lock

* lock from master

* updated lock
2022-04-04 09:33:35 +02:00
chemeandparity-processbot d6d1eb7b14 Companion to #11081, upgrade paritydb version. (#1101)
* update paritydb.

* update lockfile for {"polkadot"}

Co-authored-by: parity-processbot <>
2022-03-23 17:57:00 +00:00
cheme 183b52e7db update parity-db (#5175) 2022-03-22 09:55:57 +00:00
cheme 61f1b2e6ab Do not upgrade db_version on metadata reading failure. (#11081) 2022-03-22 09:53:49 +00:00
800cc1d4be State migration rpc (#10981)
* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* migration pallet

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* fix test

* fix child migration calls.

* useless closure

* remove remaining state_hash variables.

* Fix and add more tests

* Remove outdated comment

* useless inner hash

* fmt

* remote tests

* finally ksm works

* batches are broken

* clean the benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Update frame/state-trie-migration/src/lib.rs

* brand new version

* fix build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* new test structure

* new testing stuff from emeric

* Add commit_all, still not working

* Fix all tests

* add comment

* we have PoV tracking baby

* document stuff, but proof size is still wrong

* FUCK YEAH

* a big batch of review comments

* add more tests

* tweak test

* update config

* some remote-ext stuff

* delete some of the old stuff

* sync more files with master to minimize the diff

* Fix all tests

* make signed migration a bit more relaxed

* add witness check to signed submissions

* allow custom migration to also go above limit

* Fix these pesky tests

* ==== removal of the unsigned stuff ====

* Make all tests work again

* separate the tests from the logic so it can be reused easier

* fix overall build

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Slightly better termination

* some final tweaks

* Fix tests

* Restrict access to signed migrations

* mig rpc

* fix

* better rpc name

* Make rpc unsafe

* address most of the review comments

* fix defensive

* New simplified code

* Fix weights

* fmt

* Update frame/state-trie-migration/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make the tests correctly fail

* Fix build

* Fix build

* try and fix the benchmarks

* fix build

* Fix cargo file

* Fix runtime deposit

* make rustdoc happy

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* update rpc deps, try to process empty keys.

* move rpc crate

* move check backend out of state machine

* Add primitive crate.

* module code

* fix runtime test

* StateMigrationStatusProvider

* Pass backend to rpc.

* fmt

* review changes

* move rpc crate

* try remove primitive crate

* Update utils/frame/rpc/state-trie-migration-rpc/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
2022-03-16 11:43:24 +00:00
cheme d5ddb1a809 paritydb support for parachains db. (#4838)
* parity db subsystem without cache and no splitted column

* fmt

* fix path (auto from parity-db fail)

* lru cache for db column with cache

* Revert "lru cache for db column with cache"

This reverts commit ae177bc5e107a075eff6a21f651218ada6599b74.

* Write_lock mutex

* theoric code for bridges

* revert changes

* Revert bridge changes

* fix spec_version

* update parity db

* test purge-db

* Use specific ordered collection with paritydb.

* Revert "Use specific ordered collection with paritydb."

This reverts commit 8b66d0a4ae914cba1af0f44050d45dd6d9327c6b.

* fix chain selection tests.

* remove patch

* fix auto.

* Remove useless exists directory method

* purge chain without parity-db removal

* spellcheck

* renamings and filtering.

* fix assertion

* format

* update parity-db and fmt

* Auto keep using rocksdb when it exists.

* Revert "Auto keep using rocksdb when it exists."

This reverts commit cea49b32ae590bdce31fed5c45f3c028ae0c7564.

* Update kvdb version.
2022-03-03 11:49:38 +00:00
chemeandparity-processbot 0256fe73c0 Companion for substrate#9732 (#678)
* state-update4 branch

* new ref

* Update to latest.

* update deps

* switch to host state version

* update

* fmt

* up

* remove trie patch

* remove patch

* fmt

* update

* set state_versions in runtimes

* state version from storage

* state version from storage

* seedling compat

* restore lock

* update lockfile for substrate

* update lockfile for polkadot

Co-authored-by: parity-processbot <>
2021-12-24 11:28:43 +00:00
chemeandparity-processbot 48dc675047 Companion for substrate#9732 (#4104)
* merge master (do not compile)

* fix

* lock

* update lock

* Update to refactoring.

* runtime version

* fmt

* remove trie patch

* remove patch

* No layout alias for bridge proof.

* update depupdate depss

* No switch until migration.

* master lock

* test

* test

* Revert "test"

This reverts commit 57325ef73332bf4b054aa4a667bb716fcf8a0d89.

* Revert "test"

This reverts commit ce74d0e2062806f72c0e9e9ca07b14165f43521e.

* rename feature

* state version as parameter, use the feature only on runtimes.

* update

* update to state version in runtime

* state version from storage

* update lockfile for substrate

Co-authored-by: parity-processbot <>
2021-12-24 09:27:35 +00:00
chemeandkianenigma 4c651637f2 Inner hashing of value in state trie (runtime versioning). (#9732)
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
2021-12-24 08:54:07 +00:00
cheme ca5b07243f Fast sync child trie support. (#9239)
* state machine proofs.

* initial implementation

* Remove todo.

* Extend test and fix import.

* fix no proof, with proof ko.

* fix start at logic.

* Restore response size.

* Rework comments.

* Add explicit ref

* Use compact proof.

* ref change

* elaborato on empty change set condition.

* KeyValueState renaming.

* Do not add two time child trie with same root to sync reply.

* rust format

* Fix merge.

* fix warnings and fmt

* fmt

* update protocol id to V2
2021-11-07 13:13:02 +00:00
471e9cfbf1 Hashing proc macro utils (#9875)
* hashing macro

* fmt

* use in easy place, and fix blake sizes

* fix

* Fixes, docs.
Allow ident as input.

* fix doc tests

* update error in test (nmapkey and key are same type).

* hashing crates under sp_core

* Doc updates and format.

* use all existing hashing functions.

* return array of u8

* Update primitives/core/hashing/proc-macro/src/impls.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* ToTokeen for an array of u8

* fix

* re

* Improve impls

* complete doc tests

* fmt

* fix doctest format

* fix ui test (nmap key type alias)

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2021-11-02 09:35:23 +00:00
cheme 0a8b3ddedc Fix compact proof decoding unaccessed last child trie. (#9715)
* fix no child proof attached but root included.

* small stress test for proof of child tries.

* rust fmt
2021-09-07 16:21:42 +00:00
cheme 61941f2806 reset cache when storage possibly change (fix init of tests). (#9665)
* reset cache when storage possibly change (fix init of tests).

* remove backend_storage_mut

* fix warn

* remove remaining backend_storage_mut
2021-09-06 10:42:02 +00:00
chemeandBastian Köcher ade667f928 State machine local child root cache. (#9107)
* cache root for child api.

* minimal testing

* Reset cache on test 'set_root'.

* Update primitives/state-machine/src/trie_backend_essence.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend_essence.rs

* Update primitives/state-machine/src/trie_backend_essence.rs

* Renaming to 'reset_cache'.

* correct rust fmt

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-07-23 11:30:00 +00:00
chemeandparity-processbot b71ec24815 Companion for #9100 (Paged keys rpc for child storage.) (#3258)
* patch branch

* patch bis

* Add missing client method.

* unpatch

* update Substrate

Co-authored-by: parity-processbot <>
2021-07-07 07:53:33 +00:00
chemeandAlexander Popiak d80e1bc978 Paged keys rpc for child storage. (#9100)
* childstate_getKeysPaged rpc

* Rename `v` to `iter`.

* Update client/api/src/backend.rs

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>

* Update client/api/src/backend.rs

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>
2021-07-07 07:22:13 +00:00
cheme 0027f5e248 ensure early drop of proof memory. (#500) 2021-06-18 19:29:42 +02:00
chemeandShawn Tabrizi 693b39f43b Frame remove_all with size limit. (#9106)
* remove prefixed content with limit.

* test match

* factor comment and factor ext limit removal.

* fix benchmark

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-06-15 13:23:58 +00:00
e8b4b36446 Compact proof. (#295)
* compact, need to be made optional and look into/compress child trie
roots/state.

* proto with child trie support

* Missing set_offchain_storage overload.

* right name

* Ignore offchain indexing in validation function.

* patch trie-db

* decompress from iter

* use compressed proof

* remove wasm blob from proof (no inject plugged yet)

* change lock

* update trie

* change in toml

* Revert "change in toml"

This reverts commit e588583b79e1a64e2a54157b40968aa6b586fbbe.

* use patch to branches

* i

* i:wq

* switch branch

* ii

* ok, needed to patch the runtime by putting substrate patch in polkadot
project.

* test passing with this conf

* actual lazy code fetch

* patch issue

* Code reorg

* restore commented tests.

* update deps.

* remove polka patch

* fixes

* remove patch

* revert cargo.lock

* cargo update -p sp-trie polkadot-service

* fix collator test (using parent state root).

* Update pallets/parachain-system/src/validate_block/implementation.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove encode_witness test function.

* Update pallets/parachain-system/src/validate_block/implementation.rs

* Fix compilation

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2021-06-08 11:51:32 +02:00
cheme f01c396677 Compact proof utilities in sp_trie. (#8574)
* validation extension in sp_io

* need paths

* arc impl

* missing host function in executor

* io to pkdot

* decode function.

* encode primitive.

* trailing tab

* multiple patch

* fix child trie logic

* restore master versionning

* bench compact proof size

* trie-db 22.3 is needed

* line width

* split line

* fixes for bench (additional root may not be needed as original issue was
with empty proof).

* revert compact from block size calculation.

* New error type for compression.

* Adding test (incomplete (failing)).
Also lacking real proof checking (no good primitives in sp-trie crate).

* There is currently no proof recording utility in sp_trie, removing
test.

* small test of child root in proof without a child proof.

* remove empty test.

* remove non compact proof size

* Missing revert.

* proof method to encode decode.
2021-06-07 09:06:38 +00:00
878f7ccf6e Allow transaction for offchain indexing (#7290)
* Moving offchain change set to state machine overlay change set,
preparing use of change set internally.

* Make change set generic over key and value, and use it for offchain
indexing.

* test ui change

* remaining delta

* generating with standard method

* Remove 'drain_committed' function, and documentation.

* Default constructor for enabling offchain indexing.

* Remove offchain change specific iterators.

* remove pub accessor

* keep previous hierarchy, just expose iterator instead.

* Update primitives/state-machine/src/overlayed_changes/mod.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* fix line break

* missing renamings

* fix import

* fix new state-machine tests.

* Don't expose InnerValue type.

* Add test similar to set_storage.

* Remove conditional offchain storage (hard to instantiate correctly).

* fix

* offchain as children cannot fail if top doesn't

Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2021-01-22 12:27:43 +00:00
cheme ae9b659969 Ignore offchain indexing in validation function. (#297)
* Missing set_offchain_storage overload.

* right name

* Ignore offchain indexing in validation function.
2021-01-19 11:12:11 +01:00
cheme 1922dab050 Fix clear prefix check to avoid erasing child trie roots. (#7848)
* Fix clear prefix check to avoid erasing child trie roots.

* Renaming and extend existing test with check.

* last nitpicks.
2021-01-13 12:45:31 +00:00
cheme 378f6e070b Use state machine externalities in validation runtime. (#214)
* test

* switch to branch

* Use modified.

* Use sp_externalities environmental primitives

* Change to latest no_std version, add missing host function boilerplate.

* remove comment (was used for debugging).

* fixes

* with_externalities factor

* Read merged change from overlay.
2020-11-10 12:23:55 +01:00
chemeandBastian Köcher 232a30fdb4 state_machine no_std witness externalities (#6934)
* checkpoint before removing CT from change trie

* before trie backend without tx

* undo

* Started no transaction, but would need using a different root
calculation method, out of the scope of this pr, will roll back.

* Remove NoTransaction.

* partially address review.
dummy stats implementation for no_std.

* Remove ChangeTrieOverlay.

* modified function

* Remove witness_ext

* need noops changes root

* update from cumulus branch

* line break

* remove warning

* line break

* From review: renamings and stats active in no std (except time).

* include cache, exclude change trie cache with individual temporary bad looking
no_std check

* little test

* fuse imports and filter_map prepare_extrinsics_input_inner fold.

* put back ExtInner into Ext, awkward double proto for new function.

* Apply suggestions from code review

* Update primitives/state-machine/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-09-11 10:11:25 +00:00
chemeandBastian Köcher 186fa587cf Ignore wasm_gc for debug build. (#6962)
* Ignore gc for debug build.

* alternate implementation

* Update utils/wasm-builder/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-09-08 09:27:10 +02:00
cheme 734d30bf5d Update substrate bip39 version. (#6955)
* update bip39 version

* and lock
2020-08-31 14:16:30 +00:00
cheme 345cb13493 Fail when storage is not in proof (#205)
* fail when storage is not in proof

* Single panic.
2020-08-24 22:11:05 +02:00
cheme 424d5c722d Restrict Protected to some heap types. (#6471)
* Restrict `Protected` to some heap types.

* Comment abut Protected usage.

* Remove Protected from crypto, use secrecy crate for existing uses.

* use a parse function

* fix error convert

* Rename and move secretY string function.

* std result
2020-07-01 09:59:07 +00:00
cheme 66b5c97c80 Align trie iteration to substrate. (#90)
* Align trie iteration to substrate.

* Can use expect on trie error, remove debug_assert
2020-05-07 09:38:59 +02:00
chemeandBastian Köcher c05ec630ff Avoid changing overlay committed layer. (#5839)
* Avoid changing overlay committed layer.

* basic test

* Add some tx in the test.

* only update from backend value on missing entry in both layer.
deleted entry is replace by empty vec.

* test and review changes

* additional test and review change

* remove test on changing existing value, it does not always panic
depending on existing content

* Update primitives/state-machine/src/overlayed_changes.rs

* Update primitives/state-machine/src/overlayed_changes.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-04-30 13:59:21 +02:00
cheme 1a4db93c50 Companion to child trie api change #4857 (#950) 2020-04-21 10:39:35 +02:00
chemeandthiolliere 4ffcf98d8d Child trie api changes BREAKING (#4857)
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
2020-04-20 15:21:22 +02:00
chemeandgabriel 334e9333a1 Companion PR ci sed issue. (#5508)
* ci: fix echo sed failure

Co-authored-by: gabriel <gabriel@parity.io>
2020-04-03 17:51:21 +05:30
chemeandBastian Köcher 58578af074 State metrics possible changes (#5168)
* Registering state from overlay.

* fix

* fix2

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-04-01 19:46:40 +02:00
cheme da42d7408c Use prefixed iterator from trie. (#4858) 2020-02-11 10:00:56 +01:00
cheme 900295b020 Fix call to backend in child_storage_hash (#4815)
* Fix call to backend in child_storage_hash

* child storage test.
2020-02-03 20:28:10 +01:00
cheme 0ece5d9e17 Fix key collision for child trie (#4162)
* In progress, runtime io must switch to future proof root +
child_specific (unique id) + u32 type.

* Switch interface, sr-io seems ok, rpc could use similar interface to
sr-io, genesis json broken if there is child trie in existing encoding
genesis.

* test from previous implementation.

* fix proving test.

* Restore Keyspacedb from other branch, only apply to child trie.

* Removing unneeded child_info from child root (child info are stored
if things changed, otherwhise the root does not change).

* Switch rpc to use same format as ext: more future proof.

* use root from child info for trie backend essence.

* Breaking long lines.

* Update doc and clean pr a bit.

* fix error type

* Restore removed doc on merge and update sr-io doc.

* Switch child storage api to use directly unique id, if managed id
where to be put in place, the api will change at this time.

* Clean deprecated host interface from child.

* Removing assertion on child info (can fail depending on root
memoization).

* merging child info in the overlay when possible.

* child iteration by prefix using child_info.

* Using ChainInfo in frame support. ChainInfo gets redesign to avoid
buffers allocation on every calls.

* Add length of root to the data of child info.

* comments

* Encode compact.

* Remove child info with root.

* Fix try_update condition.

* Comment Ext child root caching.

* Replace tuples by struct with field

* remove StorageTuple alias.

* Fix doc tests, and remove StorageOverlay and ChildStorageOverlay
aliases.
2019-12-14 11:11:19 +09:00
cheme a7c04a5bce This reverts commit fdbfd0b150. (#4230)
It brings back trie-db 0.16.
2019-11-27 18:23:37 +00:00
cheme fdbfd0b150 revert (#4204) 2019-11-26 10:23:11 +01:00
cheme 05391f7e04 State machine call proof backend (#3945)
* drafting a proof extraction at the hashdb level (to include everything
for the 'call' proof case).

* use full proof by default (replace previous proof recorder).

* fix warnings.

* Cache value not found in proof recorder.

* Remove need or internal backend struct.

* fix type.

* doc
2019-11-12 16:57:37 +01:00
cheme d9cffa0bb5 Code redundancy between ext implementation and testing. (#3830)
* fix child_storage_hash

* extract common implementation for ext and testing

* cleaning impl.

* replace ExtBasisMut by actual Ext

* remove extbasis.

* Update tests to use Ext from test externalities.

* use Ext constructor for getting ext from TestExternalities.

* Add missing extensions from ext.

* fix wasmi test

* Fix merge error.
2019-10-18 09:52:25 +02:00
cheme e3d0c60a31 Implement change trie for child trie. (#3122)
* Initial implementation, some redundancy is awkward and there is
some useless computation (but there is a pending pr for that).
Next are tests.

* Minimal tests and fix extend child.

* implement iterator for change child trie.

* prune child trie.

* Fix pruning test.

* bump spec version.

* Avoid empty child trie (could also be checked before)

* tabs.

* Fix child digest overriding each others.

* Restore doc deleted on merge.

* Check correct child value on extrinsics build.

* Revert runtime version update.
2019-09-02 12:10:42 +03:00
cheme 816e132cd7 Add missing child trie rpc on_custom_message handler (#3522)
* missing on_custom for child read.

* fix and complete, add test cases.

* shorten line.

* replace vecs of key value tuple with maps.
2019-08-31 17:19:54 -07:00
cheme 95abffc8e4 Update trie crate to hashbrown usage. (#3440)
* Update trie crate to non hashmap_core one.

* bump runtime impl.
2019-08-19 15:59:36 +02:00
cheme 873b974566 Config genesis json with children. (#372) 2019-08-16 13:39:15 +02:00
cheme b0e1212d48 Child storage tests and genesis fix. (#3185)
* Using child storage, (srml-support only), test failing .

* fix simple tests.

* Enumerable by requiring owned struct (previous form only allow
&'static).
Broken tests are from genesis init.

* implement for_child_keys_with_prefix

* indent

* clear_child_prefix fix.

* clear_child_prefix fix 2.

* fix for storage_impl, if/when allowing child and not child this could be
reverted.

* Fix lot of urlinked child genesis, still need to look upon actual
genesis srml module code.
Probably still a lot of broken code needing debugging.

* switch well_known_key to their associated module child trie.
Fix a genesis init (balance).
Complete some testing.
Comment some tests before using.

* fixing test runtime child keys

* latest commit fix broken genesis init

* fix system balances child name.

* Important fix: storage_root from test externalities need children (it is
already the case for ext).

* executive root with child calculation

* Avoid empty trie on test ext.

* Symetric removal of key for system.

* commenting changes related tests.

* Remove child module specifics.

* fix issues.

* fix some formatting

* fix bench and bump runtime

* Remove extend_storage_overlays, assimilate_storage do the same as is
proper considering srml macro.

* Fix warning for assimilate.

* Removing kill as they do not impact any test cases.

* Use tuple of storage map instead of two parameters. This changes the
behavior of decl_storage genesis build closure (breaking api).

* Do not use build storage before assimilate.

* fix error

* Update core/state-machine/src/backend.rs
2019-08-08 15:05:25 +02:00
cheme da8b91ae7b Trie simplification. (#2815)
* switch to simple codec, trie broken for now

* Actualy use trie_root_noext

* align some hash, failing test on EMCH comment

* Fix trie code over layout instead of hash, revert legacy code for legacy
mainnet ??

* stub behind LayOut

* fix no_std

* temp solution for legacy trie behind feature legacy-key in various crate

* use remote project

* rc client db need prefix

* update trie deps

* bum spec runtime version

* Removing legacy as default.

* Switch mode to non legacy.

* bump runtime version

* Remove legacy trie compatibility features.

* fix warning

* bump version

* change hash on new test.

* Move dependency (#11 trie PR) patched to a parity repo.
Bench reverted to correct hasher.
Some renaming and doc improvments.

* ChildBitmap renaming to BitMap.

* Renaming of LayOut to Layout.

* formatting.

* Removing abreviation such as _ix nb_ or bm.

* Update deps and apply renaming 'Buff' -> 'Buffer'.

* Align to latest trie crates naming changes.

* Update trie dependency.

* Update trie dependency.

* change block_import test hash

* update trie deps (trie use new scale codec but it does not seems to be
an issue).

* update to use latest trie version (no mgmt of multiple radix).

* tabify

* Restoring test to 10 000.

* Use published crate, trie bench is currently down until publishing
(require another pr to update version).

* Update trie-bench.
2019-08-02 19:51:59 +02:00
cheme fb8bb00158 Update trie dependencies. (#3017)
* Update trie dependencies.

* bump rutime impl version.
2019-07-04 15:56:13 +02:00
chemeandBastian Köcher f69c48c7b8 Additional state check. (#2915)
* check resulting state root on import.

* get root from last method.

* Fix test runtime (was changing state after root calculation).

* Do reset new authorities (just take before root calculation).

* bump impl runtime version.

* Update core/sr-api-macros/tests/trybuild.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
2019-06-26 16:28:12 +02:00
chemeandTomasz Drwięga e777038418 Add storage cache for child trie and notification internals (#2639)
* child cache, and test failing notifications

* fix tests and no listen child on top wildcard

* remove useless method

* bump impl version

* Update core/client/src/notifications.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update core/client/src/notifications.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update core/client/src/notifications.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update core/client/src/notifications.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* factoring notification methods to remove some redundant code.

* test child sub removal

* HStorage implementation and some type alias.

* Remove HStorage cache: does not fit

* fix removal

* Make cache use byte length (shared) instead of number of kv

* Make use of hashes cache in rpc

* applying ratio on different lru caches

* Fix format

* break a line

* Remove per element overhead of lru cache.

* typo
2019-06-14 11:25:20 +02:00
cheme 0d8379d5d2 Child trie storage proof (#2433)
* proof on child trie

* higher level api for child storage proof

* boilerplate for proof from light fetch

* actually check proof on light fetch

* Do not break former encoding

* tabify

* tabify2

* Add child trie root tx to full_storage_root transaction.

* Shorten long lines.

* Temp rename for audit

* Make full_storage a trait method

* Name back and replace some code with full_storage where it looks fine.

* fix indentations, remove unused import

* flush child root to top when calculated

* impl +1
2019-05-10 14:31:41 +02:00
cheme 1d65e53dfa assert trie odd nibble padding on decoding (#2218)
* Do not allow malleability in odd slice padding.

* align dep
2019-04-19 14:10:24 +02:00
cheme 47c1b4b26e Fix for miscalculation of storage root (#2285) 2019-04-16 15:28:53 +08:00
cheme 669e79181e no_std support for substrate trie (#2146)
* no_std trie compile in test_runtime (require to set nightly feature due
to the way hashbrown currently works).

* No nightly with hashmap_core.

* using crate elastic-array

* switch to publish trie crates

* fix default array decl

* bump impl_version for ci

* set all semver when possible wasm, and remove redundant code.

* Actually test use_trie function

* impl version +1

* Bump impl version
2019-04-02 12:49:04 +02:00
cheme dd47be51c3 Switch contract storage to child trie (#2002) 2019-03-22 18:01:06 +01:00
cheme ea2a03b7bb Merge metadata by module (#1428)
Clean up metadata format. Merge events metadata into runtime metadata. Remove low value info (internal macro identifier and counters).
2019-01-30 19:34:08 +01:00
cheme 8ec759d32e Temporary fix for serde issue (#1511)
* Generate serde bound instead of relying on serde_derive.

* remove comment
2019-01-21 19:23:22 +01:00
cheme 753ba7306a Document when a GenesisConfig is generated. (#1437)
* Document when a GenesisConfig is generated.

* Use nocompile.
2019-01-15 11:24:59 +01:00
cheme d271a4647c Decl_storage macro : force traits. (#1317)
* Missing trait def on two calls.

Slight mcla refact on transfo.

* Allways provide GenesisConfig even when useless (for runtime module
export).

* Revert "Allways provide GenesisConfig even when useless (for runtime module"

This reverts commit 84a29bc592023d1ee107519c58693ec19dbde2d2.

* Fix Vec usage (from rstd).
2019-01-07 16:05:47 +01:00
cheme 23634b6b95 Include default values in storage metadata (#1264)
* Add a 'default' field to metadata. It contains code to generate the
default value.

* wasm update

* Make 'default' field an `Option`

* Boxed fn is not static, that won't be fine

* static fn won't do it to as it cannot get T param, will try fat trait

* Fat pointer over phantom data compatible with static instantiation

* DecodeDifferent is cool, using it for decoding.

* using once cell to do what would require copying lazy_static internals.

* Remove cache when no_std (non compatible deps)

* wasm bins update

* Fuse tooling struct and enum derive.
2018-12-20 14:31:03 +01:00
cheme 9c8e3ffb52 Reformat types string for decl_storage (#1298)
* Clean space from types when converting to string

* Missing cases.

* Add unit test for complex type, complete clean fn.
2018-12-19 17:51:46 +01:00
cheme 5f48493b9a Allow spaces in types for decl_storage proc_macro (#1245)
* Do not try to fix type display for metadata in rust.

* Fix test (adding extra space (we can see it as a canary: if at some
point quote got a better output the test will fail and js space removal
could be remove)).
2018-12-10 16:24:31 +01:00
cheme ed5b78eaf0 decl_storage as a proc_macro (#1176)
* `decl_storage` parsing of the macro (TODO change tool crate structure)

* Start formatting, for now use inner macro.
Still missing optional formating last part (genesis ...).

* Calling extra genesis macro

* decl_storage lines parsing.

* genesis macro as quote (need some cleaning reorg)

* dirty $crate substitute

* proc crate reorg.

* PR impl : skip usage of phantom data, it only applies in test and
council (others required it).

* Remaining macro of decl_storage, warning stringify behave sometime
oddly.

* Formatting code and some cleaning.

* Include line parsing to main struct (cannot use existing macro anymore).

* Remove genesis phantom data when there is already a field with type
parameter.

* Revert wasm files

* Remove old version of `decl_storage`.

* Fix false positive for phantom trait (additional type check on config
build).

* slight changes:
- return token errors instead of panic
- do not use useless intermediate vec

* Update srml/support/procedural/tools/derive/src/lib.rs

remove indent

Co-Authored-By: cheme <emericchevalier.pro@gmail.com>

* Switch iterations to fold, remove unused import.
2018-12-09 21:09:39 +01:00