Commit Graph

69 Commits

Author SHA1 Message Date
Dcompoze 002d9260f9 Fix spelling mistakes across the whole repository (#3808)
**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
2024-03-26 13:57:57 +00:00
Squirrel 1b5f4243d1 sp-std removal from substrate/primitives (#3274)
This PR removes sp-std crate from substrate/primitives sub-directories.

For now crates that have `pub use` of sp-std or export macros that would
necessitate users of the macros to `extern crate alloc` have been
excluded from this PR.

There should be no breaking changes in this PR.

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2024-03-18 05:29:35 +00:00
Bastian Köcher ecf8035da6 Set StateBackend::Transaction to PrefixedMemoryDB (#14612)
* Yep

* Try to get it working everywhere

* Make `from_raw_storage` start with an empty db

* More fixes!

* Make everything compile

* Fix `child_storage_root`

* Fix after merge

* Cleanups

* Update primitives/state-machine/src/overlayed_changes/mod.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Review comments

* Fix issues

* Silence warning

* FMT

* Clippy

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-08-17 10:49:38 +00:00
Koute 91c595bc78 Further storage iterator refactoring (#13445)
* Remove `Backend::apply_to_key_values_while`

* Add `IterArgs::start_at_exclusive`

* Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while`

* Remove `Backend::apply_to_keys_while`

* Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix`

* Remove unnecessary `to_vec` calls

* Fix unused method warning in no_std

* Remove unnecessary import

* Also check proof sizes in the test

* Iterate over both keys and values in `prove_range_read_with_size` and add a test
2023-02-27 10:45:42 +00:00
Koute f8e3bdad3d Rework storage iterators (#13284)
* Rework storage iterators

* Make sure storage iteration is also accounted for when benchmarking

* Use `trie-db` from crates.io

* Appease clippy

* Bump `trie-bench` to 0.35.0

* Fix tests' compilation

* Update comment to clarify how `IterArgs::start_at` works

* Add extra tests

* Fix iterators on `Client` so that they behave as before

* Add extra `unwrap`s in tests

* More clippy fixes

* Come on clippy, give me a break already

* Rename `allow_missing` to `stop_on_incomplete_database`

* Add `#[inline]` to `with_recorder_and_cache`

* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment

* Simplify code: use `with_trie_db` in `next_storage_key_from_root`

* Remove `expect`s in the benchmarking CLI

* Add extra doc comments

* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)

* Remove a TODO in tests

* Update comment for `StorageIterator::was_complete`

* Update `trie-db` to 0.25.1
2023-02-22 07:49:25 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Gavin Wood ecbd65fb95 Safe and sane multi-item storage removal (#11490)
* Fix overlay prefix removal result

* Second part of the overlay prefix removal fix.

* Report only items deleted from storage in clear_prefix

* Fix kill_prefix

* Formatting

* Remove unused code

* Fixes

* Fixes

* Introduce clear_prefix host function v3

* Formatting

* Use v2 for now

* Fixes

* Formatting

* Docs

* Child prefix removal should also hide v3 for now

* Fixes

* Fixes

* Formatting

* Fixes

* apply_to_keys_whle takes start_at

* apply_to_keys_whle takes start_at

* apply_to_keys_whle takes start_at

* Cursor API; force limits

* Use unsafe deprecated functions

* Formatting

* Fixes

* Grumbles

* Fixes

* Docs

* Some nitpicks 🙈

* Update primitives/externalities/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Formatting

* Fixes

* cargo fmt

* Fixes

* Update primitives/io/src/lib.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Formatting

* Fixes

Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-05-29 12:56:26 +01:00
Falco Hirschenberger b581604aa7 Apply some clippy lints (#11154)
* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2022-04-30 21:28:27 +00:00
Liu-Cheng Xu 47f96b6584 Remove unused parent_hash in OverlayedChanges::into_storage_changes (#11011)
Ref https://github.com/paritytech/substrate/pull/10922#issuecomment-1064258443

CC @cheme
2022-03-10 19:35:11 +01:00
Squirrel 3dd32d5255 Make clippy _a little_ more annoying (#10570)
* Clippy: +complexity

* Update client/cli/src/arg_enums.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/inspect/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/keystore/src/testing.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update primitives/npos-elections/fuzzer/src/reduce.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Incorporating feedback

* No need for Ok

* Additional

* Needed slice

* Wigy's suggestions on less derefs

* fix count

* reverting changes brought in by option_map_unit_fn

* add --all-targets

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-01-05 14:35:30 +00:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
cheme 4c651637f2 Inner hashing of value in state trie (runtime versioning). (#9732)
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
2021-12-24 08:54:07 +00:00
Arkadiy Paronyan 4cbbf0cf43 Kill the light client, CHTs and change tries. (#10080)
* Remove light client, change tries and CHTs

* Update tests

* fmt

* Restore changes_root

* Fixed benches

* Cargo fmt

* fmt

* fmt
2021-11-12 14:15:01 +01:00
Bastian Köcher 62187b5916 Prepare UI tests for rust 1.55 (#9637)
* Prepare UI tests for rust 1.54

* Delete wrong_page.stderr

* CI: run with a staging CI image

* Revert "CI: run with a staging CI image"

This reverts commit 66f5b00d14b50fd9d8fbf773f7e884f380697591.

* CI: debug, again

* LOG_TARGET is only used in std

* Remove unnecessary unsafe

* Fixes

* Use correct rustc locally

* FMT

* Compile with benchmarking

* Review feedback

* Some ui tests

* I know...

* Fix wasm tests

Co-authored-by: Denis P <denis.pisarev@parity.io>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
2021-09-24 10:42:08 +00:00
Kian Paimani abd08e29ce Rewrap all comments to 100 line width (#9490)
* reformat everything again

* manual formatting

* last manual fix

* Fix build
2021-08-11 14:56:55 +00:00
Bastian Köcher f07f69301a Improve state related logs to use a more uniform format (#9452)
* Improve `state` related logs to use a more uniform format

The logging before wasn't that uniform and not that great to read/parse.
Now we are using a uniform format for all the logs. Besides these
changes, there are some minor changes around the code that calls the
state machine.

* Make CI happy

* Use HexDisplay for `ext_id`
2021-07-29 11:43:03 +02:00
Squirrel aafe64315a Remove extra commas made redundent after rustfmt (#9404)
* Remove extra commas made redundent after rustfmt
2021-07-22 11:06:17 +01:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Shawn Tabrizi b42b8fc5fb Include StorageInfo in Benchmarking Pipeline (#9090)
* extend storageinfo

* extend_storage_info

* use vec

* add storage info to pipeline

* get read and written keys

* undo storageinfo move

* refactor keytracker

* return read / write count

* playing with key matching

* add basic `StorageInfo` constructor

* add whitelisted to returned info

* fix some test stuff

* pipe comments into benchmark data

* add_storage_comments

* add comments to template

* track only storage prefix

* Update frame/benchmarking/src/lib.rs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix test

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* remove test logs

* add temp benchmark script

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* remove keytracker and use trackedstoragekey

* add comment for unknown keys

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* remove duplicate comments with unknown keys

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* refactor bench tracker, and fix results

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix child tries in new tracker

* extra newline

* fix unused warning

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix master merge

* storage info usage refactor

* remove now unused

* fix refactor

* use a vec for prefix

* fix tests

* also update writer to use vec

* disable read and written keys for now

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Update frame/system/src/weights.rs

* fix test

* Delete weights.rs

* reset weights

Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2021-07-07 22:06:06 +00:00
Squirrel 07449840bd Less slices (#9176)
* Less slices

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-06-23 13:33:48 +01:00
cheme 693b39f43b Frame remove_all with size limit. (#9106)
* remove prefixed content with limit.

* test match

* factor comment and factor ext limit removal.

* fix benchmark

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-06-15 13:23:58 +00:00
Arkadiy Paronyan 84811dae00 Storage chain: Runtime module (#8624)
* Transaction storage runtime module

* WIP: Tests

* Tests, benchmarks  and docs

* Made check_proof mandatory

* Typo

* Renamed a crate

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Added weight for on_finalize

* Fixed counter mutations

* Reorganized tests

* Fixed build

* Update for the new inherent API

* Reworked for the new inherents API

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Store transactions in a Vec

* Added FeeDestination

* Get rid of constants

* Fixed node runtime build

* Fixed benches

* Update frame/transaction-storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-06-04 06:50:59 +00:00
Bastian Köcher fa22eb7361 Optimize next_storage_key (#8956)
* Optimize `next_storage_key`

- Do not rely on recursion
- Use an iterator over the overlay to not always call the same method

* Fix bug
2021-05-31 20:17:15 +02:00
Frederik Schulz ad76ac798e Removes unnecessary blank impl for Backend (#8897)
* Removes unnecessary blank impl for Backend

This commit removes a from my perspective unneccessary implementation
for &T which implement Backend.

The current implementation exists (again from my perspective) solely
to satisfy a methods &mut self parameters (i.e. allows to satisfy
this for an & reference via using &mut &Backend).

As all implementors use a RefCell with borrow_mut() where actually
calling the mentioned &mut self method and then forwad to the
{} implementation of either TrieBackend or ProvingBackend, the
current &mut self seems to be not needed.

* Fixed tests client
2021-05-25 19:50:12 +00:00
mattrutherford 6e8957b095 RPC to trace execution of specified block (#7780)
* Add filter reload handle

* add RPC, move logging module from cli to tracing

* remove dup fn

* working example

* Update client/rpc-api/src/system/mod.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Prefer "set" to "reload"

* Re-enable the commented out features of the logger

* Remove duplicate code

* cleanup

* unneeded lvar

* Bump to latest patch release

* Add new CLI option to disable log filter reloading,

Move profiling CLI options to SharedParams

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Applied suggestions from reviews

* Fix calls to init_logger()

* Handle errors when parsing logging directives

* Deny `system_setLogFilter` RPC by default

* One more time

* Don't ignore parse errors for log directives set via CLI or RPC

* Improve docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix merge errors

* include default directives with system_setLogFilter RPC,

implement system_rawSetLogFilter RPC to exclude defaults

* docs etc...

* update test

* refactor: rename fn

* Add a test for system_set_log_filter – NOTE: the code should likely change to return an error when bad directives are passed

* Update client/cli/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address review grumbles

* Add doc note on panicking behaviour

* print all invalid directives before panic

* change RPCs to: addLogFilter and resetLogFilter

* make CLI log directives default

* add comments

* restore previous behaviour to panic when hard-coded directives are invalid

* change/refactor directive parsing

* fix line width

* add test for log filter reloading

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* finish up suggestions from code review

* improve test

* change expect message

* change fn name

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* add docs, remove unused fn

* propagate Err on invalid log directive

* Update tracing-subscriber version

* Improve docs for `disable_log_reloading` CLI param

* WIP implementation: RPC and trace capturing

* WIP

* fix incorrect number of digest items

* return errors

* add From impl for Values, rename structs

* fixes

* implement option to choose targets for traces

* rename fn

* fix EnvFilter and add root span

* fix root span

* add docs, remove unnecessary traits

* fix regression on parent_id introduced in a9c73113a8

* fix line width

* remove unused

* include block hash, parent hash & targets in response

* move types from sp-tracing into sp-rpc

move block and parent hash into root of BlockTrace

* switch from log::trace to tracing::trace in state-machine

* use unsigned integer type to represent Ext::id in traces

* ensure id is unique by implementing Subscriber

tracing_subscriber::FmtSubscriber does not guarantee
unique ids

* indentation

* fix typo

* update types

* add sp_io::storage events

* Change response format

- update types
- record distinct timestamps
- sort spans by first entered

* convert to HexDisplay, refactor

* Sort out fallout from merge

* Update client/rpc-api/src/state/mod.rs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Exit early unless the node runs with --rpc-methods=Unsafe

* Better error handling

* Use wasm-timer

* revert trace alteration in `state-machine` and remove events in `sp_io::storage`

Resolve in follow-up PR

* Review feedback: less collects

* Without Arcs

* Fix span exit

* typo

* cleanup

* Add a few debug messages to tracing module

* Structure traces state-machine/ext; Dispatchable extrinsics spans not working

* Correctly encode Option storage values

* Remove test field for Put and Get

* Try out some changes to dispatch macro

* Add various log messages in dispatch

* Add span dispatch span to new proc macro

* Remove debug messages in dispatch

* Trivial clean up

* Structure remaining state-machine traces (ChangesRoot*)

* Removed unnesecary tracing targets

* Remove log

* New cargo.lock post merge

* Add logging for wasm_overrides

* remove temp logs

* remove temp logs

* remove unused dep

* remove temp logs

* add logging to wasm_overrides

* add logging to state_tracing

* add logging for spans to substrate (includes timings)

* Skip serializing some event fields; Remove most storage traces

* Bring back all ext.rs traces

* Do not skip bool values in events

* Skip serializing span values

* Serialize span values; remove some trace events in ext

* Remove more trace events

* Delete commented out traces

* Remove all unused traces

* Add event filtering

* Fix typo

* wip - change response types to be more efficient

missing import

type

* Serialize struct fields as camelCase

* Add back in event filtering

* Remove name field from event

* Sort spans by time entered

* Sort spans in ASCending order

* Add storage keys target param to rpc

* Limit payload size; improve hash fields; include storage keys

- cleanup event_key_filter
- better block hash representation
- limit payload size
- cleanup based on andrews comments

* Error when serialized payload is to big

* Import MAX_PAYLOAD from rpc-servers

* Clean up ext.rs

* Misc. cleaning and comments

* Strict ordering span Id; no span sort; adjust for rpc base payload

* Add RPC docs to rpc-api/src/state/mod

* Make params bullet points

* Update primitives/rpc/src/tracing.rs

* Put all tracing logic within trace

* Remove attr.record in new_span

* Add back value record in new_span

* restore result collection in ext

* Revert "Add back value record in new_span"

This reverts commit baf1a735f23e5eef1bf6264adfabb788910fa661.

* 🤦

* more 🤦

* Update docs; Try fix line width issues

* Improve docs

* Improve docs

* Remove default key filters + add key recs to docs

* Try restore old traces

* Add back doc comment

* Clean up newlines in ext.rs

* More new line remova;
l

* Use FxHashMap

* Try use EnvFilter directives for event filtering

* Remove directive, filter events by fields

* Use trace metadata correctly

* Try EnvFilter directive with all default targets

* Revert "Try EnvFilter directive with all default targets"

This reverts commit 4cc6ebc721d207c3b846444174f89d45038525ac.

* Clean up clippy warning

* Incorporate Niklas feedback

* Update trace/log macro calls to have better syntx

* Use Ordering::Relaxed

* Improve patch and filter doc comment

* Clean up `BlockSubscriber::new`

* Try optimize `BlockSubscriber::enabled`

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Use contains_key

* use heuristic for payload size

* Add error tupe for client::tracing::block

* Minor tweaks

* Make a note about `--features with-tracing`

* Add CURL example to RPC docs

* Link to substrate-archibe wasm

* Trivial doc clean up based on David feedback

* Explicit result type name

* Respect line length

* Use the error

* Don't print timings when spans close

* Fix failing sc-rpc-api

* Update  sp-tracing inner-line doc

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address some review grumbles

* Update primitives/state-machine/src/ext.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Use result_encoded structure fields in ext.rs

* Use value key for ext put

* Add notes about tracing key names matter

Co-authored-by: Matt <mattrutherford@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
2021-05-05 07:44:36 +00:00
Shawn Tabrizi c04b44b0a3 Add PoV Tracking to Benchmarking Pipeline (#8559)
* Added a function to estimate proof size for benchmarking

* integrate proof_size into benchmarking pipeline

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix tests

* one more test

* Update bench.rs

* Update utils/frame/benchmarking-cli/src/writer.rs

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: arkpar <arkady.paronyan@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-04-08 12:54:46 +02:00
Arkadiy Paronyan 4a0d6d9490 Storage chains: indexing, renewals and reference counting (#8265)
* Transaction indexing

* Tests and fixes

* Fixed a comment

* Style

* Build

* Style

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Code review suggestions

* Add missing impl

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* impl JoinInput

* Don't store empty slices

* JoinInput operates on slices

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-03-18 11:46:27 +00:00
Shawn Tabrizi d25229bc89 Return number of keys removed when calling storage_kill on child trie (#8166)
* Initial piping of returning amount of keys killed

* One more test for `None` limit

* forgot to update

* fix return value

* use version 3

* Update to return `KillOutcome`

* Update name to KillChildStorageResult
2021-02-22 19:24:12 +00:00
Bastian Köcher 20527aec0e Fix warning in rustdoc job (#8159)
* Fix warning in rustdoc job

* More fixes

* Remove `build-rust-doc` job

Remove this job until upstream is fixed:
https://github.com/rust-lang/rust/issues/82284

* CI: temp. remove of the publishing job, no use of it w/o build

Co-authored-by: Denis P <denis.pisarev@parity.io>
2021-02-19 15:04:23 +01:00
cheme 878f7ccf6e Allow transaction for offchain indexing (#7290)
* Moving offchain change set to state machine overlay change set,
preparing use of change set internally.

* Make change set generic over key and value, and use it for offchain
indexing.

* test ui change

* remaining delta

* generating with standard method

* Remove 'drain_committed' function, and documentation.

* Default constructor for enabling offchain indexing.

* Remove offchain change specific iterators.

* remove pub accessor

* keep previous hierarchy, just expose iterator instead.

* Update primitives/state-machine/src/overlayed_changes/mod.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* fix line break

* missing renamings

* fix import

* fix new state-machine tests.

* Don't expose InnerValue type.

* Add test similar to set_storage.

* Remove conditional offchain storage (hard to instantiate correctly).

* fix

* offchain as children cannot fail if top doesn't

Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2021-01-22 12:27:43 +00:00
Bastian Köcher cd0ad4805d Make offchain indexing work (#7940)
* Make offchain indexing work

This fixes some bugs with offchain indexing to make it actually working ;)

* Fix tests

* Fix browser build

* Update client/db/src/offchain.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Remove seperation between prefix and key

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
2021-01-21 13:12:42 +01:00
cheme 1922dab050 Fix clear prefix check to avoid erasing child trie roots. (#7848)
* Fix clear prefix check to avoid erasing child trie roots.

* Renaming and extend existing test with check.

* last nitpicks.
2021-01-13 12:45:31 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Alexander Theißen ec2c70396e Define ss58 prefix inside the runtime (#7810)
* Add SS58Prefix type to the frame_system config trait

* Remove unused chain_id runtime interface
2020-12-30 22:07:37 +00:00
Alexander Theißen 9ce24fe1f4 Allow capping the amount of work performed when deleting a child trie (#7671)
* Allow Backend::for_keys_in_child_storage to be aborted by the closure

* Ext::kill_child_storage now takes an upper limit for backend deletion

* Add Storage::storage_kill_limited() runtime interface

* review: Use a new version of kill_storage instead of a new interface

* review: Simplify boolean expression

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* review: Rename for_keys_in_child_storage

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
2020-12-09 01:17:28 +00:00
Bastian Köcher dbf4558633 Remove runtime registered extensions after execution (#7236)
* Remove runtime registered extensions after execution

This prevents a bug when an extension was registered in native, but the
native execution aborted without removing the extension again. Now, when
executing the wasm code the extension is still registered and it fails
of being registered. So, the wasm execution fails as well. This can
happen for example for the `VerificationExt`.

* Make it better
2020-10-01 09:12:05 +00:00
cheme 232a30fdb4 state_machine no_std witness externalities (#6934)
* checkpoint before removing CT from change trie

* before trie backend without tx

* undo

* Started no transaction, but would need using a different root
calculation method, out of the scope of this pr, will roll back.

* Remove NoTransaction.

* partially address review.
dummy stats implementation for no_std.

* Remove ChangeTrieOverlay.

* modified function

* Remove witness_ext

* need noops changes root

* update from cumulus branch

* line break

* remove warning

* line break

* From review: renamings and stats active in no std (except time).

* include cache, exclude change trie cache with individual temporary bad looking
no_std check

* little test

* fuse imports and filter_map prepare_extrinsics_input_inner fold.

* put back ExtInner into Ext, awkward double proto for new function.

* Apply suggestions from code review

* Update primitives/state-machine/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-09-11 10:11:25 +00:00
Guillaume Thiolliere 9b42720ac5 Fix benchmark read/write key tracker for keys in child storages. (#6905)
* WIP: read child trie and write child trie

* add test

* refactor a bit + improve log

* better naming

* trigger CI

* Revert "trigger CI"

This reverts commit d0aadaeb6a12fc6c39f01b3c1b5725d19f085865.
2020-08-24 13:40:16 +00:00
Shawn Tabrizi 481ad884d6 Dynamic Benchmarking DB Whitelist (#6815)
* Add `get_whitelist` api

* add whitelisted caller

* Whitelist caller

* remove caller 0

* initial piping of origin (not actual value yet)

* remove attempt to pass origin around

* Add whitelist for `DidUpdate` storage on `pallet_timestamp`

* fix traits

* only add to whitelist if !contains

* PassBy not implemented error

* Whitelist read/writes explicitly per key

* update docs

* reduce trait constraint

* copy pasta

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* rename functions @apopiak

* missed some renaming

* enable doc tests

* Update docs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
2020-08-19 16:15:50 +00:00
pscott 046fda914a Improve overall performance (#6699)
* Improve overall performance

* Clean up code

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-21 14:46:49 +02:00
Alexander Theißen f53e5355f8 Re-enter runtime after resetting overlay from runtime (#6513)
This still assumes that the client did not start any transactions
before calling into runtime. This is the case for benchmarking
as long as either NativeWhenPossible or AlwaysWasm exection
strategy is chosen. Using any other will result in a panic.
2020-06-25 22:52:05 +02:00
Shawn Tabrizi 935ee6f545 Add DB Read/Write Tracking to Benchmarking Pipeline (#6386)
* initial mockup

* add and wipe

* track writes

* start to add to pipeline

* return all reads/writes

* Log reads and writes from bench db

* causes panic

* Allow multiple commits

* commit before ending benchmark

* doesn't work???

* fix

* Update lib.rs

* switch to struct for `BenchmarkResults`

* add to output

* fix test

* line width

* @kianenigma review

* Add Whitelist to DB Tracking in Benchmarks Pipeline (#6405)

* hardcoded whitelist

* Add whitelist to pipeline

* Remove whitelist pipeline from CLI, add to runtime

* clean-up unused db initialized whitelist

* Add regression analysis to DB Tracking (#6475)

* Add selector

* add tests

* debug formatter for easy formula

* Update client/db/src/bench.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: arkpar <arkady.paronyan@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-06-24 21:03:55 +02:00
Alexander Theißen bb2df2122e Implement nested storage transactions (#6269)
* Add transactional storage functionality to OverlayChanges

A collection already has a natural None state. No need to
wrap it with an option.

* Add storage transactions runtime interface

* Add frame support for transactions

* Fix committed typo

* Rename 'changes' variable to 'overlay'

* Fix renaming change

* Fixed strange line break

* Rename clear to clear_where

* Add comment regarding delete value on mutation

* Add comment which changes are covered by a transaction

* Do force the arg to with_transaction return a Result

* Use rust doc comments on every documentable place

* Fix wording of insert_diry doc

* Improve doc on start_transaction

* Rename value to overlayed in close_transaction

* Inline negation

* Improve wording of close_transaction comments

* Get rid of an expect by using get_or_insert_with

* Remove trailing whitespace

* Rename should to expected in tests

* Rolling back a transaction must mark the overlay as dirty

* Protect client initiated storage tx from being droped by runtime

* Review nits

* Return Err when entering or exiting runtime fails

* Documentation fixup

* Remove close type

* Move enter/exit runtime to excute_aux in the state-machine

* Rename Discard -> Rollback

* Move child changeset creation to constructor

* Move child spawning into the closure

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fixup for code suggestion

* Unify re-exports

* Rename overlay_changes to mod.rs and move into subdir

* Change proof wording

* Adapt a new test from master to storage-tx

* Suggestions from the latest round of review

* Fix warning message

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-06-23 11:17:53 +02:00
Bastian Köcher 82a832bc3a Don't clone values when calculating storage root (#6108)
* Don't clone values when calculating storage root

Instead of cloning all the keys and values of the overlay when
calculating the storage root, we pass all the values by reference. This
should probably bring some performance improvements when calculating the
storage root.

* no cow version (#6113)

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
2020-05-23 11:28:34 +02:00
Alexander Theißen 7a5bdb896b Refactor OverlayedChanges (#5989)
* Hide internal structure of OverlayChanges

* Fix tests for OverlayChanges refactor

* Do not clone pending changes

Discarding prospective changes should be equivalent as a state machine
is not to be called with peding changes.

This will be replaced by a storage transaction that is rolled back before
executing the call the second time removing this constraint.

* Doc fixes

* Remove overlong line

* Revert "Do not clone pending changes"

This reverts commit 4799491f4ac16f8517287a0fcf4a3f84ad56f46e.

* Deduplicate chield tries returned from child_infos()

* Remove redundant type annotation

* Avoid changing the storage root in tests

* Preserve extrinsic indices in trie build test

* Swap order of comitted and prospective in fn child_infos

This is only for consistency and does not impact the result.

* Rename set_pending to replace_pending for clearity
2020-05-20 11:39:45 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Shawn Tabrizi a9523dbc37 unify spacing on state trace logs (#5932) 2020-05-07 09:24:07 +02:00
Nikolay Volf d3d0ccff12 add state trace (#5924) 2020-05-06 17:33:48 +02:00
Bastian Köcher bc9707b4c5 Use storage::append in the implementation of the storage types (#5889)
* Start improving `storage_append`

* Fix some stuff

* Fix compilation

* Update docs and add new test

* More tests

* Test kill + append
2020-05-05 15:09:07 +02:00
cheme c05ec630ff Avoid changing overlay committed layer. (#5839)
* Avoid changing overlay committed layer.

* basic test

* Add some tx in the test.

* only update from backend value on missing entry in both layer.
deleted entry is replace by empty vec.

* test and review changes

* additional test and review change

* remove test on changing existing value, it does not always panic
depending on existing content

* Update primitives/state-machine/src/overlayed_changes.rs

* Update primitives/state-machine/src/overlayed_changes.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-04-30 13:59:21 +02:00