Commit Graph

568 Commits

Author SHA1 Message Date
Robert Hambrock 09a7e69012 Revert "Optimize merkle proofs for efficient verification in Solidity (#12857)" (#14176)
* Revert "Optimize merkle proofs for efficient verification in Solidity (#12857)"

This reverts commit c077597afb since we
still require commitment to the leaves - see #12820.

* remove PartialOrd trait from mmr hash type
2023-05-22 11:55:40 +00:00
Sergej Sakac 903d62beec Improve try-state docs (#13967)
* Improve Try-State docs

* fmt

* fixes
2023-05-21 23:30:49 +02:00
Liam Aharon 5100a0376e remote-externalities: retry for get_keys (#14167)
* retry get keys request

* remove redundant clone

* remove redundant at
2023-05-19 19:26:52 +00:00
Ankan c8b66415bb divide payload into atleast one chunk (#14086) 2023-05-07 16:06:06 +00:00
Brian Anderson 4be32b174b Upgrade wasm-opt to 0.112.0 (#13574)
Co-authored-by: parity-processbot <>
2023-05-05 08:10:34 +00:00
Liam Aharon ead46b9efd try-runtime-cli: 'instant' snapshots, threading refactor, better progress logs (#14057)
* remote externalities refactor

* remove redundant logs

* use const for parallel requests

* prefer functional

* improve variable naming

* handle requests error

* use overlayedchanges

* Revert "use overlayedchanges"

This reverts commit c0ddb87a5abdd52207597f5df66cbbdf9d79badc.

* Revert "Revert "use overlayedchanges""

This reverts commit 1d49362d9b999c045c8f970a0ab8b486bc47a90a.

* Revert "Revert "Revert "use overlayedchanges"""

This reverts commit 06df786488d94f249e9abccffac4af445f76e5a7.

* backup/load raw storage values

* test raw storage drain and restore

* update snapshot tests

* improve logs

* clippy suggestions

* address comments

* fix example

* fix test

* clippy
2023-05-05 17:15:40 +10:00
Michal Kucharczyk 6a295e7c28 substrate-test-runtime migrated to "pure" frame runtime (#13737)
* substrate-test-runtime migrated to pure-frame based

* test block builder: helpers added

* simple renaming

* basic_authorship test adjusted

* block_building storage_proof test adjusted

* babe: tests: should_panic expected added

* babe: tests adjusted

ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.

* beefy: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.

* grandpa: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.

* network:bitswap: test adjusted

The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.

* runtime apis versions adjusted

* storage keys used in runtime adjusted

* wasm vs native tests removed

* rpc tests: adjusted

Transfer transaction processing was slightly improved, test was
adjusted.

* tests: sizes adjusted

Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.

Sizes were adjusted in tests.

* cargo.lock update

cargo update -p substrate-test-runtime -p substrate-test-runtime-client

* warnings fixed

* builders cleanup: includes / std

* extrinsic validation cleanup

* txpool: benches performance fixed

* fmt

* spelling

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Apply code review suggestions

* Apply code review suggestions

* get rid of 1063 const

* renaming: UncheckedExtrinsic -> Extrinsic

* test-utils-runtime: further step to pure-frame

* basic-authorship: tests OK

* CheckSubstrateCall added + tests fixes

* test::Transfer call removed

* priority / propagate / no sudo+root-testing

* fixing warnings + format

* cleanup: build2/nonce + format

* final tests fixes

all tests are passing

* logs/comments removal

* should_not_accept_old_signatures test removed

* make txpool benches work again

* Cargo.lock reset

* format

* sudo hack removed

* txpool benches fix+cleanup

* .gitignore reverted

* rebase fixing + unsigned cleanup

* Cargo.toml/Cargo.lock cleanup

* force-debug feature removed

* mmr tests fixed

* make cargo-clippy happy

* network sync test uses unsigned extrinsic

* cleanup

* ".git/.scripts/commands/fmt/fmt.sh"

* push_storage_change signed call remove

* GenesisConfig cleanup

* fix

* fix

* GenesisConfig simplified

* storage_keys_works: reworked

* storage_keys_works: expected keys in vec

* storage keys list moved to substrate-test-runtime

* substrate-test: some sanity tests + GenesisConfigBuilder rework

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

* Review suggestions

* fix

* fix

* beefy: generate_blocks_and_sync block_num sync with actaul value

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update test-utils/runtime/src/genesismap.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* cargo update -p sc-rpc -p sc-transaction-pool

* Review suggestions

* fix

* doc added

* slot_duration adjusted for Babe::slot_duration

* small doc fixes

* array_bytes::hex used instead of hex

* tiny -> medium name fix

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* TransferData::try_from_unchecked_extrinsic -> try_from

* Update Cargo.lock

---------

Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-05-04 16:20:22 +00:00
Mira Ressel 793f04fc05 Bump clap to 4.2.5 (#14061)
Co-authored-by: parity-processbot <>
2023-05-02 17:54:18 +00:00
Liam Aharon 6e5141fc23 try-runtime-cli: improve ci stability (#14030)
* hardcode 1 thread

* ci stability

* fix comment

* improve comment

* kick ci

* kick ci

* update expect message

* improve comment

* kick ci

* kick ci

* configure threads with env var

* fix threads env var

* fix threads env var
2023-04-28 00:03:47 +10:00
Liam Aharon 27f89d653b remote-externalities: batch insert key/values (#14004)
* add batch inserting into remote externalities

* use into_iter

* remove redundant comment

* redundant batch insert

* avoid extra vec allocations
2023-04-27 18:12:29 +10:00
Squirrel 01c66da036 Allow missing docs for autogen weights. (#14011) 2023-04-25 22:48:37 +02:00
Liam Aharon bc1a599284 try-runtime: dynamic storage query sizes (#13923)
* improve batch rpc error message

* wip aimd storage data fetch

* complete aimd function refactor

* make batch_request function async

* improve function name

* fix load_child_remote issue

* slight efficiency improvement

* improve logs and variable name

* remove redundant comment

* improve comment

* address pr comments

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* simplify client handling

* fix type issue

* fix clippy issue

* try to trigger ci

* try to trigger ci

---------

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
2023-04-21 09:16:40 +00:00
dependabot[bot] 890a65b87d Bump cargo_metadata from 0.15.3 to 0.15.4 (#13899)
Bumps [cargo_metadata](https://github.com/oli-obk/cargo_metadata) from 0.15.3 to 0.15.4.
- [Release notes](https://github.com/oli-obk/cargo_metadata/releases)
- [Changelog](https://github.com/oli-obk/cargo_metadata/blob/main/CHANGELOG.md)
- [Commits](https://github.com/oli-obk/cargo_metadata/compare/0.15.3...0.15.4)

---
updated-dependencies:
- dependency-name: cargo_metadata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-14 14:52:51 +02:00
Oliver Tale-Yazdi ebfe00d590 Fixes PoV over-estimation (#13766)
* Align log

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use max instead of sum

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make comment ordering deterministic

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Dont add Pov overhead when all is ignored

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update test pallet weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Re-run weights on bm2

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Actually use new weights

Fucked up the merge for this file...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update contract weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-04-13 13:11:58 +00:00
dependabot[bot] 46e2d3e5f8 Bump toml from 0.5.11 to 0.7.3 (#13619)
Bumps [toml](https://github.com/toml-rs/toml) from 0.5.11 to 0.7.3.
- [Release notes](https://github.com/toml-rs/toml/releases)
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.5.11...toml-v0.7.3)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 11:30:42 +02:00
dependabot[bot] 929a510108 Bump zstd from 0.11.2+zstd.1.5.2 to 0.12.3+zstd.1.5.2 (#13671)
Bumps [zstd](https://github.com/gyscos/zstd-rs) from 0.11.2+zstd.1.5.2 to 0.12.3+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/commits)

---
updated-dependencies:
- dependency-name: zstd
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-12 22:50:31 +00:00
Roman Useinov 376a288fb6 [Deps] Bump scale-info to match cumulus and polkadot (#13873)
* [Deps] Bump scale-info to match cumulus and polkadot

* fix Cargo.lock

* fix identity

* more fixes

* fix

* fixes

* more fixes
2023-04-12 14:14:06 +02:00
yjh d5e460b3bf refactor(sc-executor): use wasm executor builder instead of old apis (#13740)
* refactor: use builder api for all executors

* improve a lot

* remove unused args

* cleanup deps

* fix inconsistency about heap alloc

* add `heap_pages` back to try-runtime

* fix

* chore: reduce duplicated code for sc-service-test

* cleanup code

* fmt

* improve test executor

* improve

* use #[deprecated]

* set runtime_cache_size: 4

* fix and improve

* refactor builder

* fix

* fix bench

* fix tests

* fix warnings

* fix warnings

* fix

* fix

* update by suggestions

* update name
2023-04-09 23:48:40 +00:00
alexd10s 23de70a536 Update the frame-weight-template to use PhantomData from core (#13844)
* update weights template and recalculate weights from pallet template

* update template file in benchmarking-cli
2023-04-06 16:19:46 +00:00
Liam Aharon 0b3444c51d Fix try-runtime follow-chain, try-runtime upgrade tuple tests, cli test utils (#13794)
* new test for try-runtime tuple stuff

* fix

* remove development comment

* formatting

* remove todo comment

* follow-chain working test

* refactor common cli testing utils

* fix comment

* revert Cargo.lock changes

* update Cargo.lock

* improve doc comment

* fix error typo

* update Cargo.lock

* feature gate try-runtime test

* build_substrate cli test util

* feature gate follow_chain tests

* move fn start_node to test-utils

* improve test pkg name

* use tokio Child and Command

* remove redundant import

* fix ci

* fix ci

* don't leave hanging processes

* improved child process cleanup

* use existing KillChildOnDrop

* remove redundant comment

* Update test-utils/cli/src/lib.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

---------

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Koute <koute@users.noreply.github.com>
2023-04-06 19:49:53 +10:00
Bastian Köcher 495773f96d Move registration of ReadRuntimeVersionExt to ExecutionExtension (#13820)
Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.
2023-04-05 14:27:26 +02:00
Bastian Köcher 846ec8cd01 Remove deprecated batch verification (#13799)
This removes the deprecated batch verification. This was actually never really activated.
Nevertheless, we need to keep the host functions around to support old runtimes which may import
these host functions. However, we do not give access to these functions anymore. This means that any new
runtime can not call them anymore. The host function implementations we keep will not do batch verification and will
instead fall back to the always existing option of directly verifying the passed signature.
`finish_batch_verification` will return the combined result of all the batch verify calls.

This removes the `TaskExecutorExt` which only existed to support the batch verification. So, any
code that used this extension can just remove the registration of them. It also removes
`SignatureBatching` that was used by `frame-executive` to control the batch verification.
However, there wasn't any `Verify` implementation that called the batch verification functions.
2023-04-04 10:02:47 +00:00
Koute 795fcf0484 Disable sign-ext WASM feature when building runtimes (#13804)
Co-authored-by: parity-processbot <>
2023-04-03 12:00:31 +00:00
Alexander Theißen 25a616ce08 Build wasm for mvp cpu (#13758) 2023-03-30 16:20:12 +02:00
Bastian Köcher d338e9a114 Support stable rust for compiling the runtime (#13580)
* Support stable rust for compiling the runtime

This pull request brings support for compiling the runtime with stable Rust. This requires at least
rust 1.68.0 to work on stable. The code is written in a way that it is backwards compatible and
should automatically work when someone compiles with 1.68.0+ stable.

* We always support nightlies!

* 🤦

* Sort by version

* Review feedback

* Review feedback

* Fix version parsing

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-03-29 07:17:50 +00:00
Bastian Köcher 685b43c3fc try-runtime: Use configured wasm execution method (#13694)
* try-runtime: Use configured wasm execution method

* Fix compilation
2023-03-24 17:50:16 +00:00
Oliver Tale-Yazdi c6a9b81fad try-runtime: run migration checks per default (#13684)
* try-runtime: run migration checks per default

The current behaviour of having to explicetly specify --checks seems
to cause confusion. Therefore bringing back the old behaviour of
running the pre- and post-upgrade checks per default.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-03-24 15:09:34 +00:00
Liam Aharon f1f2f617e8 Fix try_runtime_cli doc broken links (#13672) 2023-03-22 21:47:44 +11:00
Davide Galassi 480396fe06 Keystore overhaul (iter 2) (#13634)
* Remove bloat about remote keystore

* Update docs and remove unused 'KeystoreRef' trait

* Use wherever possible, MemoryKeystore for testing

* Remove unrequired fully qualified method syntax for Keystore
2023-03-20 18:21:26 +00:00
Mira Ressel faaa0c2851 address clippy & rustdoc warnings
The comment was out of date anyway, State::Live no longer takes a
snapshot_path argument.
2023-03-20 16:35:53 +01:00
Davide Galassi f110941b7f Keystore overhaul (#13615)
* Remove 'supported_keys' 'sign_with_any' and 'sign_with_all' from keystore trait

* Remove the aync keystore

* Renaming:
- SyncCryptoStore -> Keystore
- SyncCryptoStorePtr -> KeystorePtr
- KeyStore -> MemoryKeystore

* Fix authority discovery worker and tests

* Rename 'insert_unknown' to 'insert'

* Remove leftover
2023-03-17 11:24:14 +00:00
cheme ef165cec3e update trie deps (#13601) 2023-03-14 21:03:50 +00:00
Vivek Pandya 2009821cde Remove use of trait Store from all pallets and deprecate it. (#13535)
* Remove use of trait Store from staking pallet

* Remove use of trait Store from bounties pallet

* Remove use of trait Store from collective pallet

* Remove use of trait Store from babe pallet

* Remove use of trait Store from assets pallet

* Remove use of trait Store from grandpa pallet

* Remove use of trait Store from balances pallet

* Remove use of trait Store from authorship pallet

* Remove use of trait Store from authority-discovery pallet

* Remove use of trait Store from atomic-swap pallet

* Remove use of trait Store from sudo pallet

* Remove use of trait Store from scheduler pallet

* Remove use of trait Store from scored-pool pallet

* Remove use of trait Store from society pallet

* Remove use of trait Store from lottery pallet

* Remove use of trait Store from executive pallet

* Remove use of trait Store from democracy pallet

* Remove use of trait Store from elections-phragmen pallet

* Remove use of trait Store from indices pallet

* Remove use of trait Store from identity pallet

* Remove use of trait Store from multisig pallet

* Remove use of trait Store from merkle-mountain-range pallet

* Remove use of trait Store from im-online pallet

* Remove use of trait Store from membership pallet

* Remove use of trait Store from nicks pallet

* Remove use of trait Store from session pallet

* Remove use of trait Store from transaction-payment pallet

* Remove use of trait Store from utility pallet

* Remove use of trait Store from child-bounties pallet

* Remove use of trait Store from nis pallet

* Remove use of trait Store from nfts pallet

* Remove use of trait Store from conviction-voting pallet

* Remove use of trait Store from treasury pallet

* Remove use of trait Store from vesting pallet

* Remove use of trait Store from preimage pallet

* Remove use of trait Store from uniques pallet

* Remove use of trait Store from ranked-collective pallet

* Remove use of trait Store from beefy-mmr pallet

* Remove use of trait Store from referenda pallet

* Remove use of trait Store from whitelist pallet

* Remove use of trait Store from alliance pallet

* Remove use of trait Store from nomination-pools pallet

* Remove use of trait Store from state-trie-migration pallet

* Remove use of trait Store from message-queue pallet

* Remove use of trait Store from root-offences pallet

* Remove use of trait Store from root-testing pallet

* Remove use of trait Store from timestamps pallet

* Remove use of trait Store from system pallet

* Remove use of trait Store from offences pallet

* Remove use of trait Store from recovery pallet

* Remove use of trait Store from node-authorization pallet

* Remove use of trait Store from proxy pallet

* Remove use of trait Store from benchmarking pallet

* Remove use of trait Store from bags-list pallet

* Add deprecated warning in store_trait

* Change warning message

* Run cargo fmt

* Fix warning and update tests

* Remove unnecessary allow deprecated

* Remove use of trait Store

* Fix mismatch in expected output

* Minor update to warning message for deprecation of generate_store with Store trait attribute

* Fixes as per review comments

* Fixes as per review suggestions

* Remove use of Store trait from core-fellowship pallet

* Fix type in store_trait.rs

* Fixes as pre review comment
2023-03-13 15:30:05 +00:00
Oliver Tale-Yazdi 9e56e1acdd Deprecate Weight::from_{ref_time, proof_size} (#13475)
* Deprecate Weight::from_{ref_time, proof_size}

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update templates

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use from_parts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use from_parts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Dont revert comment 🤦

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_balances

* Update weight files

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Adapt to Master changes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2023-03-02 21:28:17 +00:00
Koute 55263fa2a1 Speed up storage iteration from within the runtime (#13479)
* Speed up storage iteration from within the runtime

* Move the cached iterator into an `Option`

* Use `RefCell` in no_std

* Simplify the code slightly

* Use `Option::replace`

* Update doc comment for `next_storage_key_slow`
2023-03-01 08:58:18 +00:00
Oliver Tale-Yazdi 1eb0cd31b9 Fix V2 PoV benchmarking (#13485)
* Bump default 'additional_trie_layers' to two

The default here only works for extremely small runtimes, which have
no more than 16 storage prefices. This is changed to a "sane" default
of 2, which is save for runtimes with up to 4096 storage prefices (eg StorageValue).

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update tests and test weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix PoV weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_balances

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_message_queue

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton

* Fix sanity check

>0 would also do as a check, but let's try this.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2023-02-28 13:47:36 +00:00
Bastian Köcher 941288c6d0 wasm-executor: Support growing the memory (#12520)
* As always, start with something :P

* Add support for max_heap_pages

* Add support for wasmtime

* Make it compile

* Fix compilation

* Copy wrongly merged code

* Fix compilation

* Some fixes

* Fix

* Get stuff working

* More work

* More fixes

* ...

* More

* FIXEs

* Switch wasmi to use `RuntimeBlob` like wasmtime

* Removed unused stuff

* Cleanup

* More cleanups

* Introduce `CallContext`

* Fixes

* More fixes

* Add builder for creating the `WasmExecutor`

* Adds some docs

* FMT

* First round of feedback.

* Review feedback round 2

* More fixes

* Fix try-runtime

* Update client/executor/wasmtime/src/instance_wrapper.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/runtime_blob/runtime_blob.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Feedback round 3

* FMT

* Review comments

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-02-24 11:43:01 +00:00
Gonçalo Pestana c848d40775 Revert "Abstracts elections-phragmen pallet to use NposSolver (#12588)" (#13451)
This reverts commit b793666ca5.
2023-02-23 12:15:08 -06:00
Gonçalo Pestana b793666ca5 Abstracts elections-phragmen pallet to use NposSolver (#12588)
* Abstracts elections-phragmen pallet to use NposSolver

* Update frame/elections-phragmen/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections-phragmen/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* changes the name of the pallet; adds changelog

* update changelog

* Adds weight testing

* Adds log macro_rules

* renames elections-phragment dir to elections

* weights rename

* fixes typo in cargo toml

* pre/post solve weight scafolding

* refactor do_post_election

* refactors into pre and post election solve for independent benchmarking

* deconstructs PreElectionResults struct

* updates benchmarking pre and post election solve; mock weights

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* addresses PR comments

* adds pre_solve and post_sove weights

* Adds comments on election pallet id param name change

* ".git/.scripts/bench-bot.sh" pallet dev pallet_elections

* Finishes pre-post solve weights

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Addresses PR comments: no panic in on_init path; nits

* Fixes node build

* Implements approval voting to use as a `NposSolver` (#13367)

* Implements the approval voting methods in sp_npos_elections

* fmt

* remove unecessary file

* comment clarification

* re-run weights

* fix typo

* updates MaxVoters in tests for integrity_tests to pass

* Refactors election provider support benchmarks outside its own crate (#13431)

* Refactors election provider support benchmarks outside its own crate
---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: parity-processbot <>
Co-authored-by: Ross Bulat <ross@parity.io>
2023-02-23 11:21:00 +00:00
Koute f8e3bdad3d Rework storage iterators (#13284)
* Rework storage iterators

* Make sure storage iteration is also accounted for when benchmarking

* Use `trie-db` from crates.io

* Appease clippy

* Bump `trie-bench` to 0.35.0

* Fix tests' compilation

* Update comment to clarify how `IterArgs::start_at` works

* Add extra tests

* Fix iterators on `Client` so that they behave as before

* Add extra `unwrap`s in tests

* More clippy fixes

* Come on clippy, give me a break already

* Rename `allow_missing` to `stop_on_incomplete_database`

* Add `#[inline]` to `with_recorder_and_cache`

* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment

* Simplify code: use `with_trie_db` in `next_storage_key_from_root`

* Remove `expect`s in the benchmarking CLI

* Add extra doc comments

* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)

* Remove a TODO in tests

* Update comment for `StorageIterator::was_complete`

* Update `trie-db` to 0.25.1
2023-02-22 07:49:25 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Michal Kucharczyk 7a10154188 BlockId removal: runtime-api refactor (#13255)
* BlockId removal: refactor of runtime API

It changes the arguments of:
- `ApiExt` methods:  `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`

It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* BlockId removal: refactor of runtime API - tests

- tests adjusted to new runtime API,
- some tests migrated from block number to block hash

* benchmarking-cli: BlockId(0) migrated to info().genesis_hash

`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.

* trivial: api.call(BlockId) -> api.call(Hash)

- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,

* CallApiAtParams::at changed to Block::Hash

* missed doc updated

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

* BlockId removal: Benchmark::consumed_weight

Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.

* at_hash renamed

* wrong merge fixed

* beefy worker: merged with master

* beefy: tests: missing block problem fixed

* Apply review suggestion

* fix

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
2023-02-20 22:47:21 +00:00
Piotr Mikołajczyk 3595d87182 try-runtime::fast-forward (#12896)
* try-runtime::fast-forward

* Revert un`pub`ing command's fields

* Handle storage change failure

* Adjust Substrate node

* Feature-gated imports

* doc link

* Feature-gated imports in node-template

* Move trait, blanket implementation and auxiliary functions to a new module

* Distinguish between plain babe+timestamp and substrate enhanced info

* Remove uncles inherents

* Missing argument

* Add doc comment about `blocktime_millis`

* Add licenses
2023-02-17 12:21:58 +00:00
Davide Galassi ecafcd430b Fork tree prune assumptions removal v2 (#13327)
* Removed assumptions about ancestry from fork tree prune method

* Tests improvement

* Fork tree prune refactory

* Code refactory

* Correctly handle borderline, but legit, case

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Removed duplicated test

---------

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2023-02-15 11:25:26 +01:00
Shawn Tabrizi 6bbc2292e4 Improve Weight Template and API (#13355)
* improve weights template and api

* follow template
2023-02-12 15:01:06 +00:00
Koute 44cbf303c4 Bump wasmtime to 5.0.0 (and a few other deps) (#13160)
* Bump `wasmtime` to 4.0.0 (and a few other deps)

* Use `Error::msg` instead of `anyhow!`

* Bump `wasmtime` to 5.0.0

* Update `Cargo.lock`

* Add `wasmtime` feature to `sp-wasm-interface` dependency
2023-02-06 15:40:50 +00:00
Davirain 25388b267e Move beefy-merkle-tree to utils/binary-merkle-tree and make it generic (#13076)
* move BeefyMmrApi to pallet-beefy-mmr

* fix test_utils use pallet-beefy-mmr BeefyMmrApi

* Move beefy-merkle-tree to utils and Rename to Merkle-tree

* fix fmt and test

* Update merkle-tree to binary-merkle-tree and Remove Keccak256 mod from merkle-tree

* change merkle-tree name to binary-merkle-tree

* mirr fix
2023-02-02 10:07:48 +00:00
Shawn Tabrizi 1b27ae9549 Add Proof Size to Weight Output (#11637)
* initial impl

* add template test

* linear fit proof size

* always record proof when tracking storage

* calculate worst case pov

* remove duplicate worst case

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* more comment output

* add cli for worst case map size

* update name

* clap does not support underscores

* rename

* expose worst case map values

* improve some comments

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* update template

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix fmt

* more fmt

* more fmt

* Dont panic when there is no proof

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Whitelist :extrinsic_index

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use whitelist when recording proof

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add logs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add PoV testing pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Deploy PoV testing pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Storage benches reside in the PoV pallet

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Linear regress PoV per component

Splits the PoV calculation into "measured" and "estimated".
The measured part is reported by the Proof recorder and linear
regressed over all components at once.
The estimated part is calculated as worst-case by using the max
PoV size per storage access and calculating one linear regress per
component. This gives each component a (possibly) independent PoV.
For now the measured size will always be lower than the PoV on
Polkadot since it is measured on an empty snapshot. The measured
part is therefor only used as diagnostic for debugging.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Put PoV into the weight templates

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Extra alanysis choise for PoV

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add+Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make benches faster

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use same template comments

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_balances

* ".git/.scripts/bench-bot.sh" pallet dev pallet_democracy

* Update referenda mock BlockWeights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Take measured value size into account

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_scheduler

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* proof_size: None

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ugly, but works

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* wup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add pov_mode attribute to the benchmarks! macro

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use pov_mode attribute in PoV benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Scheduler, Whitelist: Add pov_mode attr

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update PoV weights

* Add CLI arg: default-pov-mode

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Update PoV weights"

This reverts commit 2f3ac2387396470b118122a6ff8fa4ee12216f4b.

* Revert "WIP"

This reverts commit c34b538cd2bc45da4544e887180184e30957904a.

* Revert first approach

This reverts commit range 8ddaa2fffe5930f225a30bee314d0b7c94c344dd^..4c84f8748e5395852a9e0e25b0404953fee1a59e

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add extra benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_alliance

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_whitelist

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_scheduler

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy 🤦

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add reference benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix doc comments

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Undo logging

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add 'Ignored' pov_mode

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Allow multiple attributes per benchmark

Turns out that the current benchmarking syntax does not support
multiple attributes per bench 🤦. Changing it to support that
since otherwise the `pov_mode` would conflict with the others.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Validate pov_mode syntax

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Ignore PoV for all contract benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bump macro recursion limit

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update contract weights

They dont have a PoV component anymore.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix test ffs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* pov_mode is unsupported in V2 syntax

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix pallet ui tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* update pallet ui

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix pallet ui tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Your Name <you@example.com>
2023-01-26 22:35:39 +00:00
zjb0807 73a9776266 Fix try-runtime with create-snapshot (#13223)
* Fix try-runtime with create-snapshot

* Apply review suggestions
2023-01-26 00:12:19 +01:00
Brian Anderson 545de88c47 Upgrade wasm-opt to 0.111.0 (#13038) 2023-01-24 18:21:52 +00:00