Commit Graph

1156 Commits

Author SHA1 Message Date
Sam Johnson c0e007b50f Add special tag to exclude runtime storage items from benchmarking (#12205)
* initial setup

* add WhitelistedStorageKeys trait

* add (A, B) tuple implementation for whitelisted_storage_keys()

* fix formatting

* implement WhitelistedStorageKeys for all tuple combinations

* impl_for_tuples up to 128 for WhitelistedStorageKeys

* refactor to #[benchmarking(cached)]

* tweak error message and mark BlockNumber as cached

* add benchmarking(cached) to the other default types

* add docs for benchmarking(cached)

* properly parse storage type declaration

* make storage_alias structs public so we can use them in this macro

* use BTreeMap since TrackedStorageKey missing Ord outside of std

* make WhitelistedStorageKeys accessible

* basic detection of benchmarking(cached) 💥

* proper parsing of #[benchmarking(cached)] from pallet parse macro

* store presence of #[benchmarking(cached)] macro on StorageDef

* will be used for later expansion

* compiling blank impl for WhitelistedStorageKeys

* move impl to expand_pallet_struct

* use frame_support::sp_std::vec::Vec properly

* successfully compiling with storage info loaded into a variable 💥

* plausible implementation for whitelisted_storage_keys()

* depends on the assumption that storage_info.encode() can be loaded
  into TrackedStorageKey::new(..)

* use Pallet::whitelisted_storage_keys() instead of hard-coded list

* AllPallets::whitelisted_storage_keys() properly working 💥

* collect storage names

* whitelisted_storage_keys() impl working 💥

* clean up

* fix compiler error

* just one compiler error

* fix doc compiler error

* use better import path

* fix comment

* whoops

* whoops again

* fix macro import issue

* cargo fmt

* mark example as ignore

* use keyword tokens instead of string parsing

* fix keyword-based parsing of benchmarking(cached)

* preliminary spec for check_whitelist()

* add additional test for benchmarking whitelist

* add TODO note

* remove irrelevant line from example

* use filter_map instead of filter and map

* simplify syntax

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* clean up

* fix test

* fix tests

* use keyword parsing instead of string parsing

* use collect() instead of a for loop

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* fix compiler error

* clean up benchmarking(cached) marking code

* use cloned()

* refactor to not use panic! and remove need for pub types in storage_alias

* remove unneeded use

Co-authored-by: Bastian Köcher <info@kchr.de>

* remove unneeded visibility changes

* don't manually hard code hash for treasury account as hex

* proper Ord, PartialOrd, and Hash impls for TrackedStorageKey

* now based just on key, and available in no-std

* use BTreeSet instead of BTreeMap

* fix comments

* cargo fmt

* switch to pallet::whitelist and re-do it basti's way :D

* make PartialOrd for TrackedStorageKey consistent with Ord

* more correct implementation of hash-related traits for TrackedStorageKey

* fix integration test

* update TODO

* remove unused keyword

* remove more unused keywords

* use into_iter()

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update frame/support/procedural/src/pallet/parse/mod.rs

Co-authored-by: Bastian Köcher <info@kchr.de>

* add docs for whitelisted

* fix comment

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2022-09-16 21:45:44 +00:00
Davide Galassi a4ebb6d31c Replace 'blake2-rfc' with rust-crypto 'blake2' crate (#12266)
* Replace 'blake2-rfc with rust-crypto 'blake2' crate
* Bump blake2 to 0.10.4
2022-09-15 17:12:55 +02:00
Oliver Tale-Yazdi 45966d509e Fix Rust features (#11976)
* Add std feature

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* WIP

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Impl function also in tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix sp-trie feature

Something makes the bench regression guard fail, maybe this?

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add runtime-benchmarks feature to sc-service

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Revert "Fix sp-trie feature"

This reverts commit f2cddfe41bc72e6f2f8133795ec9408ba0c3ec63.
Was already fixed, only needed a CI retry.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2022-09-14 16:31:58 +00:00
Shawn Tabrizi dd1f1c4d7b fix sp-weight std (#12265) 2022-09-14 15:43:19 +00:00
Keith Yeung 30e7b1e8cd Create sp-weights crate to store weight primitives (#12219)
* Create sp-weights crate to store weight primitives

* Fix templates

* Fix templates

* Fixes

* Fixes

* cargo fmt

* Fixes

* Fixes

* Use deprecated type alias instead of deprecated unit types

* Use deprecated subtraits instead of deprecated hollow new traits

* Fixes

* Allow deprecation in macro expansion

* Add missing where clause during call macro expansion

* cargo fmt

* Fixes

* cargo fmt

* Fixes

* Fixes

* Fixes

* Fixes

* Move FRAME-specific weight files back to frame_support

* Fixes

* Update frame/support/src/dispatch.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/support/src/dispatch.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/support/src/dispatch.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Add missing header

* Rewrite module docs

* Fixes

* Fixes

* Fixes

* Fixes

* cargo fmt

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2022-09-13 13:23:44 +00:00
Michal Kucharczyk 214eb25f87 Remove CanAuthorWith trait (#12213)
* Remove CanAuthorWith trait

CanAuthotWith trait removed. Also all dependencies, parameters, type
paramers were removed. This is related to removal of native runtime.

* Remove commented code

* Fix code formatting

* trigger CI job

* trigger CI job

* trigger CI job

* trigger CI job

* trigger CI job

* trigger CI job

* trigger CI job
2022-09-13 12:19:26 +00:00
yjh 472b5746e5 extract some grandpa types to Primitives crate (#12208)
* extract some grandpa types to primitives

* fmt

* fmt
2022-09-12 21:34:43 +00:00
Kian Paimani 113727950b Fuzz testing for nomination pools (#12002)
* some additional tests and stuff

* make sanity public

* add some sort of fuzz test for pools

* breaks every now and then

* breaks every now and then

* IT WORKS AND PASSES 100k TESTS

* cleanup

* safe id addition

* fix assert_eq_error_rate

* Update frame/nomination-pools/src/tests.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/nomination-pools/src/tests.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add some doc

* Fix

* ".git/.scripts/fmt.sh" 1

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2022-09-12 14:27:11 +00:00
Bastian Köcher b356a5589a Remove native call (#12201)
* Remove native call

With the recent introduction of staging runtime apis the native call wasn't supported anymore. This
removes the entire support for this as it is not used anymore.

* FMT

* Fix benchmarks

* FIX ui tests
2022-09-12 12:25:56 +02:00
MOZGIII c40ae08641 Update ss58-registry (#12220) 2022-09-08 16:37:23 +00:00
Alexander Theißen 5afe00d89c Upgrade wasm crate dependencies (#12173)
* Upgrade wasm crate dependencies

* New wasmi version changed error output a bit

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

Co-authored-by: command-bot <>
2022-09-08 12:48:19 +00:00
Bastian Köcher bec123a50f Upgrade wasmtime to 0.40.0 (#12096)
* Upgrade wasmtime to 0.40.0

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

* Update in the other Cargo.toml

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
2022-09-08 10:57:23 +02:00
Keith Yeung 99d8111c67 Move Get and bounded types to sp-core (#12203)
* Move Get and bounded types to sp-core

* Fixes

* cargo fmt

* Fixes
2022-09-07 15:41:45 +00:00
NingLin-P 56e536f6e8 staking: avoid proportional slashing leak dust into chunks that should not be slashed (#12058)
* replace slash ratio with remaining ratio

Signed-off-by: linning <linningde25@gmail.com>

* little refactor

Signed-off-by: linning <linningde25@gmail.com>

* fix test

Signed-off-by: linning <linningde25@gmail.com>

* fix typo

Signed-off-by: linning <linningde25@gmail.com>

* revert refactor

Signed-off-by: linning <linningde25@gmail.com>

* rounding up instead of remaining ratio

Signed-off-by: linning <linningde25@gmail.com>

* address comment

Signed-off-by: linning <linningde25@gmail.com>

* Update frame/nomination-pools/test-staking/src/lib.rs

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/nomination-pools/test-staking/src/lib.rs

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Roman Useinov <roman.useinov@gmail.com>

Signed-off-by: linning <linningde25@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Roman Useinov <roman.useinov@gmail.com>
Co-authored-by: parity-processbot <>
2022-09-06 12:24:31 +00:00
Davide Galassi 83aec6f93a Fetch Babe configuration from runtime state (#11760)
* Fetch babe config data from runtime state

* Some renaming

* More renaming

* Final nits

* Fix tests and benches

* Rename  to  in BabeConfiguration

* Remove duplicate babe parameter description

Already specified over the 'PRIMARY_PROBABILITY' constant value

* trigger pipeline

* trigger pipeline
2022-09-05 17:41:32 +00:00
Bastian Köcher f6f6570459 Fix flaky trie-db cache test (#12139) 2022-08-29 17:11:19 +02:00
dependabot[bot] 75a76d967b Bump serde_json from 1.0.79 to 1.0.85 (#12079)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.79 to 1.0.85.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.85)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-26 09:32:20 +02:00
dependabot[bot] 45655e485b Bump async-trait from 0.1.51 to 0.1.57 (#11948)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.51 to 0.1.57.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.51...0.1.57)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-26 09:31:56 +02:00
dependabot[bot] 5318bf53d4 Bump parking_lot from 0.12.0 to 0.12.1 (#12098)
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.12.0 to 0.12.1.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.12.0...0.12.1)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-25 23:52:19 +02:00
Bastian Köcher 7c465dfe6a Fix wasm_export_functions when using a return statement (#12107)
When used a return statement before we directly returned from the function which is obviously wrong.
2022-08-25 13:18:53 +02:00
Bastian Köcher e62e670631 Print bootnode mismatch message as warning (#12075)
* Print bootnode mismatch message as `warning`

* Fix test :thinking_face:

* Remove wrong warning
2022-08-23 22:22:58 +02:00
Bastian Köcher 73d9ae3284 Introduce trie level cache and remove state cache (#11407)
* trie state cache

* Also cache missing access on read.

* fix comp

* bis

* fix

* use has_lru

* remove local storage cache on size 0.

* No cache.

* local cache only

* trie cache and local cache

* storage cache (with local)

* trie cache no local cache

* Add state access benchmark

* Remove warnings etc

* Add trie cache benchmark

* No extra "clone" required

* Change benchmark to use multiple blocks

* Use patches

* Integrate shitty implementation

* More stuff

* Revert "Merge branch 'master' into trie_state_cache"

This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.

* Improve benchmark

* Adapt to latest changes

* Adapt to changes in trie

* Add a test that uses iterator

* Start fixing it

* Remove obsolete file

* Make it compile

* Start rewriting the trie node cache

* More work on the cache

* More docs and code etc

* Make data cache an optional

* Tests

* Remove debug stuff

* Recorder

* Some docs and a simple test for the recorder

* Compile fixes

* Make it compile

* More fixes

* More fixes

* Fix fix fix

* Make sure cache and recorder work together for basic stuff

* Test that data caching and recording works

* Test `TrieDBMut` with caching

* Try something

* Fixes, fixes, fixes

* Forward the recorder

* Make it compile

* Use recorder in more places

* Switch to new `with_optional_recorder` fn

* Refactor and cleanups

* Move `ProvingBackend` tests

* Simplify

* Move over all functionality to the essence

* Fix compilation

* Implement estimate encoded size for StorageProof

* Start using the `cache` everywhere

* Use the cache everywhere

* Fix compilation

* Fix tests

* Adds `TrieBackendBuilder` and enhances the tests

* Ensure that recorder drain checks that values are found as expected

* Switch over to `TrieBackendBuilder`

* Start fixing the problem with child tries and recording

* Fix recording of child tries

* Make it compile

* Overwrite `storage_hash` in `TrieBackend`

* Add `storage_cache` to  the benchmarks

* Fix `no_std` build

* Speed up cache lookup

* Extend the state access benchmark to also hash a runtime

* Fix build

* Fix compilation

* Rewrite value cache

* Add lru cache

* Ensure that the cache lru works

* Value cache should not be optional

* Add support for keeping the shared node cache in its bounds

* Make the cache configurable

* Check that the cache respects the bounds

* Adds a new test

* Fixes

* Docs and some renamings

* More docs

* Start using the new recorder

* Fix more code

* Take `self` argument

* Remove warnings

* Fix benchmark

* Fix accounting

* Rip off the state cache

* Start fixing fallout after removing the state cache

* Make it compile after trie changes

* Fix test

* Add some logging

* Some docs

* Some fixups and clean ups

* Fix benchmark

* Remove unneeded file

* Use git for patching

* Make CI happy

* Update primitives/trie/Cargo.toml

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Introduce new `AsTrieBackend` trait

* Make the LocalTrieCache not clonable

* Make it work in no_std and add docs

* Remove duplicate dependency

* Switch to ahash for better performance

* Speedup value cache merge

* Output errors on underflow

* Ensure the internal LRU map doesn't grow too much

* Use const fn to calculate the value cache element size

* Remove cache configuration

* Fix

* Clear the cache in between for more testing

* Try to come up with a failing test case

* Make the test fail

* Fix the child trie recording

* Make everything compile after the changes to trie

* Adapt to latest trie-db changes

* Fix on stable

* Update primitives/trie/src/cache.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fix wrong merge

* Docs

* Fix warnings

* Cargo.lock

* Bump pin-project

* Fix warnings

* Switch to released crate version

* More fixes

* Make clippy and rustdocs happy

* More clippy

* Print error when using deprecated `--state-cache-size`

* 🤦

* Fixes

* Fix storage_hash linkings

* Update client/rpc/src/dev/mod.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Review feedback

* encode bound

* Rework the shared value cache

Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.

* Improve of the cache by merging the old and new solution

* FMT

* Please stop coming back all the time :crying:

* Update primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Fixes

* Make clippy happy

* Ensure we don't deadlock

* Only use one lock to simplify the code

* Do not depend on `Hasher`

* Fix tests

* FMT

* Clippy 🤦

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2022-08-18 18:59:22 +00:00
Squirrel a68a80fbae declone and close the door (#12035)
* declone and close the door

* cargo fmt

* remove brackets
2022-08-15 19:38:36 +00:00
Jake Hemmerle d987d2b413 swap ed25519-dalek for ed25519-zebra (#11781)
* swap ed25519-dalek for ed25519-zebra; no batch verificaiton

fixed batch verificaiton tests

removed additional zero verificaiton tests

removed comments, fixed test bug, added #[derive(Clone)]

Update primitives/core/src/ed25519.rs

Co-authored-by: Squirrel <gilescope@gmail.com>

* modified assertion to allow default ed25519-zebra zero key behavior

* cargo clippy

* Update primitives/core/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/core/src/ed25519.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update primitives/core/src/ed25519.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update primitives/core/src/ed25519.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update primitives/core/src/ed25519.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* updated Cargo.lock for sp-core

* fix inaccurate comment

Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
2022-08-15 09:53:39 +02:00
Tsvetomir Dimitrov 2bff2f84e3 Runtime API versioning (#11779)
* Runtime API versioning

Related to issue #11577

Add support for multiple versions of a Runtime API. The purpose is to
have one main version of the API, which is considered stable and
multiple unstable (aka staging) ones.

How it works
===========
Some methods of the API trait can be tagged with `#[api_version(N)]`
attribute where N is version number bigger than the main one. Let's call
them **staging methods** for brevity.

The implementor of the API decides which version to implement.

Example (from https://github.com/paritytech/substrate/issues/11577#issuecomment-1145347025):

```
decl_runtime_apis! {
    #{api_version(10)]
    trait Test {
         fn something() -> Vec<u8>;
         #[api_version(11)]
         fn new_cool_function() -> u32;
    }
}
```

```
impl_runtime_apis! {
    #[api_version(11)]
    impl Test for Runtime {
         fn something() -> Vec<u8> { vec![1, 2, 3] }

         fn new_cool_function() -> u32 {
             10
         }
    }
}
```

Version safety checks (currently not implemented)
=================================================
By default in the API trait all staging methods has got default
implementation calling `unimplemented!()`. This is a problem because if
the developer wants to implement version 11 in the example above and
forgets to add `fn new_cool_function()` in `impl_runtime_apis!` the
runtime will crash when the function is executed.

Ideally a compilation error should be generated in such cases.

TODOs
=====

Things not working well at the moment:
[ ] Version safety check
[ ] Integration tests of `primitives/api` are messed up a bit. More
specifically `primitives/api/test/tests/decl_and_impl.rs`
[ ] Integration test covering the new functionality.
[ ] Some duplicated code

* Update primitives/api/proc-macro/src/impl_runtime_apis.rs

Code review feedback and formatting

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* Code review feedback

Applying suggestions from @bkchr

* fmt

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Code review feedback

* dummy trait -> versioned trait

* Implement only versioned traits (not compiling)

* Remove native API calls (still not compiling)

* fmt

* Fix compilation

* Comments

* Remove unused code

* Remove native runtime tests

* Remove unused code

* Fix UI tests

* Code review feedback

* Code review feedback

* attribute_names -> common

* Rework `append_api_version`

* Code review feedback

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Code review feedback

* Code review feedback

* Code review feedback

* Use type alias for the default trait - doesn't compile

* Fixes

* Better error for `method_api_ver < trait_api_version`

* fmt

* Rework how we call runtime functions

* Update UI tests

* Fix warnings

* Fix doctests

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix formatting and small compilation errors

* Update primitives/api/proc-macro/src/impl_runtime_apis.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2022-08-13 13:56:40 +02:00
Oliver Tale-Yazdi 8e927daa77 Add defer (#12013)
* Add defer

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Convert to macro + review fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Move into new file and add panic unwind tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix doc and Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-08-12 17:18:27 +00:00
benluelo 2bda6abae7 Add map and try_map methods to BoundedBTreeMap (#11869)
* Add map and try_map methods to BoundedBTreeMap

* Undo changes to basic_authorship.rs

* Remove unwrap and use unchecked_from instead

* Add iter_mut() method

* Remove map functions and add docs to iter_mut

* fmt

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2022-08-11 18:55:30 +00:00
yjh 9a7ff9a3a0 chore: improve runtime docs and remove unused error (#12000) 2022-08-11 11:59:38 +02:00
Bastian Köcher aa5f68a827 transactional: Wrap pallet::calls directly in storage layers (#11927)
* transactional: Wrap `pallet::calls` directly in storage layers

Before this pr we only wrapped `pallet::calls` into storage layers when executing the calls with
`dispatch`. This pr is solving that by wrapping each call function inside a storage layer.

* Teach `BasicExternalities` transactions support

* Fix crates

* FMT

* Fix benchmarking tests

* Use correct span

* Support old decl macros

* Fix test

* Apply suggestions from code review

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/state-trie-migration/src/lib.rs

* Update frame/state-trie-migration/src/lib.rs

* Update frame/state-trie-migration/src/lib.rs

* Feedback

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
2022-08-10 22:27:01 +02:00
Oliver Tale-Yazdi 85e2092d8f Add BoundedVec::sort_by_key (#11998)
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2022-08-10 08:39:24 +00:00
Nazar Mokrynskyi a685582bfd Network sync refactoring (part 6) (#11940)
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency

* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`

* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls

* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Minimize `sc-authority-discovery` dependency on `sc-network`

* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`

* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`

* Move more methods from `NetworkService` into `NetworkPeers` trait

* Move `NetworkStateInfo` trait into `sc-network-common`

* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`

* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl

* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`

* Remove dependencies on `NetworkService` from most of the methods of `sc-service`

* Address simple review comments
2022-08-09 18:28:32 +00:00
Sacha Lansky df07421904 Fix docs urls (#11966)
* Fix docs urls

* Update bin/node-template/README.md

* Update frame/aura/src/lib.rs

* Run cargo +nightly fmt

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-08-03 13:34:33 +02:00
Andronik 0cda69d34a offences: make fn slash_fraction non-static (#11956)
* offences: make fn slash_fraction non-static

* Bastifmt (inline variable)
2022-08-02 18:49:03 +00:00
Adrian Catangiu 1c6867c6ed Lean BEEFY to Full BEEFY - don't skip (older) mandatory blocks and import justifications (#11821)
* client/beefy: don't accept vote for older rounds

* client/beefy: clean up and reorg the worker struct

* client/beefy: first step towards Full BEEFY

The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.

There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.

Implementation details:

- Introduced 'VoterOracle' to separate the voting decisions logic,
  and track new/pending sessions.

- New sessions get queued up with the worker operating either:
  1. up-to-date - all mandatory blocks leading up to current GRANDPA
     finalized: queue has ONE element, the 'current session' where
     `mandatory_done == true`,
  2. lagging behind GRANDPA: queue has [1, N] elements, where all
     `mandatory_done == false`.
     In this state, everytime a session gets its mandatory block
     BEEFY finalized, the session is popped off the queue,
     eventually getting to operating mode `1. up-to-date`.

- Votes get triaged and those that fall withing the `VoterOracle`
  allowed window get processed, the others get dropped if stale,
  or buffered for later processing (when they reach the window).

- Worker general code was also updated to fall in one of two roles:
  1. react to external events and change internal 'state',
  2. generate events/votes based on internal 'state'.

Signed-off-by: acatangiu <adrian@parity.io>

* client/beefy: sketch idea for block import and sync

Signed-off-by: acatangiu <adrian@parity.io>

* client/beefy: add BEEFY block import

* client/beefy: process justifications from block import

* client/beefy: add TODOs for sync protocol

* client/beefy: add more docs and comments

* client/beefy-rpc: fix RPC error

* client/beefy: verify justification validity on block import

* client/beefy: more tests

* client/beefy: small fixes

- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.

* client/beefy: remove invalid justifications at block import

* todo: beefy block import tests

* RFC: ideas for multiple justifications per block

* Revert "RFC: ideas for multiple justifications per block"

This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.

* client/beefy: append justif to backend on block import

* client/beefy: groundwork for block import test

* client/beefy: groundwork2 for block import test

* client/beefy: groundwork3 for block import test

* client/beefy: add block import test

* client/beefy: add required trait bounds to block import builder

* remove client from beefy block import, backend gets the job done

Signed-off-by: acatangiu <adrian@parity.io>
2022-07-29 15:47:21 +00:00
Koute c4b607d4c9 Prevent double allocation of the payload when calling sp_io::storage::get (#11523)
* Expose allocation stats in `FreeingBumpHeapAllocator`

* Return allocation stats when calling into the runtime

* Bump `parity-scale-codec` to 3.1.3 (fork)

* Prevent double allocation of the payload when calling `sp_io::storage::get`

* Fix tests

* Remove unnecessary `mut`

* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Bump `parity-scale-codec` to 3.1.3

* Fix some of the UI tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-07-29 07:46:15 +00:00
Sebastian Kunert 9f409dc0b8 Prepare for rust 1.62.1 (#11903)
* Update UI test output for rust 1.62.1

* switch ci to staging image to check that everything works

* fix artifacts node-bench-regression-guard

* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning

* Fix more clippy lints

* Make clippy happy by relying on auto-deref were possible

* Add tracking issue to the comments

* pin ci image

Co-authored-by: alvicsam <alvicsam@gmail.com>
2022-07-26 14:37:05 +02:00
Qinxuan Chen 16ef0cee93 Pruned duplicated dependencies (#11900)
* Update comfy-table v5.0.1 => v6.0.0

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update strum v0.23.0 => v0.24.1

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update h2 v0.3.9 => v0.3.13

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update file-per-thread-logger v0.1.4 => v0.1.5

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update mio v0.8.0 => v0.8.4

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* revert twox-hash

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update secp256k1 v0.21.2 => v0.24.0

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2022-07-25 16:52:11 +00:00
Bastian Köcher 3d95c270e0 Disable the interest cache (#11854)
* Disable the interest cache

The feature is currently broken with the latest `tracing-core`. We will enable it again, when it is
fixed upstream.

* FMT
2022-07-18 21:25:18 +02:00
Sergej Sakac a0a806ac78 removed evaluation.rs (#11766)
* removed evaluation.rs

* no need for parent_hash

* fix
2022-07-16 23:38:38 +02:00
Adrian Catangiu 27b538222e pallet-mmr: handle forks without collisions in offchain storage (#11594)
* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: acatangiu <adrian@parity.io>

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
2022-07-07 15:02:37 +00:00
Koute 025e1d76f1 Bump wasmtime to 0.38.0 and zstd to 0.11.2 (#11720)
* Bump `wasmtime` to 0.37.0 and `zstd` to 0.11.2

* Bump `wasmtime` to 0.38.0
2022-06-24 11:25:21 +00:00
Adrian Catangiu 3018051cdd pallet-beefy-mmr: add API for BEEFY Authority Sets (#11406)
* pallet-beefy: add Config::OnNewValidatorSet type

Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.

For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.

* pallet-beefy-mmr: implement OnNewValidatorSet

Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.

Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.

* pallet-beefy-mmr: add new authority_set() API

Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.

Signed-off-by: acatangiu <adrian@parity.io>

* rename BeefyMmr exposed runtime api
2022-06-22 18:39:24 +00:00
Keith Yeung 939c31bfde Implement Serialize/Deserialize on WeakBoundedVec (#11713)
* Implement Serialize/Deserialize on WeakBoundedVec

* cargo fmt

* Warn when there are too many elements while deserializing WeakBoundedVec
2022-06-21 06:49:43 +00:00
Bastian Köcher 94976af3d5 Update syn and fix compilation (#11707)
* Update syn and fix compilation

* Bump pin-project
2022-06-20 09:53:41 +00:00
dependabot[bot] 19ce14c1ff Bump twox-hash from 1.6.2 to 1.6.3 (#11423)
Bumps [twox-hash](https://github.com/shepmaster/twox-hash) from 1.6.2 to 1.6.3.
- [Release notes](https://github.com/shepmaster/twox-hash/releases)
- [Commits](https://github.com/shepmaster/twox-hash/compare/v1.6.2...v1.6.3)

---
updated-dependencies:
- dependency-name: twox-hash
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-19 21:01:38 +02:00
yjh 3f549bad6c chore: reducing codec times (#11688) 2022-06-18 00:14:50 +02:00
Boluwatife Bakre 0108d216d2 Remove multiply_by_rational (#11598)
* Removed multiply_by_rational
Replaced with multiply_by_rational_with_rounding

* fixes

* Test Fixes

* nightly fmt

* Test Fix

* Fixed fuzzer.
2022-06-17 03:57:29 +00:00
Gavin Wood c47431118b MEL: Origin, Referenda, ConvictionVoting (#11631)
* Referenda & CV pallets ready

* Fix build

* Add mel_bound for Voting and Casting types

* Add mel_bound on Tally

* Add mel_bound on another Tally

* Add mel_bound for pallet_collective::RawOrigin

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-06-16 16:13:17 +00:00
Georges b71e180446 combine iteratons and tolerance in sp-npos-elections API (#11498)
* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Fixing rust docs by adding () to resolve ambiguity

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Fixing rust docs by adding () to resolve ambiguity

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Removing left over from rebase

* Fixing tests

* Removing unneeded import

* Removing unneeded functions

* Removing useless imports

Co-authored-by: kianenigma <kian@parity.io>
2022-06-15 23:30:22 +00:00
André Silva 06cf8ad076 grandpa: fix creation of justification with equivocating precommits in commit (#11302)
* grandpa: fix creation of justification ancestry

we would reject commits that have precommits targeting blocks lower than the
commit target. when there is an equivocation (or if it the commit is not
minimal) it is possible to have such precommits and we should assume that they
are the round base.

* grandpa: bump to 0.16.0

* grandpa: add test for justification with equivocation

* grandpa: fix failing test
2022-06-14 10:41:11 +00:00