Commit Graph

16 Commits

Author SHA1 Message Date
Kian Paimani e6c83710e6 Decouple Staking and Election - Part 3: Signed Phase (#7910)
* Base features and traits.

* pallet and unsigned phase

* add signed phase.

* remove comments

* Undo bad formattings.

* some formatting cleanup.

* Small self-cleanup.

* Add todo

* Make it all build

* self-review

* Some doc tests.

* Some changes from other PR

* Fix session test

* Update bin/node/runtime/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix name.

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* typos and verbiage

* no glob imports in signed.rs

* meaningful generic type parameters for SignedSubmission

* dedup feasibility check weight calculation

* simplify/optimize fn insert_submission

* tests: remove glob, cause to build without error

* use sp_std::vec::Vec

* maintain invariant within fn insert_submission

* fix accidentally ordering the list backward

* intentionally order the list in reverse

* get rid of unused import

* ensure signed submissions are cleared in early elect

* finalize the signed phase when appropriate

- ensure we don't leave storage lying around, even if elect called prematurely
- test that proposition
- disable the unsigned phase if a viable solution from the signed phase exists
- ensure signed phase finalization weight is accounted for

* resolve dispatch error todo

* update assumptions in submit benchmark

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* line length

* make a few more things pub

* restore missing import

* update ui test output

* update tests from master branch

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* remove duplicate definitions

* remove signed reward factor due to its attack potential

* Update frame/election-provider-multi-phase/src/signed.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* remove SignedRewardMax; no longer necessary

* compute the encoded size without actually encoding

* remove unused PostInfo

* pub use some stuff

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* ensure `pub use` things are in fact `pub`

* add event information: was another solution ejected to make room

* unconditionally run the unsigned phase even if signed was successful

* remove dead test code

* meaningful witness data name

* use errors instead of defensive `unwrap_or_default`

* get rid of a log message redundant with an event

* saturating math

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* import Saturating

* mv `fn submit` to end of call

* add log line

* Use a better data structure for SignedSubmissions instead of Vec (#8933)

* Remove: (#8748)

* `NetworkStatusSinks`
* `sc_service::SpawnTasksParams::network_status_sinks`

Also:
* `sc_service::build_network()` does not return `network_status_sinks`

* CI: fix simnet trigger (#8927)

* CI: chore

* CI: pin simnet version

* More sc-service config reexports (#8887)

* Reexport ExecutionStrategies and ExecutionStrategy

* Reexport more of the network

* Reexport the ExecutionStrategy as it's used within ExecutionStrategies

* Fix check runtime CI (#8930)

* Fix check_runtime.sh script

* contracts: Remove confusing "Related Modules" doc

* Bump parity-wasm and pwasm-utils to the newest versions everywhere (#8928)

* BROKEN: convert SignedSubmissions to BoundedBTreeSet

Eventually, once it works, this change should improve overall performance.
However, in the meantime, the trait bounds aren't playing nicely, and
this is turning into too much of a pain to handle right now as part of
/#7910. We can take care of it later.

* Simple `MaxBoundedLen` Implementations (#8793)

* implement max_values + storages info

* some formatting + doc

* sudo sanity check

* timestamp

* assets (not working)

* fix assets

* impl for proxy

* update balances

* rename StoragesInfo -> PalletStorageInfo

* merge both StorageInfoTrait and PalletStorageInfo

I think it is more future proof. In the future some storage could make
use of multiple prefix. Like one to store how much value has been
inserted, etc...

* Update frame/support/procedural/src/storage/parse.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update frame/support/procedural/src/storage/storage_struct.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix max_size using hasher information

hasher now expose `max_len` which allows to computes their maximum len.
For hasher without concatenation, it is the size of the hash part,
for hasher with concatenation, it is the size of the hash part + max
encoded len of the key.

* fix tests

* fix ui tests

* Move `MaxBoundedLen` into its own crate (#8814)

* move MaxEncodedLen into its own crate

* remove MaxEncodedLen impl from frame-support

* add to assets and balances

* try more fixes

* fix compile

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* nits

* fix compile

* line width

* fix max-values-macro merge

* Add some derive, needed for test and other purpose

* use weak bounded vec in some cases

* Update lib.rs

* move max-encoded-len crate

* fix

* remove app crypto for now

* width

* Revert "remove app crypto for now"

This reverts commit 73623e9933d50648e0e7fe90b6171a8e45d7f5a2.

* unused variable

* more unused variables

* more fixes

* Add #[max_encoded_len_crate(...)] helper attribute

The purpose of this attribute is to reduce the surface area of
max_encoded_len changes. Crates deriving `MaxEncodedLen` do not
need to add it to `Cargo.toml`; they can instead just do

```rust
\#[derive(Encode, MaxEncodedLen)]
\#[max_encoded_len_crate(frame_support::max_encoded_len)]
struct Example;
```

* fix a ui test

* use #[max_encoded_len_crate(...)] helper in app_crypto

* remove max_encoded_len import where not necessary

* update lockfile

* fix ui test

* ui

* newline

* fix merge

* try fix ui again

* Update max-encoded-len/derive/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* extract generate_crate_access_2018

* Update lib.rs

* compiler isnt smart enough

Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>

* remove duplicate Issued/Burned events (#8935)

* weather -> whether (#8938)

* make remote ext use batch ws-client (#8916)

* make remote ext use batch ws-client

* Add debug log for key length

* better assertions

* new sanity_checl

* try and make it work with batch

* update test

* remove exctra uri

* add missing at

* remove unused rpc stuff

* improve

Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>

* Make `Schedule` fields public to allow for customization (#8924)

* Make `Schedule` fields public for customization

* Fix doc typo

Co-authored-by: Andrew Jones <ascjones@gmail.com>

Co-authored-by: Andrew Jones <ascjones@gmail.com>

* Session key should be settable at genesis even for non-endowed accounts (#8942)

* Session key should be settable at genesis even for non-endowed accounts

* Docs

* Migrate pallet-scored-pool to pallet attribute macro (#8825)

* Migrate pallet-scored-pool to pallet attribute macro.

* Remove dummy event.

* Apply review suggestions.

* Bump retain_mut from 0.1.2 to 0.1.3 (#8951)

Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/upsuper/retain_mut/releases)
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.2...v0.1.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Use correct CreateInherentDataProviders impl for manual seal (#8852)

* use correct CreateInherentDataProviders impl for manual seal

* add babe inherent provider

* move client into factory fn

* Refactor code a little bit (#8932)

* Optimize `next_storage_key` (#8956)

* Optimize `next_storage_key`

- Do not rely on recursion
- Use an iterator over the overlay to not always call the same method

* Fix bug

* Add deserialize for TransactionValidityError in std. (#8961)

* Add deserialize for TransactionValidityError in std.

* Fix derives

* Bump getrandom from 0.2.2 to 0.2.3 (#8952)

Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/rust-random/getrandom/releases)
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.2...v0.2.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Allow usage of path in construct_runtime! (#8801)

* Allow usage of path in construct_runtime!

* Fix whitespace

* Fix whitespace

* Make expand_runtime_metadata accept slice instead of Iterator

* Include Call and Event in construct_runtime for testing

* Migrate impl_outer_event to proc macro

* Fix integrity_test_works

* Update UI test expectations

* Factor in module path while generating enum variant or fn names

* Use ParseStream::lookahead for more helpful error messages

* Remove generating outer_event_metadata

* Ensure pallets with different paths but same last path segment can coexist

* Remove unnecessary generated function

* Migrate decl_outer_config to proc macro

* Add default_filter test for expand_outer_origin

* Allow crate, self and super keywords to appear in pallet path

* Add UI test for specifying empty pallet paths in construct_runtime

* Reduce cargo doc warnings (#8947)

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update wasmtime to 0.27 (#8913)

* Update wasmtime to 0.27

A couple of notes:

- Now we are fair about unsafeness of runtime creation via an compiled artifact.
  This change was prompted by the change in wasmtime which made
  `deserialize` rightfully unsafe. Now `CodeSupplyMode` was hidden and
  the `create_runtime` now takes the blob again and there is now a new
  fn for creating a runtime with a compiled artifact.
- This is a big change for wasmtime. They switched to the modern backend
  for code generation. While this can bring performance improvements, it
  can also introduce some problems. In fact, 0.27 fixed a serious issue
  that could lead to sandbox escape. Hence we need a proper burn in.

This would require a change to PVF validation host as well.

* Filter regalloc logging

* Spellling corrections (no code changes) (#8971)

* Spelling corrections

* As this might break let's do as a separate PR

* Dependabot use correct label (#8973)

* Inject hashed prefix for remote-ext (#8960)

* Inject for remote-ext

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Update utils/frame/remote-externalities/src/lib.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Use `SpawnTaskHandle`s for spawning tasks in the tx pool  (#8958)

* Remove futures-diagnose

* Use `SpawnTaskHandle`s for spawning tasks in the tx pool

* Box the spawner

* Fix tests

* Use the testing task executor

* Do not spend time on verifying the signatures before calling Runtime (#8980)

* Revert "Use `SpawnTaskHandle`s for spawning tasks in the tx pool  (#8958)" (#8983)

This reverts commit 84e402389b.

* Uniques: An economically-secure basic-featured NFT pallet (#8813)

* Uniques: An economically-secure basic-featured NFT pallet

* force_transfer

* freeze/thaw

* team management

* approvals

* Fixes

* force_asset_status

* class_metadata

* instance metadata

* Fixes

* use nmap

* Fixes

* class metadata has information field

* Intiial mock/tests and a fix

* Remove impl_non_fungibles

* Docs

* Update frame/uniques/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/uniques/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/uniques/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/uniques/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Reserve, don't transfer.

* Fixes

* Tests

* Tests

* refresh_deposit

* Tests and proper handling of metdata destruction

* test burn

* Tests

* Update impl_fungibles.rs

* Initial benchmarking

* benchmark

* Fixes

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_uniques --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/uniques/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Attributes

* Attribute metadata

* Fixes

* Update frame/uniques/README.md

* Docs

* Docs

* Docs

* Simple metadata

* Use BoundedVec

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_uniques --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/uniques/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Update frame/uniques/src/lib.rs

Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>

* Update frame/uniques/src/lib.rs

Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>

* Update frame/uniques/src/lib.rs

Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>

* Update frame/uniques/src/lib.rs

Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>

* Update frame/uniques/src/lib.rs

Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>

* Fixes

* Update frame/uniques/README.md

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Update frame/uniques/README.md

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Update frame/uniques/README.md

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Docs

* Bump

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Update WeakBoundedVec's remove and swap_remove (#8985)

Co-authored-by: Boiethios <felix-dev@daudre-vignier.fr>

* Convert another instance of Into impl to From in the macros (#8986)

* Convert another instance of Into impl to From in the macros

* Convert another location

* also fix bounded vec (#8987)

* fix most compiler errors

Mostly the work so far has been in tracking down where precisely
to insert appropriate trait bounds, and updating `fn insert_submission`.
However, there's still a compiler error remaining:

```
error[E0275]: overflow evaluating the requirement `Compact<_>: Decode`
  |
  = help: consider adding a `#![recursion_limit="256"]` attribute to your crate (`pallet_election_provider_multi_phase`)
  = note: required because of the requirements on the impl of `Decode` for `Compact<_>`
  = note: 126 redundant requirements hidden
  = note: required because of the requirements on the impl of `Decode` for `Compact<_>`
```

Next up: figure out how we ended up with that recursive bound, and fix it.

* extract type SignedSubmissionsOf<T>

Weirdly, we still encounter the recursive trait definition error here,
despite removing the trait bounds. Something weird is happening.

* impl Decode bounds on BoundedBTreeMap/Set on T, not predecessor

Otherwise, Rust gets confused and decides that the trait bound is
infinitely recursive.

For that matter, it _still_ gets confused somehow and decides
that the trait bound is infinitely recursive, but at least
this should somewhat simplify the matter.

* fix recursive trait bound problem

* minor fixes

* more little fixes

* correct semantics for try_insert

* more fixes

* derive Ord for SolutionType

* tests compile

* fix most tests, rm unnecessary one

* Transactionpool: Make `ready_at` return earlier (#8995)

`ready_at` returns when we have processed the requested block. However,
on startup we already have processed the best block and there
are no transactions in the pool on startup anyway. So, we can set `updated_at`
to the best block on startup.

Besides that `ready_at` now returns early when there are no ready nor
any future transactions in the pool.

* Discard notifications if we have failed to parse handshake (#8806)

* Migrate pallet-democracy to pallet attribute macro (#8824)

* Migrate pallet-democracy to pallet attribute macro.

* Metadata fix.

* Trigger CI.

* Add ecdsa::Pair::verify_prehashed() (#8996)

* Add ecdsa::Pair::verify_prehashed()

* turn verify_prehashed() into an associated function

* add Signature::recover_prehashed()

* Non-fungible token traits (#8993)

* Non-fungible token traits

* Docs

* Fixes

* Implement non-fungible trait for Uniques

* Update frame/uniques/src/impl_nonfungibles.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/uniques/src/impl_nonfungibles.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Removes unused import (#9007)

* Add Call Filter That Prevents Nested `batch_all` (#9009)

* add filter preventing nested `batch_all`

* more tests

* fix test

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_utility --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/utility/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Parity Bot <admin@parity.io>

* Transaction pool: Ensure that we prune transactions properly (#8963)

* Transaction pool: Ensure that we prune transactions properly

There was a bug in the transaction pool that we didn't pruned
transactions properly because we called `prune_known`, instead of `prune`.

This bug was introduced by:
https://github.com/paritytech/substrate/pull/4629

This is required to have stale extrinsics being removed properly, so
that they don't fill up the tx pool.

* Fix compilation

* Fix benches

* ...

* Storage chain: Runtime module (#8624)

* Transaction storage runtime module

* WIP: Tests

* Tests, benchmarks  and docs

* Made check_proof mandatory

* Typo

* Renamed a crate

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Added weight for on_finalize

* Fixed counter mutations

* Reorganized tests

* Fixed build

* Update for the new inherent API

* Reworked for the new inherents API

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Store transactions in a Vec

* Added FeeDestination

* Get rid of constants

* Fixed node runtime build

* Fixed benches

* Update frame/transaction-storage/src/lib.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* more useful error message (#9014)

* Named reserve (#7778)

* add NamedReservableCurrency

* move currency related trait and types into a new file

* implement NamedReservableCurrency

* remove empty reserves

* Update frame/support/src/traits.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* fix build

* bump year

* add MaxReserves

* repatriate_reserved_named should put reserved fund into named reserved

* add tests

* add some docs

* fix warning

* Update lib.rs

* fix test

* fix test

* fix

* fix

* triggier CI

* Move NamedReservableCurrency.

* Use strongly bounded vec for reserves.

* Fix test.

* remove duplicated file

* trigger CI

* Make `ReserveIdentifier` assosicated type

* add helpers

* make ReserveIdentifier assosicated type

* fix

* update

* trigger CI

* Apply suggestions from code review

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* trigger CI

* Apply suggestions from code review

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Gavin Wood <i@gavwood.com>
Co-authored-by: Shaun Wang <spxwang@gmail.com>

* update ss58 type to u16 (#8955)

* Fixed build (#9021)

* Bump parity-db (#9024)

* consensus: handle justification sync for blocks authored locally (#8698)

* consensus: add trait to control justification sync process

* network: implement JustificationSyncLink for NetworkService

* slots: handle justification sync in slot worker

* babe: fix slot worker instantiation

* aura: fix slot worker instantiation

* pow: handle justification sync in miner

* babe: fix tests

* aura: fix tests

* node: fix compilation

* node-template: fix compilation

* consensus: rename justification sync link parameter

* aura: fix test compilation

* consensus: slots: move JustificationSyncLink out of on_slot

* arithmetic: fix PerThing pow (#9030)

* arithmetic: add failing test for pow

* arithmetic: fix PerThing::pow

* Revert back to previous optimisations

Co-authored-by: Gav Wood <gavin@parity.io>

* Compact proof utilities in sp_trie. (#8574)

* validation extension in sp_io

* need paths

* arc impl

* missing host function in executor

* io to pkdot

* decode function.

* encode primitive.

* trailing tab

* multiple patch

* fix child trie logic

* restore master versionning

* bench compact proof size

* trie-db 22.3 is needed

* line width

* split line

* fixes for bench (additional root may not be needed as original issue was
with empty proof).

* revert compact from block size calculation.

* New error type for compression.

* Adding test (incomplete (failing)).
Also lacking real proof checking (no good primitives in sp-trie crate).

* There is currently no proof recording utility in sp_trie, removing
test.

* small test of child root in proof without a child proof.

* remove empty test.

* remove non compact proof size

* Missing revert.

* proof method to encode decode.

* Don't inlucde nominaotrs that back no one in the snapshot. (#9017)

* fix all_in_one test which had a logic error

* use sp_std, not std

* Periodically call `Peerset::alloc_slots` on all sets (#9025)

* Periodically call alloc_slots on all slots

* Add test

* contracts: Add new `seal_call` that offers new features (#8909)

* Add new `seal_call` that offers new features

* Fix doc typo

Co-authored-by: Michael Müller <michi@parity.io>

* Fix doc typos

Co-authored-by: Michael Müller <michi@parity.io>

* Fix comment on assert

* Update CHANGELOG.md

Co-authored-by: Michael Müller <michi@parity.io>

* fix unreserve_all_named (#9042)

* Delete legacy runtime metadata macros (#9043)

* `rpc-http-threads` cli arg (#8890)

* Add optional `rpc-http-threads` cli arg

* Update `http::ServerBuilder`threads

* allow inserting equal items into bounded map/set

* refactor: only load one solution at a time

This increases the database read load, because we read one solution
at a time. On the other hand, it substantially decreases the overall
memory load, because we _only_ read one solution at a time instead
of reading all of them.

* Emit `Bonded` event when rebonding (#9040)

* Emit `Bonded` event when rebonding

* fix borrow checker

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Parity Bot <admin@parity.io>

* fix tests

* Revert "Merge remote-tracking branch 'origin/master' into prgn-election-provider-multi-phase-bounded-btree-set-signed-submissions"

This reverts commit de92b1e8e0e44a74c24e270d02b6e8e6a2c37032, reversing
changes made to dae31f2018593b60dbf1d96ec96cdc35c374bb9e.

* only derive debug when std

* write after check

* SignedSubmissions doesn't ever modify storage until .put()

This makes a true check-before-write pattern possible.

* REVERT ME: demo that Drop impl doesn't work

* Revert "REVERT ME: demo that Drop impl doesn't work"

This reverts commit 3317a4bb4de2e77d5a7fff2154552a81ec081763.

* doc note about decode_len

* rename get_submission, take_submission for clarity

* add test which fails for current incorrect behavior

* inline fn insert_submission

This fixes a tricky check-before-write error, ensuring that we
really only ever modify anything if we have in fact succeeded.

Co-authored-by: Roman Proskuryakov <humbug@deeptown.org>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Sebastian Müller <sebastian@web3.foundation>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Shaun Wang <spxwang@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Boiethios <Boiethios@users.noreply.github.com>
Co-authored-by: Boiethios <felix-dev@daudre-vignier.fr>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Dmitry Kashitsyn <korvin@deeptown.org>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Gavin Wood <i@gavwood.com>
Co-authored-by: Jakub Pánik <jakub.panik@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Michael Müller <michi@parity.io>
Co-authored-by: tgmichel <telmo@purestake.com>

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* remove duplicate weight definitions injected by benchmark bot

* check deletion overlay before getting

* clarify non-conflict between delete, insert overlays

* drain can be used wrong so is private

* update take_submission docs

* more drain improvements

* more take_submission docs

* debug assertion helps prove expectation is valid

* doc on changing SignedMaxSubmissions

* take_submission inner doc on system properties

* Apply suggestions from code review

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* get SolutionOrSnapshotSize out of the loop

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* doc which items comprise `SignedSubmissions`

* add doc about index as unique identifier

* Add debug assertions to prove drain worked properly

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* replace take_submission with swap_out_submission

* use a match to demonstrate all cases from signed_submissions.insert

* refactor signed_submissions.insert return type

* prettify test assertion

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* improve docs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* add tests that finalize_signed_phase is idempotent

* add some debug assertions to guard against misuse of storage

* log internal logic errors instead of panicing

* don't store the reward with each signed submission

The signed reward base can be treated as a constant. It can in principle
change, but even if it's updated in the middle of an election, it's
appropriate to use the current value for the winner.

* emit Rewarded, Slashed events as appropriate

Makes it easier to see who won/lost with signed submissions.

* update docs

* use a custom enum to be explicit about the outcome of insertion

* remove outdated docs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Roman Proskuryakov <humbug@deeptown.org>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Sebastian Müller <sebastian@web3.foundation>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Shaun Wang <spxwang@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
Co-authored-by: Lohann Paterno Coutinho Ferreira <developer@lohann.dev>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Boiethios <Boiethios@users.noreply.github.com>
Co-authored-by: Boiethios <felix-dev@daudre-vignier.fr>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Dmitry Kashitsyn <korvin@deeptown.org>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Gavin Wood <i@gavwood.com>
Co-authored-by: Jakub Pánik <jakub.panik@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Michael Müller <michi@parity.io>
Co-authored-by: tgmichel <telmo@purestake.com>
2021-06-28 09:20:24 +00:00
Squirrel dc88e6dd7e Reduce cargo doc warnings (#8947)
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-06-01 10:26:10 +01:00
Peter Goodspeed-Niklaus c786fb21a0 Refactor election solution trimming for efficiency (#8614)
* Refactor election solution trimming for efficiency

The previous version always trimmed the `CompactOf<T>` instance,
which was intrinsically inefficient: that's a packed data structure,
which is naturally expensive to edit. It's much easier to edit
the unpacked data structures: the `voters` and `assignments` lists.

* rework length-trim tests to work with the new interface

Test suite now compiles. Tests still don't pass because the macro
generating the compact structure still generates `unimplemented!()`
for the actual `compact_length_of` implementation.

* simplify

* add a fuzzer which can validate `Compact::encoded_size_for`

The `Compact` solution type is generated distinctly for each runtime,
and has both three type parameters and a built-in limit to the number
of candidates that each voter can vote for. Finally, they have an
optional `#[compact]` attribute which changes the encoding behavior.

The assignment truncation algorithm we're using depends on the ability
to efficiently and accurately determine how much space a `Compact`
solution will take once encoded.

Together, these two facts imply that simple unit tests are not
sufficient to validate the behavior of `Compact::encoded_size_for`.
This commit adds such a fuzzer. It is designed such that it is possible
to add a new fuzzer to the family by simply adjusting the
`generate_solution_type` macro invocation as desired, and making a
few minor documentation edits.

Of course, the fuzzer still fails for now: the generated implementation
for `encoded_size_for` is still `unimplemented!()`. However, once
the macro is updated appropriately, this fuzzer family should allow
us to gain confidence in the correctness of the generated code.

* Revert "add a fuzzer which can validate `Compact::encoded_size_for`"

This reverts commit 916038790887e64217c6a46e9a6d281386762bfb.

The design of `Compact::encoded_size_for` is flawed. When `#[compact]`
mode is enabled, every integer in the dataset is encoded using run-
length encoding. This means that it is impossible to compute the final
length faster than actually encoding the data structure, because the
encoded length of every field varies with the actual value stored.

Given that we won't be adding that method to the trait, we won't be
needing a fuzzer to validate its performance.

* revert changes to `trait CompactSolution`

If `CompactSolution::encoded_size_for` can't be implemented in the
way that we wanted, there's no point in adding it.

* WIP: restructure trim_assignments_length by actually encoding

This is not as efficient as what we'd hoped for, but it should still
be better than what it's replacing. Overall efficiency of
`fn trim_assignments_length` is now `O(edges * lg assignments.len())`.

* fix compiler errors

* don't sort voters, just assignments

Sorting the `voters` list causes lots of problems; an invariant that
we need to maintain is that an index into the voters list has a stable
meaning.

Luckily, it turns out that there is no need for the assignments list
to correspond to the voters list. That isn't an invariant, though previously
I'd thought that it was.

This simplifies things; we can just leave the voters list alone,
and sort the assignments list the way that is convenient.

* WIP: add `IndexAssignment` type to speed up repeatedly creating `Compact`

Next up: `impl<'a, T> From<&'a [IndexAssignmentOf<T>]> for Compact`,
in the proc-macro which makes `Compact`. Should be a pretty straightforward
adaptation of `from_assignment`.

* Add IndexAssignment and conversion method to CompactSolution

This involves a bit of duplication of types from
`election-provider-multi-phase`; we'll clean those up shortly.

I'm not entirely happy that we had to add a `from_index_assignments`
method to `CompactSolution`, but we couldn't define
`trait CompactSolution: TryFrom<&'a [Self::IndexAssignment]` because
that made trait lookup recursive, and I didn't want to propagate
`CompactSolutionOf<T> + TryFrom<&[IndexAssignmentOf<T>]>` everywhere
that compact solutions are specified.

* use `CompactSolution::from_index_assignment` and clean up dead code

* get rid of `from_index_assignments` in favor of `TryFrom`

* cause `pallet-election-provider-multi-phase` tests to compile successfully

Mostly that's just updating the various test functions to keep track of
refactorings elsewhere, though in a few places we needed to refactor some
test-only helpers as well.

* fix infinite binary search loop

Turns out that moving `low` and `high` into an averager function is a
bad idea, because the averager gets copies of those values, which
of course are never updated. Can't use mutable references, because
we want to read them elsewhere in the code. Just compute the average
directly; life is better that way.

* fix a test failure

* fix the rest of test failures

* remove unguarded subtraction

* fix npos-elections tests compilation

* ensure we use sp_std::vec::Vec in assignments

* add IndexAssignmentOf to sp_npos_elections

* move miner types to `unsigned`

* use stable sort

* rewrap some long comments

* use existing cache instead of building a dedicated stake map

* generalize the TryFrom bound on CompactSolution

* undo adding sp-core dependency

* consume assignments to produce index_assignments

* Add a test of Assignment -> IndexAssignment -> Compact

* fix `IndexAssignmentOf` doc

* move compact test from sp-npos-elections-compact to sp-npos-elections

This means that we can put the mocking parts of that into a proper
mock package, put the test into a test package among other tests.

Having the mocking parts in a mock package enables us to create a
benchmark (which is treated as a separate crate) import them.

* rename assignments -> sorted_assignments

* sort after reducing to avoid potential re-sort issues

* add runtime benchmark, fix critical binary search error

"Why don't you add a benchmark?", he said. "It'll be good practice,
and can help demonstrate that this isn't blowing up the runtime."

He was absolutely right.

The biggest discovery is that adding a parametric benchmark means that
you get a bunch of new test cases, for free. This is excellent, because
those test cases uncovered a binary search bug. Fixing that simplified
that part of the code nicely.

The other nice thing you get from a parametric benchmark is data about
what each parameter does. In this case, `f` is the size factor: what
percent of the votes (by size) should be removed. 0 means that we should
keep everything, 95 means that we should trim down to 5% of original size
or less.

```
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=     3846
    + v    0.015
    + t        0
    + a    0.192
    + d        0
    + f        0
              µs

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    v     t     a     d     f   mean µs  sigma µs       %
<snip>
 6000  1600  3000   800     0      4385     75.87    1.7%
 6000  1600  3000   800     9      4089     46.28    1.1%
 6000  1600  3000   800    18      3793     36.45    0.9%
 6000  1600  3000   800    27      3365     41.13    1.2%
 6000  1600  3000   800    36      3096     7.498    0.2%
 6000  1600  3000   800    45      2774     17.96    0.6%
 6000  1600  3000   800    54      2057     37.94    1.8%
 6000  1600  3000   800    63      1885     2.515    0.1%
 6000  1600  3000   800    72      1591     3.203    0.2%
 6000  1600  3000   800    81      1219     25.72    2.1%
 6000  1600  3000   800    90       859     5.295    0.6%
 6000  1600  3000   800    95     684.6     2.969    0.4%

Quality and confidence:
param     error
v         0.008
t         0.029
a         0.008
d         0.044
f         0.185

Model:
Time ~=     3957
    + v    0.009
    + t        0
    + a    0.185
    + d        0
    + f        0
              µs
```

What's nice about this is the clear negative correlation between
amount removed and total time. The more we remove, the less total
time things take.
2021-05-03 07:26:35 +00:00
Peter Goodspeed-Niklaus c2dd5e21a4 make types within generate_solution_type macro explicit (#8447)
* make types within `generate_solution_type` macro explicit

Closes #8444.

Just changes the parsing logic for that macro; does not change any
emitted code. The associated types associated with the macro now
require explicit, keyword-style declaration.

**Old**:

```rust
sp_npos_elections::generate_solution_type!(
	#[compact]
	pub struct TestCompact::<VoterIndex, TargetIndex, PerU16>(16)
);
```

**New**:

```rust
sp_npos_elections::generate_solution_type!(
	#[compact]
	pub struct TestCompact::<VoterIndex = VoterIndex, CandidateIndex = TargetIndex, Accuracy = PerU16>(16)
);
```

* un-ignore doc-tests

* use new form in bin/node/runtime/

* rename CandidateIndex -> TargetIndex

* add tests demonstrating some potential compile failures
2021-03-28 08:21:06 +00:00
Peter Goodspeed-Niklaus 23b32e7543 Improve complexity of CompactAssignments::unique_targets (#8314)
* Improve complexity of CompactAssignments::unique_targets

Original implementation was O(n**2). Current impl is O(n log n).

Avoided the original proposed mitigation because it does not retain
the de-duplicating property present in the original implementation.
This implementation does a little more work, but retains that property.

* Explicitly choose sp_std Vec and BTreeSet

Ensures that the macro still works if someone uses it in a context
in which sp_std is not imported or is renamed.

* explicitly use sp_std vectors throughout compact macro
2021-03-17 08:13:33 +00:00
Bastian Köcher c33235ac5b Update to latest proc-macro-crate version (#8294) 2021-03-09 22:22:25 +00:00
Kian Paimani ced107b355 Decouple Stkaing and Election - Part1: Support traits (#7908)
* Base features and traits.

* Fix the build

* Remove unused boxing

* Self review cleanup

* Fix build
2021-01-18 10:24:12 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Guillaume Thiolliere 8814d793c7 Fix docs (#7710)
* fix docs

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

Co-authored-by: Alexander Theißen <alex.theissen@me.com>
2020-12-10 14:38:56 +01:00
Kian Paimani 4d036e0053 Fix offchain election to respect the weight (#7215)
* Mockup

* Fix offchain election to respect the weight

* Fix builds a bit

* Update frame/staking/src/offchain_election.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/staking/src/offchain_election.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Make it build, binary search

* Fix a number of grumbles

* one more fix.

* remove unwrap.

* better alg.

* Better alg again.

* Final fixes

* Fix

* Rollback to normal

* Final touches.

* Better tests.

* Update frame/staking/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Proper maxExtWeight

* Final fix

* Final fix for the find_voter

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2020-10-02 13:45:17 +00:00
Kian Paimani 83bde4c1e6 Move Staking Weights to T::WeightInfo (#7007)
* Fix the benchmarks

* Migrate staking to weightInfo

* Fix global benchmarks

* re-calculate the submit solution weight.

* Fix some refund.

* Get rid of all the extra parameters.

* Fix staking tests.

* new values from the bench machine.

* Fix some grumbles

* better macro

* Some better doc

* Move to interpreted wasm

* Make it work temporarily

* Final fix of default ones.

* Fix payout benchmarks

* Fix payout stuff

* One last fix

* use benchmarking machine for numbers

* update weight docs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-09-16 12:25:31 +00:00
Kian Paimani e6ce3e7ac0 Fix compact npos solution edge count calculation (#7021)
This edge count is used for weighing, and it is somewhat trivial to review and verify that the current implementation was ignoring `votes16` field of the struct. As reminder, the struct is like this: 
```rust

struct Compact {
  votes1: ... ,
  votes2: ..., 
  ...,
  votes16: ...,
}
```

I already will fix this in https://github.com/paritytech/substrate/pull/7007, but since it might take a while, this one can go in asap and make it to the very next runtime.
2020-09-04 14:29:34 +02:00
Kian Paimani 775e84cc04 More renaming to move away from phragmen. (#6886) 2020-08-13 23:30:22 +02:00
Kian Paimani 1519da95d7 Custom Codec Implenetation for NPoS Election (#6720)
* Fancy compact encode/decode impl for compact solution

* Make it optional

* Remove extra file

* Update primitives/npos-elections/compact/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Final fixes.

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2020-08-11 10:33:30 +00:00
Bastian Köcher 1951962836 sp-npos-elections should not depend on itself (#6444)
This removes the `dev-dependency` onto `sp-npos-elections` from itself.
A crate should not depend on itself directly, especially not to make any
macros work.
2020-06-20 11:49:18 +02:00
Kian Paimani 8a8b4f99c3 Rename all the election operations (#6245)
* Rename and move sp-phragmen

* More renames for equalise

* Update main module doc

* Fix line width

* Line width
2020-06-05 17:33:13 +02:00