Commit Graph

63 Commits

Author SHA1 Message Date
Kian Paimani eaf1bc5633 Introduce Polkadot-Sdk developer_hub (#2102)
This PR introduces the new crate `developer_hub` into the polkadot-sdk
repo. The vision for the developer-hub crate is detailed in [this
document](https://docs.google.com/document/d/1XLLkFNE8v8HLvZpI2rzsa8N2IN1FcKntc8q-Sc4xBAk/edit?usp=sharing).

<img width="1128" alt="Screenshot 2023-11-02 at 10 45 48"
src="https://github.com/paritytech/polkadot-sdk/assets/5588131/1e12b60f-fef5-42c4-8503-a3ba234077c3">


Other than adding the new crate, it also does the following: 

* Remove the `substrate` crate, as there is now a unique umbrella crate
for multiple things in `developer_hub::polkadot_sdk`.
* (backport candidate) A minor change to `frame-support` macros that
allows `T::RuntimeOrigin` to also be acceptable as the origin type.
* (backport candidate) A minor change to `frame-system` that allows us
to deposit events at genesis because now the real genesis config is
generated via wasm, and we can safely assume `cfg!(feature = "std")`
means only testing. related to #62.
* (backport candidate) Introduces a small `read_events_for_pallet` to
`frame_system` for easier event reading in tests.
* From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it
takes action on improving the `pallet::call` docs.
* From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it
takes action on improving the `UncheckedExtrinsic` docs.

## Way Forward

First, a version of this is deployed temporarily
[here](https://blog.kianenigma.nl/polkadot-sdk/developer_hub/index.html).
I will keep it up to date on a daily basis.

### This Pull Request

I see two ways forward: 

1. We acknowledge that everything in `developer-hub` is going to be WIP,
and merge this asap. We should not yet use links to this crate anywhere.
2. We make this be the feature branch, make PRs against this, and either
gradually backport it, or only merge to master once it is done.

I am personally in favor of option 1. If we stick to option 2, we need a
better way to deploy a staging version of this to gh-pages.

### Issue Tracking

The main issues related to the future of `developer_hub` are: 

- https://github.com/paritytech/polkadot-sdk-docs/issues/31
- https://github.com/paritytech/polkadot-sdk-docs/issues/4
- https://github.com/paritytech/polkadot-sdk-docs/issues/26 
- https://github.com/paritytech/polkadot-sdk-docs/issues/32
- https://github.com/paritytech/polkadot-sdk-docs/issues/36


### After This Pull Request

- [ ] create a redirect for
https://paritytech.github.io/polkadot-sdk/master/substrate/
- [x] analytics 
- [ ] link checker
- [ ] the matter of publishing, and how all of these relative links for
when we do, that is still an open question. There is section on this in
the landing page.
- [ ] updated https://paritytech.github.io/

---------

Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: bader y <ibnbassem@gmail.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
2023-11-30 12:15:46 +01:00
Mira Ressel 447e753323 ci: bump ci image to rust 1.73.0 (#1830)
Co-authored-by: command-bot <>
2023-10-11 19:49:59 +02:00
Gonçalo Pestana 93754780b1 [NPoS] Implements dynamic number of nominators (#12970)
* Implements dynamic nominations per nominator

* Adds SnapshotBounds and ElectionSizeTracker

* Changes the ElectionDataProvider interface to receive ElectionBounds as input

* Implements get_npos_voters with ElectionBounds

* Implements get_npos_targets with ElectionBounds

* Adds comments

* tests

* Truncates nomninations that exceed nominations quota; Old tests passing

* Uses DataProviderBounds and ElectionBounds (to continue)

* Finishes conversions - tests passing

* Refactor staking in babe mocks

* Replaces MaxElectableTargets and MaxElectingVoters with ElectionBounds; Adds more tests

* Fixes nits; node compiling

* bechmarks

* removes nomination_quota extrinsic to request the nomination quota

* Lazy quota check, ie. at nominate time only

* remove non-working test (for now)

* tests lazy nominations quota when quota is lower than current number of nominated targets

* Adds runtime API and custom RPC call for clients to query the nominations quota for a given balance

* removes old rpc

* Cosmetic touches

* All mocks working

* Fixes benchmarking mocks

* nits

* more tests

* renames trait methods

* nit

* ".git/.scripts/commands/fmt/fmt.sh"

* Fix V2 PoV benchmarking (#13485)

* Bump default 'additional_trie_layers' to two

The default here only works for extremely small runtimes, which have
no more than 16 storage prefices. This is changed to a "sane" default
of 2, which is save for runtimes with up to 4096 storage prefices (eg StorageValue).

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update tests and test weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix PoV weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_balances

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_message_queue

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton

* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton

* Fix sanity check

>0 would also do as a check, but let's try this.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>

* Move BEEFY code to consensus (#13484)

* Move beefy primitives to consensus dir
* Move beefy gadget to client consensus folder
* Rename beefy crates

* chore: move genesis block builder to chain-spec crate. (#13427)

* chore: move genesis block builder to block builder crate.

* add missing file

* chore: move genesis block builder to sc-chain-spec

* Update client/chain-spec/src/genesis.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update test-utils/runtime/src/genesismap.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update test-utils/runtime/client/src/lib.rs

* fix warnings

* fix warnings

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

* Speed up storage iteration from within the runtime (#13479)

* Speed up storage iteration from within the runtime

* Move the cached iterator into an `Option`

* Use `RefCell` in no_std

* Simplify the code slightly

* Use `Option::replace`

* Update doc comment for `next_storage_key_slow`

* Make unbounded channels size warning exact (part 1) (#13490)

* Replace `futures-channel` with `async-channel` in `out_events`

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Also print the backtrace of `send()` call

* Switch from `backtrace` crate to `std::backtrace`

* Remove outdated `backtrace` dependency

* Remove `backtrace` from `Cargo.lock`

---------

Co-authored-by: Koute <koute@users.noreply.github.com>

* Removal of Prometheus alerting rules deployment in cloud-infra (#13499)

* sp-consensus: remove unused error variants (#13495)

* Expose `ChargedAmount` (#13488)

* Expose `ChargedAmount`

* Fix imports

* sc-consensus-beefy: fix metrics: use correct names (#13494)


Signed-off-by: acatangiu <adrian@parity.io>

* clippy fix

* removes NominationsQuotaExceeded event

* Update frame/staking/src/lib.rs

Co-authored-by: Ross Bulat <ross@parity.io>

* adds back the npos_max_iter

* remove duplicate imports added after merge

* fmt

* Adds comment in public struct; Refactors CountBound and SizeCount to struct

* addresses various pr comments

* PR comment reviews

* Fixes on-chain election bounds and related code

* EPM checks the size of the voter list returned by the data provider

* cosmetic changes

* updates e2e tests mock

* Adds more tests for size tracker and refactors code

* Adds back only_iterates_max_2_times_max_allowed_len test

* Refactor

* removes unecessary dependency

* empty commit -- restart all stuck CI jobs

* restarts ci jobs

* Renames ElectionBounds -> Bounds in benchmarking mocks et al

* updates mocks

* Update frame/election-provider-support/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/staking/src/pallet/impls.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/election-provider-support/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/staking/src/tests.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* more checks in api_nominations_quota in tests

* Improves docs

* fixes e2e tests

* Uses size_hint rather than mem::size_of in size tracker; Refactor size tracker to own module

* nits from reviews

* Refactors bounds to own module; improves docs

* More tests and docs

* fixes docs

* Fixes benchmarks

* Fixes rust docs

* fixes bags-list remote-ext-tests

* Simplify bound checks in create_snapshot_external

* Adds target size check in get_npos_targets

* ".git/.scripts/commands/fmt/fmt.sh"

* restart ci

* rust doc fixes and cosmetic nits

* rollback upgrade on parity-scale-codec version (unecessary)

* reset cargo lock, no need to update it

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: acatangiu <adrian@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: yjh <yjh465402634@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Anthony Lazam <lazam@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Piotr Mikołajczyk <piomiko41@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Ross Bulat <ross@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2023-08-10 07:45:55 +00:00
Bastian Köcher 83caca85b6 frame-benchmarking-cli: Remove native dispatch requirement (#14474)
* frame-benchmarking-cli: Remove native dispatch requirement

No need for this, we can just use the `WasmExecutor` directly.

* Fixes

* Pass benchmarking host functions

* Ensure we can pass custom host functions
2023-06-29 17:56:25 +02:00
Squirrel ea21a495f8 Easy PR: Fix warnings from latest nightly (#14195)
* unneeded mut

* remove needless borrows
2023-05-23 17:06:48 +02:00
Bigna Härdi 0e49ed72aa Add serde feature flag to primitives (#13027)
* add serde_full feature flag

add serde_full to sp_runtime

add space to toml

add serde_full to application-crypto

add serde_full to arithmetic

fix arithmetic

add serde full to beefy

add serde full to consensus

add serde_full to core

add serdefull to finality grandpa

add serde_full to several primitives crates

make rpc no_std compatible

add scale info to runtime

make serializer no_std compatible

add serde full to storage

add full serde to version

add serde full to weights

add all serde_full features

add . to comment

add missing impl-serde

fix no-std build

fix build

add full_crypto to serde_full

serde_full also implements crypto

full_serde does not work with full_crytpo. needs std

no no_std serde impl possible

also for crypto std is necessary

no serde full for application crypto

fix arithmetic

fix tomls

fix some things

impl fmt for Signature

add serialize to Public

add impl_maybe_marker_serde_full

fix sp-application-crypto toml

add serde feature flag

fix clippy

fix toml grandpa

fix grandpa

rename if_std to if_serde

keystore is not no_std compatible

make keystore vrf no_std compatible

fix nopos-elections

fix rpc

fix serializer

fix test-primitives

fix version

add comment

add serde full only import for format string

remove all(serde_full and full_crypot) as serde_full enforces full_crypto

make comment better readable

even better comment

clean up rpc toml

clean up toml

clean up serializer toml

clean up storage toml

fix std build

update .lock

fix sp-version

move sp_std import

test extern crate alloc

replace sp_std with core

add missing core

sp_core: serde feature do not enforce full crypto

application-crypto: serde feature do not enforce full crypto

rename serde_full to serde

add dep:serde and alloc to default feature

add full_crypto and remove unnecessary debu/fmt impls for serde

update comment

remove obolsete change in display AccountId32

remove extra changes

minimize diff

revert keystore changes

remove std from keystore

remove full-crypto feature

fix serde import

fix comment

fix feature = serde

* rename serde_full to serde

* move #[doc(hidden)] back

* remove feature = full crypto require frm MultiSigner

* reorder serde and scale_info import

* fix bs58 missing alloc import in serde feature

* add `from_string` to serde feature and add unimplemented

* remove serde feature from fixed_point display

* Remove serde/alloc

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update primitives/consensus/babe/Cargo.toml

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update primitives/arithmetic/src/fixed_point.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* revert `from_string`fixed impl back to std only

* remove duplicate runtime string impl

* use sp_std::alloc

* remove no_std compatible rpc

* remove no_std compatibility from serializer

* rename mpl_maybe_marker_serde to std_or_serde

* update .lock

* add sp-std to executor

* fix sp-std import

* fix sp_std::format import

* use crate import

* add serde feature

* Update primitives/core/src/lib.rs

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-05-17 11:31:12 +00:00
Gonçalo Pestana c848d40775 Revert "Abstracts elections-phragmen pallet to use NposSolver (#12588)" (#13451)
This reverts commit b793666ca5.
2023-02-23 12:15:08 -06:00
Gonçalo Pestana b793666ca5 Abstracts elections-phragmen pallet to use NposSolver (#12588)
* Abstracts elections-phragmen pallet to use NposSolver

* Update frame/elections-phragmen/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections-phragmen/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* changes the name of the pallet; adds changelog

* update changelog

* Adds weight testing

* Adds log macro_rules

* renames elections-phragment dir to elections

* weights rename

* fixes typo in cargo toml

* pre/post solve weight scafolding

* refactor do_post_election

* refactors into pre and post election solve for independent benchmarking

* deconstructs PreElectionResults struct

* updates benchmarking pre and post election solve; mock weights

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* addresses PR comments

* adds pre_solve and post_sove weights

* Adds comments on election pallet id param name change

* ".git/.scripts/bench-bot.sh" pallet dev pallet_elections

* Finishes pre-post solve weights

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Addresses PR comments: no panic in on_init path; nits

* Fixes node build

* Implements approval voting to use as a `NposSolver` (#13367)

* Implements the approval voting methods in sp_npos_elections

* fmt

* remove unecessary file

* comment clarification

* re-run weights

* fix typo

* updates MaxVoters in tests for integrity_tests to pass

* Refactors election provider support benchmarks outside its own crate (#13431)

* Refactors election provider support benchmarks outside its own crate
---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: parity-processbot <>
Co-authored-by: Ross Bulat <ross@parity.io>
2023-02-23 11:21:00 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Ankan 657d99202c Bound Election and Staking by MaxActiveValidators (#12436)
* bounding election provider with kian

* multi phase implement bounded election provider

* election provider blanket implementation

* staking compiles

* fix test for election provider support

* fmt

* fixing epmp tests, does not compile yet

* fix epmp tests

* fix staking tests

* fmt

* fix runtime tests

* fmt

* remove outdated wip tags

* add enum error

* sort and truncate supports

* comment

* error when unsupported number of election winners

* compiling wip after kian's suggestions

* fix TODOs

* remove,fix tags

* ensure validator count does not exceed maxwinners

* clean up

* some more clean up and todos

* handle too many winners

* rename parameter for mock

* todo

* add sort and truncate rule if there are too many winners

* fmt

* fail, not swallow emergency result bound not met

* remove too many winners resolution as it can be guaranteed to be bounded

* fix benchmark

* give MaxWinners more contextual name

* make ready solution generic over T

* kian feedback

* fix stuff

* Kian's way of solvign this

* comment fix

* fix compile

* remove use of BoundedExecution

* fmt

* comment out failing integrity test

* cap validator count increment to max winners

* dont panic

* add test for bad data provider

* Update frame/staking/src/pallet/impls.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* fix namespace conflict and add test for onchain max winners less than desired targets

* defensive unwrap

* early convert to bounded vec

* fix syntax

* fmt

* fix doc

* fix rustdoc

* fmt

* fix maxwinner count for benchmarking

* add instant election for noelection

* fmt

* fix compile

* pr feedbacks

* always error at validator count exceeding max winners

* add useful error message

* pr comments

* import fix

* add checked_desired_targets

* fmt

* fmt

* fix rust doc

Co-authored-by: parity-processbot <>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2022-11-09 09:11:51 +00:00
Ankan 6137c8707b bounding staking: BoundedElectionProvider trait (#12362)
* add a bounded election provider trait

* extract common trait election provider base

* fmt

* only bound the outer support vector

* fix tests

* docs

* fix rust docs

* fmt

* fix rustdocs

* docs

* improve docs

* small doc change
2022-09-28 20:52:16 +00:00
Squirrel a68a80fbae declone and close the door (#12035)
* declone and close the door

* cargo fmt

* remove brackets
2022-08-15 19:38:36 +00:00
Boluwatife Bakre 0108d216d2 Remove multiply_by_rational (#11598)
* Removed multiply_by_rational
Replaced with multiply_by_rational_with_rounding

* fixes

* Test Fixes

* nightly fmt

* Test Fix

* Fixed fuzzer.
2022-06-17 03:57:29 +00:00
Georges b71e180446 combine iteratons and tolerance in sp-npos-elections API (#11498)
* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Fixing rust docs by adding () to resolve ambiguity

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Fixing rust docs by adding () to resolve ambiguity

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Removing left over from rebase

* Fixing tests

* Removing unneeded import

* Removing unneeded functions

* Removing useless imports

Co-authored-by: kianenigma <kian@parity.io>
2022-06-15 23:30:22 +00:00
Kian Paimani 494167a05b Optimize offchain worker memory usage a bit. (#11454)
* add missing events to elections fallback

* Merged

* add some logs and stuff

* undo a bunch of things

* undo lock file

* remove unused err

* fix build
2022-05-24 06:23:05 +00:00
Falco Hirschenberger b581604aa7 Apply some clippy lints (#11154)
* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2022-04-30 21:28:27 +00:00
Georges ae75a371bf Adding benchmarking for new frame_election_provider_support (#11149)
* First stab at adding benchmarking for
`election-provider-support` onchain

* Adding `BoundedPhragMMS` and fixing stuff

* Fixing node runtime

* Fixing tests

* Finalising all benchmarking stuff

* better comments

* Better benchmarking config

* Better `WeightInfo` and benchmarking

* Fixing tests

* Adding some documentation

* Fixing some typos

* Incorporating review feedback

* cleanup of rustdocs

* rustdoc changes

* changes after code review

* Fixing some errors.

* Fixing dependencies post merge

* Bringing back `UnboundedExecution`

* Better rustdoc and naming

* Cargo.toml formatting
2022-04-15 10:15:01 +00:00
Georges 0e0d0a677a Enforce MaxEncodedLen impl for NposSolution (#11103)
* Fail if `MaxVoters` too small

* Fixing benchmarking test, better naming of error

* reverting accidental change

* use fully qualified syntax
no need to interate to calculate len

* Fail directly if too many voters
2022-03-31 12:54:44 +00:00
Georges 26a8c7e6b2 Moving NposSolution to frame (#11031)
* Move `sp-npos-elections-solution-type`
to `frame-election-provider-support`
First stab at it, will need to amend some more stuff

* Fixing tests

* Fixing tests

* Fixing cargo.toml for std configuration

* fmt

* Committing suggested changes
renaming, and re exporting macro.

* Removing unneeded imports

* Move `NposSolution` to frame

* Removing `npos_election` dependencies
Implementing _fpes better

* some feedback for moving NPoSSolution to frame

* fmt

* more formatting

* Fixed some imports and fmt

* Fixing docs

Co-authored-by: kianenigma <kian@parity.io>
2022-03-16 21:27:19 +00:00
Georges 644140f9da Move sp-npos-elections-solution-type to frame-election-provider-support (#11016)
* Move `sp-npos-elections-solution-type`
to `frame-election-provider-support`
First stab at it, will need to amend some more stuff

* Fixing tests

* Fixing tests

* Fixing cargo.toml for std configuration

* fmt

* Committing suggested changes
renaming, and re exporting macro.

* Removing unneeded imports
2022-03-14 11:00:41 +01:00
Kian Paimani 6b7eb2e2d7 refactor election score (#10834)
* refactor election score

* Test for ord

* remove reference

* vec -> slice

* change iter to iter_by_significance

* improve doc

* fix typo

* add explanation about [u128; 3]

* consolidate threshold and epsilon

* random fixes

* rename

* remove Into

* make iter_by_sig private

* remove vec

* Fix tests
2022-02-16 10:40:16 +00:00
Kian Paimani 38d94d6323 Use proper bounded vector type for nominations (#10601)
* Use proper bounded vector type for nominations

* add docs and tweak chill_other for cleanup purposes

* Fix the build

* remove TODO

* add a bit more doc

* even more docs
gushc

* Update frame/staking/src/pallet/mod.rs

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>

* Update frame/staking/src/pallet/mod.rs

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>

* Fix the nasty bug

* also bound the Snapshot type

* fix doc test

* document bounded_vec

* self-review

* remove unused

* Fix build

* frame-support: repetition overload for bounded_vec

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fix

* remove the need to allocate into unbounded voters etc etc

* Don't expect

* unbreal the build again

* handle macro a bit better

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2022-01-25 14:44:10 +00:00
Squirrel 3dd32d5255 Make clippy _a little_ more annoying (#10570)
* Clippy: +complexity

* Update client/cli/src/arg_enums.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/inspect/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/keystore/src/testing.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/elections/src/lib.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update primitives/npos-elections/fuzzer/src/reduce.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Incorporating feedback

* No need for Ok

* Additional

* Needed slice

* Wigy's suggestions on less derefs

* fix count

* reverting changes brought in by option_map_unit_fn

* add --all-targets

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-01-05 14:35:30 +00:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
Gavin Wood 1e24e45ea1 Remove Default bound for AccountId (#10403)
* Remove Default for AccountId

* More removals of default

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* More work

* More work

* Remove old code

* More work

* pallet-asset-tx-payment

* tips

* sc-consensus-babe

* sc-finality-grandpa

* sc-consensus-babe-rpc

* sc-cli

* make npos crates accept non-default account (#10420)

* minimal changes to make npos pallets all work

* make this pesky reduce.rs a bit cleaner

* more work

* more work

* Tests build

* Fix imonline tests

* Formatting

* Fixes

* Fixes

* Fix bench

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/finality-grandpa/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/staking/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/staking/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/runtime/src/traits.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Formatting

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>
2021-12-13 15:03:59 +01:00
Guillaume Thiolliere fcfb766366 Fix some doc link (#10329)
* fix some doc link

* fmt
2021-11-22 10:48:53 +00:00
Zeke Mostov 669972a171 Implement pallet-bags-list and its interfaces with pallet-staking (#9507)
* remove extra whitespace

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* only emit rebag event on success

* add doc explaining the term voter

* revamp/simplify rebag test

* ensure genesis accounts are placed into the correct nodes/bags

* bond_extra implicitly rebags

* types at top; doc public type

* start sketching out adjustable thresholds

* add integrity test for voter bag threshold requirements

* get rid of BagIdx

This reorganizes bag storage such that bags are always referred to
by their upper threshold. This in turn means that adding and removing
bags is cheaper; you only need to migrate certain voters, not all of them.

* implement migration logic for when the threshold list changes

* start sketching out threshold proc macros

* further refine macro signatures

* WIP: implement make_ratio macro

* start rethinking the process of producing threshold lists

The macro approach seems to be a non-starter; that only really works
if we're throwing around numeric literals everywhere, and that's just
not nice in this case.

Instead, let's write helper functions and make it really easy to
generate the tables in separate, permanent files, which humans
can then edit.

* write helper functions to emit voter bags module

* WIP: demo generating voter bags for a realistic runtime

This isn't yet done, becuase it seems to take a Very Long Time to run,
and it really shouldn't. Need to look into that.

Still, it's a lot closer than it was this morning.

* rm unnecessary arg_enum

* fix voter bags math

Turns out that when you're working in exponential space, you need
to divide, not subtract, in order to keep the math working properly.

Also neaten up the output a little bit to make it easier to read.

* add computed voter bags thresholds to node

* fixup some docs

* iter from large bags to small, fulfuilling the contract

* make tests compile

* add VoterBagThresholds to some configs

* ensure that iteration covers all voters even with implied final bag

* use sp_std::boxed::Box;

* fix unused import

* add some more voter bags tests

* file_header.txt

* integrity test to ensure min bag exceeds existential weight

* add more debug assertions about node list length

* rm unused imports

* Kian enters

* Update frame/election-provider-support/src/onchain.rs

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

* Suggestions for #9081 (Store voters in unsorted bags) (#9328)

* Add some debug asserts to node::get and remove_node

* Improve the debug asserts in remove_node

* improve debug asserts

* Space

* Remove bad assertions

* Tests: WIP take_works

* Take test

* Doc comment

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Test storage is cleaned up;

* formatting

* Switch to simpler thresholds

* Update the storage cleanup test

* Remove hardcoded values from benchmark to make it more robust

* Fix tests to acces bags properly

* Sanity check WIP; tests failing

* Update sanity checks to be more correct

* Improve storage cleanup tests

* WIP remote_ext_tests

* Some notes on next steps

* Remove some stuff that was for remote-ext tests

* Some more cleanup to reduce diff

* More :clean:

* Mo cleanin

* small fix

* A lot of changes from kian

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>

* merge fallout

* Run cargo +nightly fmt

* Fix a bunch of stuff, remove not needed runtime arg of make-bags

* add logs

* Glue the new staking bags to the election snapshot (#9415)

* Glue the new staking bags to the election snapshot

* add CheckedRem (#9412)

* add CheckedRem

* fix

* Run fmt

* Test comment

Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>

* Update node runtime with VoterSnapshotPerBlock

* Unit test for pallet-staking unsorted bags feature (targets #9081) (#9422)

* impl notional_bag_for_works

* Add tests: insert_as_works & insert_works

* Impl test: remove_works

* Trivial cleaning

* Add test: update_position_for_works

* Write out edge case; probably can delete later

* Add test: bags::get_works

* Add test: remove_node_happy_path_works

* Add test: remove_node_bad_paths_documented

* WIP: voting_data_works

* done

* Improve test voting_data_works

* Add comment

* Fill out test basic_setup_works

* Update: iteration_is_semi_sorted

* Improve remove_works

* Update update_position_for_works; create set_ledger_and_free_balance

* Improve get_works

* Improve storage clean up checks in remove test

* Test: impl rebag_works + insert_and_remove_works

* forgot file -  Test: impl rebag_works + insert_and_remove_works

* Small tweak

* Update voter_bags test to reflect unused bags are removed

* Unbond & Rebond: do_rebag

* Prevent infinite loops with duplicate tail insert

* Check iter.count on voter list in pre-migrate

* undo strang fmt comment stuff

* Add in todo

Co-authored-by: kianenigma <kian@parity.io>

* Try prepare for master merge

* Reduce diff

* Add comment for test to add

* Add in code TODO for update_position efficiency updates

* Initial impl compiles

* impl StakingVoterListStub

* Sample impl<T: Config> VoterListProvider<T> for VoterList<T>

* impl VoterListProvider for voter-bags

* WIP integrate pallet-voter-bags to staking mock

* the trait `pallet_staking::pallet::pallet::Config` is not implemented for `mock::Test`

* random

* pushing my stuff

* Mock working

* WIP voter list tests

* Add bag insert, remove tests

* Add test for bag insert and remove

* Add remaining tests for VoterList

* Add tests for node

* Add rebag works

* Add rebag extrinsic tests

* Rename to bags-list and name the list .. list!

* Rename VoterBagThresholds => BagThresholds

* Add test count_works

* Test on_update_works

* test sanity check

* a round of test fixes

* push a lot of changes

* my last changes

* all bags-list test work; fmt

* Beautify some tests

* Doc comment for bags-list

* Add insert warnings

* Setup initial benchmark

* Wire up WeightInfo

* is_terminal wip; everything broken!

* Is terminal working

* add TODOs for remove_node

* clean up remoe_node

* Fix all staking tests

* retire VoterBagFor

* commit

* bring in stashed changes

* save

* bench pipeline works now, but I can't run stuff

* sabe

* benchmarks now run, but we have a failure

* WIP: Wire up make_bags

* bags-thresholds compiles

* Fix most build issues

* This will fix all the tests

* move bag thresholds to bags-list

* Move bag-thresholds bin to within pallet-bags

* Remove some unnescary TODOs

* Impl tets wrong_rebag_is_noop

* assert remove is a noop with bad data

* Assert integrity test panics

* Return an error when inserting duplicates

* Update to handle error in staking pallet when inserting to list

* Test contains and on_insert error

* Test re-nominate does not mess up list or count

* Everything builds and works, only the benchmark...

* fuck yeah benchmarks

* more cleanup, more hardening.

* use the bags list again

* fix benhc

* Some questions and changs for List::migration

* Fix migration removed_bags and new_bags usage

* Some trivial aesthetic changes

* Some more trivial changes

* tiny changes/

* mega rename

* fix all tests and ci build

* nit

* Test and fix migration

* nit

* fmt

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fmt

* remove unused

* make a few things pub

* make node also pub.. for remote-ext test

* Fix all tests again

* Force bag changes in relevant benchmarks (targets #9507) (#9529)

* force rebag for unbond, rebond, and bond_extra

* nit

* Improve utils

* fmt

* nits

* Move generate_bags to its own pallet

* Get runtime-benchmarks feature setup with prepare_on_update_benchmark

* Withdraw unbonded kill working

* Nominate bench working

* some cleanup

* WIP

* update to check head pre & post conditions

* Add some post condition verification stuff for on_remove

* Update nominate

* fmt

* Improvements

* Fix build

* fix build with polkadot companion

* Update frame/bags-list/src/list/tests.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* move generate-bag from frame to utils

* wip

* refactor WIP

* WIP save

* Refactor working

* some variable renaming

* WIP: prepare to remove head checks

* Finish MvP refactor

* Some cleanup

* Soem more cleanup

* save

* fix a lot of stuff

* Update client/db/src/bench.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Apply suggestions from code review

* Apply suggestions from code review

* Fix some issues that came from trying to merge comments on github

* some small changes

* simplify it

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Build works

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Remove commented out debug assert

* Remove some unused deps and some unused benchmarking stuff

* Fix stakings ElectionDataProvider clear

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Improving staking pallet-bags-list migration

* fix build and some comments;

* comment

* Reduce visibility in bags list components

* make node.bag_upper only accesible to benchmarks

* Address some feedback; comments updates

* use nominator map comment

* fix vec capacity debug assert

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* clarify VoterSnapshotPerBlock

* Reduce diff on create_validators by wrapping with_seed

* Some small improvements to staking benches

* Soem comment updates

* fix vec capacity debug assert ... for real this time

* Reduce ListBags viz

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Remove supports_eq_unordered & Support eq_unordered

* Update utils/frame/generate-bags/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Make total-issuance & minimium-balance CLI args; Dont use emptry ext

* Improve docs for generate bags CLI args

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Don't use default bags weight in node runtime

* Feature gating sanity_check not working

* Feature gate sanity check by creating duplicate fns

* Fix line wrapping

* Document VoteWeightProvider

* Make bags ext-builder not a module

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* use pallet_bags_list instead of crate in mock

* Make get_bags test helper fn live in List

* use remove_from_storage_unchecked for node removal

* Remove count of ids removed in remove_many

* Add node sanity check, improve list sanity check

* Do a list sanity check after on_update

* List::migrate: clean up debug assert, exit early when no change in thresholds

* Improve public doc comments for pallet_bags_list::list::List

* Improve public doc comments for pallet_bags_list::list::List

* Update generate bags docs

* Fix grammar in bags-list benchmark

* Add benchmark case for `rebag` extrinsic

* Add count parameter to List::clear; WIP adding MaxEncodedLen to list'

* MaxEncodeLen + generate_storage_info not working for Bag or Node

* Get MaxEncodeLen derive to work

* Try to correctly feature gate SortedListProvider::clear

* Use u32::MAX, not u32::max_value

* Get up to nominators_quota noms

* SortedListProvider::clear takes an Option

* Eplicitly ignore SortedListProvider return value

* Fix doc comment

* Update node-runtime voter snapshot per block

* Add test get_max_len_voters_even_if_some_nominators_are_slashed

* Add test only_iterates_max_2_times_nominators_quota

* Fix generate bags cargo.toml

* use sp_std vec

* Remove v8 migration hooks from pallet-staking

* Update npos trait

* Try respect line width

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bags_list --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bags-list/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Update frame/bags-list/src/benchmarks.rs

* Unwrap try-runtime error; remove sortedlistprovider pre upgrade len  check

* trigger ci

* restore

* trigger ci

* restore

* trigger ci

* revert

* trigger ci

* revert

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2021-09-17 02:25:21 +00:00
Andrew Jones 49b6dfd2e5 Enrich metadata with type information (#8615)
* Cargo.lock after merge

* Restore scale-info feature

* Fully qualify TypeInfo derive

* Skip PendingSwap T

* Add missing skip_type_params attr

* metadata docs features

* Reduce pallet event attribute to struct

* Cargo.lock

* Update frame/balances/src/tests_composite.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Line widths check

* Cargo.lock

* Add scale-info/std

* Update frame/system/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Use `skip_type_params` to remove `TypeInfo` requirements on checks

* Revert "Remove unused Call metadata stuff"

This reverts commit 41311f85

* Skip BalanceSwapAction type parameter

* Remove unused event metadata macro

* Update frame-metadata

* Update primitives/npos-elections/compact/src/codec.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Manual TypeInfo for Header

* Remove TypeInfo requirement for consts in BoundedVec etc.

* Another TypeInfo bound removed

* review: fix indentation

* TypeInfo impls for Identity types

* Add some todos to add custom TypeInfo impls

* Update frame/support/procedural/src/pallet/expand/pallet_struct.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Add some todos to add custom TypeInfo impls

* Add a test for manual Data TypeInfo impl

* Add custom TypeInfo impl for Vote

* Era custom TypeInfo crimes

* Revert finality-grandpa version to 0.14.z

* review: renamed module to pallet_constants_metadata

* New line at end of file

* Add missing scale-info/std

* Update frame/support/src/storage/types/mod.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Remove StorageEntryType::Map unused flag

* Add missing scale-info dependency after merge

* SignedExtension::AdditionalSigned metadata

* Update frame-metadata, use abbreviated docs and args fields

* Update frame/example/Cargo.toml

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Add scale_info/std and remove unused scale-info dependency

* Remove scale-info dependency

* Remove treasury pallet::metadata

* Remove redundant Event test

* Add back scale-info as dev dependency

* fix error metadata when no error defined in decl_module

* Add Module3 to tests

* Fix metadata test

* Add docs feature to frame-support test

* WIP fixing pallet metadata test

* Remove redundant FunctionMetadata, FunctionArgumentMetadata as per https://github.com/paritytech/frame-metadata/pull/20

* Use main branch of frame-metadata

* Use patch of scale-info for latest changes

* Use latest patched scale-info

* Manual TypeInfo for DigestItem

* Manual TypeInfo for DigestItem

* Update scale-info

* Skip __Ignore variants for Error, depends on https://github.com/paritytech/scale-info/pull/117

* Named fields for FRAME v2 pallet Call variants

* Named fields for FRAME v1 pallet Call variants

* Add missing scale-info dependency

* WIP expand benchmark call variant

* fix benchmark with new function

create a new function for each variant of a pallet call.
This function is called by benchmarking macro in order not to break call
creation with unnamed argument

* fix tests

* more fix

* Fix staking tests

* Fix offchain workers calls

* Cherry pick rustfmt.toml from master

* cargo +nightly-2021-06-22 fmt --all

* Update to new call variant structs

* More call variant struct updates

* Remove unused import

* More call variant structs

* More call variant structs

* Even more call variant structs

* Mooar variant structs

* Evermore variant structs

* Call variant structs ad infinitum

* Fmt

* More call variants

* Last call variant

* Call variants all done?

* Fix SS58Prefix type

* Potential workaround for BitFlags<IdentityFields> TypeInfo

* Enable docs capturing for Call, Event, and Error types

* Fix IdentityFields TypeInfo

* Remove metadata-docs feature

* Add capture_docs = true for legacy Call, Event and Error types

* Fmt

* Fix metadata test type

* Update benchmarks with call struct variants

* Fmt

* More test fixes

* Fmt

* Fix benches

* Use latest capture_docs attr

* Latest scale_info

* Fmt

* review: change &Vec to &[]

* Remove pallet metadata attr

* review: remove commented out test code

* review: skip_type_params trailing comma suggestion

* Update to scale-info 0.10.0

* Update construct_runtime ui tests, different because of metadata TypeInfo impls

* Add some TypeInfo derives for UI tests

* Update storage ensure span ui stderrs

* Update call argument bound ui tests

Possibly changed because change from tuple to struct variants?

* Add scale-info dev dependency

* Update to latest finality-grandpa release

* review: missing newline

* review: missing scale-info/std

* review: remove duplicate scale-info/std

* review: remove fully qualified TypeInfo

* review: add missing scale-info/std

* review: remove unnecessary imports.

* Fmt

* Use crates.io RC version of frame-metadata

* Remove scale-info/std because it is a dev dependency

* Add missing scale_info dev-dependency for test

* Delete empty metadata folder

* Fix sp_std import

* review: improve manual UncheckedExtrinsic TypeInfo impl

* review: use full scale-info for dev-dependency

* Remove DefaultByteGetter impl

* review: derive TypeInfo for generic header

* Fmt

* Update primitives/runtime/src/generic/unchecked_extrinsic.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update primitives/runtime/src/generic/unchecked_extrinsic.rs

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* Update bin/node/executor/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/identity/src/types.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/support/src/dispatch.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove redundant derive

* Simplify scale-info dependency

* Strip underscore prefix from call variant struct names

* Another underscore field

* More underscore fields

* Another underscore field

* Update to frame-metadata 14.0.0-rc.2 with combined StorageEntryType::Map

* Fmt

* Revert weights formatting

* Fix up some tests

* Fix up some tests for StorageEntryTypeMetadata

* scale-info dev dependency

* Fix test error

* Add missing TypeInfo derives

* Add back missing scale-info dependency

* Add back missing scale-info dependency

* Fix npos compact impls

* Cargo.lock

* Fmt

* Fix errors

* Fmt

* Fix renamed raw_solution field

* Fix error

* Fmt

* Fix some benchmarks

* Fmt

* Stray R

* Fix

* Add missing TypeInfos

* ui test fix

* Fix line widths

* Revert "ui test fix"

This reverts commit 2d15ec058a216e3f92d713f1174603a2bb1eac65.

* Upgrade to scale-info 0.11.0

* Revert "Upgrade to scale-info 0.11.0"

This reverts commit 047bb179085a0059c36cd20ab405f55cf0867e28.

* Add Runtime type

* Update to scale-info 0.12

* Update to scale-info 1.0

* Update frame-metadata to version 14.0.0

* Patch finality-grandpa until release available

* Fix metadata tests

* Fix metadata tests

* Fmt

* Remove patched finality-grandpa

* Fix tests, use scale_info imports

* Fix pallet tests

* Add BlockNumber TypeInfo bound

* ui test fix

* Cargo.lock

* Remove pallet metadata

* Cargo.lock

* Add missing scale-info dependency

* Remove pallet event metadata

* Fix error

* Fix collective errors

* Semicolol

* Fmt

* Remove another metadata attribute

* Add new variant to custom digest TypeInfo

* Fmt

* Cargo.lock from master

* Remove comma lol

* Fix example call error

* Fix example call error properly

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-09-15 11:40:41 +00:00
Kian Paimani be69e4d2b2 clean the interface of supports map (#9674)
* clean the interface of supports map, make it a bit cleaner and more efficients

* Fix stiff

* fix one test

* Fix warnings
2021-09-10 18:39:48 +00:00
Zeke Mostov 6bfcfeed4c Create trait for NPoS election algorithms (#9664)
* build the template, hand it over to zeke now.

* Tests working

* save wip

* Some updates

* Some cleanup

* mo cleanin

* Link to issue

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Bound accuracy for prepare_election_result

* Use npos_election::Error for phragmms

* save

* Apply suggestions from code review

* Simplify test to use Balancing::set

* Cargo.lock after build

* Revert "Cargo.lock after build"

This reverts commit 7d726c8efa687c09e4f377196b106eb9e9760487.

* Try reduce cargo.lock diff

* Update bin/node/runtime/src/lib.rs

* Comment

* Apply suggestions from code review

* Set balancing directly

* Document som pub items

* Update frame/election-provider-multi-phase/src/unsigned.rs

* Apply suggestions from code review

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Improve some comments

* Revert accidental change to random file

* tiney

* revert

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-09-09 19:46:24 +00:00
Kian Paimani f7bcbdd261 Multi-Block Election part 0: preparation and some cleanup. (#9442)
* Partially applied

* Everything builds, need to implement compact encoding as well.

* Fix some tests, add a ui test as well.

* Fix everything and everything.

* small nits

* a bunch more rename

* more reorg

* more reorg

* last nit of self-review

* Seemingly fixed the build now

* Fix build

* make it work again

* Update primitives/npos-elections/solution-type/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Update primitives/npos-elections/solution-type/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* nits

* factor out double type

* fix try-build

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
2021-08-11 15:45:53 +00:00
Kian Paimani abd08e29ce Rewrap all comments to 100 line width (#9490)
* reformat everything again

* manual formatting

* last manual fix

* Fix build
2021-08-11 14:56:55 +00:00
Squirrel 2f4db88b41 Removing wasm unused import warnings (#9492)
* Removing wasm unused import warnings

* cargo fmt
2021-08-04 10:57:08 +00:00
Squirrel 492523f1ae post rustfmt whitespace fixup (#9436)
Taking best bits of rustfmt's format_strings

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>
2021-07-26 14:18:27 +01:00
Squirrel aafe64315a Remove extra commas made redundent after rustfmt (#9404)
* Remove extra commas made redundent after rustfmt
2021-07-22 11:06:17 +01:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Squirrel ea1f21a904 Use MAX associated const (#9196)
* Use MAX associated const
2021-06-24 11:53:49 +01:00
Peter Goodspeed-Niklaus c786fb21a0 Refactor election solution trimming for efficiency (#8614)
* Refactor election solution trimming for efficiency

The previous version always trimmed the `CompactOf<T>` instance,
which was intrinsically inefficient: that's a packed data structure,
which is naturally expensive to edit. It's much easier to edit
the unpacked data structures: the `voters` and `assignments` lists.

* rework length-trim tests to work with the new interface

Test suite now compiles. Tests still don't pass because the macro
generating the compact structure still generates `unimplemented!()`
for the actual `compact_length_of` implementation.

* simplify

* add a fuzzer which can validate `Compact::encoded_size_for`

The `Compact` solution type is generated distinctly for each runtime,
and has both three type parameters and a built-in limit to the number
of candidates that each voter can vote for. Finally, they have an
optional `#[compact]` attribute which changes the encoding behavior.

The assignment truncation algorithm we're using depends on the ability
to efficiently and accurately determine how much space a `Compact`
solution will take once encoded.

Together, these two facts imply that simple unit tests are not
sufficient to validate the behavior of `Compact::encoded_size_for`.
This commit adds such a fuzzer. It is designed such that it is possible
to add a new fuzzer to the family by simply adjusting the
`generate_solution_type` macro invocation as desired, and making a
few minor documentation edits.

Of course, the fuzzer still fails for now: the generated implementation
for `encoded_size_for` is still `unimplemented!()`. However, once
the macro is updated appropriately, this fuzzer family should allow
us to gain confidence in the correctness of the generated code.

* Revert "add a fuzzer which can validate `Compact::encoded_size_for`"

This reverts commit 916038790887e64217c6a46e9a6d281386762bfb.

The design of `Compact::encoded_size_for` is flawed. When `#[compact]`
mode is enabled, every integer in the dataset is encoded using run-
length encoding. This means that it is impossible to compute the final
length faster than actually encoding the data structure, because the
encoded length of every field varies with the actual value stored.

Given that we won't be adding that method to the trait, we won't be
needing a fuzzer to validate its performance.

* revert changes to `trait CompactSolution`

If `CompactSolution::encoded_size_for` can't be implemented in the
way that we wanted, there's no point in adding it.

* WIP: restructure trim_assignments_length by actually encoding

This is not as efficient as what we'd hoped for, but it should still
be better than what it's replacing. Overall efficiency of
`fn trim_assignments_length` is now `O(edges * lg assignments.len())`.

* fix compiler errors

* don't sort voters, just assignments

Sorting the `voters` list causes lots of problems; an invariant that
we need to maintain is that an index into the voters list has a stable
meaning.

Luckily, it turns out that there is no need for the assignments list
to correspond to the voters list. That isn't an invariant, though previously
I'd thought that it was.

This simplifies things; we can just leave the voters list alone,
and sort the assignments list the way that is convenient.

* WIP: add `IndexAssignment` type to speed up repeatedly creating `Compact`

Next up: `impl<'a, T> From<&'a [IndexAssignmentOf<T>]> for Compact`,
in the proc-macro which makes `Compact`. Should be a pretty straightforward
adaptation of `from_assignment`.

* Add IndexAssignment and conversion method to CompactSolution

This involves a bit of duplication of types from
`election-provider-multi-phase`; we'll clean those up shortly.

I'm not entirely happy that we had to add a `from_index_assignments`
method to `CompactSolution`, but we couldn't define
`trait CompactSolution: TryFrom<&'a [Self::IndexAssignment]` because
that made trait lookup recursive, and I didn't want to propagate
`CompactSolutionOf<T> + TryFrom<&[IndexAssignmentOf<T>]>` everywhere
that compact solutions are specified.

* use `CompactSolution::from_index_assignment` and clean up dead code

* get rid of `from_index_assignments` in favor of `TryFrom`

* cause `pallet-election-provider-multi-phase` tests to compile successfully

Mostly that's just updating the various test functions to keep track of
refactorings elsewhere, though in a few places we needed to refactor some
test-only helpers as well.

* fix infinite binary search loop

Turns out that moving `low` and `high` into an averager function is a
bad idea, because the averager gets copies of those values, which
of course are never updated. Can't use mutable references, because
we want to read them elsewhere in the code. Just compute the average
directly; life is better that way.

* fix a test failure

* fix the rest of test failures

* remove unguarded subtraction

* fix npos-elections tests compilation

* ensure we use sp_std::vec::Vec in assignments

* add IndexAssignmentOf to sp_npos_elections

* move miner types to `unsigned`

* use stable sort

* rewrap some long comments

* use existing cache instead of building a dedicated stake map

* generalize the TryFrom bound on CompactSolution

* undo adding sp-core dependency

* consume assignments to produce index_assignments

* Add a test of Assignment -> IndexAssignment -> Compact

* fix `IndexAssignmentOf` doc

* move compact test from sp-npos-elections-compact to sp-npos-elections

This means that we can put the mocking parts of that into a proper
mock package, put the test into a test package among other tests.

Having the mocking parts in a mock package enables us to create a
benchmark (which is treated as a separate crate) import them.

* rename assignments -> sorted_assignments

* sort after reducing to avoid potential re-sort issues

* add runtime benchmark, fix critical binary search error

"Why don't you add a benchmark?", he said. "It'll be good practice,
and can help demonstrate that this isn't blowing up the runtime."

He was absolutely right.

The biggest discovery is that adding a parametric benchmark means that
you get a bunch of new test cases, for free. This is excellent, because
those test cases uncovered a binary search bug. Fixing that simplified
that part of the code nicely.

The other nice thing you get from a parametric benchmark is data about
what each parameter does. In this case, `f` is the size factor: what
percent of the votes (by size) should be removed. 0 means that we should
keep everything, 95 means that we should trim down to 5% of original size
or less.

```
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=     3846
    + v    0.015
    + t        0
    + a    0.192
    + d        0
    + f        0
              µs

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    v     t     a     d     f   mean µs  sigma µs       %
<snip>
 6000  1600  3000   800     0      4385     75.87    1.7%
 6000  1600  3000   800     9      4089     46.28    1.1%
 6000  1600  3000   800    18      3793     36.45    0.9%
 6000  1600  3000   800    27      3365     41.13    1.2%
 6000  1600  3000   800    36      3096     7.498    0.2%
 6000  1600  3000   800    45      2774     17.96    0.6%
 6000  1600  3000   800    54      2057     37.94    1.8%
 6000  1600  3000   800    63      1885     2.515    0.1%
 6000  1600  3000   800    72      1591     3.203    0.2%
 6000  1600  3000   800    81      1219     25.72    2.1%
 6000  1600  3000   800    90       859     5.295    0.6%
 6000  1600  3000   800    95     684.6     2.969    0.4%

Quality and confidence:
param     error
v         0.008
t         0.029
a         0.008
d         0.044
f         0.185

Model:
Time ~=     3957
    + v    0.009
    + t        0
    + a    0.185
    + d        0
    + f        0
              µs
```

What's nice about this is the clear negative correlation between
amount removed and total time. The more we remove, the less total
time things take.
2021-05-03 07:26:35 +00:00
Peter Goodspeed-Niklaus c2dd5e21a4 make types within generate_solution_type macro explicit (#8447)
* make types within `generate_solution_type` macro explicit

Closes #8444.

Just changes the parsing logic for that macro; does not change any
emitted code. The associated types associated with the macro now
require explicit, keyword-style declaration.

**Old**:

```rust
sp_npos_elections::generate_solution_type!(
	#[compact]
	pub struct TestCompact::<VoterIndex, TargetIndex, PerU16>(16)
);
```

**New**:

```rust
sp_npos_elections::generate_solution_type!(
	#[compact]
	pub struct TestCompact::<VoterIndex = VoterIndex, CandidateIndex = TargetIndex, Accuracy = PerU16>(16)
);
```

* un-ignore doc-tests

* use new form in bin/node/runtime/

* rename CandidateIndex -> TargetIndex

* add tests demonstrating some potential compile failures
2021-03-28 08:21:06 +00:00
Peter Goodspeed-Niklaus 23b32e7543 Improve complexity of CompactAssignments::unique_targets (#8314)
* Improve complexity of CompactAssignments::unique_targets

Original implementation was O(n**2). Current impl is O(n log n).

Avoided the original proposed mitigation because it does not retain
the de-duplicating property present in the original implementation.
This implementation does a little more work, but retains that property.

* Explicitly choose sp_std Vec and BTreeSet

Ensures that the macro still works if someone uses it in a context
in which sp_std is not imported or is renamed.

* explicitly use sp_std vectors throughout compact macro
2021-03-17 08:13:33 +00:00
Gavin Wood 363db4f086 staking: Flexible generation of reward curve and associated tweaks (#8327)
* Initial abstraction

* Alter rest of APIs

* Fixes

* Some extra getters in Gilt pallet.

* Refactor Gilt to avoid u128 conversions

* Simplify and improve pow in per_things

* Add scalar division to per_things

* Renaming from_fraction -> from_float, drop _approximation

* Fixes

* Fixes

* Fixes

* Fixes

* Make stuff build

* Fixes

* Fixes

* Fixes

* Fixes

* Update .gitignore

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/gilt/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/gilt/src/mock.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Fixes

* Fixes

* Fixes

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-03-16 12:03:58 +00:00
Kian Paimani b6c626399e Audit fixes for election/staking decoupling part 2 (#8167)
* Base features and traits.

* pallet and unsigned phase

* Undo bad formattings.

* some formatting cleanup.

* Small self-cleanup.

* Make it all build

* self-review

* Some doc tests.

* Some changes from other PR

* Fix session test

* Update Cargo.lock

* Update frame/election-provider-multi-phase/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Some review comments

* Rename + make encode/decode

* Do an assert as well, just in case.

* Fix build

* Update frame/election-provider-multi-phase/src/unsigned.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Las comment

* fix staking fuzzer.

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Add one last layer of feasibility check as well.

* Last fixes to benchmarks

* Some more docs.

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Some nits

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Fix doc

* Mkae ci green

* Audit fixes for election-provider: part 2 signed phase.

* Fix weight

* Some grumbles.

* Try and weigh to get_npos_voters

* Fix build

* Fix line width

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Fix tests.

* Fix build

* Reorg some stuff

* More reorg.

* Reorg done.

* Fix build

* Another rename

* Fix build

* Update frame/election-provider-multi-phase/src/mock.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* nit

* better doc

* Line width

* Fix build

* Self-review

* Self-review

* Fix wan

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix build and review comments.

* Update frame/election-provider-multi-phase/src/lib.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* add comment

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2021-03-16 12:44:30 +01:00
Peter Goodspeed-Niklaus 49be0579db NPoS Challenge Mode (#8236)
* Add PJR challenge functions

- Updates the PJR check to return a counterexample if one exists
- Adds functions to cheaply check counterexamples

This is in support of off-chain PJR challenges: if a miner discovers
that an accepted election solution does not satisfy PJR, it will be
eligible for substantial rewards. This helps ensure that validator
elections have an absolute quality floor, so even if someone manages
to censor well-behaved solutions to give themselves unfair representation,
we can catch them in the act and penalize them.

* counterexample -> counter_example

* reorganize: high -> low abstraction

* reorganize challenges high -> low abstraction

* add note justifying linear search

* Simplify max_pre_score validation

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* add minor test of pjr challenge validation

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2021-03-16 09:00:03 +00:00
Peter Goodspeed-Niklaus 781f908760 Implement PJR checker (#8160)
* Apply.

* get rid of glob import

* use meaningful generic type name

* pjr_check operates on `Supports` struct used elsewhere

* improve algorithmic complexity of `prepare_pjr_input`

* fix rustdoc warnings

* improve module docs

* typo

* simplify debug assertion

* add test finding the phase-change threshold value for a constructed scenario

* add more threshold scenarios to disambiguate plausible interpretations

* add link to npos paper reference

* docs: staked_assignment -> supports

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* add utility method for generating npos inputs

* add a fuzzer which asserts that all unbalanced seq_phragmen are PJR

Note that this currently fails. I hope that this can be rectified
by calculating the threshold instead of choosing some arbitrary number.

* assert in all cases, not just debug

* leverage a native solution to choose candidates

* use existing helper methods

* add pjr-check and incorporate into the fuzzer

We should probably have one of the W3F people look at this to ensure
we're not misconstruing any definitions, but this seems like a
fairly straightforward implementation.

* fix compilation errors

* Enable manually setting iteration parameters in single run.

This gives us the ability to reproducably extract cases where
honggfuzz has discovered a panic. For example:

$ cargo run --release --bin phragmen_pjr -- --candidates 569 --voters 100
Tue 23 Feb 2021 11:23:39 AM CET
   Compiling bitflags v1.2.1
   Compiling unicode-width v0.1.8
   Compiling unicode-segmentation v1.7.1
   Compiling ansi_term v0.11.0
   Compiling strsim v0.8.0
   Compiling vec_map v0.8.2
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling textwrap v0.11.0
   Compiling atty v0.2.14
   Compiling heck v0.3.2
   Compiling clap v2.33.3
   Compiling structopt-derive v0.4.14
   Compiling structopt v0.3.21
   Compiling sp-npos-elections-fuzzer v2.0.0-alpha.5 (/home/coriolinus/Documents/Projects/paritytech/substrate/primitives/npos-elections/fuzzer)
    Finished release [optimized] target(s) in 6.15s
     Running `/home/coriolinus/Documents/Projects/paritytech/substrate/target/release/phragmen_pjr -c 569 -v 100`
thread 'main' panicked at 'unbalanced sequential phragmen must satisfy PJR', primitives/npos-elections/fuzzer/src/phragmen_pjr.rs:133:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is still not adequate proof that seq_phragmen is broken; it could
very well be that our PJR checker is doing the wrong thing, or we've
somehow missed a parameter of interest. Still, it's concerning.

* update comment verbiage for accuracy

* it is valid in PJR for an elected candidate to have 0 support

* Fix phragmen_pjr fuzzer

It turns out that the fundamental problem causing previous implementations
of the fuzzer to fail wasn't in `seq_phragmen` _or_ in `pjr_check`: it was
in the rounding errors introduced in the various conversions between the
internal data representation and the external one.

Fixing the fuzzer is then simply an issue of using the internal representation
and staying in that representation. However, that leaves the issue that
`seq_phragmen` occasionally produces an output which is technically not
PJR due to rounding errors. In the future we will need to add some kind of
"close-enough" threshold. However, that is explicitly out of scope of
this PR.

* restart ci; it appears to be stalled

* use necessary import for no-std

* use a more realistic distribution of voters and candidates

This isn't ideal; more realistic numbers would be about twice these.
However, either case generation or voting has nonlinear execution
time, and doubling these values brings iteration time from ~20s to
~180s. Fuzzing 6x as fast should make up for fuzzing cases half the size.

* identify specifically which PJR check may fail

* move candidate collection comment into correct place

* standard_threshold: use a calculation method which cannot overflow

* Apply suggestions from code review (update comments)

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* clarify the effectiveness bounds for t-pjr check

* how to spell "committee"

* reorganize: high -> low abstraction

* ensure standard threshold calc cannot panic

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Co-authored-by: kianenigma <kian.peymani@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2021-03-11 09:06:53 +00:00
Kian Paimani 0dd8cd6277 Attempt to remove the where bounds in arithmetic. (#7933)
* Attempt to remove the where bounds.

* Fix further and further.

* Format better.

* Update primitives/npos-elections/src/lib.rs

* fix build

* remove unused
2021-01-21 09:39:30 +00:00
Kian Paimani ced107b355 Decouple Stkaing and Election - Part1: Support traits (#7908)
* Base features and traits.

* Fix the build

* Remove unused boxing

* Self review cleanup

* Fix build
2021-01-18 10:24:12 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Guillaume Thiolliere 8814d793c7 Fix docs (#7710)
* fix docs

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

Co-authored-by: Alexander Theißen <alex.theissen@me.com>
2020-12-10 14:38:56 +01:00
Krishna Singh f717a20446 Changed map to filter map so that Phragmen ignores empty voters (#7378)
* Changed map to filter map so that Phragmen ignores empty voters

* Resolve flaws and added test case

* Updated test
2020-12-03 12:44:53 +00:00
Bastian Köcher 5808f467c5 Make Substrate compile with latest nightly (#7381)
* Try to get it compiling

* One more

* Make stable happy

* Make stable even more happy ;)

* Update shell.nix
2020-10-23 21:08:04 +02:00