Companion for Store voters in unsorted bags to get good stake iteration properties (#3413)

* add voter bags generation script and generated values for all runtimes

* VoterBagThresholds for test-runtime

* semicolon

* turns out the header is called file_header.txt

* perform benchmarks to get real numbers

* add note about when to re-run this script

* Suggestions for #3413 (Store voters in unsorted bags companion) (#3488)

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: CI system <>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>

* Try resolve Cargo.lock conflicts

* Add temp VoterSnapshotPerBlock value to compile

* Improve voter_bags test with nominator/validator breakdown

* Remove miner.log

* Update README

* Module doc comments

* Build working

* remote-ext-tests very  WIP

* Remove system module query

* formatting and log target

* clean up

* Save

* Fix issues with polkadot runtime after mergin

* Save

* Some updates to runtimes to make sure things compile

* save

* udpate weights

* Use UseNominatorsAndUpdateBagsList

* make remote-ext tests a bin

* Adjust remote-ext test to use pub; update some comments

* Use list_bags_get

* Try fix cargo.lock

* Gate kusama feat code

* Carg.lock .. again

* point bags-list deps to master

* Update voter-bags  CLI to take total issuance and min balance as args

* Switch polkadot runtime to UseNominatorMap

* Some tweaks to the remote-test

* Delete some stuff for polkadot; comments

* Various fixes

* Add comment for VoterSnapshotPerBlock

* Remove bags-list import from polakdot cargo

* use benchmarking feature for pallet-bags-list

* Add custom migration for westend and kusama for bags-list

* :facepalm

* Fix kusama runtime

* Small clean up

* Be more alphabetical ordered

* Use 22_500 for VoterSnapshotPerBlock

* point back to master

* remove unused type

* SortedListProvider for staking runtime

* Remove polkadot for remote-ext bags test

* Revert "Remove polkadot for remote-ext bags test"

This reverts commit 3608c466002063761cd669880a00b2e05c1ff7d1.

* Remove polkadot for remote-ext bags test

* update Substrate

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>
Co-authored-by: parity-processbot <>

polkadot/utils/remote-ext-tests/bags-list/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "remote-ext-tests-bags-list"
+version = "0.9.9"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+
+[dependencies]
+polkadot-runtime = { version = "0.9.8", path = "../../../runtime/polkadot" }
+kusama-runtime = { version = "0.9.8", path = "../../../runtime/kusama" }
+
+pallet-staking = { git = "https://github.com/paritytech/substrate", branch = "master" }
+pallet-election-provider-multi-phase = { git = "https://github.com/paritytech/substrate", branch = "master" }
+pallet-bags-list = { git = "https://github.com/paritytech/substrate", branch = "master" }
+frame-election-provider-support = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-storage = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-core = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-tracing = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-runtime = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-std = { git = "https://github.com/paritytech/substrate", branch = "master" }
+frame-support = { git = "https://github.com/paritytech/substrate", branch = "master" }
+remote-externalities = { git = "https://github.com/paritytech/substrate", branch = "master" }
+tokio = { version = "1", features = ["macros"] }
+log = { version = "0.4.14" }
+structopt = {version = "0.3.23" }