Companion for Store voters in unsorted bags to get good stake iteration properties (#3413)

* add voter bags generation script and generated values for all runtimes

* VoterBagThresholds for test-runtime

* semicolon

* turns out the header is called file_header.txt

* perform benchmarks to get real numbers

* add note about when to re-run this script

* Suggestions for #3413 (Store voters in unsorted bags companion) (#3488)

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: CI system <>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>

* Try resolve Cargo.lock conflicts

* Add temp VoterSnapshotPerBlock value to compile

* Improve voter_bags test with nominator/validator breakdown

* Remove miner.log

* Update README

* Module doc comments

* Build working

* remote-ext-tests very  WIP

* Remove system module query

* formatting and log target

* clean up

* Save

* Fix issues with polkadot runtime after mergin

* Save

* Some updates to runtimes to make sure things compile

* save

* udpate weights

* Use UseNominatorsAndUpdateBagsList

* make remote-ext tests a bin

* Adjust remote-ext test to use pub; update some comments

* Use list_bags_get

* Try fix cargo.lock

* Gate kusama feat code

* Carg.lock .. again

* point bags-list deps to master

* Update voter-bags  CLI to take total issuance and min balance as args

* Switch polkadot runtime to UseNominatorMap

* Some tweaks to the remote-test

* Delete some stuff for polkadot; comments

* Various fixes

* Add comment for VoterSnapshotPerBlock

* Remove bags-list import from polakdot cargo

* use benchmarking feature for pallet-bags-list

* Add custom migration for westend and kusama for bags-list

* :facepalm

* Fix kusama runtime

* Small clean up

* Be more alphabetical ordered

* Use 22_500 for VoterSnapshotPerBlock

* point back to master

* remove unused type

* SortedListProvider for staking runtime

* Remove polkadot for remote-ext bags test

* Revert "Remove polkadot for remote-ext bags test"

This reverts commit 3608c466002063761cd669880a00b2e05c1ff7d1.

* Remove polkadot for remote-ext bags test

* update Substrate

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>
Co-authored-by: parity-processbot <>

polkadot/runtime/common/src/elections.rs

@@ -17,11 +17,13 @@
 //! Code for elections.
 
 use super::{BlockExecutionWeight, BlockLength, BlockWeights};
+use frame_election_provider_support::{SortedListProvider, VoteWeight};
 use frame_support::{
 	parameter_types,
 	weights::{DispatchClass, Weight},
 };
 use sp_runtime::Perbill;
+use sp_std::{boxed::Box, convert::From, marker::PhantomData};
 
 parameter_types! {
 	/// A limit for off-chain phragmen unsigned solution submission.
@@ -89,3 +91,59 @@ impl frame_support::pallet_prelude::Get<Option<(usize, sp_npos_elections::Extend
 		Some((iters, 0))
 	}
 }
+
+/// Implementation of `frame_election_provider_support::SortedListProvider` that updates the
+/// bags-list but uses [`pallet_staking::Nominators`] for `iter`. This is meant to be a transitionary
+/// implementation for runtimes to "test" out the bags-list by keeping it up to date, but not yet
+/// using it for snapshot generation. In contrast, a  "complete" implementation would use bags-list
+/// for `iter`.
+pub struct UseNominatorsAndUpdateBagsList<T>(PhantomData<T>);
+impl<T: pallet_bags_list::Config + pallet_staking::Config> SortedListProvider<T::AccountId>
+	for UseNominatorsAndUpdateBagsList<T>
+{
+	type Error = pallet_bags_list::Error;
+
+	fn iter() -> Box<dyn Iterator<Item = T::AccountId>> {
+		Box::new(pallet_staking::Nominators::<T>::iter().map(|(n, _)| n))
+	}
+
+	fn count() -> u32 {
+		pallet_bags_list::Pallet::<T>::count()
+	}
+
+	fn contains(id: &T::AccountId) -> bool {
+		pallet_bags_list::Pallet::<T>::contains(id)
+	}
+
+	fn on_insert(id: T::AccountId, weight: VoteWeight) -> Result<(), Self::Error> {
+		pallet_bags_list::Pallet::<T>::on_insert(id, weight)
+	}
+
+	fn on_update(id: &T::AccountId, new_weight: VoteWeight) {
+		pallet_bags_list::Pallet::<T>::on_update(id, new_weight);
+	}
+
+	fn on_remove(id: &T::AccountId) {
+		pallet_bags_list::Pallet::<T>::on_remove(id);
+	}
+
+	fn regenerate(
+		all: impl IntoIterator<Item = T::AccountId>,
+		weight_of: Box<dyn Fn(&T::AccountId) -> VoteWeight>,
+	) -> u32 {
+		pallet_bags_list::Pallet::<T>::regenerate(all, weight_of)
+	}
+
+	fn sanity_check() -> Result<(), &'static str> {
+		pallet_bags_list::Pallet::<T>::sanity_check()
+	}
+
+	fn clear(count: Option<u32>) -> u32 {
+		pallet_bags_list::Pallet::<T>::clear(count)
+	}
+
+	#[cfg(feature = "runtime-benchmarks")]
+	fn weight_update_worst_case(who: &T::AccountId, is_increase: bool) -> VoteWeight {
+		pallet_bags_list::Pallet::<T>::weight_update_worst_case(who, is_increase)
+	}
+}