Companion for Store voters in unsorted bags to get good stake iteration properties (#3413)

* add voter bags generation script and generated values for all runtimes

* VoterBagThresholds for test-runtime

* semicolon

* turns out the header is called file_header.txt

* perform benchmarks to get real numbers

* add note about when to re-run this script

* Suggestions for #3413 (Store voters in unsorted bags companion) (#3488)

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: CI system <>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>

* Try resolve Cargo.lock conflicts

* Add temp VoterSnapshotPerBlock value to compile

* Improve voter_bags test with nominator/validator breakdown

* Remove miner.log

* Update README

* Module doc comments

* Build working

* remote-ext-tests very  WIP

* Remove system module query

* formatting and log target

* clean up

* Save

* Fix issues with polkadot runtime after mergin

* Save

* Some updates to runtimes to make sure things compile

* save

* udpate weights

* Use UseNominatorsAndUpdateBagsList

* make remote-ext tests a bin

* Adjust remote-ext test to use pub; update some comments

* Use list_bags_get

* Try fix cargo.lock

* Gate kusama feat code

* Carg.lock .. again

* point bags-list deps to master

* Update voter-bags  CLI to take total issuance and min balance as args

* Switch polkadot runtime to UseNominatorMap

* Some tweaks to the remote-test

* Delete some stuff for polkadot; comments

* Various fixes

* Add comment for VoterSnapshotPerBlock

* Remove bags-list import from polakdot cargo

* use benchmarking feature for pallet-bags-list

* Add custom migration for westend and kusama for bags-list

* :facepalm

* Fix kusama runtime

* Small clean up

* Be more alphabetical ordered

* Use 22_500 for VoterSnapshotPerBlock

* point back to master

* remove unused type

* SortedListProvider for staking runtime

* Remove polkadot for remote-ext bags test

* Revert "Remove polkadot for remote-ext bags test"

This reverts commit 3608c466002063761cd669880a00b2e05c1ff7d1.

* Remove polkadot for remote-ext bags test

* update Substrate

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>
Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com>
Co-authored-by: Seun Lanlege <seun@parity.io>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Denis Tsai <denistsai@aetheras.io>
Co-authored-by: Pierre Besson <pierre.besson@parity.io>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com>
Co-authored-by: Shaun W <spxwang@gmail.com>
Co-authored-by: parity-processbot <>

polkadot/Cargo.toml

@@ -92,6 +92,8 @@ members = [
 	"parachain/test-parachains/adder",
 	"parachain/test-parachains/adder/collator",
 	"utils/staking-miner",
+	"utils/remote-ext-tests/bags-list",
+	"utils/voter-bags",
 ]
 
 # We want to be able to build the bridge relayer without pulling it (and all of its

polkadot/runtime/common/Cargo.toml

@@ -39,6 +39,7 @@ pallet-treasury = { git = "https://github.com/paritytech/substrate", branch = "m
 pallet-election-provider-multi-phase = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-beefy-mmr = { git = "https://github.com/paritytech/grandpa-bridge-gadget", branch = "master", default-features = false }
 frame-election-provider-support = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
+pallet-bags-list = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 
 frame-benchmarking = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false, optional = true }
 pallet-babe = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false, optional = true }
@@ -96,6 +97,7 @@ std = [
 	"runtime-parachains/std",
 	"xcm/std",
 	"sp-npos-elections/std",
+	"pallet-bags-list/std"
 ]
 runtime-benchmarks = [
 	"libsecp256k1/hmac",

polkadot/runtime/common/src/elections.rs

@@ -17,11 +17,13 @@
 //! Code for elections.
 
 use super::{BlockExecutionWeight, BlockLength, BlockWeights};
+use frame_election_provider_support::{SortedListProvider, VoteWeight};
 use frame_support::{
 	parameter_types,
 	weights::{DispatchClass, Weight},
 };
 use sp_runtime::Perbill;
+use sp_std::{boxed::Box, convert::From, marker::PhantomData};
 
 parameter_types! {
 	/// A limit for off-chain phragmen unsigned solution submission.
@@ -89,3 +91,59 @@ impl frame_support::pallet_prelude::Get<Option<(usize, sp_npos_elections::Extend
 		Some((iters, 0))
 	}
 }
+
+/// Implementation of `frame_election_provider_support::SortedListProvider` that updates the
+/// bags-list but uses [`pallet_staking::Nominators`] for `iter`. This is meant to be a transitionary
+/// implementation for runtimes to "test" out the bags-list by keeping it up to date, but not yet
+/// using it for snapshot generation. In contrast, a  "complete" implementation would use bags-list
+/// for `iter`.
+pub struct UseNominatorsAndUpdateBagsList<T>(PhantomData<T>);
+impl<T: pallet_bags_list::Config + pallet_staking::Config> SortedListProvider<T::AccountId>
+	for UseNominatorsAndUpdateBagsList<T>
+{
+	type Error = pallet_bags_list::Error;
+
+	fn iter() -> Box<dyn Iterator<Item = T::AccountId>> {
+		Box::new(pallet_staking::Nominators::<T>::iter().map(|(n, _)| n))
+	}
+
+	fn count() -> u32 {
+		pallet_bags_list::Pallet::<T>::count()
+	}
+
+	fn contains(id: &T::AccountId) -> bool {
+		pallet_bags_list::Pallet::<T>::contains(id)
+	}
+
+	fn on_insert(id: T::AccountId, weight: VoteWeight) -> Result<(), Self::Error> {
+		pallet_bags_list::Pallet::<T>::on_insert(id, weight)
+	}
+
+	fn on_update(id: &T::AccountId, new_weight: VoteWeight) {
+		pallet_bags_list::Pallet::<T>::on_update(id, new_weight);
+	}
+
+	fn on_remove(id: &T::AccountId) {
+		pallet_bags_list::Pallet::<T>::on_remove(id);
+	}
+
+	fn regenerate(
+		all: impl IntoIterator<Item = T::AccountId>,
+		weight_of: Box<dyn Fn(&T::AccountId) -> VoteWeight>,
+	) -> u32 {
+		pallet_bags_list::Pallet::<T>::regenerate(all, weight_of)
+	}
+
+	fn sanity_check() -> Result<(), &'static str> {
+		pallet_bags_list::Pallet::<T>::sanity_check()
+	}
+
+	fn clear(count: Option<u32>) -> u32 {
+		pallet_bags_list::Pallet::<T>::clear(count)
+	}
+
+	#[cfg(feature = "runtime-benchmarks")]
+	fn weight_update_worst_case(who: &T::AccountId, is_increase: bool) -> VoteWeight {
+		pallet_bags_list::Pallet::<T>::weight_update_worst_case(who, is_increase)
+	}
+}

polkadot/runtime/kusama/Cargo.toml

@@ -72,6 +72,7 @@ pallet-treasury = { git = "https://github.com/paritytech/substrate", branch = "m
 pallet-utility = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-vesting = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-xcm = { path = "../../xcm/pallet-xcm", default-features = false }
+pallet-bags-list = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 frame-election-provider-support = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 
 frame-benchmarking = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false, optional = true }
@@ -174,6 +175,7 @@ std = [
 	"xcm-executor/std",
 	"xcm-builder/std",
 	"frame-election-provider-support/std",
+	"pallet-bags-list/std",
 ]
 runtime-benchmarks = [
 	"runtime-common/runtime-benchmarks",
@@ -211,6 +213,7 @@ runtime-benchmarks = [
 	"hex-literal",
 	"xcm-builder/runtime-benchmarks",
 	"frame-election-provider-support/runtime-benchmarks",
+	"pallet-bags-list/runtime-benchmarks",
 ]
 try-runtime = [
 	"frame-executive/try-runtime",

polkadot/runtime/kusama/src/lib.rs

@@ -103,6 +103,9 @@ use constants::{currency::*, fee::*, time::*};
 // Weights used in the runtime.
 mod weights;
 
+// Voter bag threshold definitions.
+mod voter_bags;
+
 #[cfg(test)]
 mod tests;
 
@@ -354,6 +357,11 @@ parameter_types! {
 
 	// miner configs
 	pub OffchainRepeat: BlockNumber = 5;
+
+	/// Whilst `UseNominatorsAndUpdateBagsList` or `UseNominatorsMap` is in use, this can still be a
+	/// very large value. Once the `BagsList` is in full motion, staking might open its door to many
+	/// more nominators, and this value should instead be what is a "safe" number (e.g. 22500).
+	pub const VoterSnapshotPerBlock: u32 = 22_500;
 }
 
 sp_npos_elections::generate_solution_type!(
@@ -399,6 +407,18 @@ impl pallet_election_provider_multi_phase::Config for Runtime {
 		pallet_collective::EnsureProportionAtLeast<_2, _3, AccountId, CouncilCollective>,
 	>;
 	type WeightInfo = weights::pallet_election_provider_multi_phase::WeightInfo<Self>;
+	type VoterSnapshotPerBlock = VoterSnapshotPerBlock;
+}
+
+parameter_types! {
+	pub const BagThresholds: &'static [u64] = &voter_bags::THRESHOLDS;
+}
+
+impl pallet_bags_list::Config for Runtime {
+	type Event = Event;
+	type VoteWeightProvider = Staking;
+	type WeightInfo = weights::pallet_bags_list::WeightInfo<Runtime>;
+	type BagThresholds = BagThresholds;
 }
 
 fn era_payout(
@@ -505,6 +525,9 @@ impl pallet_staking::Config for Runtime {
 	type EraPayout = EraPayout;
 	type NextNewSession = Session;
 	type MaxNominatorRewardedPerValidator = MaxNominatorRewardedPerValidator;
+	// Use the nominators map to iter voters, but also perform the bags-list migration and keep
+	// it up-to-date.
+	type SortedListProvider = runtime_common::elections::UseNominatorsAndUpdateBagsList<Runtime>;
 	type WeightInfo = weights::pallet_staking::WeightInfo<Runtime>;
 }
 
@@ -1462,6 +1485,9 @@ construct_runtime! {
 		// Gilts pallet.
 		Gilt: pallet_gilt::{Pallet, Call, Storage, Event<T>, Config} = 38,
 
+		// Provides a semi-sorted list of nominators for staking.
+		BagsList: pallet_bags_list::{Pallet, Call, Storage, Event<T>} = 39,
+
 		// Parachains pallets. Start indices at 50 to leave room.
 		ParachainsOrigin: parachains_origin::{Pallet, Origin} = 50,
 		Configuration: parachains_configuration::{Pallet, Call, Storage, Config<T>} = 51,
@@ -1521,6 +1547,7 @@ pub type Executive = frame_executive::Executive<
 		TechnicalCommitteeStoragePrefixMigration,
 		TechnicalMembershipStoragePrefixMigration,
 		MigrateTipsPalletPrefix,
+		StakingBagsListMigrationV8,
 	),
 >;
 /// The payload being signed in the transactions.
@@ -1638,6 +1665,25 @@ impl OnRuntimeUpgrade for MigrateTipsPalletPrefix {
 	}
 }
 
+// Migration to generate pallet staking's `SortedListProvider` from pre-existing nominators.
+pub struct StakingBagsListMigrationV8;
+
+impl OnRuntimeUpgrade for StakingBagsListMigrationV8 {
+	fn on_runtime_upgrade() -> frame_support::weights::Weight {
+		pallet_staking::migrations::v8::migrate::<Runtime>()
+	}
+
+	#[cfg(feature = "try-runtime")]
+	fn pre_upgrade() -> Result<(), &'static str> {
+		pallet_staking::migrations::v8::pre_migrate::<Runtime>()
+	}
+
+	#[cfg(feature = "try-runtime")]
+	fn post_upgrade() -> Result<(), &'static str> {
+		pallet_staking::migrations::v8::post_migrate::<Runtime>()
+	}
+}
+
 #[cfg(not(feature = "disable-runtime-api"))]
 sp_api::impl_runtime_apis! {
 	impl sp_api::Core<Block> for Runtime {
@@ -1955,6 +2001,7 @@ sp_api::impl_runtime_apis! {
 			list_benchmark!(list, extra, runtime_common::slots, Slots);
 			list_benchmark!(list, extra, runtime_common::paras_registrar, Registrar);
 			// Substrate
+			list_benchmark!(list, extra, pallet_bags_list, BagsList);
 			list_benchmark!(list, extra, pallet_balances, Balances);
 			list_benchmark!(list, extra, pallet_bounties, Bounties);
 			list_benchmark!(list, extra, pallet_collective, Council);
@@ -2029,6 +2076,7 @@ sp_api::impl_runtime_apis! {
 			add_benchmark!(params, batches, runtime_common::paras_registrar, Registrar);
 			// Substrate
 			add_benchmark!(params, batches, pallet_balances, Balances);
+			add_benchmark!(params, batches, pallet_bags_list, BagsList);
 			add_benchmark!(params, batches, pallet_bounties, Bounties);
 			add_benchmark!(params, batches, pallet_collective, Council);
 			add_benchmark!(params, batches, pallet_collective, TechnicalCommittee);

polkadot/runtime/kusama/src/voter_bags.rs

@@ -0,0 +1,234 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Autogenerated voter bag thresholds.
+//!
+//! Generated on 2021-07-05T14:34:44.453491278+00:00
+//! for the kusama runtime.
+
+/// Existential weight for this runtime.
+#[cfg(any(test, feature = "std"))]
+#[allow(unused)]
+pub const EXISTENTIAL_WEIGHT: u64 = 33_333_333;
+
+/// Constant ratio between bags for this runtime.
+#[cfg(any(test, feature = "std"))]
+#[allow(unused)]
+pub const CONSTANT_RATIO: f64 = 1.1455399939091000;
+
+/// Upper thresholds delimiting the bag list.
+pub const THRESHOLDS: [u64; 200] = [
+	33_333_333,
+	38_184_666,
+	43_742_062,
+	50_108_281,
+	57_401_040,
+	65_755_187,
+	75_325_197,
+	86_288_026,
+	98_846_385,
+	113_232_487,
+	129_712_342,
+	148_590_675,
+	170_216_561,
+	194_989_878,
+	223_368_704,
+	255_877_784,
+	293_118_235,
+	335_778_661,
+	384_647_885,
+	440_629_536,
+	504_758_756,
+	578_221_342,
+	662_375_673,
+	758_777_824,
+	869_210_344,
+	995_715_212,
+	1_140_631_598,
+	1_306_639_114,
+	1_496_807_363,
+	1_714_652_697,
+	1_964_203_240,
+	2_250_073_368,
+	2_577_549_032,
+	2_952_685_502,
+	3_382_419_332,
+	3_874_696_621,
+	4_438_619_944,
+	5_084_616_664,
+	5_824_631_742,
+	6_672_348_610,
+	7_643_442_186,
+	8_755_868_715,
+	10_030_197_794,
+	11_489_992_720,
+	13_162_246_190,
+	15_077_879_420,
+	17_272_313_899,
+	19_786_126_359,
+	22_665_799_069,
+	25_964_579_327,
+	29_743_464_044,
+	34_072_327_620,
+	39_031_213_974,
+	44_711_816_618,
+	51_219_174_136,
+	58_673_612_428,
+	67_212_969_623,
+	76_995_144_813,
+	88_201_017_720,
+	101_037_793_302,
+	115_742_833_124,
+	132_588_044_352,
+	151_884_907_519,
+	173_990_236_034,
+	199_312_773_927,
+	228_320_753_830,
+	261_550_554_952,
+	299_616_621_127,
+	343_222_822_341,
+	393_175_469_814,
+	450_398_225_296,
+	515_949_180_262,
+	591_040_420_815,
+	677_060_440_060,
+	775_599_812_382,
+	888_480_604_352,
+	1_017_790_066_098,
+	1_165_919_226_119,
+	1_335_607_103_187,
+	1_529_991_352_850,
+	1_752_666_285_025,
+	2_007_749_325_472,
+	2_299_957_150_072,
+	2_634_692_899_685,
+	3_018_146_088_258,
+	3_457_407_051_560,
+	3_960_598_052_785,
+	4_537_023_469_264,
+	5_197_341_837_346,
+	5_953_762_936_697,
+	6_820_273_558_240,
+	7_812_896_130_365,
+	8_949_984_985_591,
+	10_252_565_745_880,
+	11_744_724_102_088,
+	13_454_051_176_370,
+	15_412_153_702_632,
+	17_655_238_458_639,
+	20_224_781_756_373,
+	23_168_296_370_008,
+	26_540_210_082_583,
+	30_402_872_096_348,
+	34_827_705_916_070,
+	39_896_530_022_963,
+	45_703_070_759_499,
+	52_354_695_399_464,
+	59_974_397_449_015,
+	68_703_070_888_447,
+	78_702_115_407_088,
+	90_156_420_804_069,
+	103_277_785_738_759,
+	118_308_834_046_123,
+	135_527_501_032_588,
+	155_252_172_707_386,
+	177_847_572_977_594,
+	203_731_507_665_501,
+	233_382_590_050_230,
+	267_349_090_784_630,
+	306_259_075_829_029,
+	350_832_019_859_793,
+	401_892_109_893_305,
+	460_383_485_119_292,
+	527_387_694_739_404,
+	604_143_696_619_511,
+	692_070_766_545_736,
+	792_794_741_693_469,
+	908_178_083_570_703,
+	1_040_354_316_321_961,
+	1_191_767_477_182_765,
+	1_365_217_308_553_008,
+	1_563_911_027_324_411,
+	1_791_522_628_715_580,
+	2_052_260_821_186_860,
+	2_350_946_848_602_280,
+	2_693_103_638_628_474,
+	3_085_057_925_791_037,
+	3_534_057_237_519_885,
+	4_048_403_906_342_940,
+	4_637_608_586_213_668,
+	5_312_566_111_603_995,
+	6_085_756_951_128_531,
+	6_971_477_980_728_040,
+	7_986_106_843_580_624,
+	9_148_404_784_952_770,
+	10_479_863_561_632_778,
+	12_005_102_840_561_012,
+	13_752_325_434_854_380,
+	15_753_838_794_879_048,
+	18_046_652_397_130_688,
+	20_673_162_077_088_732,
+	23_681_933_959_870_064,
+	27_128_602_484_145_260,
+	31_076_899_124_450_156,
+	35_599_830_833_736_348,
+	40_781_029_996_443_328,
+	46_716_300_853_732_512,
+	53_515_390_995_440_424,
+	61_304_020_674_959_928,
+	70_226_207_470_596_936,
+	80_446_929_278_126_800,
+	92_155_174_875_271_168,
+	105_567_438_465_310_176,
+	120_931_722_816_550_704,
+	138_532_125_018_688_464,
+	158_694_089_650_123_072,
+	181_790_426_491_212_160,
+	208_248_204_055_475_872,
+	238_556_646_405_290_848,
+	273_276_179_270_092_192,
+	313_048_792_736_563_520,
+	358_609_912_124_694_080,
+	410_801_996_551_064_960,
+	470_590_116_626_953_088,
+	539_079_799_334_522_496,
+	617_537_470_046_187_776,
+	707_413_869_675_350_912,
+	810_370_879_959_114_368,
+	928_312_252_892_475_904,
+	1_063_418_812_524_189_696,
+	1_218_188_780_021_782_528,
+	1_395_483_967_646_286_592,
+	1_598_582_695_797_773_824,
+	1_831_240_411_607_374_592,
+	2_097_759_129_958_809_600,
+	2_403_066_980_955_773_440,
+	2_752_809_334_727_236_096,
+	3_153_453_188_536_351_744,
+	3_612_406_746_388_564_480,
+	4_138_156_402_255_148_032,
+	4_740_423_659_834_265_600,
+	5_430_344_890_413_097_984,
+	6_220_677_252_688_132_096,
+	7_126_034_582_154_840_064,
+	8_163_157_611_837_691_904,
+	9_351_223_520_943_572_992,
+	10_712_200_535_224_332_288,
+	12_271_254_135_873_939_456,
+	14_057_212_388_066_050_048,
+	16_103_098_993_404_108_800,
+	18_446_744_073_709_551_615,
+];

polkadot/runtime/kusama/src/weights/mod.rs

@@ -16,6 +16,7 @@
 //! A list of the different weight modules for our runtime.
 
 pub mod frame_system;
+pub mod pallet_bags_list;
 pub mod pallet_balances;
 pub mod pallet_bounties;
 pub mod pallet_collective_council;

polkadot/runtime/kusama/src/weights/pallet_bags_list.rs

@@ -0,0 +1,66 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2021 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Autogenerated weights for pallet_bags_list
+//!
+//! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 4.0.0-dev
+//! DATE: 2021-09-02, STEPS: `50`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("dev"), DB CACHE: 128
+
+// Executed Command:
+// target/release/substrate
+// benchmark
+// --chain=dev
+// --steps=50
+// --repeat=20
+// --pallet=pallet_bags_list
+// --extrinsic=*
+// --execution=wasm
+// --wasm-execution=compiled
+// --heap-pages=4096
+// --output=./frame/bags-list/src/weights.rs
+// --template=./.maintain/frame-weight-template.hbs
+
+// NOTE: this is copy pasta from substrate % some tweaks
+
+#![cfg_attr(rustfmt, rustfmt_skip)]
+#![allow(unused_parens)]
+#![allow(unused_imports)]
+
+use frame_support::{traits::Get, weights::{Weight, constants::RocksDbWeight}};
+use sp_std::marker::PhantomData;
+
+/// Weight functions needed for pallet_bags_list.
+
+/// Weights for pallet_bags_list using the Substrate node and recommended hardware.
+pub struct WeightInfo<T>(PhantomData<T>);
+impl<T: frame_system::Config> pallet_bags_list::WeightInfo for WeightInfo<T> {
+	fn rebag_non_terminal() -> Weight {
+		(75_718_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().writes(5 as Weight))
+	}
+	// Storage: Staking Bonded (r:1 w:0)
+	// Storage: Staking Ledger (r:1 w:0)
+	// Storage: BagsList ListNodes (r:4 w:4)
+	// Storage: BagsList ListBags (r:1 w:1)
+	fn rebag_terminal() -> Weight {
+		(75_718_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().writes(5 as Weight))
+	}
+}

polkadot/runtime/kusama/src/weights/pallet_staking.rs

@@ -16,7 +16,7 @@
 //! Autogenerated weights for `pallet_staking`
 //!
 //! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 3.0.0
-//! DATE: 2021-07-01, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! DATE: 2021-07-07, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
 //! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("kusama-dev"), DB CACHE: 128
 
 // Executed Command:
@@ -31,7 +31,7 @@
 // --wasm-execution=compiled
 // --heap-pages=4096
 // --header=./file_header.txt
-// --output=./runtime/kusama/src/weights/
+// --output=runtime/kusama/src/weights/
 
 #![allow(unused_parens)]
 #![allow(unused_imports)]
@@ -43,17 +43,17 @@ use sp_std::marker::PhantomData;
 pub struct WeightInfo<T>(PhantomData<T>);
 impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn bond() -> Weight {
-		(70_648_000 as Weight)
+		(68_673_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(5 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 	}
 	fn bond_extra() -> Weight {
-		(54_235_000 as Weight)
+		(57_515_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(3 as Weight))
+			.saturating_add(T::DbWeight::get().reads(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes(2 as Weight))
 	}
 	fn unbond() -> Weight {
-		(57_950_000 as Weight)
+		(56_018_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(6 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
@@ -73,9 +73,9 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn validate() -> Weight {
-		(31_838_000 as Weight)
+		(64_915_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(10 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 	fn kick(k: u32) -> Weight {
 		(10_120_000 as Weight)
@@ -91,18 +91,18 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add((5_240_000 as Weight).saturating_mul(n as Weight))
 			.saturating_add(T::DbWeight::get().reads(7 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(n as Weight)))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 	}
 	fn chill() -> Weight {
 		(16_594_000 as Weight).saturating_add(T::DbWeight::get().reads(3 as Weight))
 	}
 	fn set_payee() -> Weight {
-		(11_260_000 as Weight)
+		(10_848_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn set_controller() -> Weight {
-		(24_156_000 as Weight)
+		(23_658_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
@@ -121,7 +121,7 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn set_invulnerables(v: u32) -> Weight {
 		(2_524_000 as Weight)
 			// Standard Error: 0
-			.saturating_add((23_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((5_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn force_unstake(s: u32) -> Weight {
@@ -142,7 +142,7 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn payout_stakers_dead_controller(n: u32) -> Weight {
 		(108_055_000 as Weight)
 			// Standard Error: 15_000
-			.saturating_add((47_913_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((49_008_000 as Weight).saturating_mul(n as Weight))
 			.saturating_add(T::DbWeight::get().reads(10 as Weight))
 			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().writes(2 as Weight))
@@ -160,14 +160,14 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn rebond(l: u32) -> Weight {
 		(46_725_000 as Weight)
 			// Standard Error: 1_000
-			.saturating_add((62_000 as Weight).saturating_mul(l as Weight))
+			.saturating_add((61_000 as Weight).saturating_mul(l as Weight))
 			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
 	fn set_history_depth(e: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 68_000
+			// Standard Error: 70_000
-			.saturating_add((32_190_000 as Weight).saturating_mul(e as Weight))
+			.saturating_add((33_332_000 as Weight).saturating_mul(e as Weight))
 			.saturating_add(T::DbWeight::get().reads(2 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((7 as Weight).saturating_mul(e as Weight)))
@@ -182,13 +182,13 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	}
 	fn new_era(v: u32, n: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 734_000
+			// Standard Error: 765_000
-			.saturating_add((296_342_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((309_896_000 as Weight).saturating_mul(v as Weight))
-			// Standard Error: 36_000
+			// Standard Error: 38_000
-			.saturating_add((46_481_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((53_011_000 as Weight).saturating_mul(n as Weight))
-			.saturating_add(T::DbWeight::get().reads(10 as Weight))
+			.saturating_add(T::DbWeight::get().reads(192 as Weight))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((3 as Weight).saturating_mul(v as Weight)))
 	}
@@ -202,13 +202,13 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add((40_493_000 as Weight).saturating_mul(s as Weight))
 			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn get_npos_targets(v: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 27_000
+			// Standard Error: 50_000
-			.saturating_add((10_233_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((10_984_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(v as Weight)))
 	}
@@ -216,8 +216,8 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 		(5_708_000 as Weight).saturating_add(T::DbWeight::get().writes(5 as Weight))
 	}
 	fn chill_other() -> Weight {
-		(39_559_000 as Weight)
+		(84_065_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().reads(11 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 }

polkadot/runtime/polkadot/src/lib.rs

@@ -360,6 +360,11 @@ parameter_types! {
 
 	// miner configs
 	pub OffchainRepeat: BlockNumber = 5;
+
+	/// Whilst `UseNominatorsAndUpdateBagsList` or `UseNominatorsMap` is in use, this can still be a
+	/// very large value. Once the `BagsList` is in full motion, staking might open its door to many
+	/// more nominators, and this value should instead be what is a "safe" number (e.g. 22500).
+	pub const VoterSnapshotPerBlock: u32 = 22_500;
 }
 
 sp_npos_elections::generate_solution_type!(
@@ -405,6 +410,7 @@ impl pallet_election_provider_multi_phase::Config for Runtime {
 		pallet_collective::EnsureProportionAtLeast<_2, _3, AccountId, CouncilCollective>,
 	>;
 	type WeightInfo = weights::pallet_election_provider_multi_phase::WeightInfo<Self>;
+	type VoterSnapshotPerBlock = VoterSnapshotPerBlock;
 }
 
 // TODO #6469: This shouldn't be static, but a lazily cached value, not built unless needed, and
@@ -465,6 +471,9 @@ impl pallet_staking::Config for Runtime {
 	type NextNewSession = Session;
 	type ElectionProvider = ElectionProviderMultiPhase;
 	type GenesisElectionProvider = runtime_common::elections::GenesisElectionOf<Self>;
+	// Use the nominator map to iter voter AND no-ops for all SortedListProvider hooks. The migration
+	// to bags-list is a no-op, but the storage version will be updated.
+	type SortedListProvider = pallet_staking::UseNominatorsMap<Runtime>;
 	type WeightInfo = weights::pallet_staking::WeightInfo<Runtime>;
 }
 
@@ -1107,7 +1116,6 @@ construct_runtime! {
 
 		// Election pallet. Only works with staking, but placed here to maintain indices.
 		ElectionProviderMultiPhase: pallet_election_provider_multi_phase::{Pallet, Call, Storage, Event<T>, ValidateUnsigned} = 36,
-
 	}
 }
 

polkadot/runtime/polkadot/src/weights/pallet_staking.rs

@@ -16,7 +16,7 @@
 //! Autogenerated weights for `pallet_staking`
 //!
 //! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 3.0.0
-//! DATE: 2021-07-01, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! DATE: 2021-07-07, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
 //! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("polkadot-dev"), DB CACHE: 128
 
 // Executed Command:
@@ -31,7 +31,7 @@
 // --wasm-execution=compiled
 // --heap-pages=4096
 // --header=./file_header.txt
-// --output=./runtime/polkadot/src/weights/
+// --output=runtime/polkadot/src/weights/
 
 #![allow(unused_parens)]
 #![allow(unused_imports)]
@@ -43,39 +43,39 @@ use sp_std::marker::PhantomData;
 pub struct WeightInfo<T>(PhantomData<T>);
 impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn bond() -> Weight {
-		(70_259_000 as Weight)
+		(70_233_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(5 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 	}
 	fn bond_extra() -> Weight {
-		(53_621_000 as Weight)
+		(58_956_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(3 as Weight))
+			.saturating_add(T::DbWeight::get().reads(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes(2 as Weight))
 	}
 	fn unbond() -> Weight {
-		(57_038_000 as Weight)
+		(56_669_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(6 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
 	fn withdraw_unbonded_update(s: u32) -> Weight {
 		(49_182_000 as Weight)
 			// Standard Error: 0
-			.saturating_add((31_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((28_000 as Weight).saturating_mul(s as Weight))
 			.saturating_add(T::DbWeight::get().reads(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
 	fn withdraw_unbonded_kill(s: u32) -> Weight {
 		(81_006_000 as Weight)
 			// Standard Error: 1_000
-			.saturating_add((2_333_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((2_258_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(8 as Weight))
+			.saturating_add(T::DbWeight::get().reads(10 as Weight))
-			.saturating_add(T::DbWeight::get().writes(6 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn validate() -> Weight {
-		(31_525_000 as Weight)
+		(65_617_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(10 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 	fn kick(k: u32) -> Weight {
 		(10_487_000 as Weight)
@@ -91,18 +91,18 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add((5_185_000 as Weight).saturating_mul(n as Weight))
 			.saturating_add(T::DbWeight::get().reads(7 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(n as Weight)))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 	}
 	fn chill() -> Weight {
 		(16_783_000 as Weight).saturating_add(T::DbWeight::get().reads(3 as Weight))
 	}
 	fn set_payee() -> Weight {
-		(11_391_000 as Weight)
+		(11_087_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn set_controller() -> Weight {
-		(24_470_000 as Weight)
+		(24_640_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
@@ -121,15 +121,15 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn set_invulnerables(v: u32) -> Weight {
 		(2_143_000 as Weight)
 			// Standard Error: 0
-			.saturating_add((23_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((6_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn force_unstake(s: u32) -> Weight {
 		(58_264_000 as Weight)
 			// Standard Error: 1_000
-			.saturating_add((2_309_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((2_252_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(8 as Weight))
-			.saturating_add(T::DbWeight::get().writes(6 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn cancel_deferred_slash(s: u32) -> Weight {
@@ -166,8 +166,8 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	}
 	fn set_history_depth(e: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 67_000
+			// Standard Error: 75_000
-			.saturating_add((32_486_000 as Weight).saturating_mul(e as Weight))
+			.saturating_add((32_680_000 as Weight).saturating_mul(e as Weight))
 			.saturating_add(T::DbWeight::get().reads(2 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((7 as Weight).saturating_mul(e as Weight)))
@@ -182,33 +182,31 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	}
 	fn new_era(v: u32, n: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 666_000
+			// Standard Error: 1_020_000
-			.saturating_add((306_698_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((329_151_000 as Weight).saturating_mul(v as Weight))
-			// Standard Error: 33_000
+			// Standard Error: 51_000
-			.saturating_add((47_483_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((53_726_000 as Weight).saturating_mul(n as Weight))
-			.saturating_add(T::DbWeight::get().reads(10 as Weight))
+			.saturating_add(T::DbWeight::get().reads(187 as Weight))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((3 as Weight).saturating_mul(v as Weight)))
 	}
 	fn get_npos_voters(v: u32, n: u32, s: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 97_000
+			// Standard Error: 108_000
-			.saturating_add((25_109_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((24_576_000 as Weight).saturating_mul(v as Weight))
-			// Standard Error: 97_000
+			// Standard Error: 108_000
-			.saturating_add((27_162_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((33_364_000 as Weight).saturating_mul(n as Weight))
-			// Standard Error: 3_332_000
+			.saturating_add(T::DbWeight::get().reads(179 as Weight))
-			.saturating_add((50_488_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn get_npos_targets(v: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 31_000
+			// Standard Error: 26_000
-			.saturating_add((10_220_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((10_139_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(v as Weight)))
 	}
@@ -216,8 +214,8 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 		(5_584_000 as Weight).saturating_add(T::DbWeight::get().writes(5 as Weight))
 	}
 	fn chill_other() -> Weight {
-		(39_524_000 as Weight)
+		(80_902_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().reads(11 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 }

polkadot/runtime/test-runtime/src/lib.rs

@@ -340,6 +340,9 @@ impl pallet_staking::Config for Runtime {
 		frame_election_provider_support::onchain::OnChainSequentialPhragmen<Self>;
 	type GenesisElectionProvider =
 		frame_election_provider_support::onchain::OnChainSequentialPhragmen<Self>;
+	// Use the nominator map to iter voter AND no-ops for all SortedListProvider hooks. The migration
+	// to bags-list is a no-op, but the storage version will be updated.
+	type SortedListProvider = pallet_staking::UseNominatorsMap<Runtime>;
 	type WeightInfo = ();
 }
 

polkadot/runtime/westend/Cargo.toml

@@ -69,6 +69,7 @@ pallet-treasury = { git = "https://github.com/paritytech/substrate", branch = "m
 pallet-utility = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-vesting = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-xcm = { path = "../../xcm/pallet-xcm", default-features = false }
+pallet-bags-list = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 
 frame-benchmarking = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false, optional = true }
 frame-try-runtime = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false, optional = true }
@@ -149,6 +150,7 @@ std = [
 	"pallet-vesting/std",
 	"pallet-xcm/std",
 	"pallet-babe/std",
+	"pallet-bags-list/std",
 	"frame-executive/std",
 	"sp-runtime/std",
 	"sp-staking/std",
@@ -201,6 +203,7 @@ runtime-benchmarks = [
 	"hex-literal",
 	"xcm-builder/runtime-benchmarks",
 	"frame-election-provider-support/runtime-benchmarks",
+	"pallet-bags-list/runtime-benchmarks",
 ]
 try-runtime = [
 	"frame-executive/try-runtime",

polkadot/runtime/westend/src/lib.rs

@@ -20,6 +20,7 @@
 // `construct_runtime!` does a lot of recursion and requires us to increase the limit to 256.
 #![recursion_limit = "256"]
 
+use frame_support::traits::OnRuntimeUpgrade;
 use pallet_transaction_payment::CurrencyAdapter;
 use parity_scale_codec::{Decode, Encode, MaxEncodedLen};
 use primitives::v1::{
@@ -100,6 +101,9 @@ use constants::{currency::*, fee::*, time::*};
 // Weights used in the runtime
 mod weights;
 
+// Voter bag threshold definitions.
+mod voter_bags;
+
 #[cfg(test)]
 mod tests;
 
@@ -339,6 +343,11 @@ parameter_types! {
 
 	// miner configs
 	pub OffchainRepeat: BlockNumber = 5;
+
+	/// Whilst `UseNominatorsAndUpdateBagsList` or `UseNominatorsMap` is in use, this can still be a
+	/// very large value. Once the `BagsList` is in full motion, staking might open its door to many
+	/// more nominators, and this value should instead be what is a "safe" number (e.g. 22500).
+	pub const VoterSnapshotPerBlock: u32 = 22_500;
 }
 
 sp_npos_elections::generate_solution_type!(
@@ -380,6 +389,18 @@ impl pallet_election_provider_multi_phase::Config for Runtime {
 	type BenchmarkingConfig = runtime_common::elections::BenchmarkConfig;
 	type ForceOrigin = EnsureRoot<AccountId>;
 	type WeightInfo = weights::pallet_election_provider_multi_phase::WeightInfo<Self>;
+	type VoterSnapshotPerBlock = VoterSnapshotPerBlock;
+}
+
+parameter_types! {
+	pub const BagThresholds: &'static [u64] = &voter_bags::THRESHOLDS;
+}
+
+impl pallet_bags_list::Config for Runtime {
+	type Event = Event;
+	type VoteWeightProvider = Staking;
+	type WeightInfo = weights::pallet_bags_list::WeightInfo<Runtime>;
+	type BagThresholds = BagThresholds;
 }
 
 pallet_staking_reward_curve::build! {
@@ -430,6 +451,9 @@ impl pallet_staking::Config for Runtime {
 	type NextNewSession = Session;
 	type ElectionProvider = ElectionProviderMultiPhase;
 	type GenesisElectionProvider = runtime_common::elections::GenesisElectionOf<Self>;
+	// Use the nominators map to iter voters, but also perform the bags-list migration and keep
+	// it up-to-date.
+	type SortedListProvider = runtime_common::elections::UseNominatorsAndUpdateBagsList<Runtime>;
 	type WeightInfo = weights::pallet_staking::WeightInfo<Runtime>;
 }
 
@@ -1041,6 +1065,9 @@ construct_runtime! {
 		// Election pallet. Only works with staking, but placed here to maintain indices.
 		ElectionProviderMultiPhase: pallet_election_provider_multi_phase::{Pallet, Call, Storage, Event<T>, ValidateUnsigned} = 24,
 
+		// Provides a semi-sorted list of nominators for staking.
+		BagsList: pallet_bags_list::{Pallet, Call, Storage, Event<T>} = 25,
+
 		// Parachains pallets. Start indices at 40 to leave room.
 		ParachainsOrigin: parachains_origin::{Pallet, Origin} = 41,
 		Configuration: parachains_configuration::{Pallet, Call, Storage, Config<T>} = 42,
@@ -1096,11 +1123,30 @@ pub type Executive = frame_executive::Executive<
 	frame_system::ChainContext<Runtime>,
 	Runtime,
 	AllPallets,
-	(),
+	(StakingBagsListMigrationV8,),
 >;
 /// The payload being signed in transactions.
 pub type SignedPayload = generic::SignedPayload<Call, SignedExtra>;
 
+// Migration to generate pallet staking's `SortedListProvider` from pre-existing nominators.
+pub struct StakingBagsListMigrationV8;
+
+impl OnRuntimeUpgrade for StakingBagsListMigrationV8 {
+	fn on_runtime_upgrade() -> frame_support::weights::Weight {
+		pallet_staking::migrations::v8::migrate::<Runtime>()
+	}
+
+	#[cfg(feature = "try-runtime")]
+	fn pre_upgrade() -> Result<(), &'static str> {
+		pallet_staking::migrations::v8::pre_migrate::<Runtime>()
+	}
+
+	#[cfg(feature = "try-runtime")]
+	fn post_upgrade() -> Result<(), &'static str> {
+		pallet_staking::migrations::v8::post_migrate::<Runtime>()
+	}
+}
+
 #[cfg(not(feature = "disable-runtime-api"))]
 sp_api::impl_runtime_apis! {
 	impl sp_api::Core<Block> for Runtime {
@@ -1417,6 +1463,7 @@ sp_api::impl_runtime_apis! {
 			list_benchmark!(list, extra, runtime_common::paras_registrar, Registrar);
 			list_benchmark!(list, extra, runtime_common::slots, Slots);
 			// Substrate
+			list_benchmark!(list, extra, pallet_bags_list, BagsList);
 			list_benchmark!(list, extra, pallet_balances, Balances);
 			list_benchmark!(list, extra, pallet_election_provider_multi_phase, ElectionProviderMultiPhase);
 			list_benchmark!(list, extra, pallet_identity, Identity);
@@ -1481,6 +1528,7 @@ sp_api::impl_runtime_apis! {
 			add_benchmark!(params, batches, runtime_common::paras_registrar, Registrar);
 			add_benchmark!(params, batches, runtime_common::slots, Slots);
 			// Substrate
+			add_benchmark!(params, batches, pallet_bags_list, BagsList);
 			add_benchmark!(params, batches, pallet_balances, Balances);
 			add_benchmark!(params, batches, pallet_election_provider_multi_phase, ElectionProviderMultiPhase);
 			add_benchmark!(params, batches, pallet_identity, Identity);

polkadot/runtime/westend/src/voter_bags.rs

@@ -0,0 +1,234 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Autogenerated voter bag thresholds.
+//!
+//! Generated on 2021-07-05T14:35:50.538338181+00:00
+//! for the westend runtime.
+
+/// Existential weight for this runtime.
+#[cfg(any(test, feature = "std"))]
+#[allow(unused)]
+pub const EXISTENTIAL_WEIGHT: u64 = 10_000_000_000;
+
+/// Constant ratio between bags for this runtime.
+#[cfg(any(test, feature = "std"))]
+#[allow(unused)]
+pub const CONSTANT_RATIO: f64 = 1.1131723507077667;
+
+/// Upper thresholds delimiting the bag list.
+pub const THRESHOLDS: [u64; 200] = [
+	10_000_000_000,
+	11_131_723_507,
+	12_391_526_824,
+	13_793_905_044,
+	15_354_993_703,
+	17_092_754_435,
+	19_027_181_634,
+	21_180_532_507,
+	23_577_583_160,
+	26_245_913_670,
+	29_216_225_417,
+	32_522_694_326,
+	36_203_364_094,
+	40_300_583_912,
+	44_861_495_728,
+	49_938_576_656,
+	55_590_242_767,
+	61_881_521_217,
+	68_884_798_439,
+	76_680_653_006,
+	85_358_782_760,
+	95_019_036_859,
+	105_772_564_622,
+	117_743_094_401,
+	131_068_357_174,
+	145_901_671_259,
+	162_413_706_368,
+	180_794_447_305,
+	201_255_379_901,
+	224_031_924_337,
+	249_386_143_848,
+	277_609_759_981,
+	309_027_509_097,
+	344_000_878_735,
+	382_932_266_827,
+	426_269_611_626,
+	474_511_545_609,
+	528_213_132_664,
+	587_992_254_562,
+	654_536_720_209,
+	728_612_179_460,
+	811_070_932_564,
+	902_861_736_593,
+	1_005_040_721_687,
+	1_118_783_542_717,
+	1_245_398_906_179,
+	1_386_343_627_960,
+	1_543_239_395_225,
+	1_717_891_425_287,
+	1_912_309_236_147,
+	2_128_729_767_682,
+	2_369_643_119_512,
+	2_637_821_201_686,
+	2_936_349_627_828,
+	3_268_663_217_709,
+	3_638_585_517_729,
+	4_050_372_794_022,
+	4_508_763_004_364,
+	5_019_030_312_352,
+	5_587_045_771_074,
+	6_219_344_874_498,
+	6_923_202_753_807,
+	7_706_717_883_882,
+	8_578_905_263_043,
+	9_549_800_138_161,
+	10_630_573_468_586,
+	11_833_660_457_397,
+	13_172_903_628_838,
+	14_663_712_098_160,
+	16_323_238_866_411,
+	18_170_578_180_087,
+	20_226_985_226_447,
+	22_516_120_692_255,
+	25_064_322_999_817,
+	27_900_911_352_605,
+	31_058_523_077_268,
+	34_573_489_143_434,
+	38_486_252_181_966,
+	42_841_831_811_331,
+	47_690_342_626_046,
+	53_087_570_807_094,
+	59_095_615_988_698,
+	65_783_605_766_662,
+	73_228_491_069_308,
+	81_515_931_542_404,
+	90_741_281_135_191,
+	101_010_685_227_495,
+	112_442_301_921_293,
+	125_167_661_548_718,
+	139_333_180_038_781,
+	155_101_843_555_358,
+	172_655_083_789_626,
+	192_194_865_483_744,
+	213_946_010_204_502,
+	238_158_783_103_893,
+	265_111_772_429_462,
+	295_115_094_915_607,
+	328_513_963_936_552,
+	365_692_661_475_578,
+	407_078_959_611_349,
+	453_149_042_394_237,
+	504_432_984_742_966,
+	561_520_851_400_862,
+	625_069_486_125_324,
+	695_810_069_225_823,
+	774_556_530_406_243,
+	862_214_913_708_369,
+	959_793_802_308_039,
+	1_068_415_923_109_985,
+	1_189_331_064_661_951,
+	1_323_930_457_019_515,
+	1_473_762_779_014_021,
+	1_640_551_977_100_649,
+	1_826_217_100_807_404,
+	2_032_894_383_008_501,
+	2_262_961_819_074_188,
+	2_519_066_527_700_738,
+	2_804_155_208_229_882,
+	3_121_508_044_894_685,
+	3_474_776_448_088_622,
+	3_868_025_066_902_796,
+	4_305_778_556_320_752,
+	4_793_073_637_166_665,
+	5_335_517_047_800_242,
+	5_939_350_054_341_159,
+	6_611_520_261_667_250,
+	7_359_761_551_432_161,
+	8_192_683_066_856_378,
+	9_119_868_268_136_230,
+	10_151_985_198_186_376,
+	11_300_909_227_415_580,
+	12_579_859_689_817_292,
+	14_003_551_982_487_792,
+	15_588_366_878_604_342,
+	17_352_539_001_951_086,
+	19_316_366_631_550_092,
+	21_502_445_250_375_680,
+	23_935_927_525_325_748,
+	26_644_812_709_737_600,
+	29_660_268_798_266_784,
+	33_016_991_140_790_860,
+	36_753_601_641_491_664,
+	40_913_093_136_236_104,
+	45_543_324_061_189_736,
+	50_697_569_104_240_168,
+	56_435_132_174_936_472,
+	62_822_028_745_677_552,
+	69_931_745_415_056_768,
+	77_846_085_432_775_824,
+	86_656_109_914_600_688,
+	96_463_185_576_826_656,
+	107_380_151_045_315_664,
+	119_532_615_158_469_088,
+	133_060_402_202_199_856,
+	148_119_160_705_543_712,
+	164_882_154_307_451_552,
+	183_542_255_300_186_560,
+	204_314_163_786_713_728,
+	227_436_877_985_347_776,
+	253_176_444_104_585_088,
+	281_829_017_427_734_464,
+	313_724_269_827_691_328,
+	349_229_182_918_168_832,
+	388_752_270_484_770_624,
+	432_748_278_778_513_664,
+	481_723_418_752_617_984,
+	536_241_190_443_833_600,
+	596_928_866_512_693_376,
+	664_484_709_541_257_600,
+	739_686_006_129_409_280,
+	823_398_010_228_713_984,
+	916_583_898_614_395_264,
+	1_020_315_853_041_475_584,
+	1_135_787_396_594_579_584,
+	1_264_327_126_171_442_688,
+	1_407_413_999_103_859_968,
+	1_566_694_349_801_462_272,
+	1_744_000_832_209_069_824,
+	1_941_373_506_026_471_680,
+	2_161_083_309_305_266_176,
+	2_405_658_187_494_662_656,
+	2_677_912_179_572_818_944,
+	2_980_977_795_924_034_048,
+	3_318_342_060_496_414_208,
+	3_693_886_631_935_247_360,
+	4_111_932_465_319_354_368,
+	4_577_289_528_371_127_808,
+	5_095_312_144_166_932_480,
+	5_671_960_597_112_134_656,
+	6_313_869_711_009_142_784,
+	7_028_425_188_266_614_784,
+	7_823_848_588_596_424_704,
+	8_709_291_924_949_524_480,
+	9_694_942_965_096_232_960,
+	10_792_142_450_433_898_496,
+	12_013_514_580_722_579_456,
+	13_373_112_266_084_982_784,
+	14_886_578_817_516_689_408,
+	16_571_327_936_291_497_984,
+	18_446_744_073_709_551_615,
+];

polkadot/runtime/westend/src/weights/mod.rs

@@ -16,6 +16,7 @@
 //! A list of the different weight modules for our runtime.
 
 pub mod frame_system;
+pub mod pallet_bags_list;
 pub mod pallet_balances;
 pub mod pallet_election_provider_multi_phase;
 pub mod pallet_identity;

polkadot/runtime/westend/src/weights/pallet_bags_list.rs

@@ -0,0 +1,66 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2021 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Autogenerated weights for pallet_bags_list
+//!
+//! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 4.0.0-dev
+//! DATE: 2021-09-02, STEPS: `50`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("dev"), DB CACHE: 128
+
+// Executed Command:
+// target/release/substrate
+// benchmark
+// --chain=dev
+// --steps=50
+// --repeat=20
+// --pallet=pallet_bags_list
+// --extrinsic=*
+// --execution=wasm
+// --wasm-execution=compiled
+// --heap-pages=4096
+// --output=./frame/bags-list/src/weights.rs
+// --template=./.maintain/frame-weight-template.hbs
+
+// NOTE: this is copy pasta from substrate % some tweaks
+
+#![cfg_attr(rustfmt, rustfmt_skip)]
+#![allow(unused_parens)]
+#![allow(unused_imports)]
+
+use frame_support::{traits::Get, weights::{Weight, constants::RocksDbWeight}};
+use sp_std::marker::PhantomData;
+
+/// Weight functions needed for pallet_bags_list.
+
+/// Weights for pallet_bags_list using the Substrate node and recommended hardware.
+pub struct WeightInfo<T>(PhantomData<T>);
+impl<T: frame_system::Config> pallet_bags_list::WeightInfo for WeightInfo<T> {
+	fn rebag_non_terminal() -> Weight {
+		(75_718_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().writes(5 as Weight))
+	}
+	// Storage: Staking Bonded (r:1 w:0)
+	// Storage: Staking Ledger (r:1 w:0)
+	// Storage: BagsList ListNodes (r:4 w:4)
+	// Storage: BagsList ListBags (r:1 w:1)
+	fn rebag_terminal() -> Weight {
+		(75_718_000 as Weight)
+			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().writes(5 as Weight))
+	}
+}

polkadot/runtime/westend/src/weights/pallet_staking.rs

@@ -16,7 +16,7 @@
 //! Autogenerated weights for `pallet_staking`
 //!
 //! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 3.0.0
-//! DATE: 2021-07-02, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! DATE: 2021-07-07, STEPS: `[50, ]`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
 //! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("westend-dev"), DB CACHE: 128
 
 // Executed Command:
@@ -31,7 +31,7 @@
 // --wasm-execution=compiled
 // --heap-pages=4096
 // --header=./file_header.txt
-// --output=./runtime/westend/src/weights/
+// --output=runtime/westend/src/weights/
 
 #![allow(unused_parens)]
 #![allow(unused_imports)]
@@ -43,24 +43,24 @@ use sp_std::marker::PhantomData;
 pub struct WeightInfo<T>(PhantomData<T>);
 impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn bond() -> Weight {
-		(71_835_000 as Weight)
+		(70_644_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(5 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 	}
 	fn bond_extra() -> Weight {
-		(54_894_000 as Weight)
+		(57_596_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(3 as Weight))
+			.saturating_add(T::DbWeight::get().reads(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes(2 as Weight))
 	}
 	fn unbond() -> Weight {
-		(59_419_000 as Weight)
+		(57_410_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(6 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
 	fn withdraw_unbonded_update(s: u32) -> Weight {
 		(50_384_000 as Weight)
 			// Standard Error: 0
-			.saturating_add((26_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((33_000 as Weight).saturating_mul(s as Weight))
 			.saturating_add(T::DbWeight::get().reads(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
@@ -73,9 +73,9 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn validate() -> Weight {
-		(33_672_000 as Weight)
+		(69_191_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(10 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 	fn kick(k: u32) -> Weight {
 		(11_371_000 as Weight)
@@ -91,18 +91,18 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 			.saturating_add((5_341_000 as Weight).saturating_mul(n as Weight))
 			.saturating_add(T::DbWeight::get().reads(7 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(n as Weight)))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 	}
 	fn chill() -> Weight {
 		(17_740_000 as Weight).saturating_add(T::DbWeight::get().reads(3 as Weight))
 	}
 	fn set_payee() -> Weight {
-		(11_624_000 as Weight)
+		(11_867_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn set_controller() -> Weight {
-		(25_480_000 as Weight)
+		(26_102_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
@@ -121,15 +121,15 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn set_invulnerables(v: u32) -> Weight {
 		(2_352_000 as Weight)
 			// Standard Error: 0
-			.saturating_add((27_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((5_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	fn force_unstake(s: u32) -> Weight {
 		(58_375_000 as Weight)
 			// Standard Error: 1_000
-			.saturating_add((2_282_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((2_220_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(8 as Weight))
-			.saturating_add(T::DbWeight::get().writes(6 as Weight))
+			.saturating_add(T::DbWeight::get().writes(7 as Weight))
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn cancel_deferred_slash(s: u32) -> Weight {
@@ -166,8 +166,8 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	}
 	fn set_history_depth(e: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 63_000
+			// Standard Error: 76_000
-			.saturating_add((33_032_000 as Weight).saturating_mul(e as Weight))
+			.saturating_add((33_612_000 as Weight).saturating_mul(e as Weight))
 			.saturating_add(T::DbWeight::get().reads(2 as Weight))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((7 as Weight).saturating_mul(e as Weight)))
@@ -175,40 +175,38 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 	fn reap_stash(s: u32) -> Weight {
 		(69_690_000 as Weight)
 			// Standard Error: 1_000
-			.saturating_add((2_271_000 as Weight).saturating_mul(s as Weight))
+			.saturating_add((2_214_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().reads(11 as Weight))
-			.saturating_add(T::DbWeight::get().writes(8 as Weight))
+			.saturating_add(T::DbWeight::get().writes(12 as Weight))
 			.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn new_era(v: u32, n: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 787_000
+			// Standard Error: 1_065_000
-			.saturating_add((310_720_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((309_528_000 as Weight).saturating_mul(v as Weight))
-			// Standard Error: 39_000
+			// Standard Error: 53_000
-			.saturating_add((48_687_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((52_832_000 as Weight).saturating_mul(n as Weight))
-			.saturating_add(T::DbWeight::get().reads(10 as Weight))
+			.saturating_add(T::DbWeight::get().reads(187 as Weight))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().writes(4 as Weight))
 			.saturating_add(T::DbWeight::get().writes((3 as Weight).saturating_mul(v as Weight)))
 	}
 	fn get_npos_voters(v: u32, n: u32, s: u32) -> Weight {
 		(0 as Weight)
 			// Standard Error: 110_000
-			.saturating_add((25_641_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((25_393_000 as Weight).saturating_mul(v as Weight))
 			// Standard Error: 110_000
-			.saturating_add((27_889_000 as Weight).saturating_mul(n as Weight))
+			.saturating_add((34_992_000 as Weight).saturating_mul(n as Weight))
-			// Standard Error: 3_749_000
+			.saturating_add(T::DbWeight::get().reads(179 as Weight))
-			.saturating_add((15_769_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(3 as Weight))
 			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(v as Weight)))
-			.saturating_add(T::DbWeight::get().reads((3 as Weight).saturating_mul(n as Weight)))
+			.saturating_add(T::DbWeight::get().reads((4 as Weight).saturating_mul(n as Weight)))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(s as Weight)))
 	}
 	fn get_npos_targets(v: u32) -> Weight {
 		(0 as Weight)
-			// Standard Error: 36_000
+			// Standard Error: 42_000
-			.saturating_add((10_960_000 as Weight).saturating_mul(v as Weight))
+			.saturating_add((11_340_000 as Weight).saturating_mul(v as Weight))
 			.saturating_add(T::DbWeight::get().reads(1 as Weight))
 			.saturating_add(T::DbWeight::get().reads((1 as Weight).saturating_mul(v as Weight)))
 	}
@@ -216,8 +214,8 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
 		(5_696_000 as Weight).saturating_add(T::DbWeight::get().writes(5 as Weight))
 	}
 	fn chill_other() -> Weight {
-		(41_216_000 as Weight)
+		(84_796_000 as Weight)
-			.saturating_add(T::DbWeight::get().reads(7 as Weight))
+			.saturating_add(T::DbWeight::get().reads(11 as Weight))
-			.saturating_add(T::DbWeight::get().writes(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 }

polkadot/utils/remote-ext-tests/bags-list/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "remote-ext-tests-bags-list"
+version = "0.9.9"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+
+[dependencies]
+polkadot-runtime = { version = "0.9.8", path = "../../../runtime/polkadot" }
+kusama-runtime = { version = "0.9.8", path = "../../../runtime/kusama" }
+
+pallet-staking = { git = "https://github.com/paritytech/substrate", branch = "master" }
+pallet-election-provider-multi-phase = { git = "https://github.com/paritytech/substrate", branch = "master" }
+pallet-bags-list = { git = "https://github.com/paritytech/substrate", branch = "master" }
+frame-election-provider-support = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-storage = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-core = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-tracing = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-runtime = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-std = { git = "https://github.com/paritytech/substrate", branch = "master" }
+frame-support = { git = "https://github.com/paritytech/substrate", branch = "master" }
+remote-externalities = { git = "https://github.com/paritytech/substrate", branch = "master" }
+tokio = { version = "1", features = ["macros"] }
+log = { version = "0.4.14" }
+structopt = {version = "0.3.23" }

polkadot/utils/remote-ext-tests/bags-list/README.md

@@ -0,0 +1,3 @@
+# Remote Extension Tests For Pallet Bags List
+
+Integration tests that use state from live chains via remote externalities.

polkadot/utils/remote-ext-tests/bags-list/src/main.rs

@@ -0,0 +1,59 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Remote tests.
+
+use structopt::StructOpt;
+
+mod voter_bags;
+
+#[derive(StructOpt)]
+enum Runtime {
+	Kusama,
+}
+
+impl std::str::FromStr for Runtime {
+	type Err = &'static str;
+	fn from_str(s: &str) -> Result<Self, Self::Err> {
+		match s.to_lowercase().as_str() {
+			"kusama" => Ok(Runtime::Kusama),
+			_ => Err("wrong Runtime: can be 'polkadot' or 'kusama'."),
+		}
+	}
+}
+
+#[derive(StructOpt)]
+struct Cli {
+	#[structopt(long, default_value = "wss://rpc.kusama.io")]
+	uri: String,
+	#[structopt(long, short, default_value = "kusama")]
+	runtime: Runtime,
+}
+
+#[tokio::main]
+async fn main() {
+	let options = Cli::from_args();
+	match options.runtime {
+		Runtime::Kusama => {
+			use kusama_runtime::{constants::currency::UNITS, Block, Runtime};
+			voter_bags::test_voter_bags_migration::<Runtime, Block>(
+				UNITS as u64,
+				options.uri.clone(),
+			)
+			.await;
+		},
+	}
+}

polkadot/utils/remote-ext-tests/bags-list/src/voter_bags.rs

@@ -0,0 +1,148 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Generic remote tests for the voter bags module.
+
+use frame_election_provider_support::SortedListProvider;
+use frame_support::traits::Get;
+use pallet_election_provider_multi_phase as EPM;
+use pallet_staking::{BalanceOf, MinNominatorBond, Nominators};
+use remote_externalities::{Builder, Mode, OnlineConfig};
+use sp_runtime::traits::Block as BlockT;
+use sp_std::convert::TryInto;
+use sp_storage::well_known_keys;
+
+const LOG_TARGET: &'static str = "remote-ext-tests::bags-list";
+
+/// Test voter bags migration. `currency_unit` is the number of planks per the
+/// the runtimes `UNITS` (i.e. number of decimal places per DOT, KSM etc)
+pub(crate) async fn test_voter_bags_migration<
+	Runtime: pallet_staking::Config + pallet_bags_list::Config + EPM::Config,
+	Block: BlockT,
+>(
+	currency_unit: u64,
+	ws_url: String,
+) {
+	sp_tracing::try_init_simple();
+
+	let mut ext = Builder::<Block>::new()
+		.mode(Mode::Online(OnlineConfig {
+			transport: ws_url.to_string().into(),
+			modules: vec!["Staking".to_string()],
+			at: None,
+			state_snapshot: None,
+		}))
+		.inject_hashed_key(well_known_keys::CODE)
+		.build()
+		.await
+		.unwrap();
+
+	ext.execute_with(|| {
+		// set the ss58 prefix so addresses printed below are human friendly.
+		sp_core::crypto::set_default_ss58_version(Runtime::SS58Prefix::get().try_into().unwrap());
+
+		// get the nominator & validator count prior to migrating; these should be invariant.
+		let pre_migrate_nominator_count = <Nominators<Runtime>>::iter().count() as u32;
+		log::info!(target: LOG_TARGET, "Nominator count: {}", pre_migrate_nominator_count);
+
+		// run the actual migration,
+		let moved = <Runtime as pallet_staking::Config>::SortedListProvider::regenerate(
+			pallet_staking::Nominators::<Runtime>::iter().map(|(n, _)| n),
+			pallet_staking::Pallet::<Runtime>::weight_of_fn(),
+		);
+		log::info!(target: LOG_TARGET, "Moved {} nominators", moved);
+
+		let voter_list_len =
+			<Runtime as pallet_staking::Config>::SortedListProvider::iter().count() as u32;
+		let voter_list_count = <Runtime as pallet_staking::Config>::SortedListProvider::count();
+		// and confirm it is equal to the length of the `VoterList`.
+		assert_eq!(pre_migrate_nominator_count, voter_list_len);
+		assert_eq!(pre_migrate_nominator_count, voter_list_count);
+
+		let min_nominator_bond = <MinNominatorBond<Runtime>>::get();
+		log::info!(target: LOG_TARGET, "min nominator bond is {:?}", min_nominator_bond);
+
+		// go through every bag to track the total number of voters within bags
+		// and log some info about how voters are distributed within the bags.
+		let mut seen_in_bags = 0;
+		for vote_weight_thresh in <Runtime as pallet_bags_list::Config>::BagThresholds::get() {
+			// threshold in terms of UNITS (e.g. KSM, DOT etc)
+			let vote_weight_thresh_as_unit = *vote_weight_thresh as f64 / currency_unit as f64;
+			let pretty_thresh = format!("Threshold: {}.", vote_weight_thresh_as_unit);
+
+			let bag = match pallet_bags_list::Pallet::<Runtime>::list_bags_get(*vote_weight_thresh)
+			{
+				Some(bag) => bag,
+				None => {
+					log::info!(target: LOG_TARGET, "{} NO VOTERS.", pretty_thresh);
+					continue
+				},
+			};
+
+			let voters_in_bag = bag.std_iter().count() as u32;
+
+			// if this bag is below the min nominator bond print out all the members
+			let vote_weight_as_balance: BalanceOf<Runtime> =
+				(*vote_weight_thresh).try_into().map_err(|_| "should not fail").unwrap();
+			if vote_weight_as_balance <= min_nominator_bond {
+				for id in bag.std_iter().map(|node| node.std_id().clone()) {
+					log::trace!(
+						target: LOG_TARGET,
+						"{} Account found below min bond: {:?}.",
+						pretty_thresh,
+						id
+					);
+				}
+			}
+
+			// update our overall counter
+			seen_in_bags += voters_in_bag;
+
+			// percentage of all nominators
+			let percent_of_voters = percent(voters_in_bag, voter_list_count);
+
+			log::info!(
+				target: LOG_TARGET,
+				"{} Nominators: {} [%{:.3}]",
+				pretty_thresh,
+				voters_in_bag,
+				percent_of_voters,
+			);
+		}
+
+		if seen_in_bags != voter_list_count {
+			log::error!(
+				target: LOG_TARGET,
+				"bags list population ({}) not on par whoever is voter_list ({})",
+				seen_in_bags,
+				voter_list_count,
+			)
+		}
+
+		// now let's test the process of a snapshot being created..
+		EPM::Pallet::<Runtime>::create_snapshot().unwrap();
+
+		log::info!(
+			target: LOG_TARGET,
+			"a snapshot has been created using the new runtime and data, with metadata {:?}",
+			EPM::Pallet::<Runtime>::snapshot_metadata(),
+		);
+	});
+}
+
+fn percent(portion: u32, total: u32) -> f64 {
+	(portion as f64 / total as f64) * 100f64
+}

polkadot/utils/voter-bags/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "polkadot-voter-bags"
+version = "0.9.0"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+
+[dependencies]
+generate-bags = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-io = { git = "https://github.com/paritytech/substrate", branch = "master" }
+structopt = "0.3.21"
+
+westend-runtime = { path = "../../runtime/westend" }
+kusama-runtime = { path = "../../runtime/kusama" }
+polkadot-runtime = { path = "../../runtime/polkadot" }

polkadot/utils/voter-bags/src/main.rs

@@ -0,0 +1,82 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Make the set of voting bag thresholds to be used in `voter_bags.rs`.
+//!
+//! Generally speaking this script can be run once per runtime and never
+//! touched again. It can be reused to regenerate a wholly different
+//! quantity of bags, or if the existential deposit changes, etc.
+
+use generate_bags::generate_thresholds;
+use kusama_runtime::Runtime as KusamaRuntime;
+use polkadot_runtime::Runtime as PolkadotRuntime;
+use std::path::{Path, PathBuf};
+use structopt::{clap::arg_enum, StructOpt};
+use westend_runtime::Runtime as WestendRuntime;
+
+arg_enum! {
+	#[derive(Debug)]
+	enum Runtime {
+		Westend,
+		Kusama,
+		Polkadot,
+	}
+}
+
+impl Runtime {
+	fn generate_thresholds_fn(
+		&self,
+	) -> Box<dyn FnOnce(usize, &Path, u128, u128) -> Result<(), std::io::Error>> {
+		match self {
+			Runtime::Westend => Box::new(generate_thresholds::<WestendRuntime>),
+			Runtime::Kusama => Box::new(generate_thresholds::<KusamaRuntime>),
+			Runtime::Polkadot => Box::new(generate_thresholds::<PolkadotRuntime>),
+		}
+	}
+}
+
+#[derive(Debug, StructOpt)]
+struct Opt {
+	/// How many bags to generate.
+	#[structopt(long, default_value = "200")]
+	n_bags: usize,
+
+	/// Which runtime to generate.
+	#[structopt(
+		long,
+		case_insensitive = true,
+		default_value = "Polkadot",
+		possible_values = &Runtime::variants(),
+	)]
+	runtime: Runtime,
+
+	/// Where to write the output.
+	output: PathBuf,
+
+	/// The total issuance of the native currency.
+	#[structopt(short, long)]
+	total_issuance: u128,
+
+	/// The minimum account balance (i.e. existential deposit) for the native currency.
+	#[structopt(short, long)]
+	minimum_balance: u128,
+}
+
+fn main() -> Result<(), std::io::Error> {
+	let Opt { n_bags, output, runtime, total_issuance, minimum_balance } = Opt::from_args();
+
+	runtime.generate_thresholds_fn()(n_bags, &output, total_issuance, minimum_balance)
+}