[NPoS] Implements dynamic number of nominators (#12970)

* Implements dynamic nominations per nominator * Adds SnapshotBounds and ElectionSizeTracker * Changes the ElectionDataProvider interface to receive ElectionBounds as input * Implements get_npos_voters with ElectionBounds * Implements get_npos_targets with ElectionBounds * Adds comments * tests * Truncates nomninations that exceed nominations quota; Old tests passing * Uses DataProviderBounds and ElectionBounds (to continue) * Finishes conversions - tests passing * Refactor staking in babe mocks * Replaces MaxElectableTargets and MaxElectingVoters with ElectionBounds; Adds more tests * Fixes nits; node compiling * bechmarks * removes nomination_quota extrinsic to request the nomination quota * Lazy quota check, ie. at nominate time only * remove non-working test (for now) * tests lazy nominations quota when quota is lower than current number of nominated targets * Adds runtime API and custom RPC call for clients to query the nominations quota for a given balance * removes old rpc * Cosmetic touches * All mocks working * Fixes benchmarking mocks * nits * more tests * renames trait methods * nit * ".git/.scripts/commands/fmt/fmt.sh" * Fix V2 PoV benchmarking (#13485) * Bump default 'additional_trie_layers' to two The default here only works for extremely small runtimes, which have no more than 16 storage prefices. This is changed to a "sane" default of 2, which is save for runtimes with up to 4096 storage prefices (eg StorageValue). Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update tests and test weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix PoV weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_balances * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_message_queue * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton * Fix sanity check >0 would also do as a check, but let's try this. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> * Move BEEFY code to consensus (#13484) * Move beefy primitives to consensus dir * Move beefy gadget to client consensus folder * Rename beefy crates * chore: move genesis block builder to chain-spec crate. (#13427) * chore: move genesis block builder to block builder crate. * add missing file * chore: move genesis block builder to sc-chain-spec * Update client/chain-spec/src/genesis.rs Co-authored-by: Bastian Köcher <git@kchr.de> * Update test-utils/runtime/src/genesismap.rs Co-authored-by: Bastian Köcher <git@kchr.de> * Update test-utils/runtime/client/src/lib.rs * fix warnings * fix warnings --------- Co-authored-by: Bastian Köcher <git@kchr.de> * Speed up storage iteration from within the runtime (#13479) * Speed up storage iteration from within the runtime * Move the cached iterator into an `Option` * Use `RefCell` in no_std * Simplify the code slightly * Use `Option::replace` * Update doc comment for `next_storage_key_slow` * Make unbounded channels size warning exact (part 1) (#13490) * Replace `futures-channel` with `async-channel` in `out_events` * Apply suggestions from code review Co-authored-by: Koute <koute@users.noreply.github.com> * Also print the backtrace of `send()` call * Switch from `backtrace` crate to `std::backtrace` * Remove outdated `backtrace` dependency * Remove `backtrace` from `Cargo.lock` --------- Co-authored-by: Koute <koute@users.noreply.github.com> * Removal of Prometheus alerting rules deployment in cloud-infra (#13499) * sp-consensus: remove unused error variants (#13495) * Expose `ChargedAmount` (#13488) * Expose `ChargedAmount` * Fix imports * sc-consensus-beefy: fix metrics: use correct names (#13494) Signed-off-by: acatangiu <adrian@parity.io> * clippy fix * removes NominationsQuotaExceeded event * Update frame/staking/src/lib.rs Co-authored-by: Ross Bulat <ross@parity.io> * adds back the npos_max_iter * remove duplicate imports added after merge * fmt * Adds comment in public struct; Refactors CountBound and SizeCount to struct * addresses various pr comments * PR comment reviews * Fixes on-chain election bounds and related code * EPM checks the size of the voter list returned by the data provider * cosmetic changes * updates e2e tests mock * Adds more tests for size tracker and refactors code * Adds back only_iterates_max_2_times_max_allowed_len test * Refactor * removes unecessary dependency * empty commit -- restart all stuck CI jobs * restarts ci jobs * Renames ElectionBounds -> Bounds in benchmarking mocks et al * updates mocks * Update frame/election-provider-support/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/staking/src/pallet/impls.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/election-provider-support/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/staking/src/tests.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * more checks in api_nominations_quota in tests * Improves docs * fixes e2e tests * Uses size_hint rather than mem::size_of in size tracker; Refactor size tracker to own module * nits from reviews * Refactors bounds to own module; improves docs * More tests and docs * fixes docs * Fixes benchmarks * Fixes rust docs * fixes bags-list remote-ext-tests * Simplify bound checks in create_snapshot_external * Adds target size check in get_npos_targets * ".git/.scripts/commands/fmt/fmt.sh" * restart ci * rust doc fixes and cosmetic nits * rollback upgrade on parity-scale-codec version (unecessary) * reset cargo lock, no need to update it --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Co-authored-by: command-bot <> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Anthony Lazam <lazam@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Piotr Mikołajczyk <piomiko41@gmail.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2026-07-21 02:45:43 +00:00 · 2023-08-10 09:45:55 +02:00
parent 314109d87b
commit 93754780b1
30 changed files with 1415 additions and 307 deletions
@@ -20,6 +20,7 @@
 use super::*;
 use crate::{unsigned::IndexAssignmentOf, Pallet as MultiPhase};
 use frame_benchmarking::account;
+use frame_election_provider_support::bounds::DataProviderBounds;
 use frame_support::{
 	assert_ok,
 	traits::{Hooks, TryCollect},
@@ -270,8 +271,9 @@ frame_benchmarking::benchmarks! {

 		// we don't directly need the data-provider to be populated, but it is just easy to use it.
 		set_up_data_provider::<T>(v, t);
-		let targets = T::DataProvider::electable_targets(None)?;
-		let voters = T::DataProvider::electing_voters(None)?;
+		// default bounds are unbounded.
+		let targets = T::DataProvider::electable_targets(DataProviderBounds::default())?;
+		let voters = T::DataProvider::electing_voters(DataProviderBounds::default())?;
 		let desired_targets = T::DataProvider::desired_targets()?;
 		assert!(<MultiPhase<T>>::snapshot().is_none());
 	}: {
@@ -231,8 +231,9 @@

 use codec::{Decode, Encode};
 use frame_election_provider_support::{
-	BoundedSupportsOf, ElectionDataProvider, ElectionProvider, ElectionProviderBase,
-	InstantElectionProvider, NposSolution,
+	bounds::{CountBound, ElectionBounds, ElectionBoundsBuilder, SizeBound},
+	BoundedSupportsOf, DataProviderBounds, ElectionDataProvider, ElectionProvider,
+	ElectionProviderBase, InstantElectionProvider, NposSolution,
 };
 use frame_support::{
 	dispatch::DispatchClass,
@@ -659,16 +660,6 @@ pub mod pallet {
 		#[pallet::constant]
 		type SignedDepositWeight: Get<BalanceOf<Self>>;

-		/// The maximum number of electing voters to put in the snapshot. At the moment, snapshots
-		/// are only over a single block, but once multi-block elections are introduced they will
-		/// take place over multiple blocks.
-		#[pallet::constant]
-		type MaxElectingVoters: Get<SolutionVoterIndexOf<Self::MinerConfig>>;
-
-		/// The maximum number of electable targets to put in the snapshot.
-		#[pallet::constant]
-		type MaxElectableTargets: Get<SolutionTargetIndexOf<Self::MinerConfig>>;
-
 		/// The maximum number of winners that can be elected by this `ElectionProvider`
 		/// implementation.
 		///
@@ -676,6 +667,11 @@ pub mod pallet {
 		#[pallet::constant]
 		type MaxWinners: Get<u32>;

+		/// The maximum number of electing voters and electable targets to put in the snapshot.
+		/// At the moment, snapshots are only over a single block, but once multi-block elections
+		/// are introduced they will take place over multiple blocks.
+		type ElectionBounds: Get<ElectionBounds>;
+
 		/// Handler for the slashed deposits.
 		type SlashHandler: OnUnbalanced<NegativeImbalanceOf<Self>>;

@@ -1097,13 +1093,19 @@ pub mod pallet {
 			T::ForceOrigin::ensure_origin(origin)?;
 			ensure!(Self::current_phase().is_emergency(), <Error<T>>::CallNotAllowed);

-			let supports =
-				T::GovernanceFallback::instant_elect(maybe_max_voters, maybe_max_targets).map_err(
-					|e| {
-						log!(error, "GovernanceFallback failed: {:?}", e);
-						Error::<T>::FallbackFailed
-					},
-				)?;
+			let election_bounds = ElectionBoundsBuilder::default()
+				.voters_count(maybe_max_voters.unwrap_or(u32::MAX).into())
+				.targets_count(maybe_max_targets.unwrap_or(u32::MAX).into())
+				.build();
+
+			let supports = T::GovernanceFallback::instant_elect(
+				election_bounds.voters,
+				election_bounds.targets,
+			)
+			.map_err(|e| {
+				log!(error, "GovernanceFallback failed: {:?}", e);
+				Error::<T>::FallbackFailed
+			})?;

 			// transform BoundedVec<_, T::GovernanceFallback::MaxWinners> into
 			// `BoundedVec<_, T::MaxWinners>`
@@ -1426,19 +1428,28 @@ impl<T: Config> Pallet<T> {
 	/// Extracted for easier weight calculation.
 	fn create_snapshot_external(
 	) -> Result<(Vec<T::AccountId>, Vec<VoterOf<T>>, u32), ElectionError<T>> {
-		let target_limit = T::MaxElectableTargets::get().saturated_into::<usize>();
-		let voter_limit = T::MaxElectingVoters::get().saturated_into::<usize>();
+		let election_bounds = T::ElectionBounds::get();

-		let targets = T::DataProvider::electable_targets(Some(target_limit))
+		let targets = T::DataProvider::electable_targets(election_bounds.targets)
+			.and_then(|t| {
+				election_bounds.ensure_targets_limits(
+					CountBound(t.len() as u32),
+					SizeBound(t.encoded_size() as u32),
+				)?;
+				Ok(t)
+			})
 			.map_err(ElectionError::DataProvider)?;

-		let voters = T::DataProvider::electing_voters(Some(voter_limit))
+		let voters = T::DataProvider::electing_voters(election_bounds.voters)
+			.and_then(|v| {
+				election_bounds.ensure_voters_limits(
+					CountBound(v.len() as u32),
+					SizeBound(v.encoded_size() as u32),
+				)?;
+				Ok(v)
+			})
 			.map_err(ElectionError::DataProvider)?;

-		if targets.len() > target_limit || voters.len() > voter_limit {
-			return Err(ElectionError::DataProvider("Snapshot too big for submission."))
-		}
-
 		let mut desired_targets = <Pallet<T> as ElectionProviderBase>::desired_targets_checked()
 			.map_err(|e| ElectionError::DataProvider(e))?;

@@ -1544,18 +1555,25 @@ impl<T: Config> Pallet<T> {
 		// - signed phase was complete or not started, in which case finalization is idempotent and
 		//   inexpensive (1 read of an empty vector).
 		let _ = Self::finalize_signed_phase();
+
 		<QueuedSolution<T>>::take()
 			.ok_or(ElectionError::<T>::NothingQueued)
 			.or_else(|_| {
-				T::Fallback::instant_elect(None, None)
-					.map_err(|fe| ElectionError::Fallback(fe))
-					.and_then(|supports| {
-						Ok(ReadySolution {
-							supports,
-							score: Default::default(),
-							compute: ElectionCompute::Fallback,
-						})
+				// default data provider bounds are unbounded. calling `instant_elect` with
+				// unbounded data provider bounds means that the on-chain `T:Bounds` configs will
+				// *not* be overwritten.
+				T::Fallback::instant_elect(
+					DataProviderBounds::default(),
+					DataProviderBounds::default(),
+				)
+				.map_err(|fe| ElectionError::Fallback(fe))
+				.and_then(|supports| {
+					Ok(ReadySolution {
+						supports,
+						score: Default::default(),
+						compute: ElectionCompute::Fallback,
 					})
+				})
 			})
 			.map(|ReadySolution { compute, score, supports }| {
 				Self::deposit_event(Event::ElectionFinalized { compute, score });
@@ -1925,8 +1943,8 @@ mod tests {
 	use crate::{
 		mock::{
 			multi_phase_events, raw_solution, roll_to, roll_to_signed, roll_to_unsigned, AccountId,
-			ExtBuilder, MockWeightInfo, MockedWeightInfo, MultiPhase, Runtime, RuntimeOrigin,
-			SignedMaxSubmissions, System, TargetIndex, Targets,
+			ElectionsBounds, ExtBuilder, MockWeightInfo, MockedWeightInfo, MultiPhase, Runtime,
+			RuntimeOrigin, SignedMaxSubmissions, System, TargetIndex, Targets, Voters,
 		},
 		Phase,
 	};
@@ -2529,7 +2547,11 @@ mod tests {
 	fn snapshot_too_big_failure_onchain_fallback() {
 		// the `MockStaking` is designed such that if it has too many targets, it simply fails.
 		ExtBuilder::default().build_and_execute(|| {
-			Targets::set((0..(TargetIndex::max_value() as AccountId) + 1).collect::<Vec<_>>());
+			// sets bounds on number of targets.
+			let new_bounds = ElectionBoundsBuilder::default().targets_count(1_000.into()).build();
+			ElectionsBounds::set(new_bounds);
+
+			Targets::set((0..(1_000 as AccountId) + 1).collect::<Vec<_>>());

 			// Signed phase failed to open.
 			roll_to(15);
@@ -2564,9 +2586,11 @@ mod tests {
 	fn snapshot_too_big_failure_no_fallback() {
 		// and if the backup mode is nothing, we go into the emergency mode..
 		ExtBuilder::default().onchain_fallback(false).build_and_execute(|| {
-			crate::mock::Targets::set(
-				(0..(TargetIndex::max_value() as AccountId) + 1).collect::<Vec<_>>(),
-			);
+			// sets bounds on number of targets.
+			let new_bounds = ElectionBoundsBuilder::default().targets_count(1_000.into()).build();
+			ElectionsBounds::set(new_bounds);
+
+			Targets::set((0..(TargetIndex::max_value() as AccountId) + 1).collect::<Vec<_>>());

 			// Signed phase failed to open.
 			roll_to(15);
@@ -2596,9 +2620,10 @@ mod tests {
 		// but if there are too many voters, we simply truncate them.
 		ExtBuilder::default().build_and_execute(|| {
 			// we have 8 voters in total.
-			assert_eq!(crate::mock::Voters::get().len(), 8);
+			assert_eq!(Voters::get().len(), 8);
 			// but we want to take 2.
-			crate::mock::MaxElectingVoters::set(2);
+			let new_bounds = ElectionBoundsBuilder::default().voters_count(2.into()).build();
+			ElectionsBounds::set(new_bounds);

 			// Signed phase opens just fine.
 			roll_to_signed();
@@ -18,9 +18,8 @@
 use super::*;
 use crate::{self as multi_phase, unsigned::MinerConfig};
 use frame_election_provider_support::{
-	data_provider,
-	onchain::{self},
-	ElectionDataProvider, NposSolution, SequentialPhragmen,
+	bounds::{DataProviderBounds, ElectionBounds},
+	data_provider, onchain, ElectionDataProvider, NposSolution, SequentialPhragmen,
 };
 pub use frame_support::{assert_noop, assert_ok, pallet_prelude::GetDefault};
 use frame_support::{
@@ -300,7 +299,9 @@ parameter_types! {

 	#[derive(Debug)]
 	pub static MaxWinners: u32 = 200;
-
+	// `ElectionBounds` and `OnChainElectionsBounds` are defined separately to set them independently in the tests.
+	pub static ElectionsBounds: ElectionBounds = ElectionBoundsBuilder::default().build();
+	pub static OnChainElectionsBounds: ElectionBounds = ElectionBoundsBuilder::default().build();
 	pub static EpochLength: u64 = 30;
 	pub static OnChainFallback: bool = true;
 }
@@ -312,8 +313,7 @@ impl onchain::Config for OnChainSeqPhragmen {
 	type DataProvider = StakingMock;
 	type WeightInfo = ();
 	type MaxWinners = MaxWinners;
-	type VotersBound = ConstU32<{ u32::MAX }>;
-	type TargetsBound = ConstU32<{ u32::MAX }>;
+	type Bounds = OnChainElectionsBounds;
 }

 pub struct MockFallback;
@@ -327,12 +327,15 @@ impl ElectionProviderBase for MockFallback {

 impl InstantElectionProvider for MockFallback {
 	fn instant_elect(
-		max_voters: Option<u32>,
-		max_targets: Option<u32>,
+		voters_bounds: DataProviderBounds,
+		targets_bounds: DataProviderBounds,
 	) -> Result<BoundedSupportsOf<Self>, Self::Error> {
 		if OnChainFallback::get() {
-			onchain::OnChainExecution::<OnChainSeqPhragmen>::instant_elect(max_voters, max_targets)
-				.map_err(|_| "onchain::OnChainExecution failed.")
+			onchain::OnChainExecution::<OnChainSeqPhragmen>::instant_elect(
+				voters_bounds,
+				targets_bounds,
+			)
+			.map_err(|_| "onchain::OnChainExecution failed.")
 		} else {
 			Err("NoFallback.")
 		}
@@ -404,11 +407,10 @@ impl crate::Config for Runtime {
 	type GovernanceFallback =
 		frame_election_provider_support::onchain::OnChainExecution<OnChainSeqPhragmen>;
 	type ForceOrigin = frame_system::EnsureRoot<AccountId>;
-	type MaxElectingVoters = MaxElectingVoters;
-	type MaxElectableTargets = MaxElectableTargets;
 	type MaxWinners = MaxWinners;
 	type MinerConfig = Self;
 	type Solver = SequentialPhragmen<AccountId, SolutionAccuracyOf<Runtime>, Balancing>;
+	type ElectionBounds = ElectionsBounds;
 }

 impl<LocalCall> frame_system::offchain::SendTransactionTypes<LocalCall> for Runtime
@@ -436,11 +438,11 @@ impl ElectionDataProvider for StakingMock {
 	type AccountId = AccountId;
 	type MaxVotesPerVoter = MaxNominations;

-	fn electable_targets(maybe_max_len: Option<usize>) -> data_provider::Result<Vec<AccountId>> {
+	fn electable_targets(bounds: DataProviderBounds) -> data_provider::Result<Vec<AccountId>> {
 		let targets = Targets::get();

 		if !DataProviderAllowBadData::get() &&
-			maybe_max_len.map_or(false, |max_len| targets.len() > max_len)
+			bounds.count.map_or(false, |max_len| targets.len() > max_len.0 as usize)
 		{
 			return Err("Targets too big")
 		}
@@ -448,13 +450,12 @@ impl ElectionDataProvider for StakingMock {
 		Ok(targets)
 	}

-	fn electing_voters(
-		maybe_max_len: Option<usize>,
-	) -> data_provider::Result<Vec<VoterOf<Runtime>>> {
+	fn electing_voters(bounds: DataProviderBounds) -> data_provider::Result<Vec<VoterOf<Runtime>>> {
 		let mut voters = Voters::get();
+
 		if !DataProviderAllowBadData::get() {
-			if let Some(max_len) = maybe_max_len {
-				voters.truncate(max_len)
+			if let Some(max_len) = bounds.count {
+				voters.truncate(max_len.0 as usize)
 			}
 		}

@@ -536,7 +536,10 @@ impl<T: Config> Pallet<T> {
 #[cfg(test)]
 mod tests {
 	use super::*;
-	use crate::{mock::*, ElectionCompute, ElectionError, Error, Event, Perbill, Phase};
+	use crate::{
+		mock::*, ElectionBoundsBuilder, ElectionCompute, ElectionError, Error, Event, Perbill,
+		Phase,
+	};
 	use frame_support::{assert_noop, assert_ok, assert_storage_noop};

 	#[test]
@@ -565,13 +568,14 @@ mod tests {
 	fn data_provider_should_respect_target_limits() {
 		ExtBuilder::default().build_and_execute(|| {
 			// given a reduced expectation of maximum electable targets
-			MaxElectableTargets::set(2);
+			let new_bounds = ElectionBoundsBuilder::default().targets_count(2.into()).build();
+			ElectionsBounds::set(new_bounds);
 			// and a data provider that does not respect limits
 			DataProviderAllowBadData::set(true);

 			assert_noop!(
 				MultiPhase::create_snapshot(),
-				ElectionError::DataProvider("Snapshot too big for submission."),
+				ElectionError::DataProvider("Ensure targets bounds: bounds exceeded."),
 			);
 		})
 	}
@@ -580,13 +584,14 @@ mod tests {
 	fn data_provider_should_respect_voter_limits() {
 		ExtBuilder::default().build_and_execute(|| {
 			// given a reduced expectation of maximum electing voters
-			MaxElectingVoters::set(2);
+			let new_bounds = ElectionBoundsBuilder::default().voters_count(2.into()).build();
+			ElectionsBounds::set(new_bounds);
 			// and a data provider that does not respect limits
 			DataProviderAllowBadData::set(true);

 			assert_noop!(
 				MultiPhase::create_snapshot(),
-				ElectionError::DataProvider("Snapshot too big for submission."),
+				ElectionError::DataProvider("Ensure voters bounds: bounds exceeded."),
 			);
 		})
 	}