Approve multiple candidates with a single signature (#1191)

Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,

## Overall idea

When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.

This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:

- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.

## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.

## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor @burdges, let me know what you think.
- [x] More and more testing.
- [x]  Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742
- [x] Final versi burn-in before merging

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
This commit is contained in:
Alexandru Gheorghe
2023-12-13 08:43:15 +02:00
committed by GitHub
parent d18a682bf7
commit a84dd0dba5
82 changed files with 5883 additions and 1483 deletions
+576 -5
View File
@@ -37,8 +37,8 @@ use polkadot_node_subsystem_test_helpers as test_helpers;
use polkadot_node_subsystem_util::TimeoutExt;
use polkadot_overseer::HeadSupportsParachains;
use polkadot_primitives::{
vstaging::NodeFeatures, CandidateCommitments, CandidateEvent, CoreIndex, GroupIndex, Header,
Id as ParaId, IndexedVec, ValidationCode, ValidatorSignature,
vstaging::NodeFeatures, ApprovalVote, CandidateCommitments, CandidateEvent, CoreIndex,
GroupIndex, Header, Id as ParaId, IndexedVec, ValidationCode, ValidatorSignature,
};
use std::time::Duration;
@@ -56,7 +56,7 @@ use std::{
};
use super::{
approval_db::v2::StoredBlockRange,
approval_db::common::StoredBlockRange,
backend::BackendWriteOp,
import::tests::{
garbage_vrf_signature, AllowedSlots, BabeEpoch, BabeEpochConfiguration,
@@ -116,7 +116,7 @@ fn make_sync_oracle(val: bool) -> (Box<dyn SyncOracle + Send>, TestSyncOracleHan
#[cfg(test)]
pub mod test_constants {
use crate::approval_db::v2::Config as DatabaseConfig;
use crate::approval_db::common::Config as DatabaseConfig;
const DATA_COL: u32 = 0;
pub(crate) const NUM_COLUMNS: u32 = 1;
@@ -281,6 +281,7 @@ impl V1ReadBackend for TestStoreInner {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
_candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>> {
self.load_candidate_entry(candidate_hash)
}
@@ -364,6 +365,7 @@ impl V1ReadBackend for TestStore {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
_candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>> {
self.load_candidate_entry(candidate_hash)
}
@@ -446,6 +448,15 @@ fn sign_approval(
key.sign(&ApprovalVote(candidate_hash).signing_payload(session_index)).into()
}
fn sign_approval_multiple_candidates(
key: Sr25519Keyring,
candidate_hashes: Vec<CandidateHash>,
session_index: SessionIndex,
) -> ValidatorSignature {
key.sign(&ApprovalVoteMultipleCandidates(&candidate_hashes).signing_payload(session_index))
.into()
}
type VirtualOverseer = test_helpers::TestSubsystemContextHandle<ApprovalVotingMessage>;
#[derive(Default)]
@@ -641,7 +652,12 @@ async fn check_and_import_approval(
overseer,
FromOrchestra::Communication {
msg: ApprovalVotingMessage::CheckAndImportApproval(
IndirectSignedApprovalVote { block_hash, candidate_index, validator, signature },
IndirectSignedApprovalVoteV2 {
block_hash,
candidate_indices: candidate_index.into(),
validator,
signature,
},
tx,
),
},
@@ -2014,6 +2030,91 @@ fn forkful_import_at_same_height_act_on_leaf() {
});
}
#[test]
fn test_signing_a_single_candidate_is_backwards_compatible() {
let session_index = 1;
let block_hash = Hash::repeat_byte(0x01);
let candidate_descriptors = (1..10)
.into_iter()
.map(|val| make_candidate(ParaId::from(val as u32), &block_hash))
.collect::<Vec<CandidateReceipt>>();
let candidate_hashes = candidate_descriptors
.iter()
.map(|candidate_descriptor| candidate_descriptor.hash())
.collect_vec();
let first_descriptor = candidate_descriptors.first().unwrap();
let candidate_hash = first_descriptor.hash();
let sig_a = sign_approval(Sr25519Keyring::Alice, candidate_hash, session_index);
let sig_b = sign_approval(Sr25519Keyring::Alice, candidate_hash, session_index);
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_a,
)
.is_ok());
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_b,
)
.is_ok());
let sig_c = sign_approval_multiple_candidates(
Sr25519Keyring::Alice,
vec![candidate_hash],
session_index,
);
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(vec![candidate_hash])
)
.check_signature(&Sr25519Keyring::Alice.public().into(), candidate_hash, session_index, &sig_c,)
.is_ok());
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_c,
)
.is_ok());
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(vec![candidate_hash])
)
.check_signature(&Sr25519Keyring::Alice.public().into(), candidate_hash, session_index, &sig_a,)
.is_ok());
let sig_all = sign_approval_multiple_candidates(
Sr25519Keyring::Alice,
candidate_hashes.clone(),
session_index,
);
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(candidate_hashes.clone())
)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
*candidate_hashes.first().expect("test"),
session_index,
&sig_all,
)
.is_ok());
}
#[test]
fn import_checked_approval_updates_entries_and_schedules() {
let config = HarnessConfig::default();
@@ -2730,11 +2831,29 @@ async fn handle_double_assignment_import(
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 1,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(_))
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 1,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(_))
@@ -3469,3 +3588,455 @@ fn waits_until_approving_assignments_are_old_enough() {
virtual_overseer
});
}
#[test]
fn test_approval_is_sent_on_max_approval_coalesce_count() {
let assignment_criteria = Box::new(MockAssignmentCriteria(
|| {
let mut assignments = HashMap::new();
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: garbage_assignment_cert(AssignmentCertKind::RelayVRFModulo { sample: 0 })
.into(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let assignments_cert =
garbage_assignment_cert_v2(AssignmentCertKindV2::RelayVRFModuloCompact {
core_bitfield: vec![CoreIndex(0), CoreIndex(1), CoreIndex(2)]
.try_into()
.unwrap(),
});
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let _ = assignments.insert(
CoreIndex(1),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
assignments
},
|_| Ok(0),
));
let config = HarnessConfigBuilder::default().assignment_criteria(assignment_criteria).build();
let store = config.backend();
test_harness(config, |test_harness| async move {
let TestHarness { mut virtual_overseer, clock, sync_oracle_handle: _sync_oracle_handle } =
test_harness;
assert_matches!(
overseer_recv(&mut virtual_overseer).await,
AllMessages::ChainApi(ChainApiMessage::FinalizedBlockNumber(rx)) => {
rx.send(Ok(0)).unwrap();
}
);
let block_hash = Hash::repeat_byte(0x01);
let candidate_commitments = CandidateCommitments::default();
let candidate_receipt1 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(1_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let candidate_hash1 = candidate_receipt1.hash();
let candidate_receipt2 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(2_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let slot = Slot::from(1);
let candidate_index1 = 0;
let candidate_index2 = 1;
let validators = vec![
Sr25519Keyring::Alice,
Sr25519Keyring::Bob,
Sr25519Keyring::Charlie,
Sr25519Keyring::Dave,
Sr25519Keyring::Eve,
];
let session_info = SessionInfo {
validator_groups: IndexedVec::<GroupIndex, Vec<ValidatorIndex>>::from(vec![
vec![ValidatorIndex(0), ValidatorIndex(1)],
vec![ValidatorIndex(2)],
vec![ValidatorIndex(3), ValidatorIndex(4)],
]),
..session_info(&validators)
};
let candidates = Some(vec![
(candidate_receipt1.clone(), CoreIndex(0), GroupIndex(0)),
(candidate_receipt2.clone(), CoreIndex(1), GroupIndex(1)),
]);
ChainBuilder::new()
.add_block(
block_hash,
ChainBuilder::GENESIS_HASH,
1,
BlockConfig {
slot,
candidates: candidates.clone(),
session_info: Some(session_info.clone()),
},
)
.build(&mut virtual_overseer)
.await;
assert!(!clock.inner.lock().current_wakeup_is(1));
clock.inner.lock().wakeup_all(1);
assert!(clock.inner.lock().current_wakeup_is(slot_to_tick(slot)));
clock.inner.lock().wakeup_all(slot_to_tick(slot));
futures_timer::Delay::new(Duration::from_millis(200)).await;
clock.inner.lock().wakeup_all(slot_to_tick(slot + 2));
assert_eq!(clock.inner.lock().wakeups.len(), 0);
futures_timer::Delay::new(Duration::from_millis(200)).await;
let candidate_entry = store.load_candidate_entry(&candidate_hash1).unwrap().unwrap();
let our_assignment =
candidate_entry.approval_entry(&block_hash).unwrap().our_assignment().unwrap();
assert!(our_assignment.triggered());
handle_approval_on_max_coalesce_count(
&mut virtual_overseer,
vec![candidate_index1, candidate_index2],
)
.await;
virtual_overseer
});
}
async fn handle_approval_on_max_coalesce_count(
virtual_overseer: &mut VirtualOverseer,
candidate_indicies: Vec<CandidateIndex>,
) {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeAssignment(
_,
c_indices,
)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies.clone()).unwrap(), c_indices);
}
);
for _ in &candidate_indicies {
recover_available_data(virtual_overseer).await;
fetch_validation_code(virtual_overseer).await;
}
for _ in &candidate_indicies {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::CandidateValidation(CandidateValidationMessage::ValidateFromExhaustive{exec_kind, response_sender, ..}) if exec_kind == PvfExecKind::Approval => {
response_sender.send(Ok(ValidationResult::Valid(Default::default(), Default::default())))
.unwrap();
}
);
}
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 2,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 2,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(vote)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies).unwrap(), vote.candidate_indices);
}
);
// Assert that there are no more messages being sent by the subsystem
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
}
async fn handle_approval_on_max_wait_time(
virtual_overseer: &mut VirtualOverseer,
candidate_indicies: Vec<CandidateIndex>,
clock: Box<MockClock>,
) {
const TICK_NOW_BEGIN: u64 = 1;
const MAX_COALESCE_COUNT: u32 = 3;
clock.inner.lock().set_tick(TICK_NOW_BEGIN);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeAssignment(
_,
c_indices,
)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies.clone()).unwrap(), c_indices);
}
);
for _ in &candidate_indicies {
recover_available_data(virtual_overseer).await;
fetch_validation_code(virtual_overseer).await;
}
for _ in &candidate_indicies {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::CandidateValidation(CandidateValidationMessage::ValidateFromExhaustive{exec_kind, response_sender, ..}) if exec_kind == PvfExecKind::Approval => {
response_sender.send(Ok(ValidationResult::Valid(Default::default(), Default::default())))
.unwrap();
}
);
}
// First time we fetch the configuration when we are ready to approve the first candidate
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: MAX_COALESCE_COUNT,
}));
}
);
// Second time we fetch the configuration when we are ready to approve the second candidate
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: MAX_COALESCE_COUNT,
}));
}
);
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
// Move the clock just before we should send the approval
clock
.inner
.lock()
.set_tick(MAX_APPROVAL_COALESCE_WAIT_TICKS as Tick + TICK_NOW_BEGIN - 1);
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
// Move the clock tick, so we can trigger a force sending of the approvals
clock
.inner
.lock()
.set_tick(MAX_APPROVAL_COALESCE_WAIT_TICKS as Tick + TICK_NOW_BEGIN);
// Third time we fetch the configuration when timer expires and we are ready to sent the
// approval
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 3,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(vote)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies).unwrap(), vote.candidate_indices);
}
);
// Assert that there are no more messages being sent by the subsystem
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
}
#[test]
fn test_approval_is_sent_on_max_approval_coalesce_wait() {
let assignment_criteria = Box::new(MockAssignmentCriteria(
|| {
let mut assignments = HashMap::new();
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: garbage_assignment_cert(AssignmentCertKind::RelayVRFModulo { sample: 0 })
.into(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let assignments_cert =
garbage_assignment_cert_v2(AssignmentCertKindV2::RelayVRFModuloCompact {
core_bitfield: vec![CoreIndex(0), CoreIndex(1), CoreIndex(2)]
.try_into()
.unwrap(),
});
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let _ = assignments.insert(
CoreIndex(1),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
assignments
},
|_| Ok(0),
));
let config = HarnessConfigBuilder::default().assignment_criteria(assignment_criteria).build();
let store = config.backend();
test_harness(config, |test_harness| async move {
let TestHarness { mut virtual_overseer, clock, sync_oracle_handle: _sync_oracle_handle } =
test_harness;
assert_matches!(
overseer_recv(&mut virtual_overseer).await,
AllMessages::ChainApi(ChainApiMessage::FinalizedBlockNumber(rx)) => {
rx.send(Ok(0)).unwrap();
}
);
let block_hash = Hash::repeat_byte(0x01);
let candidate_commitments = CandidateCommitments::default();
let candidate_receipt1 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(1_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let candidate_hash1 = candidate_receipt1.hash();
let candidate_receipt2 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(2_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let slot = Slot::from(1);
let candidate_index1 = 0;
let candidate_index2 = 1;
let validators = vec![
Sr25519Keyring::Alice,
Sr25519Keyring::Bob,
Sr25519Keyring::Charlie,
Sr25519Keyring::Dave,
Sr25519Keyring::Eve,
];
let session_info = SessionInfo {
validator_groups: IndexedVec::<GroupIndex, Vec<ValidatorIndex>>::from(vec![
vec![ValidatorIndex(0), ValidatorIndex(1)],
vec![ValidatorIndex(2)],
vec![ValidatorIndex(3), ValidatorIndex(4)],
]),
..session_info(&validators)
};
let candidates = Some(vec![
(candidate_receipt1.clone(), CoreIndex(0), GroupIndex(0)),
(candidate_receipt2.clone(), CoreIndex(1), GroupIndex(1)),
]);
ChainBuilder::new()
.add_block(
block_hash,
ChainBuilder::GENESIS_HASH,
1,
BlockConfig {
slot,
candidates: candidates.clone(),
session_info: Some(session_info.clone()),
},
)
.build(&mut virtual_overseer)
.await;
assert!(!clock.inner.lock().current_wakeup_is(1));
clock.inner.lock().wakeup_all(1);
assert!(clock.inner.lock().current_wakeup_is(slot_to_tick(slot)));
clock.inner.lock().wakeup_all(slot_to_tick(slot));
futures_timer::Delay::new(Duration::from_millis(200)).await;
clock.inner.lock().wakeup_all(slot_to_tick(slot + 2));
assert_eq!(clock.inner.lock().wakeups.len(), 0);
futures_timer::Delay::new(Duration::from_millis(200)).await;
let candidate_entry = store.load_candidate_entry(&candidate_hash1).unwrap().unwrap();
let our_assignment =
candidate_entry.approval_entry(&block_hash).unwrap().our_assignment().unwrap();
assert!(our_assignment.triggered());
handle_approval_on_max_wait_time(
&mut virtual_overseer,
vec![candidate_index1, candidate_index2],
clock,
)
.await;
virtual_overseer
});
}