pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-06-14 13:21:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Approve multiple candidates with a single signature (#1191)

Browse Source 
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,

## Overall idea

When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.

This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:

- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.

## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.

## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor @burdges, let me know what you think.
- [x] More and more testing.
- [x]  Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742
- [x] Final versi burn-in before merging

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
This commit is contained in:
Alexandru Gheorghe
2023-12-13 08:43:15 +02:00
committed by GitHub
parent d18a682bf7
commit a84dd0dba5
82 changed files with 5883 additions and 1483 deletions
Download Patch File Download Diff File Expand all files Collapse all files
polkadot/node/core/approval-voting/src/approval_checking.rs
+129 -83
View File
@@ -25,6 +25,15 @@ use crate::{
time::Tick,
};
/// Result of counting the necessary tranches needed for approving a block.
#[derive(Debug, PartialEq, Clone)]
pub struct TranchesToApproveResult {
/// The required tranches for approving this block
pub required_tranches: RequiredTranches,
/// The total number of no_shows at the moment we are doing the counting.
pub total_observed_no_shows: usize,
}
/// The required tranches of assignments needed to determine whether a candidate is approved.
#[derive(Debug, PartialEq, Clone)]
pub enum RequiredTranches {
@@ -64,7 +73,7 @@ pub enum RequiredTranches {
}
/// The result of a check.
#[derive(Debug, Clone, Copy)]
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Check {
/// The candidate is unapproved.
Unapproved,
@@ -178,6 +187,7 @@ struct State {
next_no_show: Option<Tick>,
/// The last tick at which a considered assignment was received.
last_assignment_tick: Option<Tick>,
total_observed_no_shows: usize,
}
impl State {
@@ -187,41 +197,53 @@ impl State {
needed_approvals: usize,
n_validators: usize,
no_show_duration: Tick,
) -> RequiredTranches {
) -> TranchesToApproveResult {
let covering = if self.depth == 0 { 0 } else { self.covering };
if self.depth != 0 && self.assignments + covering + self.uncovered >= n_validators {
return RequiredTranches::All
return TranchesToApproveResult {
required_tranches: RequiredTranches::All,
total_observed_no_shows: self.total_observed_no_shows,
}
}
// If we have enough assignments and all no-shows are covered, we have reached the number
// of tranches that we need to have.
if self.assignments >= needed_approvals && (covering + self.uncovered) == 0 {
return RequiredTranches::Exact {
needed: tranche,
tolerated_missing: self.covered,
next_no_show: self.next_no_show,
last_assignment_tick: self.last_assignment_tick,
return TranchesToApproveResult {
required_tranches: RequiredTranches::Exact {
needed: tranche,
tolerated_missing: self.covered,
next_no_show: self.next_no_show,
last_assignment_tick: self.last_assignment_tick,
},
total_observed_no_shows: self.total_observed_no_shows,
}
}
// We're pending more assignments and should look at more tranches.
let clock_drift = self.clock_drift(no_show_duration);
if self.depth == 0 {
RequiredTranches::Pending {
considered: tranche,
next_no_show: self.next_no_show,
// during the initial assignment-gathering phase, we want to accept assignments
// from any tranche. Note that honest validators will still not broadcast their
// assignment until it is time to do so, regardless of this value.
maximum_broadcast: DelayTranche::max_value(),
clock_drift,
TranchesToApproveResult {
required_tranches: RequiredTranches::Pending {
considered: tranche,
next_no_show: self.next_no_show,
// during the initial assignment-gathering phase, we want to accept assignments
// from any tranche. Note that honest validators will still not broadcast their
// assignment until it is time to do so, regardless of this value.
maximum_broadcast: DelayTranche::max_value(),
clock_drift,
},
total_observed_no_shows: self.total_observed_no_shows,
}
} else {
RequiredTranches::Pending {
considered: tranche,
next_no_show: self.next_no_show,
maximum_broadcast: tranche + (covering + self.uncovered) as DelayTranche,
clock_drift,
TranchesToApproveResult {
required_tranches: RequiredTranches::Pending {
considered: tranche,
next_no_show: self.next_no_show,
maximum_broadcast: tranche + (covering + self.uncovered) as DelayTranche,
clock_drift,
},
total_observed_no_shows: self.total_observed_no_shows,
}
}
}
@@ -276,6 +298,7 @@ impl State {
uncovered,
next_no_show,
last_assignment_tick,
total_observed_no_shows: self.total_observed_no_shows + new_no_shows,
}
}
}
@@ -372,7 +395,7 @@ pub fn tranches_to_approve(
block_tick: Tick,
no_show_duration: Tick,
needed_approvals: usize,
) -> RequiredTranches {
) -> TranchesToApproveResult {
let tick_now = tranche_now as Tick + block_tick;
let n_validators = approval_entry.n_validators();
@@ -384,6 +407,7 @@ pub fn tranches_to_approve(
uncovered: 0,
next_no_show: None,
last_assignment_tick: None,
total_observed_no_shows: 0,
};
// The `ApprovalEntry` doesn't have any data for empty tranches. We still want to iterate over
@@ -434,7 +458,7 @@ pub fn tranches_to_approve(
let s = s.advance(n_assignments, no_shows, next_no_show, last_assignment_tick);
let output = s.output(tranche, needed_approvals, n_validators, no_show_duration);
*state = match output {
*state = match output.required_tranches {
RequiredTranches::Exact { .. } | RequiredTranches::All => {
// Wipe the state clean so the next iteration of this closure will terminate
// the iterator. This guarantees that we can call `last` further down to see
@@ -464,15 +488,17 @@ mod tests {
#[test]
fn pending_is_not_approved() {
let candidate = approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: BitVec::default(),
}
.into();
let candidate = CandidateEntry::from_v1(
approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: BitVec::default(),
},
0,
);
let approval_entry = approval_db::v2::ApprovalEntry {
let approval_entry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: BitVec::default(),
our_assignment: None,
@@ -497,29 +523,31 @@ mod tests {
#[test]
fn exact_takes_only_assignments_up_to() {
let mut candidate: CandidateEntry = approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 10],
}
.into();
let mut candidate: CandidateEntry = CandidateEntry::from_v1(
approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 10],
},
0,
);
for i in 0..3 {
candidate.mark_approval(ValidatorIndex(i));
}
let approval_entry = approval_db::v2::ApprovalEntry {
let approval_entry = approval_db::v3::ApprovalEntry {
tranches: vec![
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 0,
assignments: (0..2).map(|i| (ValidatorIndex(i), 0.into())).collect(),
},
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 1,
assignments: (2..5).map(|i| (ValidatorIndex(i), 1.into())).collect(),
},
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 2,
assignments: (5..10).map(|i| (ValidatorIndex(i), 0.into())).collect(),
},
@@ -569,29 +597,31 @@ mod tests {
#[test]
fn one_honest_node_always_approves() {
let mut candidate: CandidateEntry = approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 10],
}
.into();
let mut candidate: CandidateEntry = CandidateEntry::from_v1(
approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 10],
},
0,
);
for i in 0..3 {
candidate.mark_approval(ValidatorIndex(i));
}
let approval_entry = approval_db::v2::ApprovalEntry {
let approval_entry = approval_db::v3::ApprovalEntry {
tranches: vec![
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 0,
assignments: (0..4).map(|i| (ValidatorIndex(i), 0.into())).collect(),
},
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 1,
assignments: (4..6).map(|i| (ValidatorIndex(i), 1.into())).collect(),
},
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 2,
assignments: (6..10).map(|i| (ValidatorIndex(i), 0.into())).collect(),
},
@@ -647,7 +677,7 @@ mod tests {
let no_show_duration = 10;
let needed_approvals = 4;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; 5],
our_assignment: None,
@@ -675,7 +705,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Exact {
needed: 1,
tolerated_missing: 0,
@@ -691,7 +722,7 @@ mod tests {
let no_show_duration = 10;
let needed_approvals = 4;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; 10],
our_assignment: None,
@@ -715,7 +746,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 2,
next_no_show: Some(block_tick + no_show_duration),
@@ -731,7 +763,7 @@ mod tests {
let no_show_duration = 10;
let needed_approvals = 4;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; 10],
our_assignment: None,
@@ -759,7 +791,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 11,
next_no_show: None,
@@ -776,7 +809,7 @@ mod tests {
let needed_approvals = 4;
let n_validators = 8;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; n_validators],
our_assignment: None,
@@ -807,7 +840,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 1,
next_no_show: None,
@@ -826,7 +860,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 1,
next_no_show: None,
@@ -843,7 +878,7 @@ mod tests {
let needed_approvals = 4;
let n_validators = 8;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; n_validators],
our_assignment: None,
@@ -879,7 +914,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Exact {
needed: 1,
tolerated_missing: 0,
@@ -898,7 +934,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Exact {
needed: 2,
tolerated_missing: 1,
@@ -917,7 +954,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 2,
next_no_show: None,
@@ -934,7 +972,7 @@ mod tests {
let needed_approvals = 4;
let n_validators = 8;
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
assigned_validators: bitvec![u8, BitOrderLsb0; 0; n_validators],
our_assignment: None,
@@ -970,7 +1008,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Exact {
needed: 2,
tolerated_missing: 1,
@@ -992,7 +1031,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 2,
next_no_show: None,
@@ -1013,7 +1053,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Exact {
needed: 3,
tolerated_missing: 2,
@@ -1029,22 +1070,24 @@ mod tests {
let no_show_duration = 10;
let needed_approvals = 3;
let mut candidate: CandidateEntry = approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 3],
}
.into();
let mut candidate: CandidateEntry = CandidateEntry::from_v1(
approval_db::v1::CandidateEntry {
candidate: dummy_candidate_receipt(dummy_hash()),
session: 0,
block_assignments: BTreeMap::default(),
approvals: bitvec![u8, BitOrderLsb0; 0; 3],
},
0,
);
for i in 0..3 {
candidate.mark_approval(ValidatorIndex(i));
}
let approval_entry = approval_db::v2::ApprovalEntry {
let approval_entry = approval_db::v3::ApprovalEntry {
tranches: vec![
// Assignments with invalid validator indexes.
approval_db::v2::TrancheEntry {
approval_db::v3::TrancheEntry {
tranche: 1,
assignments: (2..5).map(|i| (ValidatorIndex(i), 1.into())).collect(),
},
@@ -1068,7 +1111,8 @@ mod tests {
block_tick,
no_show_duration,
needed_approvals,
),
)
.required_tranches,
RequiredTranches::Pending {
considered: 10,
next_no_show: None,
@@ -1094,7 +1138,7 @@ mod tests {
];
for test_tranche in test_tranches {
let mut approval_entry: ApprovalEntry = approval_db::v2::ApprovalEntry {
let mut approval_entry: ApprovalEntry = approval_db::v3::ApprovalEntry {
tranches: Vec::new(),
backing_group: GroupIndex(0),
our_assignment: None,
@@ -1345,10 +1389,11 @@ mod tests {
uncovered: 0,
next_no_show: None,
last_assignment_tick: None,
total_observed_no_shows: 0,
};
assert_eq!(
state.output(0, 10, 10, 20),
state.output(0, 10, 10, 20).required_tranches,
RequiredTranches::Pending {
considered: 0,
next_no_show: None,
@@ -1368,10 +1413,11 @@ mod tests {
uncovered: 0,
next_no_show: None,
last_assignment_tick: None,
total_observed_no_shows: 0,
};
assert_eq!(
state.output(0, 10, 10, 20),
state.output(0, 10, 10, 20).required_tranches,
RequiredTranches::Exact {
needed: 0,
tolerated_missing: 0,
polkadot/node/core/approval-voting/src/approval_db/common/migration_helpers.rs
+39
View File
@@ -0,0 +1,39 @@
// Copyright (C) Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
use bitvec::{order::Lsb0 as BitOrderLsb0, vec::BitVec};
use polkadot_node_primitives::approval::{
v1::{AssignmentCert, AssignmentCertKind, VrfProof, VrfSignature, RELAY_VRF_MODULO_CONTEXT},
v2::VrfPreOutput,
};
pub fn make_bitvec(len: usize) -> BitVec<u8, BitOrderLsb0> {
bitvec::bitvec![u8, BitOrderLsb0; 0; len]
}
pub fn dummy_assignment_cert(kind: AssignmentCertKind) -> AssignmentCert {
let ctx = schnorrkel::signing_context(RELAY_VRF_MODULO_CONTEXT);
let msg = b"test-garbage";
let mut prng = rand_core::OsRng;
let keypair = schnorrkel::Keypair::generate_with(&mut prng);
let (inout, proof, _) = keypair.vrf_sign(ctx.bytes(msg));
let out = inout.to_output();
AssignmentCert {
kind,
vrf: VrfSignature { pre_output: VrfPreOutput(out), proof: VrfProof(proof) },
}
}
polkadot/node/core/approval-voting/src/approval_db/common/mod.rs
+293
View File
@@ -0,0 +1,293 @@
// Copyright (C) Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Common helper functions for all versions of approval-voting database.
use std::sync::Arc;
use parity_scale_codec::{Decode, Encode};
use polkadot_node_subsystem::{SubsystemError, SubsystemResult};
use polkadot_node_subsystem_util::database::{DBTransaction, Database};
use polkadot_primitives::{BlockNumber, CandidateHash, CandidateIndex, Hash};
use crate::{
backend::{Backend, BackendWriteOp, V1ReadBackend, V2ReadBackend},
persisted_entries,
};
use super::{
v2::{load_block_entry_v1, load_candidate_entry_v1},
v3::{load_block_entry_v2, load_candidate_entry_v2, BlockEntry, CandidateEntry},
};
pub mod migration_helpers;
const STORED_BLOCKS_KEY: &[u8] = b"Approvals_StoredBlocks";
/// A range from earliest..last block number stored within the DB.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct StoredBlockRange(pub BlockNumber, pub BlockNumber);
/// The database config.
#[derive(Debug, Clone, Copy)]
pub struct Config {
/// The column family in the database where data is stored.
pub col_approval_data: u32,
}
/// `DbBackend` is a concrete implementation of the higher-level Backend trait
pub struct DbBackend {
inner: Arc<dyn Database>,
config: Config,
}
impl DbBackend {
/// Create a new [`DbBackend`] with the supplied key-value store and
/// config.
pub fn new(db: Arc<dyn Database>, config: Config) -> Self {
DbBackend { inner: db, config }
}
}
/// Errors while accessing things from the DB.
#[derive(Debug, derive_more::From, derive_more::Display)]
pub enum Error {
Io(std::io::Error),
InvalidDecoding(parity_scale_codec::Error),
InternalError(SubsystemError),
}
impl std::error::Error for Error {}
/// Result alias for DB errors.
pub type Result<T> = std::result::Result<T, Error>;
impl Backend for DbBackend {
fn load_block_entry(
&self,
block_hash: &Hash,
) -> SubsystemResult<Option<persisted_entries::BlockEntry>> {
load_block_entry(&*self.inner, &self.config, block_hash).map(|e| e.map(Into::into))
}
fn load_candidate_entry(
&self,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<persisted_entries::CandidateEntry>> {
load_candidate_entry(&*self.inner, &self.config, candidate_hash).map(|e| e.map(Into::into))
}
fn load_blocks_at_height(&self, block_height: &BlockNumber) -> SubsystemResult<Vec<Hash>> {
load_blocks_at_height(&*self.inner, &self.config, block_height)
}
fn load_all_blocks(&self) -> SubsystemResult<Vec<Hash>> {
load_all_blocks(&*self.inner, &self.config)
}
fn load_stored_blocks(&self) -> SubsystemResult<Option<StoredBlockRange>> {
load_stored_blocks(&*self.inner, &self.config)
}
/// Atomically write the list of operations, with later operations taking precedence over prior.
fn write<I>(&mut self, ops: I) -> SubsystemResult<()>
where
I: IntoIterator<Item = BackendWriteOp>,
{
let mut tx = DBTransaction::new();
for op in ops {
match op {
BackendWriteOp::WriteStoredBlockRange(stored_block_range) => {
tx.put_vec(
self.config.col_approval_data,
&STORED_BLOCKS_KEY,
stored_block_range.encode(),
);
},
BackendWriteOp::DeleteStoredBlockRange => {
tx.delete(self.config.col_approval_data, &STORED_BLOCKS_KEY);
},
BackendWriteOp::WriteBlocksAtHeight(h, blocks) => {
tx.put_vec(
self.config.col_approval_data,
&blocks_at_height_key(h),
blocks.encode(),
);
},
BackendWriteOp::DeleteBlocksAtHeight(h) => {
tx.delete(self.config.col_approval_data, &blocks_at_height_key(h));
},
BackendWriteOp::WriteBlockEntry(block_entry) => {
let block_entry: BlockEntry = block_entry.into();
tx.put_vec(
self.config.col_approval_data,
&block_entry_key(&block_entry.block_hash),
block_entry.encode(),
);
},
BackendWriteOp::DeleteBlockEntry(hash) => {
tx.delete(self.config.col_approval_data, &block_entry_key(&hash));
},
BackendWriteOp::WriteCandidateEntry(candidate_entry) => {
let candidate_entry: CandidateEntry = candidate_entry.into();
tx.put_vec(
self.config.col_approval_data,
&candidate_entry_key(&candidate_entry.candidate.hash()),
candidate_entry.encode(),
);
},
BackendWriteOp::DeleteCandidateEntry(candidate_hash) => {
tx.delete(self.config.col_approval_data, &candidate_entry_key(&candidate_hash));
},
}
}
self.inner.write(tx).map_err(|e| e.into())
}
}
impl V1ReadBackend for DbBackend {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
candidate_index: CandidateIndex,
) -> SubsystemResult<Option<persisted_entries::CandidateEntry>> {
load_candidate_entry_v1(&*self.inner, &self.config, candidate_hash)
.map(|e| e.map(|e| persisted_entries::CandidateEntry::from_v1(e, candidate_index)))
}
fn load_block_entry_v1(
&self,
block_hash: &Hash,
) -> SubsystemResult<Option<persisted_entries::BlockEntry>> {
load_block_entry_v1(&*self.inner, &self.config, block_hash).map(|e| e.map(Into::into))
}
}
impl V2ReadBackend for DbBackend {
fn load_candidate_entry_v2(
&self,
candidate_hash: &CandidateHash,
candidate_index: CandidateIndex,
) -> SubsystemResult<Option<persisted_entries::CandidateEntry>> {
load_candidate_entry_v2(&*self.inner, &self.config, candidate_hash)
.map(|e| e.map(|e| persisted_entries::CandidateEntry::from_v2(e, candidate_index)))
}
fn load_block_entry_v2(
&self,
block_hash: &Hash,
) -> SubsystemResult<Option<persisted_entries::BlockEntry>> {
load_block_entry_v2(&*self.inner, &self.config, block_hash).map(|e| e.map(Into::into))
}
}
pub(crate) fn load_decode<D: Decode>(
store: &dyn Database,
col_approval_data: u32,
key: &[u8],
) -> Result<Option<D>> {
match store.get(col_approval_data, key)? {
None => Ok(None),
Some(raw) => D::decode(&mut &raw[..]).map(Some).map_err(Into::into),
}
}
/// The key a given block entry is stored under.
pub(crate) fn block_entry_key(block_hash: &Hash) -> [u8; 46] {
const BLOCK_ENTRY_PREFIX: [u8; 14] = *b"Approvals_blck";
let mut key = [0u8; 14 + 32];
key[0..14].copy_from_slice(&BLOCK_ENTRY_PREFIX);
key[14..][..32].copy_from_slice(block_hash.as_ref());
key
}
/// The key a given candidate entry is stored under.
pub(crate) fn candidate_entry_key(candidate_hash: &CandidateHash) -> [u8; 46] {
const CANDIDATE_ENTRY_PREFIX: [u8; 14] = *b"Approvals_cand";
let mut key = [0u8; 14 + 32];
key[0..14].copy_from_slice(&CANDIDATE_ENTRY_PREFIX);
key[14..][..32].copy_from_slice(candidate_hash.0.as_ref());
key
}
/// The key a set of block hashes corresponding to a block number is stored under.
pub(crate) fn blocks_at_height_key(block_number: BlockNumber) -> [u8; 16] {
const BLOCKS_AT_HEIGHT_PREFIX: [u8; 12] = *b"Approvals_at";
let mut key = [0u8; 12 + 4];
key[0..12].copy_from_slice(&BLOCKS_AT_HEIGHT_PREFIX);
block_number.using_encoded(|s| key[12..16].copy_from_slice(s));
key
}
/// Return all blocks which have entries in the DB, ascending, by height.
pub fn load_all_blocks(store: &dyn Database, config: &Config) -> SubsystemResult<Vec<Hash>> {
let mut hashes = Vec::new();
if let Some(stored_blocks) = load_stored_blocks(store, config)? {
for height in stored_blocks.0..stored_blocks.1 {
let blocks = load_blocks_at_height(store, config, &height)?;
hashes.extend(blocks);
}
}
Ok(hashes)
}
/// Load the stored-blocks key from the state.
pub fn load_stored_blocks(
store: &dyn Database,
config: &Config,
) -> SubsystemResult<Option<StoredBlockRange>> {
load_decode(store, config.col_approval_data, STORED_BLOCKS_KEY)
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a blocks-at-height entry for a given block number.
pub fn load_blocks_at_height(
store: &dyn Database,
config: &Config,
block_number: &BlockNumber,
) -> SubsystemResult<Vec<Hash>> {
load_decode(store, config.col_approval_data, &blocks_at_height_key(*block_number))
.map(|x| x.unwrap_or_default())
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a block entry from the aux store.
pub fn load_block_entry(
store: &dyn Database,
config: &Config,
block_hash: &Hash,
) -> SubsystemResult<Option<BlockEntry>> {
load_decode(store, config.col_approval_data, &block_entry_key(block_hash))
.map(|u: Option<BlockEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a candidate entry from the aux store in current version format.
pub fn load_candidate_entry(
store: &dyn Database,
config: &Config,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<CandidateEntry>> {
load_decode(store, config.col_approval_data, &candidate_entry_key(candidate_hash))
.map(|u: Option<CandidateEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
polkadot/node/core/approval-voting/src/approval_db/mod.rs
+2
View File
@@ -30,5 +30,7 @@
//! In the future, we may use a temporary DB which doesn't need to be wiped, but for the
//! time being we share the same DB with the rest of Substrate.
pub mod common;
pub mod v1;
pub mod v2;
pub mod v3;
polkadot/node/core/approval-voting/src/approval_db/v1/tests.rs
-4
View File
@@ -40,10 +40,6 @@ fn make_db() -> (DbBackend, Arc<dyn Database>) {
(DbBackend::new(db_writer.clone(), TEST_CONFIG), db_writer)
}
fn make_bitvec(len: usize) -> BitVec<u8, BitOrderLsb0> {
bitvec::bitvec![u8, BitOrderLsb0; 0; len]
}
fn make_block_entry(
block_hash: Hash,
parent_hash: Hash,
polkadot/node/core/approval-voting/src/approval_db/v2/migration_helpers.rs
+14 -60
View File
@@ -16,29 +16,19 @@
//! Approval DB migration helpers.
use super::*;
use crate::backend::Backend;
use polkadot_node_primitives::approval::v1::{
AssignmentCert, AssignmentCertKind, VrfPreOutput, VrfProof, VrfSignature,
RELAY_VRF_MODULO_CONTEXT,
use crate::{
approval_db::common::{
migration_helpers::{dummy_assignment_cert, make_bitvec},
Error, Result, StoredBlockRange,
},
backend::Backend,
};
use polkadot_node_primitives::approval::v1::AssignmentCertKind;
use polkadot_node_subsystem_util::database::Database;
use sp_application_crypto::sp_core::H256;
use std::{collections::HashSet, sync::Arc};
fn dummy_assignment_cert(kind: AssignmentCertKind) -> AssignmentCert {
let ctx = schnorrkel::signing_context(RELAY_VRF_MODULO_CONTEXT);
let msg = b"test-garbage";
let mut prng = rand_core::OsRng;
let keypair = schnorrkel::Keypair::generate_with(&mut prng);
let (inout, proof, _) = keypair.vrf_sign(ctx.bytes(msg));
let preout = inout.to_output();
AssignmentCert {
kind,
vrf: VrfSignature { pre_output: VrfPreOutput(preout), proof: VrfProof(proof) },
}
}
fn make_block_entry_v1(
block_hash: Hash,
parent_hash: Hash,
@@ -58,14 +48,10 @@ fn make_block_entry_v1(
}
}
fn make_bitvec(len: usize) -> BitVec<u8, BitOrderLsb0> {
bitvec::bitvec![u8, BitOrderLsb0; 0; len]
}
/// Migrates `OurAssignment`, `CandidateEntry` and `ApprovalEntry` to version 2.
/// Returns on any error.
/// Must only be used in parachains DB migration code - `polkadot-service` crate.
pub fn v1_to_v2(db: Arc<dyn Database>, config: Config) -> Result<()> {
pub fn v1_to_latest(db: Arc<dyn Database>, config: Config) -> Result<()> {
let mut backend = crate::DbBackend::new(db, config);
let all_blocks = backend
.load_all_blocks()
@@ -89,11 +75,13 @@ pub fn v1_to_v2(db: Arc<dyn Database>, config: Config) -> Result<()> {
let mut counter = 0;
// Get all candidate entries, approval entries and convert each of them.
for block in all_blocks {
for (_core_index, candidate_hash) in block.candidates() {
for (candidate_index, (_core_index, candidate_hash)) in
block.candidates().iter().enumerate()
{
// Loading the candidate will also perform the conversion to the updated format and
// return that represantation.
if let Some(candidate_entry) = backend
.load_candidate_entry_v1(&candidate_hash)
.load_candidate_entry_v1(&candidate_hash, candidate_index as CandidateIndex)
.map_err(|e| Error::InternalError(e))?
{
// Write the updated representation.
@@ -113,42 +101,8 @@ pub fn v1_to_v2(db: Arc<dyn Database>, config: Config) -> Result<()> {
Ok(())
}
// Checks if the migration doesn't leave the DB in an unsane state.
// This function is to be used in tests.
pub fn v1_to_v2_sanity_check(
db: Arc<dyn Database>,
config: Config,
expected_candidates: HashSet<CandidateHash>,
) -> Result<()> {
let backend = crate::DbBackend::new(db, config);
let all_blocks = backend
.load_all_blocks()
.unwrap()
.iter()
.map(|block_hash| backend.load_block_entry(block_hash).unwrap().unwrap())
.collect::<Vec<_>>();
let mut candidates = HashSet::new();
// Iterate all blocks and approval entries.
for block in all_blocks {
for (_core_index, candidate_hash) in block.candidates() {
// Loading the candidate will also perform the conversion to the updated format and
// return that represantation.
if let Some(candidate_entry) = backend.load_candidate_entry(&candidate_hash).unwrap() {
candidates.insert(candidate_entry.candidate.hash());
}
}
}
assert_eq!(candidates, expected_candidates);
Ok(())
}
// Fills the db with dummy data in v1 scheme.
pub fn v1_to_v2_fill_test_data<F>(
pub fn v1_fill_test_data<F>(
db: Arc<dyn Database>,
config: Config,
dummy_candidate_create: F,
polkadot/node/core/approval-voting/src/approval_db/v2/mod.rs
+5 -246
View File
@@ -21,145 +21,23 @@ use polkadot_node_primitives::approval::{v1::DelayTranche, v2::AssignmentCertV2}
use polkadot_node_subsystem::{SubsystemError, SubsystemResult};
use polkadot_node_subsystem_util::database::{DBTransaction, Database};
use polkadot_primitives::{
BlockNumber, CandidateHash, CandidateReceipt, CoreIndex, GroupIndex, Hash, SessionIndex,
ValidatorIndex, ValidatorSignature,
BlockNumber, CandidateHash, CandidateIndex, CandidateReceipt, CoreIndex, GroupIndex, Hash,
SessionIndex, ValidatorIndex, ValidatorSignature,
};
use sp_consensus_slots::Slot;
use bitvec::{order::Lsb0 as BitOrderLsb0, vec::BitVec};
use std::{collections::BTreeMap, sync::Arc};
use std::collections::BTreeMap;
use crate::{
backend::{Backend, BackendWriteOp, V1ReadBackend},
persisted_entries,
};
use crate::backend::V1ReadBackend;
const STORED_BLOCKS_KEY: &[u8] = b"Approvals_StoredBlocks";
use super::common::{block_entry_key, candidate_entry_key, load_decode, Config};
pub mod migration_helpers;
#[cfg(test)]
pub mod tests;
/// `DbBackend` is a concrete implementation of the higher-level Backend trait
pub struct DbBackend {
inner: Arc<dyn Database>,
config: Config,
}
impl DbBackend {
/// Create a new [`DbBackend`] with the supplied key-value store and
/// config.
pub fn new(db: Arc<dyn Database>, config: Config) -> Self {
DbBackend { inner: db, config }
}
}
impl V1ReadBackend for DbBackend {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<persisted_entries::CandidateEntry>> {
load_candidate_entry_v1(&*self.inner, &self.config, candidate_hash)
.map(|e| e.map(Into::into))
}
fn load_block_entry_v1(
&self,
block_hash: &Hash,
) -> SubsystemResult<Option<persisted_entries::BlockEntry>> {
load_block_entry_v1(&*self.inner, &self.config, block_hash).map(|e| e.map(Into::into))
}
}
impl Backend for DbBackend {
fn load_block_entry(
&self,
block_hash: &Hash,
) -> SubsystemResult<Option<persisted_entries::BlockEntry>> {
load_block_entry(&*self.inner, &self.config, block_hash).map(|e| e.map(Into::into))
}
fn load_candidate_entry(
&self,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<persisted_entries::CandidateEntry>> {
load_candidate_entry(&*self.inner, &self.config, candidate_hash).map(|e| e.map(Into::into))
}
fn load_blocks_at_height(&self, block_height: &BlockNumber) -> SubsystemResult<Vec<Hash>> {
load_blocks_at_height(&*self.inner, &self.config, block_height)
}
fn load_all_blocks(&self) -> SubsystemResult<Vec<Hash>> {
load_all_blocks(&*self.inner, &self.config)
}
fn load_stored_blocks(&self) -> SubsystemResult<Option<StoredBlockRange>> {
load_stored_blocks(&*self.inner, &self.config)
}
/// Atomically write the list of operations, with later operations taking precedence over prior.
fn write<I>(&mut self, ops: I) -> SubsystemResult<()>
where
I: IntoIterator<Item = BackendWriteOp>,
{
let mut tx = DBTransaction::new();
for op in ops {
match op {
BackendWriteOp::WriteStoredBlockRange(stored_block_range) => {
tx.put_vec(
self.config.col_approval_data,
&STORED_BLOCKS_KEY,
stored_block_range.encode(),
);
},
BackendWriteOp::DeleteStoredBlockRange => {
tx.delete(self.config.col_approval_data, &STORED_BLOCKS_KEY);
},
BackendWriteOp::WriteBlocksAtHeight(h, blocks) => {
tx.put_vec(
self.config.col_approval_data,
&blocks_at_height_key(h),
blocks.encode(),
);
},
BackendWriteOp::DeleteBlocksAtHeight(h) => {
tx.delete(self.config.col_approval_data, &blocks_at_height_key(h));
},
BackendWriteOp::WriteBlockEntry(block_entry) => {
let block_entry: BlockEntry = block_entry.into();
tx.put_vec(
self.config.col_approval_data,
&block_entry_key(&block_entry.block_hash),
block_entry.encode(),
);
},
BackendWriteOp::DeleteBlockEntry(hash) => {
tx.delete(self.config.col_approval_data, &block_entry_key(&hash));
},
BackendWriteOp::WriteCandidateEntry(candidate_entry) => {
let candidate_entry: CandidateEntry = candidate_entry.into();
tx.put_vec(
self.config.col_approval_data,
&candidate_entry_key(&candidate_entry.candidate.hash()),
candidate_entry.encode(),
);
},
BackendWriteOp::DeleteCandidateEntry(candidate_hash) => {
tx.delete(self.config.col_approval_data, &candidate_entry_key(&candidate_hash));
},
}
}
self.inner.write(tx).map_err(|e| e.into())
}
}
/// A range from earliest..last block number stored within the DB.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct StoredBlockRange(pub BlockNumber, pub BlockNumber);
// slot_duration * 2 + DelayTranche gives the number of delay tranches since the
// unix epoch.
#[derive(Encode, Decode, Clone, Copy, Debug, PartialEq)]
@@ -168,13 +46,6 @@ pub struct Tick(u64);
/// Convenience type definition
pub type Bitfield = BitVec<u8, BitOrderLsb0>;
/// The database config.
#[derive(Debug, Clone, Copy)]
pub struct Config {
/// The column family in the database where data is stored.
pub col_approval_data: u32,
}
/// Details pertaining to our assignment on a block.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct OurAssignment {
@@ -259,118 +130,6 @@ impl From<Tick> for crate::Tick {
}
}
/// Errors while accessing things from the DB.
#[derive(Debug, derive_more::From, derive_more::Display)]
pub enum Error {
Io(std::io::Error),
InvalidDecoding(parity_scale_codec::Error),
InternalError(SubsystemError),
}
impl std::error::Error for Error {}
/// Result alias for DB errors.
pub type Result<T> = std::result::Result<T, Error>;
pub(crate) fn load_decode<D: Decode>(
store: &dyn Database,
col_approval_data: u32,
key: &[u8],
) -> Result<Option<D>> {
match store.get(col_approval_data, key)? {
None => Ok(None),
Some(raw) => D::decode(&mut &raw[..]).map(Some).map_err(Into::into),
}
}
/// The key a given block entry is stored under.
pub(crate) fn block_entry_key(block_hash: &Hash) -> [u8; 46] {
const BLOCK_ENTRY_PREFIX: [u8; 14] = *b"Approvals_blck";
let mut key = [0u8; 14 + 32];
key[0..14].copy_from_slice(&BLOCK_ENTRY_PREFIX);
key[14..][..32].copy_from_slice(block_hash.as_ref());
key
}
/// The key a given candidate entry is stored under.
pub(crate) fn candidate_entry_key(candidate_hash: &CandidateHash) -> [u8; 46] {
const CANDIDATE_ENTRY_PREFIX: [u8; 14] = *b"Approvals_cand";
let mut key = [0u8; 14 + 32];
key[0..14].copy_from_slice(&CANDIDATE_ENTRY_PREFIX);
key[14..][..32].copy_from_slice(candidate_hash.0.as_ref());
key
}
/// The key a set of block hashes corresponding to a block number is stored under.
pub(crate) fn blocks_at_height_key(block_number: BlockNumber) -> [u8; 16] {
const BLOCKS_AT_HEIGHT_PREFIX: [u8; 12] = *b"Approvals_at";
let mut key = [0u8; 12 + 4];
key[0..12].copy_from_slice(&BLOCKS_AT_HEIGHT_PREFIX);
block_number.using_encoded(|s| key[12..16].copy_from_slice(s));
key
}
/// Return all blocks which have entries in the DB, ascending, by height.
pub fn load_all_blocks(store: &dyn Database, config: &Config) -> SubsystemResult<Vec<Hash>> {
let mut hashes = Vec::new();
if let Some(stored_blocks) = load_stored_blocks(store, config)? {
for height in stored_blocks.0..stored_blocks.1 {
let blocks = load_blocks_at_height(store, config, &height)?;
hashes.extend(blocks);
}
}
Ok(hashes)
}
/// Load the stored-blocks key from the state.
pub fn load_stored_blocks(
store: &dyn Database,
config: &Config,
) -> SubsystemResult<Option<StoredBlockRange>> {
load_decode(store, config.col_approval_data, STORED_BLOCKS_KEY)
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a blocks-at-height entry for a given block number.
pub fn load_blocks_at_height(
store: &dyn Database,
config: &Config,
block_number: &BlockNumber,
) -> SubsystemResult<Vec<Hash>> {
load_decode(store, config.col_approval_data, &blocks_at_height_key(*block_number))
.map(|x| x.unwrap_or_default())
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a block entry from the aux store.
pub fn load_block_entry(
store: &dyn Database,
config: &Config,
block_hash: &Hash,
) -> SubsystemResult<Option<BlockEntry>> {
load_decode(store, config.col_approval_data, &block_entry_key(block_hash))
.map(|u: Option<BlockEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a candidate entry from the aux store in current version format.
pub fn load_candidate_entry(
store: &dyn Database,
config: &Config,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<CandidateEntry>> {
load_decode(store, config.col_approval_data, &candidate_entry_key(candidate_hash))
.map(|u: Option<CandidateEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a candidate entry from the aux store in v1 format.
pub fn load_candidate_entry_v1(
store: &dyn Database,
polkadot/node/core/approval-voting/src/approval_db/v2/tests.rs
+40 -28
View File
@@ -16,13 +16,22 @@
//! Tests for the aux-schema of approval voting.
use super::{DbBackend, StoredBlockRange, *};
use crate::{
approval_db::{
common::{migration_helpers::make_bitvec, DbBackend, StoredBlockRange, *},
v2::*,
v3::{load_block_entry_v2, load_candidate_entry_v2},
},
backend::{Backend, OverlayedBackend},
ops::{add_block_entry, canonicalize, force_approve, NewCandidateInfo},
};
use polkadot_primitives::{
BlockNumber, CandidateHash, CandidateReceipt, CoreIndex, GroupIndex, Hash,
};
use polkadot_node_subsystem_util::database::Database;
use polkadot_primitives::Id as ParaId;
use sp_consensus_slots::Slot;
use std::{collections::HashMap, sync::Arc};
use ::test_helpers::{dummy_candidate_receipt, dummy_candidate_receipt_bad_sig, dummy_hash};
@@ -60,10 +69,6 @@ fn make_block_entry(
}
}
fn make_bitvec(len: usize) -> BitVec<u8, BitOrderLsb0> {
bitvec::bitvec![u8, BitOrderLsb0; 0; len]
}
fn make_candidate(para_id: ParaId, relay_parent: Hash) -> CandidateReceipt {
let mut c = dummy_candidate_receipt(dummy_hash());
@@ -110,7 +115,10 @@ fn read_write() {
overlay_db.write_stored_block_range(range.clone());
overlay_db.write_blocks_at_height(1, at_height.clone());
overlay_db.write_block_entry(block_entry.clone().into());
overlay_db.write_candidate_entry(candidate_entry.clone().into());
overlay_db.write_candidate_entry(crate::persisted_entries::CandidateEntry::from_v2(
candidate_entry.clone(),
0,
));
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
@@ -118,11 +126,11 @@ fn read_write() {
assert_eq!(load_stored_blocks(store.as_ref(), &TEST_CONFIG).unwrap(), Some(range));
assert_eq!(load_blocks_at_height(store.as_ref(), &TEST_CONFIG, &1).unwrap(), at_height);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap(),
load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap(),
Some(block_entry.into())
);
assert_eq!(
load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash).unwrap(),
load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &candidate_hash).unwrap(),
Some(candidate_entry.into()),
);
@@ -134,8 +142,8 @@ fn read_write() {
db.write(write_ops).unwrap();
assert!(load_blocks_at_height(store.as_ref(), &TEST_CONFIG, &1).unwrap().is_empty());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap().is_none());
assert!(load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap().is_none());
assert!(load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &candidate_hash)
.unwrap()
.is_none());
}
@@ -196,25 +204,27 @@ fn add_block_entry_works() {
db.write(write_ops).unwrap();
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
Some(block_entry_a.into())
);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
Some(block_entry_b.into())
);
let candidate_entry_a = load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash_a)
.unwrap()
.unwrap();
let candidate_entry_a =
load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &candidate_hash_a)
.unwrap()
.unwrap();
assert_eq!(
candidate_entry_a.block_assignments.keys().collect::<Vec<_>>(),
vec![&block_hash_a, &block_hash_b]
);
let candidate_entry_b = load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash_b)
.unwrap()
.unwrap();
let candidate_entry_b =
load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &candidate_hash_b)
.unwrap()
.unwrap();
assert_eq!(candidate_entry_b.block_assignments.keys().collect::<Vec<_>>(), vec![&block_hash_b]);
}
@@ -243,11 +253,11 @@ fn add_block_entry_adds_child() {
block_entry_a.children.push(block_hash_b);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
Some(block_entry_a.into())
);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
Some(block_entry_b.into())
);
}
@@ -365,13 +375,15 @@ fn canonicalize_works() {
for (c_hash, in_blocks) in expected {
let (entry, in_blocks) = match in_blocks {
None => {
assert!(load_candidate_entry(store.as_ref(), &TEST_CONFIG, &c_hash)
assert!(load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &c_hash)
.unwrap()
.is_none());
continue
},
Some(i) => (
load_candidate_entry(store.as_ref(), &TEST_CONFIG, &c_hash).unwrap().unwrap(),
load_candidate_entry_v2(store.as_ref(), &TEST_CONFIG, &c_hash)
.unwrap()
.unwrap(),
i,
),
};
@@ -388,13 +400,13 @@ fn canonicalize_works() {
for (hash, with_candidates) in expected {
let (entry, with_candidates) = match with_candidates {
None => {
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &hash)
.unwrap()
.is_none());
continue
},
Some(i) =>
(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash).unwrap().unwrap(), i),
(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &hash).unwrap().unwrap(), i),
};
assert_eq!(entry.candidates.len(), with_candidates.len());
@@ -510,22 +522,22 @@ fn force_approve_works() {
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a,)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_a,)
.unwrap()
.unwrap()
.approved_bitfield
.all());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b,)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_b,)
.unwrap()
.unwrap()
.approved_bitfield
.all());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_c,)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_c,)
.unwrap()
.unwrap()
.approved_bitfield
.not_any());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_d,)
assert!(load_block_entry_v2(store.as_ref(), &TEST_CONFIG, &block_hash_d,)
.unwrap()
.unwrap()
.approved_bitfield
polkadot/node/core/approval-voting/src/approval_db/v3/migration_helpers.rs
+237
View File
@@ -0,0 +1,237 @@
// Copyright (C) Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Approval DB migration helpers.
use super::*;
use crate::{
approval_db::common::{
block_entry_key, candidate_entry_key,
migration_helpers::{dummy_assignment_cert, make_bitvec},
Config, Error, Result, StoredBlockRange,
},
backend::{Backend, V2ReadBackend},
};
use polkadot_node_primitives::approval::v1::AssignmentCertKind;
use polkadot_node_subsystem_util::database::Database;
use sp_application_crypto::sp_core::H256;
use std::{collections::HashSet, sync::Arc};
/// Migrates `BlockEntry`, `CandidateEntry`, `ApprovalEntry` and `OurApproval` to version 3.
/// Returns on any error.
/// Must only be used in parachains DB migration code - `polkadot-service` crate.
pub fn v2_to_latest(db: Arc<dyn Database>, config: Config) -> Result<()> {
let mut backend = crate::DbBackend::new(db, config);
let all_blocks = backend
.load_all_blocks()
.map_err(|e| Error::InternalError(e))?
.iter()
.filter_map(|block_hash| {
backend
.load_block_entry_v2(block_hash)
.map_err(|e| Error::InternalError(e))
.ok()?
})
.collect::<Vec<_>>();
gum::info!(
target: crate::LOG_TARGET,
"Migrating candidate entries on top of {} blocks",
all_blocks.len()
);
let mut overlay = crate::OverlayedBackend::new(&backend);
let mut counter = 0;
// Get all candidate entries, approval entries and convert each of them.
for block in all_blocks {
for (candidate_index, (_core_index, candidate_hash)) in
block.candidates().iter().enumerate()
{
// Loading the candidate will also perform the conversion to the updated format and
// return that represantation.
if let Some(candidate_entry) = backend
.load_candidate_entry_v2(&candidate_hash, candidate_index as CandidateIndex)
.map_err(|e| Error::InternalError(e))?
{
// Write the updated representation.
overlay.write_candidate_entry(candidate_entry);
counter += 1;
}
}
overlay.write_block_entry(block);
}
gum::info!(target: crate::LOG_TARGET, "Migrated {} entries", counter);
// Commit all changes to DB.
let write_ops = overlay.into_write_ops();
backend.write(write_ops).unwrap();
Ok(())
}
// Checks if the migration doesn't leave the DB in an unsane state.
// This function is to be used in tests.
pub fn v1_to_latest_sanity_check(
db: Arc<dyn Database>,
config: Config,
expected_candidates: HashSet<CandidateHash>,
) -> Result<()> {
let backend = crate::DbBackend::new(db, config);
let all_blocks = backend
.load_all_blocks()
.unwrap()
.iter()
.map(|block_hash| backend.load_block_entry(block_hash).unwrap().unwrap())
.collect::<Vec<_>>();
let mut candidates = HashSet::new();
// Iterate all blocks and approval entries.
for block in all_blocks {
for (_core_index, candidate_hash) in block.candidates() {
// Loading the candidate will also perform the conversion to the updated format and
// return that represantation.
if let Some(candidate_entry) = backend.load_candidate_entry(&candidate_hash).unwrap() {
candidates.insert(candidate_entry.candidate.hash());
}
}
}
assert_eq!(candidates, expected_candidates);
Ok(())
}
// Fills the db with dummy data in v2 scheme.
pub fn v2_fill_test_data<F>(
db: Arc<dyn Database>,
config: Config,
dummy_candidate_create: F,
) -> Result<HashSet<CandidateHash>>
where
F: Fn(H256) -> CandidateReceipt<H256>,
{
let mut backend = crate::DbBackend::new(db.clone(), config);
let mut overlay_db = crate::OverlayedBackend::new(&backend);
let mut expected_candidates = HashSet::new();
const RELAY_BLOCK_COUNT: u32 = 10;
let range = StoredBlockRange(1, 11);
overlay_db.write_stored_block_range(range.clone());
for relay_number in 1..=RELAY_BLOCK_COUNT {
let relay_hash = Hash::repeat_byte(relay_number as u8);
let assignment_core_index = CoreIndex(relay_number);
let candidate = dummy_candidate_create(relay_hash);
let candidate_hash = candidate.hash();
let at_height = vec![relay_hash];
let block_entry = make_block_entry_v2(
relay_hash,
Default::default(),
relay_number,
vec![(assignment_core_index, candidate_hash)],
);
let dummy_assignment = crate::approval_db::v2::OurAssignment {
cert: dummy_assignment_cert(AssignmentCertKind::RelayVRFModulo { sample: 0 }).into(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
};
let candidate_entry = crate::approval_db::v2::CandidateEntry {
candidate,
session: 123,
block_assignments: vec![(
relay_hash,
crate::approval_db::v2::ApprovalEntry {
tranches: Vec::new(),
backing_group: GroupIndex(1),
our_assignment: Some(dummy_assignment),
our_approval_sig: None,
approved: false,
assigned_validators: make_bitvec(1),
},
)]
.into_iter()
.collect(),
approvals: Default::default(),
};
overlay_db.write_blocks_at_height(relay_number, at_height.clone());
expected_candidates.insert(candidate_entry.candidate.hash());
db.write(write_candidate_entry_v2(candidate_entry, config)).unwrap();
db.write(write_block_entry_v2(block_entry, config)).unwrap();
}
let write_ops = overlay_db.into_write_ops();
backend.write(write_ops).unwrap();
Ok(expected_candidates)
}
fn make_block_entry_v2(
block_hash: Hash,
parent_hash: Hash,
block_number: BlockNumber,
candidates: Vec<(CoreIndex, CandidateHash)>,
) -> crate::approval_db::v2::BlockEntry {
crate::approval_db::v2::BlockEntry {
block_hash,
parent_hash,
block_number,
session: 1,
slot: Slot::from(1),
relay_vrf_story: [0u8; 32],
approved_bitfield: make_bitvec(candidates.len()),
distributed_assignments: make_bitvec(candidates.len()),
candidates,
children: Vec::new(),
}
}
// Low level DB helper to write a candidate entry in v1 scheme.
fn write_candidate_entry_v2(
candidate_entry: crate::approval_db::v2::CandidateEntry,
config: Config,
) -> DBTransaction {
let mut tx = DBTransaction::new();
tx.put_vec(
config.col_approval_data,
&candidate_entry_key(&candidate_entry.candidate.hash()),
candidate_entry.encode(),
);
tx
}
// Low level DB helper to write a block entry in v1 scheme.
fn write_block_entry_v2(
block_entry: crate::approval_db::v2::BlockEntry,
config: Config,
) -> DBTransaction {
let mut tx = DBTransaction::new();
tx.put_vec(
config.col_approval_data,
&block_entry_key(&block_entry.block_hash),
block_entry.encode(),
);
tx
}
polkadot/node/core/approval-voting/src/approval_db/v3/mod.rs
+137
View File
@@ -0,0 +1,137 @@
// Copyright (C) Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Version 3 of the DB schema.
//!
//! Version 3 modifies the `our_approval` format of `ApprovalEntry`
//! and adds a new field `pending_signatures` for `BlockEntry`
use parity_scale_codec::{Decode, Encode};
use polkadot_node_primitives::approval::v2::CandidateBitfield;
use polkadot_node_subsystem::SubsystemResult;
use polkadot_node_subsystem_util::database::{DBTransaction, Database};
use polkadot_overseer::SubsystemError;
use polkadot_primitives::{
BlockNumber, CandidateHash, CandidateIndex, CandidateReceipt, CoreIndex, GroupIndex, Hash,
SessionIndex, ValidatorIndex, ValidatorSignature,
};
use sp_consensus_slots::Slot;
use std::collections::BTreeMap;
use super::common::{block_entry_key, candidate_entry_key, load_decode, Config};
/// Re-export this structs as v3 since they did not change between v2 and v3.
pub use super::v2::{Bitfield, OurAssignment, Tick, TrancheEntry};
pub mod migration_helpers;
#[cfg(test)]
pub mod tests;
/// Metadata about our approval signature
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct OurApproval {
/// The signature for the candidates hashes pointed by indices.
pub signature: ValidatorSignature,
/// The indices of the candidates signed in this approval.
pub signed_candidates_indices: CandidateBitfield,
}
/// Metadata regarding approval of a particular candidate within the context of some
/// particular block.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct ApprovalEntry {
pub tranches: Vec<TrancheEntry>,
pub backing_group: GroupIndex,
pub our_assignment: Option<OurAssignment>,
pub our_approval_sig: Option<OurApproval>,
// `n_validators` bits.
pub assigned_validators: Bitfield,
pub approved: bool,
}
/// Metadata regarding approval of a particular candidate.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct CandidateEntry {
pub candidate: CandidateReceipt,
pub session: SessionIndex,
// Assignments are based on blocks, so we need to track assignments separately
// based on the block we are looking at.
pub block_assignments: BTreeMap<Hash, ApprovalEntry>,
pub approvals: Bitfield,
}
/// Metadata regarding approval of a particular block, by way of approval of the
/// candidates contained within it.
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
pub struct BlockEntry {
pub block_hash: Hash,
pub block_number: BlockNumber,
pub parent_hash: Hash,
pub session: SessionIndex,
pub slot: Slot,
/// Random bytes derived from the VRF submitted within the block by the block
/// author as a credential and used as input to approval assignment criteria.
pub relay_vrf_story: [u8; 32],
// The candidates included as-of this block and the index of the core they are
// leaving. Sorted ascending by core index.
pub candidates: Vec<(CoreIndex, CandidateHash)>,
// A bitfield where the i'th bit corresponds to the i'th candidate in `candidates`.
// The i'th bit is `true` iff the candidate has been approved in the context of this
// block. The block can be considered approved if the bitfield has all bits set to `true`.
pub approved_bitfield: Bitfield,
pub children: Vec<Hash>,
// A list of candidates we have checked, but didn't not sign and
// advertise the vote yet.
pub candidates_pending_signature: BTreeMap<CandidateIndex, CandidateSigningContext>,
// Assignments we already distributed. A 1 bit means the candidate index for which
// we already have sent out an assignment. We need this to avoid distributing
// multiple core assignments more than once.
pub distributed_assignments: Bitfield,
}
#[derive(Encode, Decode, Debug, Clone, PartialEq)]
/// Context needed for creating an approval signature for a given candidate.
pub struct CandidateSigningContext {
/// The candidate hash, to be included in the signature.
pub candidate_hash: CandidateHash,
/// The latest tick we have to create and send the approval.
pub sign_no_later_than_tick: Tick,
}
/// Load a candidate entry from the aux store in v2 format.
pub fn load_candidate_entry_v2(
store: &dyn Database,
config: &Config,
candidate_hash: &CandidateHash,
) -> SubsystemResult<Option<super::v2::CandidateEntry>> {
load_decode(store, config.col_approval_data, &candidate_entry_key(candidate_hash))
.map(|u: Option<super::v2::CandidateEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
/// Load a block entry from the aux store in v2 format.
pub fn load_block_entry_v2(
store: &dyn Database,
config: &Config,
block_hash: &Hash,
) -> SubsystemResult<Option<super::v2::BlockEntry>> {
load_decode(store, config.col_approval_data, &block_entry_key(block_hash))
.map(|u: Option<super::v2::BlockEntry>| u.map(|v| v.into()))
.map_err(|e| SubsystemError::with_origin("approval-voting", e))
}
polkadot/node/core/approval-voting/src/approval_db/v3/tests.rs
+575
View File
@@ -0,0 +1,575 @@
// Copyright (C) Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Tests for the aux-schema of approval voting.
use crate::{
approval_db::{
common::{migration_helpers::make_bitvec, DbBackend, StoredBlockRange, *},
v3::*,
},
backend::{Backend, OverlayedBackend},
ops::{add_block_entry, canonicalize, force_approve, NewCandidateInfo},
};
use polkadot_primitives::{
BlockNumber, CandidateHash, CandidateReceipt, CoreIndex, GroupIndex, Hash,
};
use polkadot_node_subsystem_util::database::Database;
use polkadot_primitives::Id as ParaId;
use sp_consensus_slots::Slot;
use std::{collections::HashMap, sync::Arc};
use ::test_helpers::{dummy_candidate_receipt, dummy_candidate_receipt_bad_sig, dummy_hash};
const DATA_COL: u32 = 0;
const NUM_COLUMNS: u32 = 1;
const TEST_CONFIG: Config = Config { col_approval_data: DATA_COL };
fn make_db() -> (DbBackend, Arc<dyn Database>) {
let db = kvdb_memorydb::create(NUM_COLUMNS);
let db = polkadot_node_subsystem_util::database::kvdb_impl::DbAdapter::new(db, &[]);
let db_writer: Arc<dyn Database> = Arc::new(db);
(DbBackend::new(db_writer.clone(), TEST_CONFIG), db_writer)
}
fn make_block_entry(
block_hash: Hash,
parent_hash: Hash,
block_number: BlockNumber,
candidates: Vec<(CoreIndex, CandidateHash)>,
) -> BlockEntry {
BlockEntry {
block_hash,
parent_hash,
block_number,
session: 1,
slot: Slot::from(1),
relay_vrf_story: [0u8; 32],
approved_bitfield: make_bitvec(candidates.len()),
candidates,
children: Vec::new(),
candidates_pending_signature: Default::default(),
distributed_assignments: Default::default(),
}
}
fn make_candidate(para_id: ParaId, relay_parent: Hash) -> CandidateReceipt {
let mut c = dummy_candidate_receipt(dummy_hash());
c.descriptor.para_id = para_id;
c.descriptor.relay_parent = relay_parent;
c
}
#[test]
fn read_write() {
let (mut db, store) = make_db();
let hash_a = Hash::repeat_byte(1);
let hash_b = Hash::repeat_byte(2);
let candidate_hash = dummy_candidate_receipt_bad_sig(dummy_hash(), None).hash();
let range = StoredBlockRange(10, 20);
let at_height = vec![hash_a, hash_b];
let block_entry =
make_block_entry(hash_a, Default::default(), 1, vec![(CoreIndex(0), candidate_hash)]);
let candidate_entry = CandidateEntry {
candidate: dummy_candidate_receipt_bad_sig(dummy_hash(), None),
session: 5,
block_assignments: vec![(
hash_a,
ApprovalEntry {
tranches: Vec::new(),
backing_group: GroupIndex(1),
our_assignment: None,
our_approval_sig: None,
assigned_validators: Default::default(),
approved: false,
},
)]
.into_iter()
.collect(),
approvals: Default::default(),
};
let mut overlay_db = OverlayedBackend::new(&db);
overlay_db.write_stored_block_range(range.clone());
overlay_db.write_blocks_at_height(1, at_height.clone());
overlay_db.write_block_entry(block_entry.clone().into());
overlay_db.write_candidate_entry(candidate_entry.clone().into());
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert_eq!(load_stored_blocks(store.as_ref(), &TEST_CONFIG).unwrap(), Some(range));
assert_eq!(load_blocks_at_height(store.as_ref(), &TEST_CONFIG, &1).unwrap(), at_height);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap(),
Some(block_entry.into())
);
assert_eq!(
load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash).unwrap(),
Some(candidate_entry.into()),
);
let mut overlay_db = OverlayedBackend::new(&db);
overlay_db.delete_blocks_at_height(1);
overlay_db.delete_block_entry(&hash_a);
overlay_db.delete_candidate_entry(&candidate_hash);
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert!(load_blocks_at_height(store.as_ref(), &TEST_CONFIG, &1).unwrap().is_empty());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash_a).unwrap().is_none());
assert!(load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash)
.unwrap()
.is_none());
}
#[test]
fn add_block_entry_works() {
let (mut db, store) = make_db();
let parent_hash = Hash::repeat_byte(1);
let block_hash_a = Hash::repeat_byte(2);
let block_hash_b = Hash::repeat_byte(69);
let candidate_receipt_a = make_candidate(ParaId::from(1_u32), parent_hash);
let candidate_receipt_b = make_candidate(ParaId::from(2_u32), parent_hash);
let candidate_hash_a = candidate_receipt_a.hash();
let candidate_hash_b = candidate_receipt_b.hash();
let block_number = 10;
let block_entry_a = make_block_entry(
block_hash_a,
parent_hash,
block_number,
vec![(CoreIndex(0), candidate_hash_a)],
);
let block_entry_b = make_block_entry(
block_hash_b,
parent_hash,
block_number,
vec![(CoreIndex(0), candidate_hash_a), (CoreIndex(1), candidate_hash_b)],
);
let n_validators = 10;
let mut new_candidate_info = HashMap::new();
new_candidate_info
.insert(candidate_hash_a, NewCandidateInfo::new(candidate_receipt_a, GroupIndex(0), None));
let mut overlay_db = OverlayedBackend::new(&db);
add_block_entry(&mut overlay_db, block_entry_a.clone().into(), n_validators, |h| {
new_candidate_info.get(h).map(|x| x.clone())
})
.unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
new_candidate_info
.insert(candidate_hash_b, NewCandidateInfo::new(candidate_receipt_b, GroupIndex(1), None));
let mut overlay_db = OverlayedBackend::new(&db);
add_block_entry(&mut overlay_db, block_entry_b.clone().into(), n_validators, |h| {
new_candidate_info.get(h).map(|x| x.clone())
})
.unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
Some(block_entry_a.into())
);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
Some(block_entry_b.into())
);
let candidate_entry_a = load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash_a)
.unwrap()
.unwrap();
assert_eq!(
candidate_entry_a.block_assignments.keys().collect::<Vec<_>>(),
vec![&block_hash_a, &block_hash_b]
);
let candidate_entry_b = load_candidate_entry(store.as_ref(), &TEST_CONFIG, &candidate_hash_b)
.unwrap()
.unwrap();
assert_eq!(candidate_entry_b.block_assignments.keys().collect::<Vec<_>>(), vec![&block_hash_b]);
}
#[test]
fn add_block_entry_adds_child() {
let (mut db, store) = make_db();
let parent_hash = Hash::repeat_byte(1);
let block_hash_a = Hash::repeat_byte(2);
let block_hash_b = Hash::repeat_byte(69);
let mut block_entry_a = make_block_entry(block_hash_a, parent_hash, 1, Vec::new());
let block_entry_b = make_block_entry(block_hash_b, block_hash_a, 2, Vec::new());
let n_validators = 10;
let mut overlay_db = OverlayedBackend::new(&db);
add_block_entry(&mut overlay_db, block_entry_a.clone().into(), n_validators, |_| None).unwrap();
add_block_entry(&mut overlay_db, block_entry_b.clone().into(), n_validators, |_| None).unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
block_entry_a.children.push(block_hash_b);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a).unwrap(),
Some(block_entry_a.into())
);
assert_eq!(
load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b).unwrap(),
Some(block_entry_b.into())
);
}
#[test]
fn canonicalize_works() {
let (mut db, store) = make_db();
// -> B1 -> C1 -> D1
// A -> B2 -> C2 -> D2
//
// We'll canonicalize C1. Everytning except D1 should disappear.
//
// Candidates:
// Cand1 in B2
// Cand2 in C2
// Cand3 in C2 and D1
// Cand4 in D1
// Cand5 in D2
// Only Cand3 and Cand4 should remain after canonicalize.
let n_validators = 10;
let mut overlay_db = OverlayedBackend::new(&db);
overlay_db.write_stored_block_range(StoredBlockRange(1, 5));
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
let genesis = Hash::repeat_byte(0);
let block_hash_a = Hash::repeat_byte(1);
let block_hash_b1 = Hash::repeat_byte(2);
let block_hash_b2 = Hash::repeat_byte(3);
let block_hash_c1 = Hash::repeat_byte(4);
let block_hash_c2 = Hash::repeat_byte(5);
let block_hash_d1 = Hash::repeat_byte(6);
let block_hash_d2 = Hash::repeat_byte(7);
let candidate_receipt_genesis = make_candidate(ParaId::from(1_u32), genesis);
let candidate_receipt_a = make_candidate(ParaId::from(2_u32), block_hash_a);
let candidate_receipt_b = make_candidate(ParaId::from(3_u32), block_hash_a);
let candidate_receipt_b1 = make_candidate(ParaId::from(4_u32), block_hash_b1);
let candidate_receipt_c1 = make_candidate(ParaId::from(5_u32), block_hash_c1);
let cand_hash_1 = candidate_receipt_genesis.hash();
let cand_hash_2 = candidate_receipt_a.hash();
let cand_hash_3 = candidate_receipt_b.hash();
let cand_hash_4 = candidate_receipt_b1.hash();
let cand_hash_5 = candidate_receipt_c1.hash();
let block_entry_a = make_block_entry(block_hash_a, genesis, 1, Vec::new());
let block_entry_b1 = make_block_entry(block_hash_b1, block_hash_a, 2, Vec::new());
let block_entry_b2 =
make_block_entry(block_hash_b2, block_hash_a, 2, vec![(CoreIndex(0), cand_hash_1)]);
let block_entry_c1 = make_block_entry(block_hash_c1, block_hash_b1, 3, Vec::new());
let block_entry_c2 = make_block_entry(
block_hash_c2,
block_hash_b2,
3,
vec![(CoreIndex(0), cand_hash_2), (CoreIndex(1), cand_hash_3)],
);
let block_entry_d1 = make_block_entry(
block_hash_d1,
block_hash_c1,
4,
vec![(CoreIndex(0), cand_hash_3), (CoreIndex(1), cand_hash_4)],
);
let block_entry_d2 =
make_block_entry(block_hash_d2, block_hash_c2, 4, vec![(CoreIndex(0), cand_hash_5)]);
let candidate_info = {
let mut candidate_info = HashMap::new();
candidate_info.insert(
cand_hash_1,
NewCandidateInfo::new(candidate_receipt_genesis, GroupIndex(1), None),
);
candidate_info
.insert(cand_hash_2, NewCandidateInfo::new(candidate_receipt_a, GroupIndex(2), None));
candidate_info
.insert(cand_hash_3, NewCandidateInfo::new(candidate_receipt_b, GroupIndex(3), None));
candidate_info
.insert(cand_hash_4, NewCandidateInfo::new(candidate_receipt_b1, GroupIndex(4), None));
candidate_info
.insert(cand_hash_5, NewCandidateInfo::new(candidate_receipt_c1, GroupIndex(5), None));
candidate_info
};
// now insert all the blocks.
let blocks = vec![
block_entry_a.clone(),
block_entry_b1.clone(),
block_entry_b2.clone(),
block_entry_c1.clone(),
block_entry_c2.clone(),
block_entry_d1.clone(),
block_entry_d2.clone(),
];
let mut overlay_db = OverlayedBackend::new(&db);
for block_entry in blocks {
add_block_entry(&mut overlay_db, block_entry.into(), n_validators, |h| {
candidate_info.get(h).map(|x| x.clone())
})
.unwrap();
}
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
let check_candidates_in_store = |expected: Vec<(CandidateHash, Option<Vec<_>>)>| {
for (c_hash, in_blocks) in expected {
let (entry, in_blocks) = match in_blocks {
None => {
assert!(load_candidate_entry(store.as_ref(), &TEST_CONFIG, &c_hash)
.unwrap()
.is_none());
continue
},
Some(i) => (
load_candidate_entry(store.as_ref(), &TEST_CONFIG, &c_hash).unwrap().unwrap(),
i,
),
};
assert_eq!(entry.block_assignments.len(), in_blocks.len());
for x in in_blocks {
assert!(entry.block_assignments.contains_key(&x));
}
}
};
let check_blocks_in_store = |expected: Vec<(Hash, Option<Vec<_>>)>| {
for (hash, with_candidates) in expected {
let (entry, with_candidates) = match with_candidates {
None => {
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash)
.unwrap()
.is_none());
continue
},
Some(i) =>
(load_block_entry(store.as_ref(), &TEST_CONFIG, &hash).unwrap().unwrap(), i),
};
assert_eq!(entry.candidates.len(), with_candidates.len());
for x in with_candidates {
assert!(entry.candidates.iter().any(|(_, c)| c == &x));
}
}
};
check_candidates_in_store(vec![
(cand_hash_1, Some(vec![block_hash_b2])),
(cand_hash_2, Some(vec![block_hash_c2])),
(cand_hash_3, Some(vec![block_hash_c2, block_hash_d1])),
(cand_hash_4, Some(vec![block_hash_d1])),
(cand_hash_5, Some(vec![block_hash_d2])),
]);
check_blocks_in_store(vec![
(block_hash_a, Some(vec![])),
(block_hash_b1, Some(vec![])),
(block_hash_b2, Some(vec![cand_hash_1])),
(block_hash_c1, Some(vec![])),
(block_hash_c2, Some(vec![cand_hash_2, cand_hash_3])),
(block_hash_d1, Some(vec![cand_hash_3, cand_hash_4])),
(block_hash_d2, Some(vec![cand_hash_5])),
]);
let mut overlay_db = OverlayedBackend::new(&db);
canonicalize(&mut overlay_db, 3, block_hash_c1).unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert_eq!(
load_stored_blocks(store.as_ref(), &TEST_CONFIG).unwrap().unwrap(),
StoredBlockRange(4, 5)
);
check_candidates_in_store(vec![
(cand_hash_1, None),
(cand_hash_2, None),
(cand_hash_3, Some(vec![block_hash_d1])),
(cand_hash_4, Some(vec![block_hash_d1])),
(cand_hash_5, None),
]);
check_blocks_in_store(vec![
(block_hash_a, None),
(block_hash_b1, None),
(block_hash_b2, None),
(block_hash_c1, None),
(block_hash_c2, None),
(block_hash_d1, Some(vec![cand_hash_3, cand_hash_4])),
(block_hash_d2, None),
]);
}
#[test]
fn force_approve_works() {
let (mut db, store) = make_db();
let n_validators = 10;
let mut overlay_db = OverlayedBackend::new(&db);
overlay_db.write_stored_block_range(StoredBlockRange(1, 4));
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
let candidate_hash = CandidateHash(Hash::repeat_byte(42));
let single_candidate_vec = vec![(CoreIndex(0), candidate_hash)];
let candidate_info = {
let mut candidate_info = HashMap::new();
candidate_info.insert(
candidate_hash,
NewCandidateInfo::new(
make_candidate(ParaId::from(1_u32), Default::default()),
GroupIndex(1),
None,
),
);
candidate_info
};
let block_hash_a = Hash::repeat_byte(1); // 1
let block_hash_b = Hash::repeat_byte(2);
let block_hash_c = Hash::repeat_byte(3);
let block_hash_d = Hash::repeat_byte(4); // 4
let block_entry_a =
make_block_entry(block_hash_a, Default::default(), 1, single_candidate_vec.clone());
let block_entry_b =
make_block_entry(block_hash_b, block_hash_a, 2, single_candidate_vec.clone());
let block_entry_c =
make_block_entry(block_hash_c, block_hash_b, 3, single_candidate_vec.clone());
let block_entry_d =
make_block_entry(block_hash_d, block_hash_c, 4, single_candidate_vec.clone());
let blocks = vec![
block_entry_a.clone(),
block_entry_b.clone(),
block_entry_c.clone(),
block_entry_d.clone(),
];
let mut overlay_db = OverlayedBackend::new(&db);
for block_entry in blocks {
add_block_entry(&mut overlay_db, block_entry.into(), n_validators, |h| {
candidate_info.get(h).map(|x| x.clone())
})
.unwrap();
}
let approved_hashes = force_approve(&mut overlay_db, block_hash_d, 2).unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_a,)
.unwrap()
.unwrap()
.approved_bitfield
.all());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_b,)
.unwrap()
.unwrap()
.approved_bitfield
.all());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_c,)
.unwrap()
.unwrap()
.approved_bitfield
.not_any());
assert!(load_block_entry(store.as_ref(), &TEST_CONFIG, &block_hash_d,)
.unwrap()
.unwrap()
.approved_bitfield
.not_any());
assert_eq!(approved_hashes, vec![block_hash_b, block_hash_a]);
}
#[test]
fn load_all_blocks_works() {
let (mut db, store) = make_db();
let parent_hash = Hash::repeat_byte(1);
let block_hash_a = Hash::repeat_byte(2);
let block_hash_b = Hash::repeat_byte(69);
let block_hash_c = Hash::repeat_byte(42);
let block_number = 10;
let block_entry_a = make_block_entry(block_hash_a, parent_hash, block_number, vec![]);
let block_entry_b = make_block_entry(block_hash_b, parent_hash, block_number, vec![]);
let block_entry_c = make_block_entry(block_hash_c, block_hash_a, block_number + 1, vec![]);
let n_validators = 10;
let mut overlay_db = OverlayedBackend::new(&db);
add_block_entry(&mut overlay_db, block_entry_a.clone().into(), n_validators, |_| None).unwrap();
// add C before B to test sorting.
add_block_entry(&mut overlay_db, block_entry_c.clone().into(), n_validators, |_| None).unwrap();
add_block_entry(&mut overlay_db, block_entry_b.clone().into(), n_validators, |_| None).unwrap();
let write_ops = overlay_db.into_write_ops();
db.write(write_ops).unwrap();
assert_eq!(
load_all_blocks(store.as_ref(), &TEST_CONFIG).unwrap(),
vec![block_hash_a, block_hash_b, block_hash_c],
)
}
polkadot/node/core/approval-voting/src/backend.rs
+16 -2
View File
@@ -22,12 +22,12 @@
//! before any commit to the underlying storage is made.
use polkadot_node_subsystem::SubsystemResult;
use polkadot_primitives::{BlockNumber, CandidateHash, Hash};
use polkadot_primitives::{BlockNumber, CandidateHash, CandidateIndex, Hash};
use std::collections::HashMap;
use super::{
approval_db::v2::StoredBlockRange,
approval_db::common::StoredBlockRange,
persisted_entries::{BlockEntry, CandidateEntry},
};
@@ -72,12 +72,26 @@ pub trait V1ReadBackend: Backend {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>>;
/// Load a block entry from the DB with scheme version 1.
fn load_block_entry_v1(&self, block_hash: &Hash) -> SubsystemResult<Option<BlockEntry>>;
}
/// A read only backend to enable db migration from version 2 of DB.
pub trait V2ReadBackend: Backend {
/// Load a candidate entry from the DB with scheme version 1.
fn load_candidate_entry_v2(
&self,
candidate_hash: &CandidateHash,
candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>>;
/// Load a block entry from the DB with scheme version 1.
fn load_block_entry_v2(&self, block_hash: &Hash) -> SubsystemResult<Option<BlockEntry>>;
}
// Status of block range in the `OverlayedBackend`.
#[derive(PartialEq)]
enum BlockRangeStatus {
polkadot/node/core/approval-voting/src/import.rs
+14 -10
View File
@@ -56,7 +56,7 @@ use futures::{channel::oneshot, prelude::*};
use std::collections::HashMap;
use super::approval_db::v2;
use super::approval_db::v3;
use crate::{
backend::{Backend, OverlayedBackend},
criteria::{AssignmentCriteria, OurAssignment},
@@ -512,7 +512,7 @@ pub(crate) async fn handle_new_head<Context, B: Backend>(
ctx.send_message(ChainSelectionMessage::Approved(block_hash)).await;
}
let block_entry = v2::BlockEntry {
let block_entry = v3::BlockEntry {
block_hash,
parent_hash: block_header.parent_hash,
block_number: block_header.number,
@@ -525,6 +525,7 @@ pub(crate) async fn handle_new_head<Context, B: Backend>(
.collect(),
approved_bitfield,
children: Vec::new(),
candidates_pending_signature: Default::default(),
distributed_assignments: Default::default(),
};
@@ -604,7 +605,10 @@ pub(crate) async fn handle_new_head<Context, B: Backend>(
#[cfg(test)]
pub(crate) mod tests {
use super::*;
use crate::{approval_db::v2::DbBackend, RuntimeInfo, RuntimeInfoConfig};
use crate::{
approval_db::common::{load_block_entry, DbBackend},
RuntimeInfo, RuntimeInfoConfig,
};
use ::test_helpers::{dummy_candidate_receipt, dummy_hash};
use assert_matches::assert_matches;
use polkadot_node_primitives::{
@@ -627,7 +631,7 @@ pub(crate) mod tests {
pub(crate) use sp_runtime::{Digest, DigestItem};
use std::{pin::Pin, sync::Arc};
use crate::{approval_db::v2::Config as DatabaseConfig, criteria, BlockEntry};
use crate::{approval_db::common::Config as DatabaseConfig, criteria, BlockEntry};
const DATA_COL: u32 = 0;
@@ -1347,7 +1351,7 @@ pub(crate) mod tests {
let (state, mut session_info_provider) = single_session_state();
overlay_db.write_block_entry(
v2::BlockEntry {
v3::BlockEntry {
block_hash: parent_hash,
parent_hash: Default::default(),
block_number: 4,
@@ -1357,6 +1361,7 @@ pub(crate) mod tests {
candidates: Vec::new(),
approved_bitfield: Default::default(),
children: Vec::new(),
candidates_pending_signature: Default::default(),
distributed_assignments: Default::default(),
}
.into(),
@@ -1389,11 +1394,10 @@ pub(crate) mod tests {
assert_eq!(candidates[1].1.approvals().len(), 6);
// the first candidate should be insta-approved
// the second should not
let entry: BlockEntry =
v2::load_block_entry(db_writer.as_ref(), &TEST_CONFIG, &hash)
.unwrap()
.unwrap()
.into();
let entry: BlockEntry = load_block_entry(db_writer.as_ref(), &TEST_CONFIG, &hash)
.unwrap()
.unwrap()
.into();
assert!(entry.is_candidate_approved(&candidates[0].0));
assert!(!entry.is_candidate_approved(&candidates[1].0));
})
polkadot/node/core/approval-voting/src/lib.rs
+676 -190
View File
File diff suppressed because it is too large Load Diff
polkadot/node/core/approval-voting/src/ops.rs
+1 -1
View File
@@ -25,7 +25,7 @@ use polkadot_primitives::{BlockNumber, CandidateHash, CandidateReceipt, GroupInd
use std::collections::{hash_map::Entry, BTreeMap, HashMap};
use super::{
approval_db::v2::{OurAssignment, StoredBlockRange},
approval_db::{common::StoredBlockRange, v2::OurAssignment},
backend::{Backend, OverlayedBackend},
persisted_entries::{ApprovalEntry, BlockEntry, CandidateEntry},
LOG_TARGET,
polkadot/node/core/approval-voting/src/persisted_entries.rs
+257 -35
View File
@@ -20,13 +20,14 @@
//! Within that context, things are plain-old-data. Within this module,
//! data and logic are intertwined.
use itertools::Itertools;
use polkadot_node_primitives::approval::{
v1::{DelayTranche, RelayVRFStory},
v2::{AssignmentCertV2, CandidateBitfield},
};
use polkadot_primitives::{
BlockNumber, CandidateHash, CandidateReceipt, CoreIndex, GroupIndex, Hash, SessionIndex,
ValidatorIndex, ValidatorSignature,
BlockNumber, CandidateHash, CandidateIndex, CandidateReceipt, CoreIndex, GroupIndex, Hash,
SessionIndex, ValidatorIndex, ValidatorSignature,
};
use sp_consensus_slots::Slot;
@@ -76,6 +77,45 @@ impl From<TrancheEntry> for crate::approval_db::v2::TrancheEntry {
}
}
impl From<crate::approval_db::v3::OurApproval> for OurApproval {
fn from(approval: crate::approval_db::v3::OurApproval) -> Self {
Self {
signature: approval.signature,
signed_candidates_indices: approval.signed_candidates_indices,
}
}
}
impl From<OurApproval> for crate::approval_db::v3::OurApproval {
fn from(approval: OurApproval) -> Self {
Self {
signature: approval.signature,
signed_candidates_indices: approval.signed_candidates_indices,
}
}
}
/// Metadata about our approval signature
#[derive(Debug, Clone, PartialEq)]
pub struct OurApproval {
/// The signature for the candidates hashes pointed by indices.
pub signature: ValidatorSignature,
/// The indices of the candidates signed in this approval.
pub signed_candidates_indices: CandidateBitfield,
}
impl OurApproval {
/// Converts a ValidatorSignature to an OurApproval.
/// It used in converting the database from v1 to latest.
pub fn from_v1(value: ValidatorSignature, candidate_index: CandidateIndex) -> Self {
Self { signature: value, signed_candidates_indices: candidate_index.into() }
}
/// Converts a ValidatorSignature to an OurApproval.
/// It used in converting the database from v2 to latest.
pub fn from_v2(value: ValidatorSignature, candidate_index: CandidateIndex) -> Self {
Self::from_v1(value, candidate_index)
}
}
/// Metadata regarding approval of a particular candidate within the context of some
/// particular block.
#[derive(Debug, Clone, PartialEq)]
@@ -83,7 +123,7 @@ pub struct ApprovalEntry {
tranches: Vec<TrancheEntry>,
backing_group: GroupIndex,
our_assignment: Option<OurAssignment>,
our_approval_sig: Option<ValidatorSignature>,
our_approval_sig: Option<OurApproval>,
// `n_validators` bits.
assigned_validators: Bitfield,
approved: bool,
@@ -95,7 +135,7 @@ impl ApprovalEntry {
tranches: Vec<TrancheEntry>,
backing_group: GroupIndex,
our_assignment: Option<OurAssignment>,
our_approval_sig: Option<ValidatorSignature>,
our_approval_sig: Option<OurApproval>,
// `n_validators` bits.
assigned_validators: Bitfield,
approved: bool,
@@ -137,7 +177,7 @@ impl ApprovalEntry {
}
/// Import our local approval vote signature for this candidate.
pub fn import_approval_sig(&mut self, approval_sig: ValidatorSignature) {
pub fn import_approval_sig(&mut self, approval_sig: OurApproval) {
self.our_approval_sig = Some(approval_sig);
}
@@ -224,7 +264,7 @@ impl ApprovalEntry {
/// Get the assignment cert & approval signature.
///
/// The approval signature will only be `Some` if the assignment is too.
pub fn local_statements(&self) -> (Option<OurAssignment>, Option<ValidatorSignature>) {
pub fn local_statements(&self) -> (Option<OurAssignment>, Option<OurApproval>) {
let approval_sig = self.our_approval_sig.clone();
if let Some(our_assignment) = self.our_assignment.as_ref().filter(|a| a.triggered()) {
(Some(our_assignment.clone()), approval_sig)
@@ -232,10 +272,44 @@ impl ApprovalEntry {
(None, None)
}
}
// Convert an ApprovalEntry from v1 version to latest version
pub fn from_v1(
value: crate::approval_db::v1::ApprovalEntry,
candidate_index: CandidateIndex,
) -> Self {
ApprovalEntry {
tranches: value.tranches.into_iter().map(|tranche| tranche.into()).collect(),
backing_group: value.backing_group,
our_assignment: value.our_assignment.map(|assignment| assignment.into()),
our_approval_sig: value
.our_approval_sig
.map(|sig| OurApproval::from_v1(sig, candidate_index)),
assigned_validators: value.assignments,
approved: value.approved,
}
}
// Convert an ApprovalEntry from v1 version to latest version
pub fn from_v2(
value: crate::approval_db::v2::ApprovalEntry,
candidate_index: CandidateIndex,
) -> Self {
ApprovalEntry {
tranches: value.tranches.into_iter().map(|tranche| tranche.into()).collect(),
backing_group: value.backing_group,
our_assignment: value.our_assignment.map(|assignment| assignment.into()),
our_approval_sig: value
.our_approval_sig
.map(|sig| OurApproval::from_v2(sig, candidate_index)),
assigned_validators: value.assigned_validators,
approved: value.approved,
}
}
}
impl From<crate::approval_db::v2::ApprovalEntry> for ApprovalEntry {
fn from(entry: crate::approval_db::v2::ApprovalEntry) -> Self {
impl From<crate::approval_db::v3::ApprovalEntry> for ApprovalEntry {
fn from(entry: crate::approval_db::v3::ApprovalEntry) -> Self {
ApprovalEntry {
tranches: entry.tranches.into_iter().map(Into::into).collect(),
backing_group: entry.backing_group,
@@ -247,7 +321,7 @@ impl From<crate::approval_db::v2::ApprovalEntry> for ApprovalEntry {
}
}
impl From<ApprovalEntry> for crate::approval_db::v2::ApprovalEntry {
impl From<ApprovalEntry> for crate::approval_db::v3::ApprovalEntry {
fn from(entry: ApprovalEntry) -> Self {
Self {
tranches: entry.tranches.into_iter().map(Into::into).collect(),
@@ -303,10 +377,44 @@ impl CandidateEntry {
pub fn approval_entry(&self, block_hash: &Hash) -> Option<&ApprovalEntry> {
self.block_assignments.get(block_hash)
}
/// Convert a CandidateEntry from a v1 to its latest equivalent.
pub fn from_v1(
value: crate::approval_db::v1::CandidateEntry,
candidate_index: CandidateIndex,
) -> Self {
Self {
approvals: value.approvals,
block_assignments: value
.block_assignments
.into_iter()
.map(|(h, ae)| (h, ApprovalEntry::from_v1(ae, candidate_index)))
.collect(),
candidate: value.candidate,
session: value.session,
}
}
/// Convert a CandidateEntry from a v2 to its latest equivalent.
pub fn from_v2(
value: crate::approval_db::v2::CandidateEntry,
candidate_index: CandidateIndex,
) -> Self {
Self {
approvals: value.approvals,
block_assignments: value
.block_assignments
.into_iter()
.map(|(h, ae)| (h, ApprovalEntry::from_v2(ae, candidate_index)))
.collect(),
candidate: value.candidate,
session: value.session,
}
}
}
impl From<crate::approval_db::v2::CandidateEntry> for CandidateEntry {
fn from(entry: crate::approval_db::v2::CandidateEntry) -> Self {
impl From<crate::approval_db::v3::CandidateEntry> for CandidateEntry {
fn from(entry: crate::approval_db::v3::CandidateEntry) -> Self {
CandidateEntry {
candidate: entry.candidate,
session: entry.session,
@@ -320,7 +428,7 @@ impl From<crate::approval_db::v2::CandidateEntry> for CandidateEntry {
}
}
impl From<CandidateEntry> for crate::approval_db::v2::CandidateEntry {
impl From<CandidateEntry> for crate::approval_db::v3::CandidateEntry {
fn from(entry: CandidateEntry) -> Self {
Self {
candidate: entry.candidate,
@@ -353,12 +461,21 @@ pub struct BlockEntry {
// block. The block can be considered approved if the bitfield has all bits set to `true`.
pub approved_bitfield: Bitfield,
pub children: Vec<Hash>,
// A list of candidates we have checked, but didn't not sign and
// advertise the vote yet.
candidates_pending_signature: BTreeMap<CandidateIndex, CandidateSigningContext>,
// A list of assignments for which we already distributed the assignment.
// We use this to ensure we don't distribute multiple core assignments twice as we track
// individual wakeups for each core.
distributed_assignments: Bitfield,
}
#[derive(Debug, Clone, PartialEq)]
pub struct CandidateSigningContext {
pub candidate_hash: CandidateHash,
pub sign_no_later_than_tick: Tick,
}
impl BlockEntry {
/// Mark a candidate as fully approved in the bitfield.
pub fn mark_approved_by_hash(&mut self, candidate_hash: &CandidateHash) {
@@ -447,10 +564,97 @@ impl BlockEntry {
distributed
}
/// Defer signing and issuing an approval for a candidate no later than the specified tick
pub fn defer_candidate_signature(
&mut self,
candidate_index: CandidateIndex,
candidate_hash: CandidateHash,
sign_no_later_than_tick: Tick,
) -> Option<CandidateSigningContext> {
self.candidates_pending_signature.insert(
candidate_index,
CandidateSigningContext { candidate_hash, sign_no_later_than_tick },
)
}
/// Returns the number of candidates waiting for an approval to be issued.
pub fn num_candidates_pending_signature(&self) -> usize {
self.candidates_pending_signature.len()
}
/// Return if we have candidates waiting for signature to be issued
pub fn has_candidates_pending_signature(&self) -> bool {
!self.candidates_pending_signature.is_empty()
}
/// Candidate hashes for candidates pending signatures
fn candidate_hashes_pending_signature(&self) -> Vec<CandidateHash> {
self.candidates_pending_signature
.values()
.map(|unsigned_approval| unsigned_approval.candidate_hash)
.collect()
}
/// Candidate indices for candidates pending signature
fn candidate_indices_pending_signature(&self) -> Option<CandidateBitfield> {
self.candidates_pending_signature
.keys()
.map(|val| *val)
.collect_vec()
.try_into()
.ok()
}
/// Returns a list of candidates hashes that need need signature created at the current tick:
/// This might happen in other of the two reasons:
/// 1. We queued more than max_approval_coalesce_count candidates.
/// 2. We have candidates that waiting in the queue past their `sign_no_later_than_tick`
///
/// Additionally, we also return the first tick when we will have to create a signature,
/// so that the caller can arm the timer if it is not already armed.
pub fn get_candidates_that_need_signature(
&self,
tick_now: Tick,
max_approval_coalesce_count: u32,
) -> (Option<(Vec<CandidateHash>, CandidateBitfield)>, Option<Tick>) {
let sign_no_later_than_tick = self
.candidates_pending_signature
.values()
.min_by(|a, b| a.sign_no_later_than_tick.cmp(&b.sign_no_later_than_tick))
.map(|val| val.sign_no_later_than_tick);
if let Some(sign_no_later_than_tick) = sign_no_later_than_tick {
if sign_no_later_than_tick <= tick_now ||
self.num_candidates_pending_signature() >= max_approval_coalesce_count as usize
{
(
self.candidate_indices_pending_signature().and_then(|candidate_indices| {
Some((self.candidate_hashes_pending_signature(), candidate_indices))
}),
Some(sign_no_later_than_tick),
)
} else {
// We can still wait for other candidates to queue in, so just make sure
// we wake up at the tick we have to sign the longest waiting candidate.
(Default::default(), Some(sign_no_later_than_tick))
}
} else {
// No cached candidates, nothing to do here, this just means the timer fired,
// but the signatures were already sent because we gathered more than
// max_approval_coalesce_count.
(Default::default(), sign_no_later_than_tick)
}
}
/// Clears the candidates pending signature because the approval was issued.
pub fn issued_approval(&mut self) {
self.candidates_pending_signature.clear();
}
}
impl From<crate::approval_db::v2::BlockEntry> for BlockEntry {
fn from(entry: crate::approval_db::v2::BlockEntry) -> Self {
impl From<crate::approval_db::v3::BlockEntry> for BlockEntry {
fn from(entry: crate::approval_db::v3::BlockEntry) -> Self {
BlockEntry {
block_hash: entry.block_hash,
parent_hash: entry.parent_hash,
@@ -461,6 +665,11 @@ impl From<crate::approval_db::v2::BlockEntry> for BlockEntry {
candidates: entry.candidates,
approved_bitfield: entry.approved_bitfield,
children: entry.children,
candidates_pending_signature: entry
.candidates_pending_signature
.into_iter()
.map(|(candidate_index, signing_context)| (candidate_index, signing_context.into()))
.collect(),
distributed_assignments: entry.distributed_assignments,
}
}
@@ -479,11 +688,30 @@ impl From<crate::approval_db::v1::BlockEntry> for BlockEntry {
approved_bitfield: entry.approved_bitfield,
children: entry.children,
distributed_assignments: Default::default(),
candidates_pending_signature: Default::default(),
}
}
}
impl From<BlockEntry> for crate::approval_db::v2::BlockEntry {
impl From<crate::approval_db::v2::BlockEntry> for BlockEntry {
fn from(entry: crate::approval_db::v2::BlockEntry) -> Self {
BlockEntry {
block_hash: entry.block_hash,
parent_hash: entry.parent_hash,
block_number: entry.block_number,
session: entry.session,
slot: entry.slot,
relay_vrf_story: RelayVRFStory(entry.relay_vrf_story),
candidates: entry.candidates,
approved_bitfield: entry.approved_bitfield,
children: entry.children,
distributed_assignments: entry.distributed_assignments,
candidates_pending_signature: Default::default(),
}
}
}
impl From<BlockEntry> for crate::approval_db::v3::BlockEntry {
fn from(entry: BlockEntry) -> Self {
Self {
block_hash: entry.block_hash,
@@ -495,36 +723,30 @@ impl From<BlockEntry> for crate::approval_db::v2::BlockEntry {
candidates: entry.candidates,
approved_bitfield: entry.approved_bitfield,
children: entry.children,
candidates_pending_signature: entry
.candidates_pending_signature
.into_iter()
.map(|(candidate_index, signing_context)| (candidate_index, signing_context.into()))
.collect(),
distributed_assignments: entry.distributed_assignments,
}
}
}
/// Migration helpers.
impl From<crate::approval_db::v1::CandidateEntry> for CandidateEntry {
fn from(value: crate::approval_db::v1::CandidateEntry) -> Self {
impl From<crate::approval_db::v3::CandidateSigningContext> for CandidateSigningContext {
fn from(signing_context: crate::approval_db::v3::CandidateSigningContext) -> Self {
Self {
approvals: value.approvals,
block_assignments: value
.block_assignments
.into_iter()
.map(|(h, ae)| (h, ae.into()))
.collect(),
candidate: value.candidate,
session: value.session,
candidate_hash: signing_context.candidate_hash,
sign_no_later_than_tick: signing_context.sign_no_later_than_tick.into(),
}
}
}
impl From<crate::approval_db::v1::ApprovalEntry> for ApprovalEntry {
fn from(value: crate::approval_db::v1::ApprovalEntry) -> Self {
ApprovalEntry {
tranches: value.tranches.into_iter().map(|tranche| tranche.into()).collect(),
backing_group: value.backing_group,
our_assignment: value.our_assignment.map(|assignment| assignment.into()),
our_approval_sig: value.our_approval_sig,
assigned_validators: value.assignments,
approved: value.approved,
impl From<CandidateSigningContext> for crate::approval_db::v3::CandidateSigningContext {
fn from(signing_context: CandidateSigningContext) -> Self {
Self {
candidate_hash: signing_context.candidate_hash,
sign_no_later_than_tick: signing_context.sign_no_later_than_tick.into(),
}
}
}
polkadot/node/core/approval-voting/src/tests.rs
+576 -5
View File
@@ -37,8 +37,8 @@ use polkadot_node_subsystem_test_helpers as test_helpers;
use polkadot_node_subsystem_util::TimeoutExt;
use polkadot_overseer::HeadSupportsParachains;
use polkadot_primitives::{
vstaging::NodeFeatures, CandidateCommitments, CandidateEvent, CoreIndex, GroupIndex, Header,
Id as ParaId, IndexedVec, ValidationCode, ValidatorSignature,
vstaging::NodeFeatures, ApprovalVote, CandidateCommitments, CandidateEvent, CoreIndex,
GroupIndex, Header, Id as ParaId, IndexedVec, ValidationCode, ValidatorSignature,
};
use std::time::Duration;
@@ -56,7 +56,7 @@ use std::{
};
use super::{
approval_db::v2::StoredBlockRange,
approval_db::common::StoredBlockRange,
backend::BackendWriteOp,
import::tests::{
garbage_vrf_signature, AllowedSlots, BabeEpoch, BabeEpochConfiguration,
@@ -116,7 +116,7 @@ fn make_sync_oracle(val: bool) -> (Box<dyn SyncOracle + Send>, TestSyncOracleHan
#[cfg(test)]
pub mod test_constants {
use crate::approval_db::v2::Config as DatabaseConfig;
use crate::approval_db::common::Config as DatabaseConfig;
const DATA_COL: u32 = 0;
pub(crate) const NUM_COLUMNS: u32 = 1;
@@ -281,6 +281,7 @@ impl V1ReadBackend for TestStoreInner {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
_candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>> {
self.load_candidate_entry(candidate_hash)
}
@@ -364,6 +365,7 @@ impl V1ReadBackend for TestStore {
fn load_candidate_entry_v1(
&self,
candidate_hash: &CandidateHash,
_candidate_index: CandidateIndex,
) -> SubsystemResult<Option<CandidateEntry>> {
self.load_candidate_entry(candidate_hash)
}
@@ -446,6 +448,15 @@ fn sign_approval(
key.sign(&ApprovalVote(candidate_hash).signing_payload(session_index)).into()
}
fn sign_approval_multiple_candidates(
key: Sr25519Keyring,
candidate_hashes: Vec<CandidateHash>,
session_index: SessionIndex,
) -> ValidatorSignature {
key.sign(&ApprovalVoteMultipleCandidates(&candidate_hashes).signing_payload(session_index))
.into()
}
type VirtualOverseer = test_helpers::TestSubsystemContextHandle<ApprovalVotingMessage>;
#[derive(Default)]
@@ -641,7 +652,12 @@ async fn check_and_import_approval(
overseer,
FromOrchestra::Communication {
msg: ApprovalVotingMessage::CheckAndImportApproval(
IndirectSignedApprovalVote { block_hash, candidate_index, validator, signature },
IndirectSignedApprovalVoteV2 {
block_hash,
candidate_indices: candidate_index.into(),
validator,
signature,
},
tx,
),
},
@@ -2014,6 +2030,91 @@ fn forkful_import_at_same_height_act_on_leaf() {
});
}
#[test]
fn test_signing_a_single_candidate_is_backwards_compatible() {
let session_index = 1;
let block_hash = Hash::repeat_byte(0x01);
let candidate_descriptors = (1..10)
.into_iter()
.map(|val| make_candidate(ParaId::from(val as u32), &block_hash))
.collect::<Vec<CandidateReceipt>>();
let candidate_hashes = candidate_descriptors
.iter()
.map(|candidate_descriptor| candidate_descriptor.hash())
.collect_vec();
let first_descriptor = candidate_descriptors.first().unwrap();
let candidate_hash = first_descriptor.hash();
let sig_a = sign_approval(Sr25519Keyring::Alice, candidate_hash, session_index);
let sig_b = sign_approval(Sr25519Keyring::Alice, candidate_hash, session_index);
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_a,
)
.is_ok());
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_b,
)
.is_ok());
let sig_c = sign_approval_multiple_candidates(
Sr25519Keyring::Alice,
vec![candidate_hash],
session_index,
);
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(vec![candidate_hash])
)
.check_signature(&Sr25519Keyring::Alice.public().into(), candidate_hash, session_index, &sig_c,)
.is_ok());
assert!(DisputeStatement::Valid(ValidDisputeStatementKind::ApprovalChecking)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
candidate_hash,
session_index,
&sig_c,
)
.is_ok());
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(vec![candidate_hash])
)
.check_signature(&Sr25519Keyring::Alice.public().into(), candidate_hash, session_index, &sig_a,)
.is_ok());
let sig_all = sign_approval_multiple_candidates(
Sr25519Keyring::Alice,
candidate_hashes.clone(),
session_index,
);
assert!(DisputeStatement::Valid(
ValidDisputeStatementKind::ApprovalCheckingMultipleCandidates(candidate_hashes.clone())
)
.check_signature(
&Sr25519Keyring::Alice.public().into(),
*candidate_hashes.first().expect("test"),
session_index,
&sig_all,
)
.is_ok());
}
#[test]
fn import_checked_approval_updates_entries_and_schedules() {
let config = HarnessConfig::default();
@@ -2730,11 +2831,29 @@ async fn handle_double_assignment_import(
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 1,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(_))
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 1,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(_))
@@ -3469,3 +3588,455 @@ fn waits_until_approving_assignments_are_old_enough() {
virtual_overseer
});
}
#[test]
fn test_approval_is_sent_on_max_approval_coalesce_count() {
let assignment_criteria = Box::new(MockAssignmentCriteria(
|| {
let mut assignments = HashMap::new();
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: garbage_assignment_cert(AssignmentCertKind::RelayVRFModulo { sample: 0 })
.into(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let assignments_cert =
garbage_assignment_cert_v2(AssignmentCertKindV2::RelayVRFModuloCompact {
core_bitfield: vec![CoreIndex(0), CoreIndex(1), CoreIndex(2)]
.try_into()
.unwrap(),
});
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let _ = assignments.insert(
CoreIndex(1),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
assignments
},
|_| Ok(0),
));
let config = HarnessConfigBuilder::default().assignment_criteria(assignment_criteria).build();
let store = config.backend();
test_harness(config, |test_harness| async move {
let TestHarness { mut virtual_overseer, clock, sync_oracle_handle: _sync_oracle_handle } =
test_harness;
assert_matches!(
overseer_recv(&mut virtual_overseer).await,
AllMessages::ChainApi(ChainApiMessage::FinalizedBlockNumber(rx)) => {
rx.send(Ok(0)).unwrap();
}
);
let block_hash = Hash::repeat_byte(0x01);
let candidate_commitments = CandidateCommitments::default();
let candidate_receipt1 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(1_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let candidate_hash1 = candidate_receipt1.hash();
let candidate_receipt2 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(2_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let slot = Slot::from(1);
let candidate_index1 = 0;
let candidate_index2 = 1;
let validators = vec![
Sr25519Keyring::Alice,
Sr25519Keyring::Bob,
Sr25519Keyring::Charlie,
Sr25519Keyring::Dave,
Sr25519Keyring::Eve,
];
let session_info = SessionInfo {
validator_groups: IndexedVec::<GroupIndex, Vec<ValidatorIndex>>::from(vec![
vec![ValidatorIndex(0), ValidatorIndex(1)],
vec![ValidatorIndex(2)],
vec![ValidatorIndex(3), ValidatorIndex(4)],
]),
..session_info(&validators)
};
let candidates = Some(vec![
(candidate_receipt1.clone(), CoreIndex(0), GroupIndex(0)),
(candidate_receipt2.clone(), CoreIndex(1), GroupIndex(1)),
]);
ChainBuilder::new()
.add_block(
block_hash,
ChainBuilder::GENESIS_HASH,
1,
BlockConfig {
slot,
candidates: candidates.clone(),
session_info: Some(session_info.clone()),
},
)
.build(&mut virtual_overseer)
.await;
assert!(!clock.inner.lock().current_wakeup_is(1));
clock.inner.lock().wakeup_all(1);
assert!(clock.inner.lock().current_wakeup_is(slot_to_tick(slot)));
clock.inner.lock().wakeup_all(slot_to_tick(slot));
futures_timer::Delay::new(Duration::from_millis(200)).await;
clock.inner.lock().wakeup_all(slot_to_tick(slot + 2));
assert_eq!(clock.inner.lock().wakeups.len(), 0);
futures_timer::Delay::new(Duration::from_millis(200)).await;
let candidate_entry = store.load_candidate_entry(&candidate_hash1).unwrap().unwrap();
let our_assignment =
candidate_entry.approval_entry(&block_hash).unwrap().our_assignment().unwrap();
assert!(our_assignment.triggered());
handle_approval_on_max_coalesce_count(
&mut virtual_overseer,
vec![candidate_index1, candidate_index2],
)
.await;
virtual_overseer
});
}
async fn handle_approval_on_max_coalesce_count(
virtual_overseer: &mut VirtualOverseer,
candidate_indicies: Vec<CandidateIndex>,
) {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeAssignment(
_,
c_indices,
)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies.clone()).unwrap(), c_indices);
}
);
for _ in &candidate_indicies {
recover_available_data(virtual_overseer).await;
fetch_validation_code(virtual_overseer).await;
}
for _ in &candidate_indicies {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::CandidateValidation(CandidateValidationMessage::ValidateFromExhaustive{exec_kind, response_sender, ..}) if exec_kind == PvfExecKind::Approval => {
response_sender.send(Ok(ValidationResult::Valid(Default::default(), Default::default())))
.unwrap();
}
);
}
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 2,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 2,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(vote)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies).unwrap(), vote.candidate_indices);
}
);
// Assert that there are no more messages being sent by the subsystem
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
}
async fn handle_approval_on_max_wait_time(
virtual_overseer: &mut VirtualOverseer,
candidate_indicies: Vec<CandidateIndex>,
clock: Box<MockClock>,
) {
const TICK_NOW_BEGIN: u64 = 1;
const MAX_COALESCE_COUNT: u32 = 3;
clock.inner.lock().set_tick(TICK_NOW_BEGIN);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeAssignment(
_,
c_indices,
)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies.clone()).unwrap(), c_indices);
}
);
for _ in &candidate_indicies {
recover_available_data(virtual_overseer).await;
fetch_validation_code(virtual_overseer).await;
}
for _ in &candidate_indicies {
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::CandidateValidation(CandidateValidationMessage::ValidateFromExhaustive{exec_kind, response_sender, ..}) if exec_kind == PvfExecKind::Approval => {
response_sender.send(Ok(ValidationResult::Valid(Default::default(), Default::default())))
.unwrap();
}
);
}
// First time we fetch the configuration when we are ready to approve the first candidate
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: MAX_COALESCE_COUNT,
}));
}
);
// Second time we fetch the configuration when we are ready to approve the second candidate
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: MAX_COALESCE_COUNT,
}));
}
);
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
// Move the clock just before we should send the approval
clock
.inner
.lock()
.set_tick(MAX_APPROVAL_COALESCE_WAIT_TICKS as Tick + TICK_NOW_BEGIN - 1);
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
// Move the clock tick, so we can trigger a force sending of the approvals
clock
.inner
.lock()
.set_tick(MAX_APPROVAL_COALESCE_WAIT_TICKS as Tick + TICK_NOW_BEGIN);
// Third time we fetch the configuration when timer expires and we are ready to sent the
// approval
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::RuntimeApi(RuntimeApiMessage::Request(_, RuntimeApiRequest::ApprovalVotingParams(_, sender))) => {
let _ = sender.send(Ok(ApprovalVotingParams {
max_approval_coalesce_count: 3,
}));
}
);
assert_matches!(
overseer_recv(virtual_overseer).await,
AllMessages::ApprovalDistribution(ApprovalDistributionMessage::DistributeApproval(vote)) => {
assert_eq!(TryInto::<CandidateBitfield>::try_into(candidate_indicies).unwrap(), vote.candidate_indices);
}
);
// Assert that there are no more messages being sent by the subsystem
assert!(overseer_recv(virtual_overseer).timeout(TIMEOUT / 2).await.is_none());
}
#[test]
fn test_approval_is_sent_on_max_approval_coalesce_wait() {
let assignment_criteria = Box::new(MockAssignmentCriteria(
|| {
let mut assignments = HashMap::new();
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: garbage_assignment_cert(AssignmentCertKind::RelayVRFModulo { sample: 0 })
.into(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let assignments_cert =
garbage_assignment_cert_v2(AssignmentCertKindV2::RelayVRFModuloCompact {
core_bitfield: vec![CoreIndex(0), CoreIndex(1), CoreIndex(2)]
.try_into()
.unwrap(),
});
let _ = assignments.insert(
CoreIndex(0),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
let _ = assignments.insert(
CoreIndex(1),
approval_db::v2::OurAssignment {
cert: assignments_cert.clone(),
tranche: 0,
validator_index: ValidatorIndex(0),
triggered: false,
}
.into(),
);
assignments
},
|_| Ok(0),
));
let config = HarnessConfigBuilder::default().assignment_criteria(assignment_criteria).build();
let store = config.backend();
test_harness(config, |test_harness| async move {
let TestHarness { mut virtual_overseer, clock, sync_oracle_handle: _sync_oracle_handle } =
test_harness;
assert_matches!(
overseer_recv(&mut virtual_overseer).await,
AllMessages::ChainApi(ChainApiMessage::FinalizedBlockNumber(rx)) => {
rx.send(Ok(0)).unwrap();
}
);
let block_hash = Hash::repeat_byte(0x01);
let candidate_commitments = CandidateCommitments::default();
let candidate_receipt1 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(1_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let candidate_hash1 = candidate_receipt1.hash();
let candidate_receipt2 = {
let mut receipt = dummy_candidate_receipt(block_hash);
receipt.descriptor.para_id = ParaId::from(2_u32);
receipt.commitments_hash = candidate_commitments.hash();
receipt
};
let slot = Slot::from(1);
let candidate_index1 = 0;
let candidate_index2 = 1;
let validators = vec![
Sr25519Keyring::Alice,
Sr25519Keyring::Bob,
Sr25519Keyring::Charlie,
Sr25519Keyring::Dave,
Sr25519Keyring::Eve,
];
let session_info = SessionInfo {
validator_groups: IndexedVec::<GroupIndex, Vec<ValidatorIndex>>::from(vec![
vec![ValidatorIndex(0), ValidatorIndex(1)],
vec![ValidatorIndex(2)],
vec![ValidatorIndex(3), ValidatorIndex(4)],
]),
..session_info(&validators)
};
let candidates = Some(vec![
(candidate_receipt1.clone(), CoreIndex(0), GroupIndex(0)),
(candidate_receipt2.clone(), CoreIndex(1), GroupIndex(1)),
]);
ChainBuilder::new()
.add_block(
block_hash,
ChainBuilder::GENESIS_HASH,
1,
BlockConfig {
slot,
candidates: candidates.clone(),
session_info: Some(session_info.clone()),
},
)
.build(&mut virtual_overseer)
.await;
assert!(!clock.inner.lock().current_wakeup_is(1));
clock.inner.lock().wakeup_all(1);
assert!(clock.inner.lock().current_wakeup_is(slot_to_tick(slot)));
clock.inner.lock().wakeup_all(slot_to_tick(slot));
futures_timer::Delay::new(Duration::from_millis(200)).await;
clock.inner.lock().wakeup_all(slot_to_tick(slot + 2));
assert_eq!(clock.inner.lock().wakeups.len(), 0);
futures_timer::Delay::new(Duration::from_millis(200)).await;
let candidate_entry = store.load_candidate_entry(&candidate_hash1).unwrap().unwrap();
let our_assignment =
candidate_entry.approval_entry(&block_hash).unwrap().our_assignment().unwrap();
assert!(our_assignment.triggered());
handle_approval_on_max_wait_time(
&mut virtual_overseer,
vec![candidate_index1, candidate_index2],
clock,
)
.await;
virtual_overseer
});
}
polkadot/node/core/approval-voting/src/time.rs
+164 -1
View File
@@ -16,14 +16,23 @@
//! Time utilities for approval voting.
use futures::prelude::*;
use futures::{
future::BoxFuture,
prelude::*,
stream::{FusedStream, FuturesUnordered},
Stream, StreamExt,
};
use polkadot_node_primitives::approval::v1::DelayTranche;
use sp_consensus_slots::Slot;
use std::{
collections::HashSet,
pin::Pin,
task::Poll,
time::{Duration, SystemTime},
};
use polkadot_primitives::{Hash, ValidatorIndex};
const TICK_DURATION_MILLIS: u64 = 500;
/// A base unit of time, starting from the Unix epoch, split into half-second intervals.
@@ -88,3 +97,157 @@ pub(crate) fn slot_number_to_tick(slot_duration_millis: u64, slot: Slot) -> Tick
let ticks_per_slot = slot_duration_millis / TICK_DURATION_MILLIS;
u64::from(slot) * ticks_per_slot
}
/// A list of delayed futures that gets triggered when the waiting time has expired and it is
/// time to sign the candidate.
/// We have a timer per relay-chain block.
#[derive(Default)]
pub struct DelayedApprovalTimer {
timers: FuturesUnordered<BoxFuture<'static, (Hash, ValidatorIndex)>>,
blocks: HashSet<Hash>,
}
impl DelayedApprovalTimer {
/// Starts a single timer per block hash
///
/// Guarantees that if a timer already exits for the give block hash,
/// no additional timer is started.
pub(crate) fn maybe_arm_timer(
&mut self,
wait_untill: Tick,
clock: &dyn Clock,
block_hash: Hash,
validator_index: ValidatorIndex,
) {
if self.blocks.insert(block_hash) {
let clock_wait = clock.wait(wait_untill);
self.timers.push(Box::pin(async move {
clock_wait.await;
(block_hash, validator_index)
}));
}
}
}
impl Stream for DelayedApprovalTimer {
type Item = (Hash, ValidatorIndex);
fn poll_next(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Self::Item>> {
let poll_result = self.timers.poll_next_unpin(cx);
match poll_result {
Poll::Ready(Some(result)) => {
self.blocks.remove(&result.0);
Poll::Ready(Some(result))
},
_ => poll_result,
}
}
}
impl FusedStream for DelayedApprovalTimer {
fn is_terminated(&self) -> bool {
self.timers.is_terminated()
}
}
#[cfg(test)]
mod tests {
use std::time::Duration;
use futures::{executor::block_on, FutureExt, StreamExt};
use futures_timer::Delay;
use polkadot_primitives::{Hash, ValidatorIndex};
use crate::time::{Clock, SystemClock};
use super::DelayedApprovalTimer;
#[test]
fn test_select_empty_timer() {
block_on(async move {
let mut timer = DelayedApprovalTimer::default();
for _ in 1..10 {
let result = futures::select!(
_ = timer.select_next_some() => {
0
}
// Only this arm should fire
_ = Delay::new(Duration::from_millis(100)).fuse() => {
1
}
);
assert_eq!(result, 1);
}
});
}
#[test]
fn test_timer_functionality() {
block_on(async move {
let mut timer = DelayedApprovalTimer::default();
let test_hashes =
vec![Hash::repeat_byte(0x01), Hash::repeat_byte(0x02), Hash::repeat_byte(0x03)];
for (index, hash) in test_hashes.iter().enumerate() {
timer.maybe_arm_timer(
SystemClock.tick_now() + index as u64,
&SystemClock,
*hash,
ValidatorIndex::from(2),
);
timer.maybe_arm_timer(
SystemClock.tick_now() + index as u64,
&SystemClock,
*hash,
ValidatorIndex::from(2),
);
}
let timeout_hash = Hash::repeat_byte(0x02);
for i in 0..test_hashes.len() * 2 {
let result = futures::select!(
(hash, _) = timer.select_next_some() => {
hash
}
// Timers should fire only once, so for the rest of the iterations we should timeout through here.
_ = Delay::new(Duration::from_secs(2)).fuse() => {
timeout_hash
}
);
assert_eq!(test_hashes.get(i).cloned().unwrap_or(timeout_hash), result);
}
// Now check timer can be restarted if already fired
for (index, hash) in test_hashes.iter().enumerate() {
timer.maybe_arm_timer(
SystemClock.tick_now() + index as u64,
&SystemClock,
*hash,
ValidatorIndex::from(2),
);
timer.maybe_arm_timer(
SystemClock.tick_now() + index as u64,
&SystemClock,
*hash,
ValidatorIndex::from(2),
);
}
for i in 0..test_hashes.len() * 2 {
let result = futures::select!(
(hash, _) = timer.select_next_some() => {
hash
}
// Timers should fire only once, so for the rest of the iterations we should timeout through here.
_ = Delay::new(Duration::from_secs(2)).fuse() => {
timeout_hash
}
);
assert_eq!(test_hashes.get(i).cloned().unwrap_or(timeout_hash), result);
}
});
}
}