Fixing BABE epochs to change between blocks (#3583)

* always fetch epoch from runtime

* node integration tests don't test light nodes

* give stand-in full node a FULL role

* rejig babe APIs

* introduce next-epoch-descriptor type

* overhaul srml-BABE epoch logic

* ensure VRF outputs end up in the right epoch-randomness

* rewrite `do_initialize` to remove unnecessary loop

* begin accounting for next epoch in epoch function

* slots passes header to epoch_data

* pass slot_number to SlotWorker::epoch_data

* begin extracting epoch-change logic into its own module

* aux methods for block weight

* aux methods for genesis configuration

* comment-out most, refactor header-check pipeline

* mostly flesh out verifier again

* reinstantiate babe BlockImport implementation

* reinstate import-queue instantiation

* reintroduce slot-worker implementation

* reinstate pretty much all the rest

* move fork-choice logic to BlockImport

* fix some, but not all errors

* patch test-runtime

* make is_descendent of slightly more generic

* get skeleton compiling when passing is_descendent_of

* make descendent-of-builder more succinct

* restore ordering of authority_index / slot_number

* start fiddling with tests

* fix warnings

* improve initialization architecture and handle genesis

* tests use correct block-import

* fix BABE tests

* fix some compiler errors

* fix node-cli compilation

* all crates compile

* bump runtime versions and fix some warnings

* tweak fork-tree search implementation

* do backtracking search in fork-tree

* node-cli integration tests now work

* fix broken assumption in test_connectivity

* babe tests fail for the right reasons.

* test genesis epoch logic for epoch_changes

* test that epochs can change between blocks

* First BABE SRML test

* Testing infrastructure for BABE

Also includes a trivial additional test.

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* A little more test progress

* More work on BABE testing

* Try to get the tests working

* Implement `UintAuthorityId`-based test mocks

* Fix compilation errors

* Adjust to upstream changes

* Block numbers are ignored in BABE epoch calculation

* authority_index() should ignore invalid authorities

* Fix compile error

* Add tests that session transitions happen

* Check if BABE produces logs

It currently does not.

* Fix test suite

This was really nasty, due to a type confusion that showed up as an
off-by-1 buffer error.

* Add additional tests

Most of these were derived from the current output, so they are only
useful to guard against regressions.

* Make the tests more readable

Also bump impl_version.

* Fix excessive line width

* Remove unused imports

* Update srml/babe/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* try to fix imports

* Fix build errors in test suite

* tests did not pass

* Try to get at least one digest to be output

Currently, the code emits either no digests (if I don’t call
`Session::rotate_session()` or two digests (if I do), which is wrong.

* More tests

They still don’t work, but this should help debugging.

* fix silly error

* Don’t even try to compile a broken test

* remove broken check_epoch test and add one for genesis epoch

* Check that the length of the pre-digests is correct

* Bump `impl_version`

* use epoch_for_descendent_of even for genesis

* account for competing block 1s

* finish srml-babe docs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* address grumbles
This commit is contained in:
Robert Habermeier
2019-09-23 16:03:05 +02:00
committed by GitHub
parent e6d4a76521
commit c200ce757b
34 changed files with 2168 additions and 989 deletions
+84 -94
View File
@@ -18,32 +18,36 @@
//! from VRF outputs and manages epoch transitions.
#![cfg_attr(not(feature = "std"), no_std)]
#![forbid(unused_must_use, unsafe_code, unused_variables)]
// TODO: @marcio uncomment this when BabeEquivocation is integrated.
// #![forbid(dead_code)]
#![forbid(unused_must_use, unsafe_code, unused_variables, unused_must_use)]
#![deny(unused_imports)]
pub use timestamp;
use rstd::{result, prelude::*};
use support::{decl_storage, decl_module, StorageValue, StorageMap, traits::FindAuthor, traits::Get};
use timestamp::{OnTimestampSet};
use timestamp::OnTimestampSet;
use sr_primitives::{generic::DigestItem, ConsensusEngineId, Perbill};
use sr_primitives::traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon};
use sr_staking_primitives::{
SessionIndex,
offence::{Offence, Kind},
};
use sr_primitives::weights::SimpleDispatchInfo;
#[cfg(feature = "std")]
use timestamp::TimestampInherentData;
use codec::{Encode, Decode};
use inherents::{RuntimeString, InherentIdentifier, InherentData, ProvideInherent, MakeFatalError};
#[cfg(feature = "std")]
use inherents::{InherentDataProviders, ProvideInherentData};
use babe_primitives::{BABE_ENGINE_ID, ConsensusLog, BabeAuthorityWeight, Epoch, RawBabePreDigest};
use babe_primitives::{
BABE_ENGINE_ID, ConsensusLog, BabeAuthorityWeight, NextEpochDescriptor, RawBabePreDigest,
SlotNumber,
};
pub use babe_primitives::{AuthorityId, VRF_OUTPUT_LENGTH, PUBLIC_KEY_LENGTH};
use system::ensure_root;
#[cfg(all(feature = "std", test))]
mod tests;
#[cfg(all(feature = "std", test))]
mod mock;
/// The BABE inherent identifier.
pub const INHERENT_IDENTIFIER: InherentIdentifier = *b"babeslot";
@@ -118,7 +122,7 @@ impl ProvideInherentData for InherentDataProvider {
}
pub trait Trait: timestamp::Trait {
type EpochDuration: Get<u64>;
type EpochDuration: Get<SlotNumber>;
type ExpectedBlockTime: Get<Self::Moment>;
}
@@ -127,6 +131,8 @@ pub const RANDOMNESS_LENGTH: usize = 32;
const UNDER_CONSTRUCTION_SEGMENT_LENGTH: usize = 256;
type MaybeVrf = Option<[u8; 32 /* VRF_OUTPUT_LENGTH */]>;
decl_storage! {
trait Store for Module<T: Trait> as Babe {
/// Current epoch index.
@@ -135,22 +141,13 @@ decl_storage! {
/// Current epoch authorities.
pub Authorities get(authorities): Vec<(AuthorityId, BabeAuthorityWeight)>;
/// Slot at which the current epoch started. It is possible that no
/// block was authored at the given slot and the epoch change was
/// signalled later than this.
pub EpochStartSlot get(epoch_start_slot): u64;
/// The slot at which the first epoch actually started. This is 0
/// until the first block of the chain.
pub GenesisSlot get(genesis_slot): u64;
/// Current slot number.
pub CurrentSlot get(current_slot): u64;
/// Whether secondary slots are enabled in case the VRF-based slot is
/// empty for the current epoch and the next epoch, respectively.
pub SecondarySlots get(secondary_slots): (bool, bool) = (true, true);
/// Pending change to enable/disable secondary slots which will be
/// triggered at `current_epoch + 2`.
pub PendingSecondarySlotsChange get(pending_secondary_slots_change): Option<bool> = None;
/// The epoch randomness for the *current* epoch.
///
/// # Security
@@ -181,9 +178,9 @@ decl_storage! {
SegmentIndex build(|_| 0): u32;
UnderConstruction: map u32 => Vec<[u8; 32 /* VRF_OUTPUT_LENGTH */]>;
/// Temporary value (cleared at block finalization) which is true
/// Temporary value (cleared at block finalization) which is `Some`
/// if per-block initialization has already been called for current block.
Initialized get(initialized): Option<bool>;
Initialized get(initialized): Option<MaybeVrf>;
}
add_extra_genesis {
config(authorities): Vec<(AuthorityId, BabeAuthorityWeight)>;
@@ -212,20 +209,13 @@ decl_module! {
/// Block finalization
fn on_finalize() {
Initialized::kill();
}
/// Sets a pending change to enable / disable secondary slot assignment.
/// The pending change will be set at the end of the current epoch and
/// will be enacted at `current_epoch + 2`.
#[weight = SimpleDispatchInfo::FixedOperational(10_000)]
fn set_pending_secondary_slots_change(origin, change: Option<bool>) {
ensure_root(origin)?;
match change {
Some(change) => PendingSecondarySlotsChange::put(change),
None => {
PendingSecondarySlotsChange::take();
},
// at the end of the block, we can safely include the new VRF output
// from this block into the under-construction randomness. If we've determined
// that this block was the first in a new epoch, the changeover logic has
// already occurred at this point, so the under-construction randomness
// will only contain outputs from the right epoch.
if let Some(Some(vrf_output)) = Initialized::take() {
Self::deposit_vrf_output(&vrf_output);
}
}
}
@@ -269,15 +259,25 @@ impl<T: Trait> IsMember<AuthorityId> for Module<T> {
}
impl<T: Trait> session::ShouldEndSession<T::BlockNumber> for Module<T> {
fn should_end_session(_: T::BlockNumber) -> bool {
fn should_end_session(now: T::BlockNumber) -> bool {
// it might be (and it is in current implementation) that session module is calling
// should_end_session() from it's own on_initialize() handler
// => because session on_initialize() is called earlier than ours, let's ensure
// that we have synced with digest before checking if session should be ended
Self::do_initialize();
let diff = CurrentSlot::get().saturating_sub(EpochStartSlot::get());
diff >= T::EpochDuration::get()
// The session has technically ended during the passage of time
// between this block and the last, but we have to "end" the session now,
// since there is no earlier possible block we could have done it.
//
// The exception is for block 1: the genesis has slot 0, so we treat
// epoch 0 as having started at the slot of block 1. We want to use
// the same randomness and validator set as signalled in the genesis,
// so we don't rotate the session.
now != sr_primitives::traits::One::one() && {
let diff = CurrentSlot::get().saturating_sub(Self::current_epoch_start());
diff >= T::EpochDuration::get()
}
}
}
@@ -336,15 +336,18 @@ impl<T: Trait> Module<T> {
<T as timestamp::Trait>::MinimumPeriod::get().saturating_mul(2.into())
}
// finds the start slot of the current epoch. only guaranteed to
// give correct results after `do_initialize` of the first block
// in the chain (as its result is based off of `GenesisSlot`).
fn current_epoch_start() -> SlotNumber {
(EpochIndex::get() * T::EpochDuration::get()) + GenesisSlot::get()
}
fn deposit_consensus<U: Encode>(new: U) {
let log: DigestItem<T::Hash> = DigestItem::Consensus(BABE_ENGINE_ID, new.encode());
<system::Module<T>>::deposit_log(log.into())
}
fn get_inherent_digests() -> system::DigestOf<T> {
<system::Module<T>>::digest()
}
fn deposit_vrf_output(vrf_output: &[u8; VRF_OUTPUT_LENGTH]) {
let segment_idx = <SegmentIndex>::get();
let mut segment = <UnderConstruction>::get(&segment_idx);
@@ -363,13 +366,12 @@ impl<T: Trait> Module<T> {
fn do_initialize() {
// since do_initialize can be called twice (if session module is present)
// => let's ensure that we only modify the storage once per block
let initialized = Self::initialized().unwrap_or(false);
let initialized = Self::initialized().is_some();
if initialized {
return;
}
Initialized::put(true);
for digest in Self::get_inherent_digests()
let maybe_pre_digest = <system::Module<T>>::digest()
.logs
.iter()
.filter_map(|s| s.as_pre_runtime())
@@ -378,19 +380,40 @@ impl<T: Trait> Module<T> {
} else {
None
})
{
if EpochStartSlot::get() == 0 {
EpochStartSlot::put(digest.slot_number());
.next();
let maybe_vrf = maybe_pre_digest.and_then(|digest| {
// on the first non-zero block (i.e. block #1)
// this is where the first epoch (epoch #0) actually starts.
// we need to adjust internal storage accordingly.
if GenesisSlot::get() == 0 {
GenesisSlot::put(digest.slot_number());
debug_assert_ne!(GenesisSlot::get(), 0);
// deposit a log because this is the first block in epoch #0
// we use the same values as genesis because we haven't collected any
// randomness yet.
let next = NextEpochDescriptor {
authorities: Self::authorities(),
randomness: Self::randomness(),
};
Self::deposit_consensus(ConsensusLog::NextEpochData(next))
}
CurrentSlot::put(digest.slot_number());
if let RawBabePreDigest::Primary { vrf_output, .. } = digest {
Self::deposit_vrf_output(&vrf_output);
// place the VRF output into the `Initialized` storage item
// and it'll be put onto the under-construction randomness
// later, once we've decided which epoch this block is in.
Some(vrf_output)
} else {
None
}
});
return;
}
Initialized::put(maybe_vrf);
}
/// Call this function exactly once when an epoch changes, to update the
@@ -437,7 +460,12 @@ impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
fn on_new_session<'a, I: 'a>(_changed: bool, validators: I, queued_validators: I)
where I: Iterator<Item=(&'a T::AccountId, AuthorityId)>
{
Self::do_initialize();
// PRECONDITION: `should_end_session` has done initialization and is guaranteed
// by the session module to be called before this.
#[cfg(debug_assertions)]
{
assert!(Self::initialized().is_some())
}
// Update epoch index
let epoch_index = EpochIndex::get()
@@ -453,21 +481,6 @@ impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
Authorities::put(authorities);
// Update epoch start slot.
let now = CurrentSlot::get();
EpochStartSlot::mutate(|previous| {
loop {
// on the first epoch we must account for skipping at least one
// whole epoch, in case the first block is authored with a slot
// number far in the past.
if now.saturating_sub(*previous) < T::EpochDuration::get() {
break;
}
*previous = previous.saturating_add(T::EpochDuration::get());
}
});
// Update epoch randomness.
let next_epoch_index = epoch_index
.checked_add(1)
@@ -484,34 +497,11 @@ impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
(k, 1)
}).collect::<Vec<_>>();
let next_epoch_start_slot = EpochStartSlot::get().saturating_add(T::EpochDuration::get());
let next_randomness = NextRandomness::get();
// Update any pending secondary slots change
let mut secondary_slots = SecondarySlots::get();
// change for E + 1 now becomes change at E
secondary_slots.0 = secondary_slots.1;
if let Some(change) = PendingSecondarySlotsChange::take() {
// if there's a pending change schedule it for E + 1
secondary_slots.1 = change;
} else {
// otherwise E + 1 will have the same value as E
secondary_slots.1 = secondary_slots.0;
}
SecondarySlots::mutate(|secondary| {
*secondary = secondary_slots;
});
let next = Epoch {
epoch_index: next_epoch_index,
start_slot: next_epoch_start_slot,
duration: T::EpochDuration::get(),
let next = NextEpochDescriptor {
authorities: next_authorities,
randomness: next_randomness,
secondary_slots: secondary_slots.1,
};
Self::deposit_consensus(ConsensusLog::NextEpochData(next))
+112
View File
@@ -0,0 +1,112 @@
// Copyright 2019 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
//! Test utilities
#![allow(dead_code, unused_imports)]
use super::{Trait, Module, GenesisConfig};
use babe_primitives::AuthorityId;
use sr_primitives::{
traits::IdentityLookup, Perbill,
testing::{Header, UintAuthorityId},
impl_opaque_keys, key_types::DUMMY,
};
use sr_version::RuntimeVersion;
use support::{impl_outer_origin, parameter_types};
use runtime_io;
use primitives::{H256, Blake2Hasher};
impl_outer_origin!{
pub enum Origin for Test {}
}
type DummyValidatorId = u64;
// Workaround for https://github.com/rust-lang/rust/issues/26925 . Remove when sorted.
#[derive(Clone, PartialEq, Eq, Debug)]
pub struct Test;
parameter_types! {
pub const BlockHashCount: u64 = 250;
pub const MaximumBlockWeight: u32 = 1024;
pub const MaximumBlockLength: u32 = 2 * 1024;
pub const AvailableBlockRatio: Perbill = Perbill::one();
pub const MinimumPeriod: u64 = 1;
pub const EpochDuration: u64 = 3;
pub const ExpectedBlockTime: u64 = 1;
pub const Version: RuntimeVersion = test_runtime::VERSION;
pub const DisabledValidatorsThreshold: Perbill = Perbill::from_percent(16);
}
impl system::Trait for Test {
type Origin = Origin;
type Index = u64;
type BlockNumber = u64;
type Call = ();
type Hash = H256;
type Version = Version;
type Hashing = sr_primitives::traits::BlakeTwo256;
type AccountId = DummyValidatorId;
type Lookup = IdentityLookup<Self::AccountId>;
type Header = Header;
type WeightMultiplierUpdate = ();
type Event = ();
type BlockHashCount = BlockHashCount;
type MaximumBlockWeight = MaximumBlockWeight;
type AvailableBlockRatio = AvailableBlockRatio;
type MaximumBlockLength = MaximumBlockLength;
}
impl_opaque_keys! {
pub struct MockSessionKeys {
#[id(DUMMY)]
pub dummy: UintAuthorityId,
}
}
impl session::Trait for Test {
type Event = ();
type ValidatorId = <Self as system::Trait>::AccountId;
type ShouldEndSession = Babe;
type SessionHandler = (Babe,Babe,);
type OnSessionEnding = ();
type ValidatorIdOf = ();
type SelectInitialValidators = ();
type Keys = MockSessionKeys;
type DisabledValidatorsThreshold = DisabledValidatorsThreshold;
}
impl timestamp::Trait for Test {
type Moment = u64;
type OnTimestampSet = Babe;
type MinimumPeriod = MinimumPeriod;
}
impl Trait for Test {
type EpochDuration = EpochDuration;
type ExpectedBlockTime = ExpectedBlockTime;
}
pub fn new_test_ext(authorities: Vec<DummyValidatorId>) -> runtime_io::TestExternalities<Blake2Hasher> {
let mut t = system::GenesisConfig::default().build_storage::<Test>().unwrap();
GenesisConfig {
authorities: authorities.into_iter().map(|a| (UintAuthorityId(a).to_public_key(), 1)).collect(),
}.assimilate_storage::<Test>(&mut t).unwrap();
t.into()
}
pub type System = system::Module<Test>;
pub type Babe = Module<Test>;
+127
View File
@@ -0,0 +1,127 @@
// Copyright 2019 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
//! Consensus extension module tests for BABE consensus.
use super::*;
use runtime_io::with_externalities;
use mock::{new_test_ext, Babe, Test};
use sr_primitives::{traits::OnFinalize, testing::{Digest, DigestItem}};
use session::ShouldEndSession;
const EMPTY_RANDOMNESS: [u8; 32] = [
74, 25, 49, 128, 53, 97, 244, 49,
222, 202, 176, 2, 231, 66, 95, 10,
133, 49, 213, 228, 86, 161, 164, 127,
217, 153, 138, 37, 48, 192, 248, 0,
];
fn make_pre_digest(
authority_index: babe_primitives::AuthorityIndex,
slot_number: babe_primitives::SlotNumber,
vrf_output: [u8; babe_primitives::VRF_OUTPUT_LENGTH],
vrf_proof: [u8; babe_primitives::VRF_PROOF_LENGTH],
) -> Digest {
let digest_data = babe_primitives::RawBabePreDigest::Primary {
authority_index,
slot_number,
vrf_output,
vrf_proof,
};
let log = DigestItem::PreRuntime(babe_primitives::BABE_ENGINE_ID, digest_data.encode());
Digest { logs: vec![log] }
}
#[test]
fn empty_randomness_is_correct() {
let s = compute_randomness([0; RANDOMNESS_LENGTH], 0, std::iter::empty(), None);
assert_eq!(s, EMPTY_RANDOMNESS);
}
#[test]
fn initial_values() {
with_externalities(&mut new_test_ext(vec![0, 1, 2, 3]), || {
assert_eq!(Babe::authorities().len(), 4)
})
}
#[test]
fn check_module() {
with_externalities(&mut new_test_ext(vec![0, 1, 2, 3]), || {
assert!(!Babe::should_end_session(0), "Genesis does not change sessions");
assert!(!Babe::should_end_session(200000),
"BABE does not include the block number in epoch calculations");
})
}
type System = system::Module<Test>;
#[test]
fn first_block_epoch_zero_start() {
with_externalities(&mut new_test_ext(vec![0, 1, 2, 3]), || {
let genesis_slot = 100;
let first_vrf = [1; 32];
let pre_digest = make_pre_digest(
0,
genesis_slot,
first_vrf,
[0xff; 64],
);
assert_eq!(Babe::genesis_slot(), 0);
System::initialize(&1, &Default::default(), &Default::default(), &pre_digest);
// see implementation of the function for details why: we issue an
// epoch-change digest but don't do it via the normal session mechanism.
assert!(!Babe::should_end_session(1));
assert_eq!(Babe::genesis_slot(), genesis_slot);
assert_eq!(Babe::current_slot(), genesis_slot);
assert_eq!(Babe::epoch_index(), 0);
Babe::on_finalize(1);
let header = System::finalize();
assert_eq!(SegmentIndex::get(), 0);
assert_eq!(UnderConstruction::get(0), vec![first_vrf]);
assert_eq!(Babe::randomness(), [0; 32]);
assert_eq!(NextRandomness::get(), [0; 32]);
assert_eq!(header.digest.logs.len(), 2);
assert_eq!(pre_digest.logs.len(), 1);
assert_eq!(header.digest.logs[0], pre_digest.logs[0]);
let authorities = Babe::authorities();
let consensus_log = babe_primitives::ConsensusLog::NextEpochData(
babe_primitives::NextEpochDescriptor {
authorities,
randomness: Babe::randomness(),
}
);
let consensus_digest = DigestItem::Consensus(BABE_ENGINE_ID, consensus_log.encode());
// first epoch descriptor has same info as last.
assert_eq!(header.digest.logs[1], consensus_digest.clone())
})
}
#[test]
fn authority_index() {
with_externalities(&mut new_test_ext(vec![0, 1, 2, 3]), || {
assert_eq!(
Babe::find_author((&[(BABE_ENGINE_ID, &[][..])]).into_iter().cloned()), None,
"Trivially invalid authorities are ignored")
})
}