BABE Epochs (#3028)

* Add `epoch` field to `SlotInfo`

* Add slot calculations

* More work on epochs in BABE

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Typo: `/` not `%` for division

* Delete useless `LastSlotInEpoch::put(false)`

* Bump `spec_version`

* Make test suite pass again

* Implement BABE epoch randomness signing

* Try to fix compilation

Currently causes a stack overflow in the compiler

* Fix rustc stack overflow

* Add missing `PartialEq` and `Eq` implementations

* Fix compile errors in test suite

* Another silly compile error

* Clone `epoch`

* Fix compile error in benchmarks

* Implement `clone` for `Epoch`

* Merge master

* AUTHORING TEST PASSES!!!

* Fix compilation

* Bump `spec_version`

* Fix compilation

* Fix compilation (again)

* Remove an outdated FIXME

* Fix run.sh and move it to scripts/

* Delete commented-out code

* Fix documentation

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Fix BABE initialization and refactor

* Respond to review

* typo

* Remove useless data in `CheckedHeader::Deferred`

* Remove `slot_number` from Epoch

It is not needed, and only served to waste space and cause confusion.

* Remove epoch from BABE digests

* Move digest.rs to primitives

* Fix incorrect warning names

* Fix compile error

* Consistent field naming for BABE digests

* More compiler error fixex

* Unbound variable

* more compile errors

* another compile error

* Fix compile errors in runtime

* another compile error

* Another compile error

* Fix wasm build

* missing import

* Fix more compile errors

* yet another compile error

* compile fix in test runtime

* Fix and simplify the BABE runtime

The BABE runtime was massively overcomplicated and also wrong.  It
assumed it needed to:

1. delay new authorities taking effect until the next epoch
2. not delay emitting `Consensus` digests to mark epoch changes

However, the first is handled by the `srml_session` crate, and the
second is flat-out incorrect: `Consensus` digests take effect
immediately.  Furthermore, `srml_babe` tried to duplicate the
functionality of `srml_session::PeriodicSession`, but did it both
clumsily and incorrectly.  Fortunately, the new code is simpler and far
more likely to be correct.

* Use `system` to get the test authorities

The genesis block used by tests defines no authorities.  Only the test
suite is affected.

* Fix test runtime impl for BabeApi::epoch() with std

* Fix compilation

* Cached authorities are in the form of an epoch

not a `Vec<AuthorityId>`.

* `slots_per_epoch` is not fixed in general

The BABE code previously assumed `slots_per_epoch` to be a constant,
but that assumption is false in general.  Furthermore, removing this
assumption also allows a lot of code to go away.

* fix compile error

* Implement epoch checker

* Fix runtime compilation

* fork-tree: add method for finding a node in the tree

* babe: register epoch transitions in fork tree and validate them

* fork-tree: add method for arbitrary pruning

* Expose the queued validator set to SRML modules

BABE needs to know not only what the current validator set is, but also
what the next validator set will be.  Expose this to clients of the
session module.

* Bump hex-literal

Hopefully this will fix the panic

* babe: prune epoch change fork tree on finality

* babe: validate epoch index on transition

* babe: persist epoch changes tree

* Fix compile error in tests

* Fix compile error in tests

* Another compile error in tests

* Fix compilation of tests

* core: move grandpa::is_descendent_of to client utils

* babe: use is_descendent_of from client utils

* babe: extract slot_number from pre_digest in import_block

* Move BABE testsuite to its own file

* Initial part of test code

* Missing `WeightMultiplierUpdate` in test-runtime

* bump `spec_version`

* Add a test that a very bogus is rejected

* Run the tests again

* Fix compiler diagnostics

* Bump `spec_version`

* Initial infrastructure for mutation testing

* Mutation testing of block import

* babe: revert epoch changes in case of block import error

* babe: fix logging target

* babe: BabeBlockImport doesn't box inner BlockImport

* babe: fix epoch check in block import

* babe: populate authorities cache on block authorship

* babe: remove unused functions

* babe: use RANDOMNESS_LENGTH const

* babe: remove unneeded config parameters

* core: revert change to hex dependency version

* cleanup gitignore

* babe: add docs to aux_schema

* babe: remove useless drops in tests

* babe: remove annoying macos smart quotes

* fork-tree: docs

* fork-tree: add tests

* babe: style

* babe: rename randomness config variable

* babe: remove randomness helper function

* babe: style fixes

* babe: add docs

* babe: fix tests

* node: bump spec_version

* babe: fix tests

substrate/srml/babe/src/lib.rs

@@ -23,26 +23,22 @@ pub use timestamp;
 use rstd::{result, prelude::*};
 use srml_support::{decl_storage, decl_module, StorageValue, traits::FindAuthor, traits::Get};
 use timestamp::{OnTimestampSet, Trait};
-use primitives::{
-	generic::DigestItem,
-	traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon}
-};
-use primitives::ConsensusEngineId;
+use primitives::{generic::DigestItem, ConsensusEngineId};
+use primitives::traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon, Convert};
 #[cfg(feature = "std")]
 use timestamp::TimestampInherentData;
 use parity_codec::{Encode, Decode};
 use inherents::{RuntimeString, InherentIdentifier, InherentData, ProvideInherent, MakeFatalError};
 #[cfg(feature = "std")]
 use inherents::{InherentDataProviders, ProvideInherentData};
-use babe_primitives::{BABE_ENGINE_ID, ConsensusLog};
-pub use babe_primitives::{AuthorityId, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH, PUBLIC_KEY_LENGTH};
+use babe_primitives::{BABE_ENGINE_ID, ConsensusLog, Weight, Epoch, RawBabePreDigest};
+pub use babe_primitives::{AuthorityId, VRF_OUTPUT_LENGTH, PUBLIC_KEY_LENGTH};
 
 /// The BABE inherent identifier.
 pub const INHERENT_IDENTIFIER: InherentIdentifier = *b"babeslot";
 
 /// The type of the BABE inherent.
 pub type InherentType = u64;
-
 /// Auxiliary trait to extract BABE inherent data.
 pub trait BabeInherentData {
 	/// Get BABE inherent data.
@@ -101,8 +97,8 @@ impl ProvideInherentData for InherentDataProvider {
 		inherent_data: &mut InherentData,
 	) -> result::Result<(), RuntimeString> {
 		let timestamp = inherent_data.timestamp_inherent_data()?;
-		let slot_num = timestamp / self.slot_duration;
-		inherent_data.put_data(INHERENT_IDENTIFIER, &slot_num)
+		let slot_number = timestamp / self.slot_duration;
+		inherent_data.put_data(INHERENT_IDENTIFIER, &slot_number)
 	}
 
 	fn error_to_string(&self, error: &[u8]) -> Option<String> {
@@ -115,13 +111,15 @@ pub const RANDOMNESS_LENGTH: usize = 32;
 
 decl_storage! {
 	trait Store for Module<T: Trait> as Babe {
-		/// The last timestamp.
-		LastTimestamp get(last): T::Moment;
+		NextRandomness: [u8; RANDOMNESS_LENGTH];
 
-		/// The current authorities set.
-		Authorities get(authorities): Vec<AuthorityId>;
+		/// Randomness under construction
+		UnderConstruction: [u8; VRF_OUTPUT_LENGTH];
 
-		/// The epoch randomness.
+		/// Current epoch
+		pub Authorities get(authorities) config(): Vec<(AuthorityId, Weight)>;
+
+		/// The epoch randomness for the *current* epoch.
 		///
 		/// # Security
 		///
@@ -131,16 +129,10 @@ decl_storage! {
 		/// (like everything else on-chain) it is public. For example, it can be
 		/// used where a number is needed that cannot have been chosen by an
 		/// adversary, for purposes such as public-coin zero-knowledge proofs.
-		EpochRandomness get(epoch_randomness): [u8; VRF_OUTPUT_LENGTH];
+		pub Randomness get(randomness): [u8; RANDOMNESS_LENGTH];
 
-		/// The randomness under construction
-		UnderConstruction: [u8; VRF_OUTPUT_LENGTH];
-
-		/// The randomness for the next epoch
-		NextEpochRandomness: [u8; VRF_OUTPUT_LENGTH];
-
-		/// The current epoch
-		EpochIndex get(epoch_index): u64;
+		/// Current epoch index
+		EpochIndex: u64;
 	}
 }
 
@@ -154,11 +146,13 @@ decl_module! {
 				.iter()
 				.filter_map(|s| s.as_pre_runtime())
 				.filter_map(|(id, mut data)| if id == BABE_ENGINE_ID {
-					<[u8; VRF_OUTPUT_LENGTH]>::decode(&mut data)
+					RawBabePreDigest::decode(&mut data)
 				} else {
 					None
 				}) {
-				Self::deposit_vrf_output(&i);
+
+				Self::deposit_vrf_output(&i.vrf_output);
+				return;
 			}
 		}
 	}
@@ -166,7 +160,7 @@ decl_module! {
 
 impl<T: Trait> RandomnessBeacon for Module<T> {
 	fn random() -> [u8; VRF_OUTPUT_LENGTH] {
-		Self::epoch_randomness()
+		Self::randomness()
 	}
 }
 
@@ -179,15 +173,10 @@ impl<T: Trait> FindAuthor<u64> for Module<T> {
 	{
 		for (id, mut data) in digests.into_iter() {
 			if id == BABE_ENGINE_ID {
-				let (_, _, i): (
-					[u8; VRF_OUTPUT_LENGTH],
-					[u8; VRF_PROOF_LENGTH],
-					u64,
-				) = Decode::decode(&mut data)?;
-				return Some(i)
+				return Some(RawBabePreDigest::decode(&mut data)?.authority_index);
 			}
 		}
-		return None
+		return None;
 	}
 }
 
@@ -195,7 +184,7 @@ impl<T: timestamp::Trait> IsMember<AuthorityId> for Module<T> {
 	fn is_member(authority_id: &AuthorityId) -> bool {
 		<Module<T>>::authorities()
 			.iter()
-			.any(|id| id == authority_id)
+			.any(|id| &id.0 == authority_id)
 	}
 }
 
@@ -207,65 +196,97 @@ impl<T: Trait> Module<T> {
 		<T as timestamp::Trait>::MinimumPeriod::get().saturating_mul(2.into())
 	}
 
-	fn change_authorities(new: Vec<AuthorityId>) {
-		Authorities::put(&new);
-
+	fn deposit_consensus<U: Encode>(new: U) {
 		let log: DigestItem<T::Hash> = DigestItem::Consensus(BABE_ENGINE_ID, new.encode());
-		<system::Module<T>>::deposit_log(log.into());
+		<system::Module<T>>::deposit_log(log.into())
+	}
+
+	fn get_inherent_digests() -> system::DigestOf<T> {
+		<system::Module<T>>::digest()
+	}
+
+	fn change_epoch(new: Epoch) {
+		Authorities::put(&new.authorities);
+		Randomness::put(&new.randomness);
+		Self::deposit_consensus(ConsensusLog::NextEpochData(new))
 	}
 
 	fn deposit_vrf_output(vrf_output: &[u8; VRF_OUTPUT_LENGTH]) {
 		UnderConstruction::mutate(|z| z.iter_mut().zip(vrf_output).for_each(|(x, y)| *x^=y))
 	}
 
-	fn get_inherent_digests() -> system::DigestOf<T> {
-		<system::Module<T>>::digest()
+	/// Call this function exactly once when an epoch changes, to update the
+	/// randomness. Returns the new randomness.
+	fn randomness_change_epoch(epoch_index: u64) -> [u8; RANDOMNESS_LENGTH] {
+		let this_randomness = NextRandomness::get();
+		let next_randomness = compute_randomness(
+			this_randomness,
+			epoch_index,
+			UnderConstruction::get(),
+		);
+		UnderConstruction::put(&[0; RANDOMNESS_LENGTH]);
+		NextRandomness::put(&next_randomness);
+		this_randomness
 	}
+
 }
 
 impl<T: Trait> OnTimestampSet<T::Moment> for Module<T> {
 	fn on_timestamp_set(_moment: T::Moment) { }
 }
 
-impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
-	type Key = AuthorityId;
+pub trait Duration {
+	fn babe_epoch_duration() -> u64;
+}
 
-	fn on_new_session<'a, I: 'a>(changed: bool, validators: I)
+impl<T: Trait + staking::Trait + Duration> session::OneSessionHandler<T::AccountId> for Module<T> {
+	type Key = AuthorityId;
+	fn on_new_session<'a, I: 'a>(_changed: bool, _validators: I, queued_validators: I)
 		where I: Iterator<Item=(&'a T::AccountId, AuthorityId)>
 	{
-		// instant changes
-		if changed {
-			let next_authorities = validators.map(|(_, k)| k).collect::<Vec<_>>();
-			let last_authorities = <Module<T>>::authorities();
-			if next_authorities != last_authorities {
-				Self::change_authorities(next_authorities);
-			}
-		}
+		use staking::BalanceOf;
+		let to_votes = |b: BalanceOf<T>| {
+			<T::CurrencyToVote as Convert<BalanceOf<T>, u64>>::convert(b)
+		};
 
-		let rho = UnderConstruction::get();
-		UnderConstruction::put([0; 32]);
-		let last_epoch_randomness = EpochRandomness::get();
 		let epoch_index = EpochIndex::get()
 			.checked_add(1)
 			.expect("epoch indices will never reach 2^64 before the death of the universe; qed");
+
 		EpochIndex::put(epoch_index);
-		EpochRandomness::put(NextEpochRandomness::get());
-		let mut s = [0; 72];
-		s[..32].copy_from_slice(&last_epoch_randomness);
-		s[32..40].copy_from_slice(&epoch_index.to_le_bytes());
-		s[40..].copy_from_slice(&rho);
-		NextEpochRandomness::put(runtime_io::blake2_256(&s))
+
+		// *Next* epoch's authorities.
+		let authorities = queued_validators.map(|(account, k)| {
+			(k, to_votes(staking::Module::<T>::stakers(account).total))
+		}).collect::<Vec<_>>();
+
+		// What was the next epoch is now the current epoch
+		let randomness = Self::randomness_change_epoch(epoch_index);
+		Self::change_epoch(Epoch {
+			randomness,
+			authorities,
+			epoch_index,
+			duration: T::babe_epoch_duration(),
+		})
 	}
 
 	fn on_disabled(i: usize) {
-		let log: DigestItem<T::Hash> = DigestItem::Consensus(
-			BABE_ENGINE_ID,
-			ConsensusLog::OnDisabled(i as u64).encode(),
-		);
-		<system::Module<T>>::deposit_log(log.into());
+		Self::deposit_consensus(ConsensusLog::OnDisabled(i as u64))
 	}
 }
 
+fn compute_randomness(
+	last_epoch_randomness: [u8; RANDOMNESS_LENGTH],
+	epoch_index: u64,
+	rho: [u8; VRF_OUTPUT_LENGTH],
+) -> [u8; RANDOMNESS_LENGTH] {
+	let mut s = [0; 40 + VRF_OUTPUT_LENGTH];
+	s[..32].copy_from_slice(&last_epoch_randomness);
+	s[32..40].copy_from_slice(&epoch_index.to_le_bytes());
+	s[40..].copy_from_slice(&rho);
+	runtime_io::blake2_256(&s)
+}
+
 impl<T: Trait> ProvideInherent for Module<T> {
 	type Call = timestamp::Call<T>;
 	type Error = MakeFatalError<RuntimeString>;
@@ -286,7 +307,7 @@ impl<T: Trait> ProvideInherent for Module<T> {
 		if timestamp_based_slot == seal_slot {
 			Ok(())
 		} else {
-			Err(RuntimeString::from("timestamp set in block doesn’t match slot in seal").into())
+			Err(RuntimeString::from("timestamp set in block doesn't match slot in seal").into())
 		}
 	}
 }