BABE Epochs (#3028)

* Add `epoch` field to `SlotInfo`

* Add slot calculations

* More work on epochs in BABE

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Typo: `/` not `%` for division

* Delete useless `LastSlotInEpoch::put(false)`

* Bump `spec_version`

* Make test suite pass again

* Implement BABE epoch randomness signing

* Try to fix compilation

Currently causes a stack overflow in the compiler

* Fix rustc stack overflow

* Add missing `PartialEq` and `Eq` implementations

* Fix compile errors in test suite

* Another silly compile error

* Clone `epoch`

* Fix compile error in benchmarks

* Implement `clone` for `Epoch`

* Merge master

* AUTHORING TEST PASSES!!!

* Fix compilation

* Bump `spec_version`

* Fix compilation

* Fix compilation (again)

* Remove an outdated FIXME

* Fix run.sh and move it to scripts/

* Delete commented-out code

* Fix documentation

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Fix BABE initialization and refactor

* Respond to review

* typo

* Remove useless data in `CheckedHeader::Deferred`

* Remove `slot_number` from Epoch

It is not needed, and only served to waste space and cause confusion.

* Remove epoch from BABE digests

* Move digest.rs to primitives

* Fix incorrect warning names

* Fix compile error

* Consistent field naming for BABE digests

* More compiler error fixex

* Unbound variable

* more compile errors

* another compile error

* Fix compile errors in runtime

* another compile error

* Another compile error

* Fix wasm build

* missing import

* Fix more compile errors

* yet another compile error

* compile fix in test runtime

* Fix and simplify the BABE runtime

The BABE runtime was massively overcomplicated and also wrong.  It
assumed it needed to:

1. delay new authorities taking effect until the next epoch
2. not delay emitting `Consensus` digests to mark epoch changes

However, the first is handled by the `srml_session` crate, and the
second is flat-out incorrect: `Consensus` digests take effect
immediately.  Furthermore, `srml_babe` tried to duplicate the
functionality of `srml_session::PeriodicSession`, but did it both
clumsily and incorrectly.  Fortunately, the new code is simpler and far
more likely to be correct.

* Use `system` to get the test authorities

The genesis block used by tests defines no authorities.  Only the test
suite is affected.

* Fix test runtime impl for BabeApi::epoch() with std

* Fix compilation

* Cached authorities are in the form of an epoch

not a `Vec<AuthorityId>`.

* `slots_per_epoch` is not fixed in general

The BABE code previously assumed `slots_per_epoch` to be a constant,
but that assumption is false in general.  Furthermore, removing this
assumption also allows a lot of code to go away.

* fix compile error

* Implement epoch checker

* Fix runtime compilation

* fork-tree: add method for finding a node in the tree

* babe: register epoch transitions in fork tree and validate them

* fork-tree: add method for arbitrary pruning

* Expose the queued validator set to SRML modules

BABE needs to know not only what the current validator set is, but also
what the next validator set will be.  Expose this to clients of the
session module.

* Bump hex-literal

Hopefully this will fix the panic

* babe: prune epoch change fork tree on finality

* babe: validate epoch index on transition

* babe: persist epoch changes tree

* Fix compile error in tests

* Fix compile error in tests

* Another compile error in tests

* Fix compilation of tests

* core: move grandpa::is_descendent_of to client utils

* babe: use is_descendent_of from client utils

* babe: extract slot_number from pre_digest in import_block

* Move BABE testsuite to its own file

* Initial part of test code

* Missing `WeightMultiplierUpdate` in test-runtime

* bump `spec_version`

* Add a test that a very bogus is rejected

* Run the tests again

* Fix compiler diagnostics

* Bump `spec_version`

* Initial infrastructure for mutation testing

* Mutation testing of block import

* babe: revert epoch changes in case of block import error

* babe: fix logging target

* babe: BabeBlockImport doesn't box inner BlockImport

* babe: fix epoch check in block import

* babe: populate authorities cache on block authorship

* babe: remove unused functions

* babe: use RANDOMNESS_LENGTH const

* babe: remove unneeded config parameters

* core: revert change to hex dependency version

* cleanup gitignore

* babe: add docs to aux_schema

* babe: remove useless drops in tests

* babe: remove annoying macos smart quotes

* fork-tree: docs

* fork-tree: add tests

* babe: style

* babe: rename randomness config variable

* babe: remove randomness helper function

* babe: style fixes

* babe: add docs

* babe: fix tests

* node: bump spec_version

* babe: fix tests

substrate/Cargo.lock

@@ -3672,6 +3672,7 @@ dependencies = [
  "sr-primitives 2.0.0",
  "sr-std 2.0.0",
  "srml-session 2.0.0",
+ "srml-staking 2.0.0",
  "srml-support 2.0.0",
  "srml-system 2.0.0",
  "srml-timestamp 2.0.0",
@@ -4318,6 +4319,7 @@ name = "substrate-consensus-babe"
 version = "2.0.0"
 dependencies = [
  "env_logger 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)",
+ "fork-tree 2.0.0",
  "futures 0.1.28 (registry+https://github.com/rust-lang/crates.io-index)",
  "futures-preview 0.3.0-alpha.17 (registry+https://github.com/rust-lang/crates.io-index)",
  "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -4352,6 +4354,7 @@ name = "substrate-consensus-babe-primitives"
 version = "2.0.0"
 dependencies = [
  "parity-codec 4.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
+ "schnorrkel 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
  "sr-primitives 2.0.0",
  "sr-std 2.0.0",
  "substrate-client 2.0.0",
@@ -4854,8 +4857,11 @@ dependencies = [
  "sr-primitives 2.0.0",
  "sr-std 2.0.0",
  "sr-version 2.0.0",
+ "srml-babe 2.0.0",
  "srml-executive 2.0.0",
  "srml-support 2.0.0",
+ "srml-system 2.0.0",
+ "srml-timestamp 2.0.0",
  "substrate-client 2.0.0",
  "substrate-consensus-aura-primitives 2.0.0",
  "substrate-consensus-babe-primitives 2.0.0",

substrate/Dockerfile

@@ -11,7 +11,7 @@ WORKDIR /substrate
 COPY . /substrate
 
 RUN apt-get update && \
-	apt-get upgrade -y && \
+	apt-get dist-upgrade -y && \
 	apt-get install -y cmake pkg-config libssl-dev git clang
 
 RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && \
@@ -21,7 +21,7 @@ RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && \
 	cargo install --git https://github.com/alexcrichton/wasm-gc && \
 	rustup default nightly && \
 	rustup default stable && \
-	cargo build --$PROFILE
+	cargo build "--$PROFILE"
 
 # ===== SECOND STAGE ======
 

substrate/core/client/src/client.rs

@@ -1799,6 +1799,53 @@ impl<B, E, Block, RA> backend::AuxStore for Client<B, E, Block, RA>
 		crate::backend::AuxStore::get_aux(&*self.backend, key)
 	}
 }
+
+/// Utility methods for the client.
+pub mod utils {
+	use super::*;
+	use crate::{backend::Backend, blockchain, error};
+	use primitives::H256;
+
+	/// Returns a function for checking block ancestry, the returned function will
+	/// return `true` if the given hash (second parameter) is a descendent of the
+	/// base (first parameter). If the `current` parameter is defined, it should
+	/// represent the current block `hash` and its `parent hash`, if given the
+	/// function that's returned will assume that `hash` isn't part of the local DB
+	/// yet, and all searches in the DB will instead reference the parent.
+	pub fn is_descendent_of<'a, B, E, Block: BlockT<Hash=H256>, RA>(
+		client: &'a Client<B, E, Block, RA>,
+		current: Option<(&'a H256, &'a H256)>,
+	) -> impl Fn(&H256, &H256) -> Result<bool, error::Error> + 'a
+		where B: Backend<Block, Blake2Hasher>,
+			  E: CallExecutor<Block, Blake2Hasher> + Send + Sync,
+	{
+		move |base, hash| {
+			if base == hash { return Ok(false); }
+
+			let mut hash = hash;
+			if let Some((current_hash, current_parent_hash)) = current {
+				if base == current_hash { return Ok(false); }
+				if hash == current_hash {
+					if base == current_parent_hash {
+						return Ok(true);
+					} else {
+						hash = current_parent_hash;
+					}
+				}
+			}
+
+			let tree_route = blockchain::tree_route(
+				#[allow(deprecated)]
+				client.backend().blockchain(),
+				BlockId::Hash(*hash),
+				BlockId::Hash(*base),
+			)?;
+
+			Ok(tree_route.common_block().hash == *base)
+		}
+	}
+}
+
 #[cfg(test)]
 pub(crate) mod tests {
 	use std::collections::HashMap;

substrate/core/client/src/lib.rs

@@ -60,6 +60,7 @@ pub use crate::client::{
 	BlockBody, BlockStatus, ImportNotifications, FinalityNotifications, BlockchainEvents,
 	BlockImportNotification, Client, ClientInfo, ExecutionStrategies, FinalityNotification,
 	LongestChain,
+	utils,
 };
 #[cfg(feature = "std")]
 pub use crate::notifications::{StorageEventStream, StorageChangeSet};

substrate/core/consensus/babe/Cargo.toml

@@ -19,7 +19,9 @@ client = { package = "substrate-client", path = "../../client" }
 consensus_common = { package = "substrate-consensus-common", path = "../common" }
 slots = { package = "substrate-consensus-slots", path = "../slots"  }
 runtime_primitives = { package = "sr-primitives", path = "../../sr-primitives" }
+fork-tree = { path = "../../utils/fork-tree" }
 futures = "0.1.26"
+futures03 = { package = "futures-preview", version = "0.3.0-alpha.17", features = ["compat"] }
 tokio-timer = "0.2.11"
 parking_lot = "0.8.0"
 log = "0.4.6"
@@ -28,7 +30,6 @@ rand = "0.6.5"
 merlin = "1.0.3"
 
 [dev-dependencies]
-futures03 = { package = "futures-preview", version = "0.3.0-alpha.17", features = ["compat"] }
 keyring = { package = "substrate-keyring", path = "../../keyring" }
 substrate-executor = { path = "../../executor" }
 network = { package = "substrate-network", path = "../../network", features = ["test-helpers"]}

substrate/core/consensus/babe/primitives/Cargo.toml

@@ -12,6 +12,7 @@ runtime_primitives = { package = "sr-primitives", path = "../../../sr-primitives
 substrate-primitives = { path = "../../../primitives", default-features = false }
 slots = { package = "substrate-consensus-slots", path = "../../slots", optional = true }
 parity-codec = { version = "4.1.1", default-features = false }
+schnorrkel = { version = "0.1.1", optional = true }
 
 [features]
 default = ["std"]
@@ -20,5 +21,6 @@ std = [
 	"runtime_primitives/std",
 	"substrate-client/std",
 	"parity-codec/std",
+	"schnorrkel",
 	"slots",
 ]

substrate/core/consensus/babe/primitives/src/digest.rs

@@ -16,66 +16,85 @@
 
 //! Private implementation details of BABE digests.
 
-use primitives::sr25519::Signature;
-use babe_primitives::{self, BABE_ENGINE_ID, SlotNumber};
+#[cfg(feature = "std")]
+use substrate_primitives::sr25519::Signature;
+#[cfg(feature = "std")]
+use super::{BABE_ENGINE_ID, Epoch};
+#[cfg(not(feature = "std"))]
+use super::{VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH};
+use super::SlotNumber;
+#[cfg(feature = "std")]
 use runtime_primitives::{DigestItem, generic::OpaqueDigestItemId};
+#[cfg(feature = "std")]
 use std::fmt::Debug;
-use parity_codec::{Decode, Encode, Codec, Input};
-use schnorrkel::{vrf::{VRFProof, VRFOutput, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH}};
+use parity_codec::{Decode, Encode};
+#[cfg(feature = "std")]
+use parity_codec::{Codec, Input};
+#[cfg(feature = "std")]
+use schnorrkel::vrf::{VRFProof, VRFOutput, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH};
 
-/// A BABE pre-digest.  It includes:
-///
-/// * The public key of the author.
-/// * The VRF proof.
-/// * The VRF output.
-/// * The slot number.
-#[derive(Clone, Debug, PartialEq, Eq)]
+/// A BABE pre-digest
+#[cfg(feature = "std")]
+#[derive(Clone, Debug)]
 pub struct BabePreDigest {
-	pub(super) vrf_output: VRFOutput,
-	pub(super) proof: VRFProof,
-	pub(super) index: babe_primitives::AuthorityIndex,
-	pub(super) slot_num: SlotNumber,
+	/// VRF output
+	pub vrf_output: VRFOutput,
+	/// VRF proof
+	pub vrf_proof: VRFProof,
+	/// Authority index
+	pub authority_index: super::AuthorityIndex,
+	/// Slot number
+	pub slot_number: SlotNumber,
 }
 
 /// The prefix used by BABE for its VRF keys.
 pub const BABE_VRF_PREFIX: &'static [u8] = b"substrate-babe-vrf";
 
-type RawBabePreDigest = (
-	[u8; VRF_OUTPUT_LENGTH],
-	[u8; VRF_PROOF_LENGTH],
-	u64,
-	u64,
-);
+/// A raw version of `BabePreDigest`, usable on `no_std`.
+#[derive(Copy, Clone, Encode, Decode)]
+pub struct RawBabePreDigest {
+	/// Slot number
+	pub slot_number: SlotNumber,
+	/// Authority index
+	pub authority_index: super::AuthorityIndex,
+	/// VRF output
+	pub vrf_output: [u8; VRF_OUTPUT_LENGTH],
+	/// VRF proof
+	pub vrf_proof: [u8; VRF_PROOF_LENGTH],
+}
 
+#[cfg(feature = "std")]
 impl Encode for BabePreDigest {
 	fn encode(&self) -> Vec<u8> {
-		let tmp: RawBabePreDigest = (
-			*self.vrf_output.as_bytes(),
-			self.proof.to_bytes(),
-			self.index,
-			self.slot_num,
-		);
+		let tmp =  RawBabePreDigest {
+			vrf_output: *self.vrf_output.as_bytes(),
+			vrf_proof: self.vrf_proof.to_bytes(),
+			authority_index: self.authority_index,
+			slot_number: self.slot_number,
+		};
 		parity_codec::Encode::encode(&tmp)
 	}
 }
 
+#[cfg(feature = "std")]
 impl Decode for BabePreDigest {
 	fn decode<R: Input>(i: &mut R) -> Option<Self> {
-		let (output, proof, index, slot_num): RawBabePreDigest = Decode::decode(i)?;
+		let RawBabePreDigest { vrf_output, vrf_proof, authority_index, slot_number } = Decode::decode(i)?;
 
 		// Verify (at compile time) that the sizes in babe_primitives are correct
-		let _: [u8; babe_primitives::VRF_OUTPUT_LENGTH] = output;
-		let _: [u8; babe_primitives::VRF_PROOF_LENGTH] = proof;
+		let _: [u8; super::VRF_OUTPUT_LENGTH] = vrf_output;
+		let _: [u8; super::VRF_PROOF_LENGTH] = vrf_proof;
 		Some(BabePreDigest {
-			proof: VRFProof::from_bytes(&proof).ok()?,
-			vrf_output: VRFOutput::from_bytes(&output).ok()?,
-			index,
-			slot_num,
+			vrf_proof: VRFProof::from_bytes(&vrf_proof).ok()?,
+			vrf_output: VRFOutput::from_bytes(&vrf_output).ok()?,
+			authority_index,
+			slot_number,
 		})
 	}
 }
 
 /// A digest item which is usable with BABE consensus.
+#[cfg(feature = "std")]
 pub trait CompatibleDigestItem: Sized {
 	/// Construct a digest item which contains a BABE pre-digest.
 	fn babe_pre_digest(seal: BabePreDigest) -> Self;
@@ -88,8 +107,12 @@ pub trait CompatibleDigestItem: Sized {
 
 	/// If this item is a BABE signature, return the signature.
 	fn as_babe_seal(&self) -> Option<Signature>;
+
+	/// If this item is a BABE epoch, return it.
+	fn as_babe_epoch(&self) -> Option<Epoch>;
 }
 
+#[cfg(feature = "std")]
 impl<Hash> CompatibleDigestItem for DigestItem<Hash> where
 	Hash: Debug + Send + Sync + Eq + Clone + Codec + 'static
 {
@@ -108,4 +131,8 @@ impl<Hash> CompatibleDigestItem for DigestItem<Hash> where
 	fn as_babe_seal(&self) -> Option<Signature> {
 		self.try_to(OpaqueDigestItemId::Seal(&BABE_ENGINE_ID))
 	}
+
+	fn as_babe_epoch(&self) -> Option<Epoch> {
+		self.try_to(OpaqueDigestItemId::Consensus(&BABE_ENGINE_ID))
+	}
 }

substrate/core/consensus/babe/primitives/src/lib.rs

@@ -15,15 +15,22 @@
 // along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
 
 //! Primitives for BABE.
-#![deny(warnings, unsafe_code, missing_docs)]
+#![deny(warnings)]
+#![forbid(unsafe_code, missing_docs, unused_variables, unused_imports)]
 #![cfg_attr(not(feature = "std"), no_std)]
 
+mod digest;
+
 use parity_codec::{Encode, Decode};
 use rstd::vec::Vec;
 use runtime_primitives::ConsensusEngineId;
 use substrate_primitives::sr25519::Public;
 use substrate_client::decl_runtime_apis;
 
+#[cfg(feature = "std")]
+pub use digest::{BabePreDigest, CompatibleDigestItem};
+pub use digest::{BABE_VRF_PREFIX, RawBabePreDigest};
+
 /// A Babe authority identifier. Necessarily equivalent to the schnorrkel public key used in
 /// the main Babe module. If that ever changes, then this must, too.
 pub type AuthorityId = Public;
@@ -49,15 +56,29 @@ pub type SlotNumber = u64;
 /// The weight of an authority.
 pub type Weight = u64;
 
+/// BABE epoch information
+#[derive(Decode, Encode, Default, PartialEq, Eq, Clone)]
+#[cfg_attr(any(feature = "std", test), derive(Debug))]
+pub struct Epoch {
+	/// The authorities and their weights
+	pub authorities: Vec<(AuthorityId, Weight)>,
+	/// The epoch index
+	pub epoch_index: u64,
+	/// Randomness for this epoch
+	pub randomness: [u8; VRF_OUTPUT_LENGTH],
+	/// The duration of this epoch
+	pub duration: SlotNumber,
+}
+
 /// An consensus log item for BABE.
-#[derive(Decode, Encode)]
+#[derive(Decode, Encode, Clone, PartialEq, Eq)]
 pub enum ConsensusLog {
 	/// The epoch has changed. This provides information about the
 	/// epoch _after_ next: what slot number it will start at, who are the authorities (and their weights)
 	/// and the next epoch randomness. The information for the _next_ epoch should already
 	/// be available.
 	#[codec(index = "1")]
-	NextEpochData(SlotNumber, Vec<(AuthorityId, Weight)>, [u8; VRF_OUTPUT_LENGTH]),
+	NextEpochData(Epoch),
 	/// Disable the authority with given index.
 	#[codec(index = "2")]
 	OnDisabled(AuthorityIndex),
@@ -72,6 +93,12 @@ pub struct BabeConfiguration {
 	/// Dynamic slot duration may be supported in the future.
 	pub slot_duration: u64,
 
+	/// The number of slots per BABE epoch. Currently, only
+	/// the value provided by this type at genesis will be used.
+	///
+	/// Dynamic slot duration may be supported in the future.
+	pub slots_per_epoch: u64,
+
 	/// The expected block time in milliseconds for BABE. Currently,
 	/// only the value provided by this type at genesis will be used.
 	///
@@ -116,7 +143,7 @@ decl_runtime_apis! {
 		/// Dynamic configuration may be supported in the future.
 		fn startup_data() -> BabeConfiguration;
 
-		/// Get the current authorites for Babe.
-		fn authorities() -> Vec<AuthorityId>;
+		/// Get the current epoch data for Babe.
+		fn epoch() -> Epoch;
 	}
 }

substrate/core/consensus/babe/src/aux_schema.rs

@@ -0,0 +1,69 @@
+// Copyright 2019 Parity Technologies (UK) Ltd.
+// This file is part of Substrate.
+
+// Substrate is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Substrate is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Schema for BABE epoch changes in the aux-db.
+
+use log::info;
+use parity_codec::{Decode, Encode};
+
+use client::backend::AuxStore;
+use client::error::{Result as ClientResult, Error as ClientError};
+use runtime_primitives::traits::Block as BlockT;
+
+use super::{EpochChanges, SharedEpochChanges};
+
+const BABE_EPOCH_CHANGES: &[u8] = b"babe_epoch_changes";
+
+fn load_decode<B, T>(backend: &B, key: &[u8]) -> ClientResult<Option<T>>
+	where
+		B: AuxStore,
+		T: Decode,
+{
+	let corrupt = || ClientError::Backend(format!("BABE DB is corrupted.")).into();
+	match backend.get_aux(key)? {
+		None => Ok(None),
+		Some(t) => T::decode(&mut &t[..]).ok_or_else(corrupt).map(Some)
+	}
+}
+
+/// Load or initialize persistent epoch change data from backend.
+pub(crate) fn load_epoch_changes<Block: BlockT, B: AuxStore>(
+	backend: &B,
+) -> ClientResult<SharedEpochChanges<Block>> {
+	let epoch_changes = load_decode::<_, EpochChanges<Block>>(backend, BABE_EPOCH_CHANGES)?
+		.map(Into::into)
+		.unwrap_or_else(|| {
+			info!(target: "babe",
+				"Creating empty BABE epoch changes on what appears to be first startup."
+			);
+			SharedEpochChanges::new()
+		});
+
+	Ok(epoch_changes)
+}
+
+/// Update the epoch changes on disk after a change.
+pub(crate) fn write_epoch_changes<Block: BlockT, F, R>(
+	epoch_changes: &EpochChanges<Block>,
+	write_aux: F,
+) -> R where
+	F: FnOnce(&[(&'static [u8], &[u8])]) -> R,
+{
+	let encoded_epoch_changes = epoch_changes.encode();
+	write_aux(
+		&[(BABE_EPOCH_CHANGES, encoded_epoch_changes.as_slice())],
+	)
+}

substrate/core/consensus/babe/src/tests.rs

@@ -0,0 +1,340 @@
+// Copyright 2019 Parity Technologies (UK) Ltd.
+// This file is part of Substrate.
+
+// Substrate is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Substrate is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
+
+//! BABE testsuite
+
+// FIXME #2532: need to allow deprecated until refactor is done
+// https://github.com/paritytech/substrate/issues/2532
+#![allow(deprecated)]
+use super::*;
+
+use client::{LongestChain, block_builder::BlockBuilder};
+use consensus_common::NoNetwork as DummyOracle;
+use network::test::*;
+use network::test::{Block as TestBlock, PeersClient};
+use runtime_primitives::traits::{Block as BlockT, DigestFor};
+use network::config::ProtocolConfig;
+use tokio::runtime::current_thread;
+use keyring::sr25519::Keyring;
+use super::generic::DigestItem;
+use client::BlockchainEvents;
+use test_client;
+use futures::Async;
+use log::debug;
+use std::{time::Duration, borrow::Borrow, cell::RefCell};
+type Item = generic::DigestItem<Hash>;
+
+type Error = client::error::Error;
+
+type TestClient = client::Client<
+	test_client::Backend,
+	test_client::Executor,
+	TestBlock,
+	test_client::runtime::RuntimeApi,
+>;
+
+struct DummyFactory(Arc<TestClient>);
+struct DummyProposer(u64, Arc<TestClient>);
+
+impl Environment<TestBlock> for DummyFactory {
+	type Proposer = DummyProposer;
+	type Error = Error;
+
+	fn init(&self, parent_header: &<TestBlock as BlockT>::Header)
+		-> Result<DummyProposer, Error>
+	{
+		Ok(DummyProposer(parent_header.number + 1, self.0.clone()))
+	}
+}
+
+impl Proposer<TestBlock> for DummyProposer {
+	type Error = Error;
+	type Create = Result<TestBlock, Error>;
+
+	fn propose(&self, _: InherentData, digests: DigestFor<TestBlock>, _: Duration) -> Result<TestBlock, Error> {
+		self.1.new_block(digests).unwrap().bake().map_err(|e| e.into())
+	}
+}
+
+type Mutator = Arc<dyn for<'r> Fn(&'r mut TestHeader) + Send + Sync>;
+
+thread_local! {
+	static MUTATOR: RefCell<Mutator> = RefCell::new(Arc::new(|_|()));
+}
+
+pub struct BabeTestNet {
+	peers: Vec<Peer<(), DummySpecialization>>,
+}
+
+type TestHeader = <TestBlock as BlockT>::Header;
+type TestExtrinsic = <TestBlock as BlockT>::Extrinsic;
+
+pub struct TestVerifier {
+	inner: BabeVerifier<PeersFullClient>,
+	mutator: Mutator,
+}
+
+impl Verifier<TestBlock> for TestVerifier {
+	/// Verify the given data and return the BlockImportParams and an optional
+	/// new set of validators to import. If not, err with an Error-Message
+	/// presented to the User in the logs.
+	fn verify(
+		&self,
+		origin: BlockOrigin,
+		mut header: TestHeader,
+		justification: Option<Justification>,
+		body: Option<Vec<TestExtrinsic>>,
+	) -> Result<(BlockImportParams<TestBlock>, Option<Vec<(CacheKeyId, Vec<u8>)>>), String> {
+		let cb: &(dyn Fn(&mut TestHeader) + Send + Sync) = self.mutator.borrow();
+		cb(&mut header);
+		Ok(self.inner.verify(origin, header, justification, body).expect("verification failed!"))
+	}
+}
+
+impl TestNetFactory for BabeTestNet {
+	type Specialization = DummySpecialization;
+	type Verifier = TestVerifier;
+	type PeerData = ();
+
+	/// Create new test network with peers and given config.
+	fn from_config(_config: &ProtocolConfig) -> Self {
+		debug!(target: "babe", "Creating test network from config");
+		BabeTestNet {
+			peers: Vec::new(),
+		}
+	}
+
+	/// KLUDGE: this function gets the mutator from thread-local storage.
+	fn make_verifier(&self, client: PeersClient, _cfg: &ProtocolConfig)
+		-> Arc<Self::Verifier>
+	{
+		let api = client.as_full().expect("only full clients are used in test");
+		trace!(target: "babe", "Creating a verifier");
+		let config = Config::get_or_compute(&*api)
+			.expect("slot duration available");
+		let inherent_data_providers = InherentDataProviders::new();
+		register_babe_inherent_data_provider(
+			&inherent_data_providers,
+			config.get()
+		).expect("Registers babe inherent data provider");
+		trace!(target: "babe", "Provider registered");
+
+		Arc::new(TestVerifier {
+			inner: BabeVerifier {
+				api,
+				inherent_data_providers,
+				config,
+				time_source: Default::default(),
+			},
+			mutator: MUTATOR.with(|s| s.borrow().clone()),
+		})
+	}
+
+	fn peer(&mut self, i: usize) -> &mut Peer<Self::PeerData, DummySpecialization> {
+		trace!(target: "babe", "Retreiving a peer");
+		&mut self.peers[i]
+	}
+
+	fn peers(&self) -> &Vec<Peer<Self::PeerData, DummySpecialization>> {
+		trace!(target: "babe", "Retreiving peers");
+		&self.peers
+	}
+
+	fn mut_peers<F: FnOnce(&mut Vec<Peer<Self::PeerData, DummySpecialization>>)>(
+		&mut self,
+		closure: F,
+	) {
+		closure(&mut self.peers);
+	}
+}
+
+#[test]
+fn can_serialize_block() {
+	let _ = env_logger::try_init();
+	assert!(BabePreDigest::decode(&mut &b""[..]).is_none());
+}
+
+#[test]
+#[should_panic]
+fn rejects_empty_block() {
+	env_logger::try_init().unwrap();
+	let mut net = BabeTestNet::new(3);
+	let block_builder = |builder: BlockBuilder<_, _>| {
+		builder.bake().unwrap()
+	};
+	net.mut_peers(|peer| {
+		peer[0].generate_blocks(1, BlockOrigin::NetworkInitialSync, block_builder);
+	})
+}
+
+fn run_one_test() {
+	let _ = env_logger::try_init();
+	let net = BabeTestNet::new(3);
+
+	let peers = &[
+		(0, Keyring::Alice),
+		(1, Keyring::Bob),
+		(2, Keyring::Charlie),
+	];
+
+	let net = Arc::new(Mutex::new(net));
+	let mut import_notifications = Vec::new();
+	let mut runtime = current_thread::Runtime::new().unwrap();
+	for (peer_id, key) in peers {
+		let client = net.lock().peer(*peer_id).client().as_full().unwrap();
+		let environ = Arc::new(DummyFactory(client.clone()));
+		import_notifications.push(
+			client.import_notification_stream()
+				.map(|v| Ok::<_, ()>(v)).compat()
+				.take_while(|n| Ok(n.header.number() < &5))
+				.for_each(move |_| Ok(()))
+		);
+
+		let config = Config::get_or_compute(&*client)
+			.expect("slot duration available");
+
+		let inherent_data_providers = InherentDataProviders::new();
+		register_babe_inherent_data_provider(
+			&inherent_data_providers, config.get()
+		).expect("Registers babe inherent data provider");
+
+
+		#[allow(deprecated)]
+		let select_chain = LongestChain::new(client.backend().clone());
+
+		runtime.spawn(start_babe(BabeParams {
+			config,
+			local_key: Arc::new(key.clone().into()),
+			block_import: client.clone(),
+			select_chain,
+			client,
+			env: environ.clone(),
+			sync_oracle: DummyOracle,
+			inherent_data_providers,
+			force_authoring: false,
+			time_source: Default::default(),
+		}).expect("Starts babe"));
+	}
+
+	// wait for all finalized on each.
+	let wait_for = futures::future::join_all(import_notifications);
+
+	let drive_to_completion = futures::future::poll_fn(|| { net.lock().poll(); Ok(Async::NotReady) });
+	runtime.block_on(wait_for.select(drive_to_completion).map_err(|_| ())).unwrap();
+}
+
+#[test]
+fn authoring_blocks() { run_one_test() }
+
+#[test]
+#[should_panic]
+fn rejects_missing_inherent_digest() {
+	MUTATOR.with(|s| *s.borrow_mut() = Arc::new(move |header: &mut TestHeader| {
+		let v = std::mem::replace(&mut header.digest_mut().logs, vec![]);
+		header.digest_mut().logs = v.into_iter()
+			.filter(|v| v.as_babe_pre_digest().is_none())
+			.collect()
+	}));
+	run_one_test()
+}
+
+#[test]
+#[should_panic]
+fn rejects_missing_seals() {
+	MUTATOR.with(|s| *s.borrow_mut() = Arc::new(move |header: &mut TestHeader| {
+		let v = std::mem::replace(&mut header.digest_mut().logs, vec![]);
+		header.digest_mut().logs = v.into_iter()
+			.filter(|v| v.as_babe_seal().is_none())
+			.collect()
+	}));
+	run_one_test()
+}
+
+// TODO: this test assumes that the test runtime will trigger epoch changes
+// which isn't the case since it doesn't include the session module.
+#[test]
+#[should_panic]
+#[ignore]
+fn rejects_missing_consensus_digests() {
+	MUTATOR.with(|s| *s.borrow_mut() = Arc::new(move |header: &mut TestHeader| {
+		let v = std::mem::replace(&mut header.digest_mut().logs, vec![]);
+		header.digest_mut().logs = v.into_iter()
+			.filter(|v| v.as_babe_epoch().is_none())
+			.collect()
+	}));
+	run_one_test()
+}
+
+#[test]
+fn wrong_consensus_engine_id_rejected() {
+	let _ = env_logger::try_init();
+	let sig = sr25519::Pair::generate().0.sign(b"");
+	let bad_seal: Item = DigestItem::Seal([0; 4], sig.0.to_vec());
+	assert!(bad_seal.as_babe_pre_digest().is_none());
+	assert!(bad_seal.as_babe_seal().is_none())
+}
+
+#[test]
+fn malformed_pre_digest_rejected() {
+	let _ = env_logger::try_init();
+	let bad_seal: Item = DigestItem::Seal(BABE_ENGINE_ID, [0; 64].to_vec());
+	assert!(bad_seal.as_babe_pre_digest().is_none());
+}
+
+#[test]
+fn sig_is_not_pre_digest() {
+	let _ = env_logger::try_init();
+	let sig = sr25519::Pair::generate().0.sign(b"");
+	let bad_seal: Item = DigestItem::Seal(BABE_ENGINE_ID, sig.0.to_vec());
+	assert!(bad_seal.as_babe_pre_digest().is_none());
+	assert!(bad_seal.as_babe_seal().is_some())
+}
+
+#[test]
+fn can_author_block() {
+	let _ = env_logger::try_init();
+	let randomness = &[];
+	let (pair, _) = sr25519::Pair::generate();
+	let mut i = 0;
+	let epoch = Epoch {
+		authorities: vec![(pair.public(), 0)],
+		randomness: [0; 32],
+		epoch_index: 1,
+		duration: 100,
+	};
+	loop {
+		match claim_slot(randomness, i, 0, epoch.clone(), &pair, u64::MAX / 10) {
+			None => i += 1,
+			Some(s) => {
+				debug!(target: "babe", "Authored block {:?}", s);
+				break
+			}
+		}
+	}
+}
+
+#[test]
+fn authorities_call_works() {
+	let _ = env_logger::try_init();
+	let client = test_client::new();
+
+	assert_eq!(client.info().chain.best_number, 0);
+	assert_eq!(epoch(&client, &BlockId::Number(0)).unwrap().authorities, vec![
+		(Keyring::Alice.into(), 1),
+		(Keyring::Bob.into(), 1),
+		(Keyring::Charlie.into(), 1),
+	]);
+}

substrate/core/consensus/slots/src/lib.rs

@@ -20,12 +20,14 @@
 //! time during which certain events can and/or must occur.  This crate
 //! provides generic functionality for slots.
 
-#![forbid(warnings, unsafe_code, missing_docs)]
+#![deny(warnings)]
+#![forbid(unsafe_code, missing_docs)]
 
 mod slots;
 mod aux_schema;
 
-pub use slots::{SignedDuration, SlotInfo, Slots};
+pub use slots::{SignedDuration, SlotInfo};
+use slots::Slots;
 pub use aux_schema::{check_equivocation, MAX_SLOT_CAPACITY, PRUNING_BOUND};
 
 use codec::{Decode, Encode};

substrate/core/consensus/slots/src/slots.rs

@@ -96,7 +96,7 @@ impl SlotInfo {
 }
 
 /// A stream that returns every time there is a new slot.
-pub struct Slots<SC> {
+pub(crate) struct Slots<SC> {
 	last_slot: u64,
 	slot_duration: u64,
 	inner_delay: Option<Delay>,

substrate/core/finality-grandpa/src/environment.rs

@@ -26,7 +26,8 @@ use tokio_timer::Delay;
 use parking_lot::RwLock;
 
 use client::{
-	backend::Backend, BlockchainEvents, CallExecutor, Client, error::Error as ClientError
+	backend::Backend, BlockchainEvents, CallExecutor, Client, error::Error as ClientError,
+	utils::is_descendent_of,
 };
 use grandpa::{
 	BlockNumberOps, Equivocation, Error as GrandpaError, round::State as RoundState,
@@ -990,42 +991,3 @@ pub(crate) fn canonical_at_height<B, E, Block: BlockT<Hash=H256>, RA>(
 
 	Ok(Some(current.hash()))
 }
-
-/// Returns a function for checking block ancestry, the returned function will
-/// return `true` if the given hash (second parameter) is a descendent of the
-/// base (first parameter). If the `current` parameter is defined, it should
-/// represent the current block `hash` and its `parent hash`, if given the
-/// function that's returned will assume that `hash` isn't part of the local DB
-/// yet, and all searches in the DB will instead reference the parent.
-pub fn is_descendent_of<'a, B, E, Block: BlockT<Hash=H256>, RA>(
-	client: &'a Client<B, E, Block, RA>,
-	current: Option<(&'a H256, &'a H256)>,
-) -> impl Fn(&H256, &H256) -> Result<bool, client::error::Error> + 'a
-where B: Backend<Block, Blake2Hasher>,
-	  E: CallExecutor<Block, Blake2Hasher> + Send + Sync,
-{
-	move |base, hash| {
-		if base == hash { return Ok(false); }
-
-		let mut hash = hash;
-		if let Some((current_hash, current_parent_hash)) = current {
-			if base == current_hash { return Ok(false); }
-			if hash == current_hash {
-				if base == current_parent_hash {
-					return Ok(true);
-				} else {
-					hash = current_parent_hash;
-				}
-			}
-		}
-
-		let tree_route = client::blockchain::tree_route(
-			#[allow(deprecated)]
-			client.backend().blockchain(),
-			BlockId::Hash(*hash),
-			BlockId::Hash(*base),
-		)?;
-
-		Ok(tree_route.common_block().hash == *base)
-	}
-}

substrate/core/finality-grandpa/src/import.rs

@@ -25,6 +25,7 @@ use client::{blockchain, CallExecutor, Client};
 use client::blockchain::HeaderBackend;
 use client::backend::Backend;
 use client::runtime_api::ApiExt;
+use client::utils::is_descendent_of;
 use consensus_common::{
 	BlockImport, Error as ConsensusError,
 	BlockImportParams, ImportResult, JustificationImport, well_known_cache_keys,
@@ -42,7 +43,7 @@ use substrate_primitives::{H256, Blake2Hasher};
 use crate::{Error, CommandOrError, NewAuthoritySet, VoterCommand};
 use crate::authorities::{AuthoritySet, SharedAuthoritySet, DelayKind, PendingChange};
 use crate::consensus_changes::SharedConsensusChanges;
-use crate::environment::{finalize_block, is_descendent_of};
+use crate::environment::finalize_block;
 use crate::justification::GrandpaJustification;
 
 /// A block-import handler for GRANDPA.

substrate/core/network/src/protocol/sync.rs

@@ -1189,4 +1189,3 @@ fn peer_block_request<B: BlockT>(
 		None
 	}
 }
-

substrate/core/sr-api-macros/tests/runtime_calls.rs

@@ -104,7 +104,7 @@ fn calling_with_both_strategy_and_fail_on_native_should_work() {
 
 
 #[test]
-fn calling_with_native_else_wasm_and_faild_on_wasm_should_work() {
+fn calling_with_native_else_wasm_and_fail_on_wasm_should_work() {
 	let client = TestClientBuilder::new().set_execution_strategy(ExecutionStrategy::NativeElseWasm).build();
 	let runtime_api = client.runtime_api();
 	let block_id = BlockId::Number(client.info().chain.best_number);

substrate/core/test-runtime/Cargo.toml

@@ -25,6 +25,9 @@ trie-db = { version = "0.14.0", default-features = false }
 offchain-primitives = { package = "substrate-offchain-primitives", path = "../offchain/primitives", default-features = false}
 executive = { package = "srml-executive", path = "../../srml/executive", default-features = false }
 cfg-if = "0.1.6"
+srml-babe = { path = "../../srml/babe", default-features = false }
+srml-timestamp = { path = "../../srml/timestamp", default-features = false }
+srml-system = { path = "../../srml/system", default-features = false }
 
 [dev-dependencies]
 substrate-executor = { path = "../executor" }
@@ -58,4 +61,7 @@ std = [
 	"trie-db/std",
 	"offchain-primitives/std",
 	"executive/std",
+	"srml-babe/std",
+	"srml-timestamp/std",
+	"srml-system/std",
 ]

substrate/core/test-runtime/src/lib.rs

@@ -39,7 +39,7 @@ use runtime_primitives::{
 	transaction_validity::{TransactionValidity, ValidTransaction},
 	traits::{
 		BlindCheckable, BlakeTwo256, Block as BlockT, Extrinsic as ExtrinsicT,
-		GetNodeBlockType, GetRuntimeBlockType, Verify
+		GetNodeBlockType, GetRuntimeBlockType, Verify, IdentityLookup
 	},
 };
 use runtime_version::RuntimeVersion;
@@ -47,10 +47,10 @@ pub use primitives::hash::H256;
 use primitives::{sr25519, OpaqueMetadata};
 #[cfg(any(feature = "std", test))]
 use runtime_version::NativeVersion;
+use runtime_support::{impl_outer_origin, parameter_types};
 use inherents::{CheckInherentsResult, InherentData};
 use cfg_if::cfg_if;
 pub use consensus_babe::AuthorityId;
-
 // Ensure Babe and Aura use the same crypto to simplify things a bit.
 pub type AuraId = AuthorityId;
 // Ensure Babe and Aura use the same crypto to simplify things a bit.
@@ -301,6 +301,7 @@ cfg_if! {
 	}
 }
 
+#[derive(Clone, Eq, PartialEq)]
 pub struct Runtime;
 
 impl GetNodeBlockType for Runtime {
@@ -311,6 +312,46 @@ impl GetRuntimeBlockType for Runtime {
 	type RuntimeBlock = Block;
 }
 
+impl_outer_origin!{
+	pub enum Origin for Runtime where system = srml_system {}
+}
+
+#[derive(Clone, Encode, Decode, Eq, PartialEq)]
+#[cfg_attr(feature = "std", derive(Debug))]
+pub struct Event;
+
+impl From<srml_system::Event> for Event {
+	fn from(_evt: srml_system::Event) -> Self {
+		unimplemented!("Not required in tests!")
+	}
+}
+
+parameter_types! {
+	pub const BlockHashCount: BlockNumber = 250;
+	pub const MinimumPeriod: u64 = 5;
+}
+
+impl srml_system::Trait for Runtime {
+	type Origin = Origin;
+	type Index = u64;
+	type BlockNumber = u64;
+	type Hash = H256;
+	type Hashing = BlakeTwo256;
+	type AccountId = u64;
+	type Lookup = IdentityLookup<Self::AccountId>;
+	type Header = Header;
+	type Event = Event;
+	type WeightMultiplierUpdate = ();
+	type BlockHashCount = BlockHashCount;
+}
+
+impl srml_timestamp::Trait for Runtime {
+	/// A timestamp: seconds since the unix epoch.
+	type Moment = u64;
+	type OnTimestampSet = ();
+	type MinimumPeriod = MinimumPeriod;
+}
+
 /// Adds one to the given input and returns the final result.
 #[inline(never)]
 fn benchmark_add_one(i: u64) -> u64 {
@@ -477,13 +518,23 @@ cfg_if! {
 			impl consensus_babe::BabeApi<Block> for Runtime {
 				fn startup_data() -> consensus_babe::BabeConfiguration {
 					consensus_babe::BabeConfiguration {
-						slot_duration: 1,
+						median_required_blocks: 0,
+						slot_duration: 3,
 						expected_block_time: 1,
-						threshold: std::u64::MAX,
-						median_required_blocks: 100,
+						threshold: core::u64::MAX,
+						slots_per_epoch: 6,
+					}
+				}
+				fn epoch() -> consensus_babe::Epoch {
+					let authorities = system::authorities();
+					let authorities: Vec<_> = authorities.into_iter().map(|x|(x, 1)).collect();
+					consensus_babe::Epoch {
+						authorities,
+						randomness: <srml_babe::Module<Runtime>>::randomness(),
+						epoch_index: 1,
+						duration: 6,
 					}
 				}
-				fn authorities() -> Vec<BabeId> { system::authorities() }
 			}
 
 			impl offchain_primitives::OffchainWorkerApi<Block> for Runtime {
@@ -626,9 +677,20 @@ cfg_if! {
 						slot_duration: 1,
 						expected_block_time: 1,
 						threshold: core::u64::MAX,
+						slots_per_epoch: 6,
+					}
+				}
+
+				fn epoch() -> consensus_babe::Epoch {
+					let authorities = system::authorities();
+					let authorities: Vec<_> = authorities.into_iter().map(|x|(x, 1)).collect();
+					consensus_babe::Epoch {
+						authorities,
+						randomness: <srml_babe::Module<Runtime>>::randomness(),
+						epoch_index: 1,
+						duration: 6,
 					}
 				}
-				fn authorities() -> Vec<BabeId> { system::authorities() }
 			}
 
 			impl offchain_primitives::OffchainWorkerApi<Block> for Runtime {

substrate/core/utils/fork-tree/src/lib.rs

@@ -81,6 +81,62 @@ pub struct ForkTree<H, N, V> {
 	best_finalized_number: Option<N>,
 }
 
+impl<H, N, V> ForkTree<H, N, V> where
+	H: PartialEq + Clone,
+	N: Ord + Clone,
+	V: Clone,
+{
+	/// Prune all nodes that are not descendents of `hash` according to
+	/// `is_descendent_of`. The given function `is_descendent_of` should return
+	/// `true` if the second hash (target) is a descendent of the first hash
+	/// (base). After pruning the tree it should have one or zero roots. The
+	/// number and order of calls to `is_descendent_of` is unspecified and
+	/// subject to change.
+	pub fn prune<F, E>(
+		&mut self,
+		hash: &H,
+		number: N,
+		is_descendent_of: &F
+	) -> Result<(), Error<E>>
+		where E: std::error::Error,
+			  F: Fn(&H, &H) -> Result<bool, E>
+	{
+		let mut new_root = None;
+		for node in self.node_iter() {
+			// if the node has a lower number than the one being finalized then
+			// we only keep if it has no children and the finalized block is a
+			// descendent of this node
+			if node.number < number  {
+				if !node.children.is_empty() || !is_descendent_of(&node.hash, hash)? {
+					continue;
+				}
+			}
+
+			// if the node has the same number as the finalized block then it
+			// must have the same hash
+			if node.number == number && node.hash != *hash {
+				continue;
+			}
+
+			// if the node has a higher number then we keep it if it is a
+			// descendent of the finalized block
+			if node.number > number && !is_descendent_of(hash, &node.hash)? {
+				continue;
+			}
+
+			new_root = Some(node);
+			break;
+		}
+
+		if let Some(root) = new_root {
+			self.roots = vec![root.clone()];
+		}
+
+		Ok(())
+	}
+
+}
+
 impl<H, N, V> ForkTree<H, N, V> where
 	H: PartialEq,
 	N: Ord,
@@ -152,6 +208,34 @@ impl<H, N, V> ForkTree<H, N, V> where
 		self.node_iter().map(|node| (&node.hash, &node.number, &node.data))
 	}
 
+	/// Find a node in the tree that is the lowest ancestor of the given
+	/// block hash and which passes the given predicate. The given function
+	/// `is_descendent_of` should return `true` if the second hash (target)
+	/// is a descendent of the first hash (base).
+	pub fn find_node_where<F, E, P>(
+		&self,
+		hash: &H,
+		number: &N,
+		is_descendent_of: &F,
+		predicate: &P,
+	) -> Result<Option<&Node<H, N, V>>, Error<E>>
+		where E: std::error::Error,
+			  F: Fn(&H, &H) -> Result<bool, E>,
+			  P: Fn(&V) -> bool,
+	{
+		// search for node starting from all roots
+		for root in self.roots.iter() {
+			let node = root.find_node_where(hash, number, is_descendent_of, predicate)?;
+
+			// found the node, early exit
+			if node.is_some() {
+				return Ok(node);
+			}
+		}
+
+		Ok(None)
+	}
+
 	/// Finalize a root in the tree and return it, return `None` in case no root
 	/// with the given hash exists. All other roots are pruned, and the children
 	/// of the finalized node become the new roots.
@@ -167,7 +251,7 @@ impl<H, N, V> ForkTree<H, N, V> where
 	}
 
 	/// Finalize a node in the tree. This method will make sure that the node
-	/// being finalized is either an existing root (an return its data), or a
+	/// being finalized is either an existing root (and return its data), or a
 	/// node from a competing branch (not in the tree), tree pruning is done
 	/// accordingly. The given function `is_descendent_of` should return `true`
 	/// if the second hash (target) is a descendent of the first hash (base).
@@ -400,6 +484,49 @@ mod node_implementation {
 				Ok(Some((hash, number, data)))
 			}
 		}
+
+		/// Find a node in the tree that is the lowest ancestor of the given
+		/// block hash and which passes the given predicate. The given function
+		/// `is_descendent_of` should return `true` if the second hash (target)
+		/// is a descendent of the first hash (base).
+		// FIXME: it would be useful if this returned a mutable reference but
+		// rustc can't deal with lifetimes properly. an option would be to try
+		// an iterative definition instead.
+		pub fn find_node_where<F, P, E>(
+			&self,
+			hash: &H,
+			number: &N,
+			is_descendent_of: &F,
+			predicate: &P,
+		) -> Result<Option<&Node<H, N, V>>, Error<E>>
+			where E: std::error::Error,
+				  F: Fn(&H, &H) -> Result<bool, E>,
+				  P: Fn(&V) -> bool,
+		{
+			// stop searching this branch
+			if *number < self.number {
+				return Ok(None);
+			}
+
+			// continue depth-first search through all children
+			for node in self.children.iter() {
+				let node = node.find_node_where(hash, number, is_descendent_of, predicate)?;
+
+				// found node, early exit
+				if node.is_some() {
+					return Ok(node);
+				}
+			}
+
+			// node not found in any of the descendents, if the node we're
+			// searching for is a descendent of this node and it passes the
+			// predicate, then it is this one.
+			if predicate(&self.data) && is_descendent_of(&self.hash, hash)? {
+				Ok(Some(self))
+			} else {
+				Ok(None)
+			}
+		}
 	}
 }
 
@@ -870,4 +997,40 @@ mod test {
 			);
 		}
 	}
+
+	#[test]
+	fn find_node_works() {
+		let (tree, is_descendent_of) = test_fork_tree();
+
+		let node = tree.find_node_where(
+			&"D",
+			&4,
+			&is_descendent_of,
+			&|_| true,
+		).unwrap().unwrap();
+
+		assert_eq!(node.hash, "C");
+		assert_eq!(node.number, 3);
+	}
+
+	#[test]
+	fn prune_works() {
+		let (mut tree, is_descendent_of) = test_fork_tree();
+
+		tree.prune(
+			&"C",
+			3,
+			&is_descendent_of,
+		).unwrap();
+
+		assert_eq!(
+			tree.roots.iter().map(|node| node.hash).collect::<Vec<_>>(),
+			vec!["C"],
+		);
+
+		assert_eq!(
+			tree.iter().map(|(hash, _, _)| *hash).collect::<Vec<_>>(),
+			vec!["C", "D", "E"],
+		);
+	}
 }

substrate/srml/aura/src/lib.rs

@@ -188,7 +188,7 @@ impl<T: Trait> Module<T> {
 impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
 	type Key = T::AuthorityId;
 
-	fn on_new_session<'a, I: 'a>(changed: bool, validators: I)
+	fn on_new_session<'a, I: 'a>(changed: bool, validators: I, _queued_validators: I)
 		where I: Iterator<Item=(&'a T::AccountId, T::AuthorityId)>
 	{
 		// instant changes

substrate/srml/babe/Cargo.toml

@@ -11,6 +11,7 @@ serde = { version = "1.0.93", optional = true }
 inherents = { package = "substrate-inherents", path = "../../core/inherents", default-features = false }
 rstd = { package = "sr-std", path = "../../core/sr-std", default-features = false }
 primitives = { package = "sr-primitives", path = "../../core/sr-primitives", default-features = false }
+staking = { package = "srml-staking", path = "../staking", default-features = false }
 srml-support = { path = "../support", default-features = false }
 system = { package = "srml-system", path = "../system", default-features = false }
 timestamp = { package = "srml-timestamp", path = "../timestamp", default-features = false }
@@ -37,4 +38,5 @@ std = [
 	"babe-primitives/std",
 	"session/std",
 	"runtime_io/std",
+	"staking/std",
 ]

substrate/srml/babe/src/lib.rs

@@ -23,26 +23,22 @@ pub use timestamp;
 use rstd::{result, prelude::*};
 use srml_support::{decl_storage, decl_module, StorageValue, traits::FindAuthor, traits::Get};
 use timestamp::{OnTimestampSet, Trait};
-use primitives::{
-	generic::DigestItem,
-	traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon}
-};
-use primitives::ConsensusEngineId;
+use primitives::{generic::DigestItem, ConsensusEngineId};
+use primitives::traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon, Convert};
 #[cfg(feature = "std")]
 use timestamp::TimestampInherentData;
 use parity_codec::{Encode, Decode};
 use inherents::{RuntimeString, InherentIdentifier, InherentData, ProvideInherent, MakeFatalError};
 #[cfg(feature = "std")]
 use inherents::{InherentDataProviders, ProvideInherentData};
-use babe_primitives::{BABE_ENGINE_ID, ConsensusLog};
-pub use babe_primitives::{AuthorityId, VRF_OUTPUT_LENGTH, VRF_PROOF_LENGTH, PUBLIC_KEY_LENGTH};
+use babe_primitives::{BABE_ENGINE_ID, ConsensusLog, Weight, Epoch, RawBabePreDigest};
+pub use babe_primitives::{AuthorityId, VRF_OUTPUT_LENGTH, PUBLIC_KEY_LENGTH};
 
 /// The BABE inherent identifier.
 pub const INHERENT_IDENTIFIER: InherentIdentifier = *b"babeslot";
 
 /// The type of the BABE inherent.
 pub type InherentType = u64;
-
 /// Auxiliary trait to extract BABE inherent data.
 pub trait BabeInherentData {
 	/// Get BABE inherent data.
@@ -101,8 +97,8 @@ impl ProvideInherentData for InherentDataProvider {
 		inherent_data: &mut InherentData,
 	) -> result::Result<(), RuntimeString> {
 		let timestamp = inherent_data.timestamp_inherent_data()?;
-		let slot_num = timestamp / self.slot_duration;
-		inherent_data.put_data(INHERENT_IDENTIFIER, &slot_num)
+		let slot_number = timestamp / self.slot_duration;
+		inherent_data.put_data(INHERENT_IDENTIFIER, &slot_number)
 	}
 
 	fn error_to_string(&self, error: &[u8]) -> Option<String> {
@@ -115,13 +111,15 @@ pub const RANDOMNESS_LENGTH: usize = 32;
 
 decl_storage! {
 	trait Store for Module<T: Trait> as Babe {
-		/// The last timestamp.
-		LastTimestamp get(last): T::Moment;
+		NextRandomness: [u8; RANDOMNESS_LENGTH];
 
-		/// The current authorities set.
-		Authorities get(authorities): Vec<AuthorityId>;
+		/// Randomness under construction
+		UnderConstruction: [u8; VRF_OUTPUT_LENGTH];
 
-		/// The epoch randomness.
+		/// Current epoch
+		pub Authorities get(authorities) config(): Vec<(AuthorityId, Weight)>;
+
+		/// The epoch randomness for the *current* epoch.
 		///
 		/// # Security
 		///
@@ -131,16 +129,10 @@ decl_storage! {
 		/// (like everything else on-chain) it is public. For example, it can be
 		/// used where a number is needed that cannot have been chosen by an
 		/// adversary, for purposes such as public-coin zero-knowledge proofs.
-		EpochRandomness get(epoch_randomness): [u8; VRF_OUTPUT_LENGTH];
+		pub Randomness get(randomness): [u8; RANDOMNESS_LENGTH];
 
-		/// The randomness under construction
-		UnderConstruction: [u8; VRF_OUTPUT_LENGTH];
-
-		/// The randomness for the next epoch
-		NextEpochRandomness: [u8; VRF_OUTPUT_LENGTH];
-
-		/// The current epoch
-		EpochIndex get(epoch_index): u64;
+		/// Current epoch index
+		EpochIndex: u64;
 	}
 }
 
@@ -154,11 +146,13 @@ decl_module! {
 				.iter()
 				.filter_map(|s| s.as_pre_runtime())
 				.filter_map(|(id, mut data)| if id == BABE_ENGINE_ID {
-					<[u8; VRF_OUTPUT_LENGTH]>::decode(&mut data)
+					RawBabePreDigest::decode(&mut data)
 				} else {
 					None
 				}) {
-				Self::deposit_vrf_output(&i);
+
+				Self::deposit_vrf_output(&i.vrf_output);
+				return;
 			}
 		}
 	}
@@ -166,7 +160,7 @@ decl_module! {
 
 impl<T: Trait> RandomnessBeacon for Module<T> {
 	fn random() -> [u8; VRF_OUTPUT_LENGTH] {
-		Self::epoch_randomness()
+		Self::randomness()
 	}
 }
 
@@ -179,15 +173,10 @@ impl<T: Trait> FindAuthor<u64> for Module<T> {
 	{
 		for (id, mut data) in digests.into_iter() {
 			if id == BABE_ENGINE_ID {
-				let (_, _, i): (
-					[u8; VRF_OUTPUT_LENGTH],
-					[u8; VRF_PROOF_LENGTH],
-					u64,
-				) = Decode::decode(&mut data)?;
-				return Some(i)
+				return Some(RawBabePreDigest::decode(&mut data)?.authority_index);
 			}
 		}
-		return None
+		return None;
 	}
 }
 
@@ -195,7 +184,7 @@ impl<T: timestamp::Trait> IsMember<AuthorityId> for Module<T> {
 	fn is_member(authority_id: &AuthorityId) -> bool {
 		<Module<T>>::authorities()
 			.iter()
-			.any(|id| id == authority_id)
+			.any(|id| &id.0 == authority_id)
 	}
 }
 
@@ -207,65 +196,97 @@ impl<T: Trait> Module<T> {
 		<T as timestamp::Trait>::MinimumPeriod::get().saturating_mul(2.into())
 	}
 
-	fn change_authorities(new: Vec<AuthorityId>) {
-		Authorities::put(&new);
-
+	fn deposit_consensus<U: Encode>(new: U) {
 		let log: DigestItem<T::Hash> = DigestItem::Consensus(BABE_ENGINE_ID, new.encode());
-		<system::Module<T>>::deposit_log(log.into());
+		<system::Module<T>>::deposit_log(log.into())
+	}
+
+	fn get_inherent_digests() -> system::DigestOf<T> {
+		<system::Module<T>>::digest()
+	}
+
+	fn change_epoch(new: Epoch) {
+		Authorities::put(&new.authorities);
+		Randomness::put(&new.randomness);
+		Self::deposit_consensus(ConsensusLog::NextEpochData(new))
 	}
 
 	fn deposit_vrf_output(vrf_output: &[u8; VRF_OUTPUT_LENGTH]) {
 		UnderConstruction::mutate(|z| z.iter_mut().zip(vrf_output).for_each(|(x, y)| *x^=y))
 	}
 
-	fn get_inherent_digests() -> system::DigestOf<T> {
-		<system::Module<T>>::digest()
+	/// Call this function exactly once when an epoch changes, to update the
+	/// randomness. Returns the new randomness.
+	fn randomness_change_epoch(epoch_index: u64) -> [u8; RANDOMNESS_LENGTH] {
+		let this_randomness = NextRandomness::get();
+		let next_randomness = compute_randomness(
+			this_randomness,
+			epoch_index,
+			UnderConstruction::get(),
+		);
+		UnderConstruction::put(&[0; RANDOMNESS_LENGTH]);
+		NextRandomness::put(&next_randomness);
+		this_randomness
 	}
+
 }
 
 impl<T: Trait> OnTimestampSet<T::Moment> for Module<T> {
 	fn on_timestamp_set(_moment: T::Moment) { }
 }
 
-impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
-	type Key = AuthorityId;
+pub trait Duration {
+	fn babe_epoch_duration() -> u64;
+}
 
-	fn on_new_session<'a, I: 'a>(changed: bool, validators: I)
+impl<T: Trait + staking::Trait + Duration> session::OneSessionHandler<T::AccountId> for Module<T> {
+	type Key = AuthorityId;
+	fn on_new_session<'a, I: 'a>(_changed: bool, _validators: I, queued_validators: I)
 		where I: Iterator<Item=(&'a T::AccountId, AuthorityId)>
 	{
-		// instant changes
-		if changed {
-			let next_authorities = validators.map(|(_, k)| k).collect::<Vec<_>>();
-			let last_authorities = <Module<T>>::authorities();
-			if next_authorities != last_authorities {
-				Self::change_authorities(next_authorities);
-			}
-		}
+		use staking::BalanceOf;
+		let to_votes = |b: BalanceOf<T>| {
+			<T::CurrencyToVote as Convert<BalanceOf<T>, u64>>::convert(b)
+		};
 
-		let rho = UnderConstruction::get();
-		UnderConstruction::put([0; 32]);
-		let last_epoch_randomness = EpochRandomness::get();
 		let epoch_index = EpochIndex::get()
 			.checked_add(1)
 			.expect("epoch indices will never reach 2^64 before the death of the universe; qed");
+
 		EpochIndex::put(epoch_index);
-		EpochRandomness::put(NextEpochRandomness::get());
-		let mut s = [0; 72];
-		s[..32].copy_from_slice(&last_epoch_randomness);
-		s[32..40].copy_from_slice(&epoch_index.to_le_bytes());
-		s[40..].copy_from_slice(&rho);
-		NextEpochRandomness::put(runtime_io::blake2_256(&s))
+
+		// *Next* epoch's authorities.
+		let authorities = queued_validators.map(|(account, k)| {
+			(k, to_votes(staking::Module::<T>::stakers(account).total))
+		}).collect::<Vec<_>>();
+
+		// What was the next epoch is now the current epoch
+		let randomness = Self::randomness_change_epoch(epoch_index);
+		Self::change_epoch(Epoch {
+			randomness,
+			authorities,
+			epoch_index,
+			duration: T::babe_epoch_duration(),
+		})
 	}
 
 	fn on_disabled(i: usize) {
-		let log: DigestItem<T::Hash> = DigestItem::Consensus(
-			BABE_ENGINE_ID,
-			ConsensusLog::OnDisabled(i as u64).encode(),
-		);
-		<system::Module<T>>::deposit_log(log.into());
+		Self::deposit_consensus(ConsensusLog::OnDisabled(i as u64))
 	}
 }
 
+fn compute_randomness(
+	last_epoch_randomness: [u8; RANDOMNESS_LENGTH],
+	epoch_index: u64,
+	rho: [u8; VRF_OUTPUT_LENGTH],
+) -> [u8; RANDOMNESS_LENGTH] {
+	let mut s = [0; 40 + VRF_OUTPUT_LENGTH];
+	s[..32].copy_from_slice(&last_epoch_randomness);
+	s[32..40].copy_from_slice(&epoch_index.to_le_bytes());
+	s[40..].copy_from_slice(&rho);
+	runtime_io::blake2_256(&s)
+}
+
 impl<T: Trait> ProvideInherent for Module<T> {
 	type Call = timestamp::Call<T>;
 	type Error = MakeFatalError<RuntimeString>;
@@ -286,7 +307,7 @@ impl<T: Trait> ProvideInherent for Module<T> {
 		if timestamp_based_slot == seal_slot {
 			Ok(())
 		} else {
-			Err(RuntimeString::from("timestamp set in block doesn’t match slot in seal").into())
+			Err(RuntimeString::from("timestamp set in block doesn't match slot in seal").into())
 		}
 	}
 }

substrate/srml/grandpa/src/lib.rs

@@ -346,7 +346,7 @@ impl<T: Trait> Module<T> {
 impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
 	type Key = AuthorityId;
 
-	fn on_new_session<'a, I: 'a>(changed: bool, validators: I)
+	fn on_new_session<'a, I: 'a>(changed: bool, validators: I, _queued_validators: I)
 		where I: Iterator<Item=(&'a T::AccountId, AuthorityId)>
 	{
 		// instant changes

substrate/srml/im-online/src/lib.rs

@@ -356,7 +356,7 @@ impl<T: Trait> Module<T> {
 impl<T: Trait> session::OneSessionHandler<T::AccountId> for Module<T> {
 	type Key = <T as Trait>::AuthorityId;
 
-	fn on_new_session<'a, I: 'a>(_changed: bool, _validators: I) {
+	fn on_new_session<'a, I: 'a>(_changed: bool, _validators: I, _next_validators: I) {
 		Self::new_session();
 	}
 

substrate/srml/session/src/lib.rs

@@ -186,7 +186,11 @@ impl<A> OnSessionEnding<A> for () {
 /// Handler for when a session keys set changes.
 pub trait SessionHandler<ValidatorId> {
 	/// Session set has changed; act appropriately.
-	fn on_new_session<Ks: OpaqueKeys>(changed: bool, validators: &[(ValidatorId, Ks)]);
+	fn on_new_session<Ks: OpaqueKeys>(
+		changed: bool,
+		validators: &[(ValidatorId, Ks)],
+		queued_validators: &[(ValidatorId, Ks)],
+	);
 
 	/// A validator got disabled. Act accordingly until a new session begins.
 	fn on_disabled(validator_index: usize);
@@ -197,7 +201,7 @@ pub trait OneSessionHandler<ValidatorId> {
 	/// The key type expected.
 	type Key: Decode + Default + TypedKey;
 
-	fn on_new_session<'a, I: 'a>(changed: bool, validators: I)
+	fn on_new_session<'a, I: 'a>(changed: bool, validators: I, queued_validators: I)
 		where I: Iterator<Item=(&'a ValidatorId, Self::Key)>, ValidatorId: 'a;
 	fn on_disabled(i: usize);
 }
@@ -205,19 +209,26 @@ pub trait OneSessionHandler<ValidatorId> {
 macro_rules! impl_session_handlers {
 	() => (
 		impl<AId> SessionHandler<AId> for () {
-			fn on_new_session<Ks: OpaqueKeys>(_: bool, _: &[(AId, Ks)]) {}
+			fn on_new_session<Ks: OpaqueKeys>(_: bool, _: &[(AId, Ks)], _: &[(AId, Ks)]) {}
 			fn on_disabled(_: usize) {}
 		}
 	);
 
 	( $($t:ident)* ) => {
 		impl<AId, $( $t: OneSessionHandler<AId> ),*> SessionHandler<AId> for ( $( $t , )* ) {
-			fn on_new_session<Ks: OpaqueKeys>(changed: bool, validators: &[(AId, Ks)]) {
+			fn on_new_session<Ks: OpaqueKeys>(
+				changed: bool,
+				validators: &[(AId, Ks)],
+				queued_validators: &[(AId, Ks)],
+			) {
 				$(
-					let our_keys = validators.iter()
+					let our_keys: Box<dyn Iterator<Item=_>> = Box::new(validators.iter()
 						.map(|k| (&k.0, k.1.get::<$t::Key>(<$t::Key as TypedKey>::KEY_TYPE)
-							.unwrap_or_default()));
-					$t::on_new_session(changed, our_keys);
+							.unwrap_or_default())));
+					let queued_keys: Box<dyn Iterator<Item=_>> = Box::new(queued_validators.iter()
+						.map(|k| (&k.0, k.1.get::<$t::Key>(<$t::Key as TypedKey>::KEY_TYPE)
+							.unwrap_or_default())));
+					$t::on_new_session(changed, our_keys, queued_keys);
 				)*
 			}
 			fn on_disabled(i: usize) {
@@ -420,14 +431,14 @@ impl<T: Trait> Module<T> {
 			.map(|a| { let k = Self::load_keys(&a).unwrap_or_default(); (a, k) })
 			.collect::<Vec<_>>();
 
-		<QueuedKeys<T>>::put(queued_amalgamated);
+		<QueuedKeys<T>>::put(queued_amalgamated.clone());
 		QueuedChanged::put(next_changed);
 
 		// Record that this happened.
 		Self::deposit_event(Event::NewSession(session_index));
 
 		// Tell everyone about the new session keys.
-		T::SessionHandler::on_new_session::<T::Keys>(changed, &session_keys);
+		T::SessionHandler::on_new_session::<T::Keys>(changed, &session_keys, &queued_amalgamated);
 	}
 
 	/// Disable the validator of index `i`.

substrate/srml/session/src/mock.rs

@@ -50,7 +50,11 @@ impl ShouldEndSession<u64> for TestShouldEndSession {
 
 pub struct TestSessionHandler;
 impl SessionHandler<u64> for TestSessionHandler {
-	fn on_new_session<T: OpaqueKeys>(changed: bool, validators: &[(u64, T)]) {
+	fn on_new_session<T: OpaqueKeys>(
+		changed: bool,
+		validators: &[(u64, T)],
+		_queued_validators: &[(u64, T)],
+	) {
 		SESSION_CHANGED.with(|l| *l.borrow_mut() = changed);
 		AUTHORITIES.with(|l|
 			*l.borrow_mut() = validators.iter().map(|(_, id)| id.get::<UintAuthorityId>(0).unwrap_or_default()).collect()

substrate/srml/staking/src/lib.rs

@@ -429,7 +429,7 @@ pub struct Exposure<AccountId, Balance: HasCompact> {
 	pub others: Vec<IndividualExposure<AccountId, Balance>>,
 }
 
-type BalanceOf<T> = <<T as Trait>::Currency as Currency<<T as system::Trait>::AccountId>>::Balance;
+pub type BalanceOf<T> = <<T as Trait>::Currency as Currency<<T as system::Trait>::AccountId>>::Balance;
 type PositiveImbalanceOf<T> =
 <<T as Trait>::Currency as Currency<<T as system::Trait>::AccountId>>::PositiveImbalance;
 type NegativeImbalanceOf<T> =

substrate/srml/staking/src/mock.rs

@@ -51,7 +51,11 @@ thread_local! {
 
 pub struct TestSessionHandler;
 impl session::SessionHandler<AccountId> for TestSessionHandler {
-	fn on_new_session<Ks: OpaqueKeys>(_changed: bool, validators: &[(AccountId, Ks)]) {
+	fn on_new_session<Ks: OpaqueKeys>(
+		_changed: bool,
+		validators: &[(AccountId, Ks)],
+		_queued_validators: &[(AccountId, Ks)],
+	) {
 		SESSION.with(|x|
 			*x.borrow_mut() = (validators.iter().map(|x| x.0.clone()).collect(), HashSet::new())
 		);

substrate/srml/system/benches/bench.rs

@@ -20,7 +20,7 @@ use srml_support::{decl_module, decl_event, impl_outer_origin, impl_outer_event}
 use runtime_io::{with_externalities, Blake2Hasher};
 use substrate_primitives::H256;
 use primitives::{
-	BuildStorage, traits::{BlakeTwo256, IdentityLookup},
+	traits::{BlakeTwo256, IdentityLookup},
 	testing::Header,
 };
 
@@ -54,6 +54,9 @@ impl_outer_event! {
 	}
 }
 
+srml_support::parameter_types! {
+	pub const BlockHashCount: u64 = 250;
+}
 #[derive(Clone, Eq, PartialEq)]
 pub struct Runtime;
 impl system::Trait for Runtime {
@@ -67,6 +70,7 @@ impl system::Trait for Runtime {
 	type Header = Header;
 	type WeightMultiplierUpdate = ();
 	type Event = Event;
+	type BlockHashCount = BlockHashCount;
 }
 
 impl module::Trait for Runtime {
@@ -74,7 +78,7 @@ impl module::Trait for Runtime {
 }
 
 fn new_test_ext() -> runtime_io::TestExternalities<Blake2Hasher> {
-	system::GenesisConfig::<Runtime>::default().build_storage().unwrap().0.into()
+	system::GenesisConfig::default().build_storage::<Runtime>().unwrap().0.into()
 }
 
 fn deposit_events(n: usize) {