BABE signing and verification (#2270)

* Add basic BABE consensus type

* Update core/consensus/babe/slots/Cargo.toml

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Fix parameterization and run `rustfmt`

* Respond to review comments

* Update various Cargo.lock files

* Revert "Update various Cargo.lock files"

This reverts commit af53d7624752a744320e9cbb25749fdd8e6f46d2.

* `BabeSealSignature` → `BabeSeal`

* Move slot code to its own crate

This was highly non-trivial, due to cyclic dependencies.

* Remove redundancy between AuRa and BABE

Some of the code duplication was removed using a macro.

* Fix build error

* Avoid non-`#[doc(hidden)]` re-exports

Also, bump some library versions in `Cargo.toml`.

* Remove dead code in AuRa

* Remove impl_slot macro

It was more trouble than it was worth.

Also, delete useless dependencies on Serde.

* AuRa and BABE need different DB keys

* Bring back `aura::Network`, but deprecate it.

* Improve docs and add `slot_duration` inherent method

* Add docs to `substrate_consensus_aura::SlotDuration`

* Add missing documentation and #![forbid(missing_docs, unsafe_code)]

* Add a #![forbid(missing_docs)]

* Remove dependency of `test-runtime` on `slots`

* Update core/consensus/babe/src/lib.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Remove wrongly added file

* Fix copyright notice

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Bump `impl_version` and `spec_version`

* Add more code to BABE

Most of it is copied from AuRa code, but at least the initial core is
there.

* Stuck on horrible compiler error message

* add missing files

* Spaces → tabs

* Simplify code

* Fix compilation

This involved fixing dependencies and adding a `Mutex`.

* More work on BABE

* Fix deprecation version

* Fix deprecation version; remove spurious carets

* Fix Cargo.toml

* Implement VRF signing logic

* The import queue code compiles, though it probably doesn’t work.

* Add VRF verification

* Update Cargo.lock

* Update dependencies

* Move test network to sr25519 authority keys

* Fix accidental build bustage

* Trying to get the tests to work

* Add logging messages and remove dead code

There seems to be a problem with the test network.  Since AuRa and BABE
are both affected, this is most likely due to the switch from ed25519 to
sr25519.

* Trying to get the tests to work

* Add logging messages and remove dead code

There seems to be a problem with the test network.  Since AuRa and BABE
are both affected, this is most likely due to the switch from ed25519 to
sr25519.

* Working testsuite at last!

The problem was with serialization and deserialization.  Normally,
those functions are generated automatically, but those for `BabeSeal`
had to be written manually.  The hand-written versions were not
correct, however, as shown by the decoder not being able to decode the
output of the encoder.

* Enable BabeSeal::Encode asserts in --release tests

* Bump runtime and dependency versions

* Fix wasm compilation

The wasm build was broken because of a typo in
`core/test-runtime/src/lib.rs`, and missing gates on the `std` feature
in `core/consensus/{aura,babe}/primitives/Cargo.toml`.  Additionally,
improve the quotation in the build scripts.

* Merge Cargo.lock

* Change expected JSON string

The test was also broken on `master`, so I suspect that the test was
incorrect.

* Responded to review

* Remove hard-coded threshold from production code

A hard-coded threshold is now only used in tests.

* Fix swapped doc comments

* Fix unused import warnings

* fix ci error

* fix typo

* Fix spacing in docs

* Minor changes suggested by @joepetrowski

on https://github.com/paritytech/substrate/pull/2372

* Remove unnecessary getters

* fix compile error

* Fix silly unused-variable error

* Improve documentation formatting

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Add issue links

* Revert excess verbosity and #![forbid(warnings)]

* Apply suggestions from code review

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Reformat some comments

* Threshold should depend on number of validators

Also, respond to code review

* Fix silly compilation errors

* Reduce logging verbosity

* Fix missing import

substrate/core/finality-grandpa/src/import.rs

@@ -33,10 +33,10 @@ use fg_primitives::GrandpaApi;
 use runtime_primitives::Justification;
 use runtime_primitives::generic::BlockId;
 use runtime_primitives::traits::{
-	Block as BlockT, DigestFor, DigestItemFor, DigestItem,
+	Block as BlockT, DigestFor,
 	Header as HeaderT, NumberFor, ProvideRuntimeApi,
 };
-use substrate_primitives::{H256, ed25519, Blake2Hasher};
+use substrate_primitives::{H256, Blake2Hasher};
 
 use crate::{Error, CommandOrError, NewAuthoritySet, VoterCommand};
 use crate::authorities::{AuthoritySet, SharedAuthoritySet, DelayKind, PendingChange};
@@ -44,8 +44,6 @@ use crate::consensus_changes::SharedConsensusChanges;
 use crate::environment::{finalize_block, is_descendent_of};
 use crate::justification::GrandpaJustification;
 
-use ed25519::Public as AuthorityId;
-
 /// A block-import handler for GRANDPA.
 ///
 /// This scans each imported block for signals of changing authority set.
@@ -69,7 +67,6 @@ impl<B, E, Block: BlockT<Hash=H256>, RA, PRA> JustificationImport<Block>
 		B: Backend<Block, Blake2Hasher> + 'static,
 		E: CallExecutor<Block, Blake2Hasher> + 'static + Clone + Send + Sync,
 		DigestFor<Block>: Encode,
-		DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 		RA: Send + Sync,
 		PRA: ProvideRuntimeApi,
 		PRA::Api: GrandpaApi<Block>,
@@ -163,7 +160,6 @@ impl<B, E, Block: BlockT<Hash=H256>, RA, PRA> GrandpaBlockImport<B, E, Block, RA
 	B: Backend<Block, Blake2Hasher> + 'static,
 	E: CallExecutor<Block, Blake2Hasher> + 'static + Clone + Send + Sync,
 	DigestFor<Block>: Encode,
-	DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 	RA: Send + Sync,
 	PRA: ProvideRuntimeApi,
 	PRA::Api: GrandpaApi<Block>,
@@ -381,7 +377,6 @@ impl<B, E, Block: BlockT<Hash=H256>, RA, PRA> BlockImport<Block>
 		B: Backend<Block, Blake2Hasher> + 'static,
 		E: CallExecutor<Block, Blake2Hasher> + 'static + Clone + Send + Sync,
 		DigestFor<Block>: Encode,
-		DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 		RA: Send + Sync,
 		PRA: ProvideRuntimeApi,
 		PRA::Api: GrandpaApi<Block>,

substrate/core/finality-grandpa/src/lib.rs

@@ -51,7 +51,7 @@
 //! number (this is num(signal) + N). When finalizing a block, we either apply
 //! or prune any signaled changes based on whether the signaling block is
 //! included in the newly-finalized chain.
-
+#![forbid(warnings)]
 use futures::prelude::*;
 use log::{debug, info, warn};
 use futures::sync::mpsc;
@@ -62,7 +62,7 @@ use client::{
 use client::blockchain::HeaderBackend;
 use parity_codec::Encode;
 use runtime_primitives::traits::{
-	NumberFor, Block as BlockT, DigestFor, ProvideRuntimeApi, DigestItemFor, DigestItem,
+	NumberFor, Block as BlockT, DigestFor, ProvideRuntimeApi,
 };
 use fg_primitives::GrandpaApi;
 use inherents::InherentDataProviders;
@@ -362,7 +362,6 @@ fn global_communication<Block: BlockT<Hash=H256>, B, E, N, RA>(
 	N: Network<Block>,
 	RA: Send + Sync,
 	NumberFor<Block>: BlockNumberOps,
-	DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 {
 
 	let is_voter = local_key
@@ -463,7 +462,6 @@ pub fn run_grandpa_voter<B, E, Block: BlockT<Hash=H256>, N, RA, X>(
 	N::In: Send + 'static,
 	NumberFor<Block>: BlockNumberOps,
 	DigestFor<Block>: Encode,
-	DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 	RA: Send + Sync + 'static,
 	X: Future<Item=(),Error=()> + Clone + Send + 'static,
 {
@@ -716,7 +714,6 @@ pub fn run_grandpa<B, E, Block: BlockT<Hash=H256>, N, RA, X>(
 	N::In: Send + 'static,
 	NumberFor<Block>: BlockNumberOps,
 	DigestFor<Block>: Encode,
-	DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 	RA: Send + Sync + 'static,
 	X: Future<Item=(),Error=()> + Clone + Send + 'static,
 {

substrate/core/finality-grandpa/src/observer.rs

@@ -25,9 +25,8 @@ use grandpa::{
 use log::{debug, info, warn};
 
 use client::{CallExecutor, Client, backend::Backend};
-use ed25519::Public as AuthorityId;
-use runtime_primitives::traits::{NumberFor, Block as BlockT, DigestItemFor, DigestItem};
-use substrate_primitives::{ed25519, H256, Blake2Hasher};
+use runtime_primitives::traits::{NumberFor, Block as BlockT};
+use substrate_primitives::{ed25519::Public as AuthorityId, H256, Blake2Hasher};
 
 use crate::{
 	AuthoritySignature, global_communication, CommandOrError, Config, environment,
@@ -155,7 +154,6 @@ pub fn run_grandpa_observer<B, E, Block: BlockT<Hash=H256>, N, RA>(
 	N: Network<Block> + Send + Sync + 'static,
 	N::In: Send + 'static,
 	NumberFor<Block>: BlockNumberOps,
-	DigestItemFor<Block>: DigestItem<AuthorityId=AuthorityId>,
 	RA: Send + Sync + 'static,
 {
 	let LinkHalf {

substrate/core/finality-grandpa/src/tests.rs

@@ -23,7 +23,7 @@ use network::config::{ProtocolConfig, Roles};
 use network::consensus_gossip as network_gossip;
 use parking_lot::Mutex;
 use tokio::runtime::current_thread;
-use keyring::AuthorityKeyring;
+use keyring::ed25519::{Keyring as AuthorityKeyring};
 use client::{
 	BlockchainEvents, error::Result,
 	blockchain::Backend as BlockchainBackend,
@@ -36,7 +36,7 @@ use std::collections::{HashMap, HashSet};
 use std::result;
 use runtime_primitives::traits::{ApiRef, ProvideRuntimeApi, Header as HeaderT};
 use runtime_primitives::generic::BlockId;
-use substrate_primitives::{NativeOrEncoded, ExecutionContext};
+use substrate_primitives::{NativeOrEncoded, ExecutionContext, ed25519::Public as AuthorityId};
 
 use authorities::AuthoritySet;
 use communication::GRANDPA_ENGINE_ID;
@@ -290,7 +290,7 @@ impl Core<Block> for RuntimeApi {
 		_: ExecutionContext,
 		_: Option<()>,
 		_: Vec<u8>,
-	) -> Result<NativeOrEncoded<Vec<AuthorityId>>> {
+	) -> Result<NativeOrEncoded<Vec<substrate_primitives::sr25519::Public>>> {
 		unimplemented!("Not required for testing!")
 	}
 }
@@ -323,7 +323,7 @@ impl GrandpaApi<Block> for RuntimeApi {
 		_: ExecutionContext,
 		_: Option<()>,
 		_: Vec<u8>,
-	) -> Result<NativeOrEncoded<Vec<(AuthorityId, u64)>>> {
+	) -> Result<NativeOrEncoded<Vec<(substrate_primitives::ed25519::Public, u64)>>> {
 		if at == &BlockId::Number(0) {
 			Ok(self.inner.genesis_authorities.clone()).map(NativeOrEncoded::Native)
 		} else {
@@ -370,7 +370,7 @@ impl GrandpaApi<Block> for RuntimeApi {
 const TEST_GOSSIP_DURATION: Duration = Duration::from_millis(500);
 const TEST_ROUTING_INTERVAL: Duration = Duration::from_millis(50);
 
-fn make_ids(keys: &[AuthorityKeyring]) -> Vec<(AuthorityId, u64)> {
+fn make_ids(keys: &[AuthorityKeyring]) -> Vec<(substrate_primitives::ed25519::Public, u64)> {
 	keys.iter()
 		.map(|key| AuthorityId(key.to_raw_public()))
 		.map(|id| (id, 1))
@@ -734,7 +734,7 @@ fn justification_is_emitted_when_consensus_data_changes() {
 	let mut net = GrandpaTestNet::new(TestApi::new(make_ids(peers)), 3);
 
 	// import block#1 WITH consensus data change
-	let new_authorities = vec![AuthorityId::from_raw([42; 32])];
+	let new_authorities = vec![substrate_primitives::sr25519::Public::from_raw([42; 32])];
 	net.peer(0).push_authorities_change_block(new_authorities);
 	net.sync();
 	let net = Arc::new(Mutex::new(net));