BABE signing and verification (#2270)

* Add basic BABE consensus type

* Update core/consensus/babe/slots/Cargo.toml

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Fix parameterization and run `rustfmt`

* Respond to review comments

* Update various Cargo.lock files

* Revert "Update various Cargo.lock files"

This reverts commit af53d7624752a744320e9cbb25749fdd8e6f46d2.

* `BabeSealSignature` → `BabeSeal`

* Move slot code to its own crate

This was highly non-trivial, due to cyclic dependencies.

* Remove redundancy between AuRa and BABE

Some of the code duplication was removed using a macro.

* Fix build error

* Avoid non-`#[doc(hidden)]` re-exports

Also, bump some library versions in `Cargo.toml`.

* Remove dead code in AuRa

* Remove impl_slot macro

It was more trouble than it was worth.

Also, delete useless dependencies on Serde.

* AuRa and BABE need different DB keys

* Bring back `aura::Network`, but deprecate it.

* Improve docs and add `slot_duration` inherent method

* Add docs to `substrate_consensus_aura::SlotDuration`

* Add missing documentation and #![forbid(missing_docs, unsafe_code)]

* Add a #![forbid(missing_docs)]

* Remove dependency of `test-runtime` on `slots`

* Update core/consensus/babe/src/lib.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Remove wrongly added file

* Fix copyright notice

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Bump `impl_version` and `spec_version`

* Add more code to BABE

Most of it is copied from AuRa code, but at least the initial core is
there.

* Stuck on horrible compiler error message

* add missing files

* Spaces → tabs

* Simplify code

* Fix compilation

This involved fixing dependencies and adding a `Mutex`.

* More work on BABE

* Fix deprecation version

* Fix deprecation version; remove spurious carets

* Fix Cargo.toml

* Implement VRF signing logic

* The import queue code compiles, though it probably doesn’t work.

* Add VRF verification

* Update Cargo.lock

* Update dependencies

* Move test network to sr25519 authority keys

* Fix accidental build bustage

* Trying to get the tests to work

* Add logging messages and remove dead code

There seems to be a problem with the test network.  Since AuRa and BABE
are both affected, this is most likely due to the switch from ed25519 to
sr25519.

* Trying to get the tests to work

* Add logging messages and remove dead code

There seems to be a problem with the test network.  Since AuRa and BABE
are both affected, this is most likely due to the switch from ed25519 to
sr25519.

* Working testsuite at last!

The problem was with serialization and deserialization.  Normally,
those functions are generated automatically, but those for `BabeSeal`
had to be written manually.  The hand-written versions were not
correct, however, as shown by the decoder not being able to decode the
output of the encoder.

* Enable BabeSeal::Encode asserts in --release tests

* Bump runtime and dependency versions

* Fix wasm compilation

The wasm build was broken because of a typo in
`core/test-runtime/src/lib.rs`, and missing gates on the `std` feature
in `core/consensus/{aura,babe}/primitives/Cargo.toml`.  Additionally,
improve the quotation in the build scripts.

* Merge Cargo.lock

* Change expected JSON string

The test was also broken on `master`, so I suspect that the test was
incorrect.

* Responded to review

* Remove hard-coded threshold from production code

A hard-coded threshold is now only used in tests.

* Fix swapped doc comments

* Fix unused import warnings

* fix ci error

* fix typo

* Fix spacing in docs

* Minor changes suggested by @joepetrowski

on https://github.com/paritytech/substrate/pull/2372

* Remove unnecessary getters

* fix compile error

* Fix silly unused-variable error

* Improve documentation formatting

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Add issue links

* Revert excess verbosity and #![forbid(warnings)]

* Apply suggestions from code review

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Reformat some comments

* Threshold should depend on number of validators

Also, respond to code review

* Fix silly compilation errors

* Reduce logging verbosity

* Fix missing import

substrate/core/network/src/protocol.rs

@@ -435,7 +435,10 @@ impl<B: BlockT, S: NetworkSpecialization<B>, H: ExHashT> Protocol<B, S, H> {
 				return false;
 			},
 			#[cfg(any(test, feature = "test-helpers"))]
-			ProtocolMsg::Synchronize => self.network_chan.send(NetworkMsg::Synchronized),
+			ProtocolMsg::Synchronize => {
+				trace!(target: "sync", "handle_client_msg: received Synchronize msg");
+				self.network_chan.send(NetworkMsg::Synchronized)
+			}
 		}
 		true
 	}
@@ -755,6 +758,11 @@ impl<B: BlockT, S: NetworkSpecialization<B>, H: ExHashT> Protocol<B, S, H> {
 					"Peer is on different chain (our genesis: {} theirs: {})",
 					self.genesis_hash, status.genesis_hash
 				);
+				trace!(
+					target: "protocol",
+					"Peer is on different chain (our genesis: {} theirs: {})",
+					self.genesis_hash, status.genesis_hash
+				);
 				self.network_chan.send(NetworkMsg::ReportPeer(
 					who,
 					Severity::Bad(reason),
@@ -763,6 +771,7 @@ impl<B: BlockT, S: NetworkSpecialization<B>, H: ExHashT> Protocol<B, S, H> {
 			}
 			if status.version < MIN_VERSION && CURRENT_VERSION < status.min_supported_version {
 				let reason = format!("Peer using unsupported protocol version {}", status.version);
+				trace!(target: "protocol", "Peer {:?} using unsupported protocol version {}", who, status.version);
 				self.network_chan.send(NetworkMsg::ReportPeer(
 					who,
 					Severity::Bad(reason),

substrate/core/network/src/service.rs

@@ -589,7 +589,7 @@ fn run_thread<B: BlockT + 'static>(
 				}
 			}
 		}
-		
+
 		Ok(Async::NotReady)
 	})
 }

substrate/core/network/src/sync.rs

@@ -848,8 +848,9 @@ impl<B: BlockT> ChainSync<B> {
 	/// Handle new block announcement.
 	pub(crate) fn on_block_announce(&mut self, protocol: &mut Context<B>, who: PeerId, hash: B::Hash, header: &B::Header) {
 		let number = *header.number();
+		debug!(target: "sync", "Received block announcement with number {:?}", number);
 		if number <= As::sa(0) {
-			trace!(target: "sync", "Ignored invalid block announcement from {}: {}", who, hash);
+			warn!(target: "sync", "Ignored invalid block announcement from {}: {}", who, hash);
 			return;
 		}
 		let parent_status = block_status(&*protocol.client(), &self.queue_blocks, header.parent_hash().clone()).ok()

substrate/core/network/src/test/mod.rs

@@ -40,7 +40,7 @@ use futures::sync::{mpsc, oneshot};
 use crate::message::Message;
 use network_libp2p::PeerId;
 use parking_lot::{Mutex, RwLock};
-use primitives::{H256, ed25519::Public as AuthorityId};
+use primitives::{H256, sr25519::Public as AuthorityId};
 use crate::protocol::{ConnectedPeer, Context, FromNetworkMsg, Protocol, ProtocolMsg};
 use runtime_primitives::generic::BlockId;
 use runtime_primitives::traits::{AuthorityIdFor, Block as BlockT, Digest, DigestItem, Header, NumberFor};
@@ -48,6 +48,7 @@ use runtime_primitives::{Justification, ConsensusEngineId};
 use crate::service::{network_channel, NetworkChan, NetworkLink, NetworkMsg, NetworkPort, TransactionPool};
 use crate::specialization::NetworkSpecialization;
 use test_client::{self, AccountKeyring};
+use log::debug;
 
 pub use test_client::runtime::{Block, Extrinsic, Hash, Transfer};
 pub use test_client::TestClient;
@@ -117,17 +118,20 @@ pub type PeersClient = client::Client<test_client::Backend, test_client::Executo
 /// A Link that can wait for a block to have been imported.
 pub struct TestLink<S: NetworkSpecialization<Block>> {
 	link: NetworkLink<Block, S>,
+
+	#[cfg(any(test, feature = "test-helpers"))]
 	network_to_protocol_sender: Sender<FromNetworkMsg<Block>>,
 }
 
 impl<S: NetworkSpecialization<Block>> TestLink<S> {
 	fn new(
 		protocol_sender: Sender<ProtocolMsg<Block, S>>,
-		network_to_protocol_sender: Sender<FromNetworkMsg<Block>>,
+		_network_to_protocol_sender: Sender<FromNetworkMsg<Block>>,
 		network_sender: NetworkChan<Block>
 	) -> TestLink<S> {
 		TestLink {
-			network_to_protocol_sender,
+			#[cfg(any(test, feature = "test-helpers"))]
+			network_to_protocol_sender: _network_to_protocol_sender,
 			link: NetworkLink {
 				protocol_sender,
 				network_sender,
@@ -165,8 +169,14 @@ impl<S: NetworkSpecialization<Block>> Link<Block> for TestLink<S> {
 		self.link.restart();
 	}
 
+	/// Send synchronization request to the block import channel.
+	///
+	/// The caller should wait for the `Link::synchronized` call to ensure that it has synchronized
+	/// with `ImportQueue`.
+	#[cfg(any(test, feature = "test-helpers"))]
 	fn synchronized(&self) {
-		let _ = self.network_to_protocol_sender.send(FromNetworkMsg::Synchronize);
+		trace!(target: "test_network", "Synchronizing");
+		drop(self.network_to_protocol_sender.send(FromNetworkMsg::Synchronize))
 	}
 }
 
@@ -225,12 +235,14 @@ impl<S: NetworkSpecialization<Block>> ProtocolChannel<S> {
 
 	/// Wait until synchronization response is generated by the protocol.
 	pub fn wait_sync(&self) -> Result<(), RecvError> {
+		trace!(target: "test_network", "Waiting for sync");
 		loop {
 			match self.protocol_to_network_receiver.receiver().recv() {
 				Ok(NetworkMsg::Synchronized) => return Ok(()),
 				Err(error) => return Err(error),
 				Ok(msg) => self.buffered_messages.lock().push_back(msg),
 			}
+			trace!(target: "test_network", "Retrying sync");
 		}
 	}
 
@@ -370,8 +382,11 @@ impl<D, S: NetworkSpecialization<Block>> Peer<D, S> {
 	}
 
 	/// Synchronize with import queue.
+	#[cfg(any(test, feature = "test-helpers"))]
 	fn import_queue_sync(&self) {
+		trace!(target: "test_network", "syncing this queue");
 		self.import_queue.synchronize();
+		trace!(target: "test_network", "wating for sync to finish");
 		let _ = self.net_proto_channel.wait_sync();
 	}
 
@@ -491,6 +506,7 @@ impl<D, S: NetworkSpecialization<Block>> Peer<D, S> {
 			let block = edit_block(builder);
 			let hash = block.header.hash();
 			trace!(
+				target: "test_network",
 				"Generating {}, (#{}, parent={})",
 				hash,
 				block.header.number,
@@ -610,10 +626,12 @@ pub trait TestNetFactory: Sized {
 
 	/// Create new test network with this many peers.
 	fn new(n: usize) -> Self {
+		trace!(target: "test_network", "Creating test network");
 		let config = Self::default_config();
 		let mut net = Self::from_config(&config);
 
-		for _ in 0..n {
+		for i in 0..n {
+			trace!(target: "test_network", "Adding peer {}", i);
 			net.add_peer(&config);
 		}
 		net
@@ -681,6 +699,7 @@ pub trait TestNetFactory: Sized {
 		}
 
 		loop {
+			debug!(target: "test_network", "loop iteration");
 			// we only deliver Status messages during start
 			let need_continue = self.route_single(true, None, &|msg| match *msg {
 				NetworkMsg::Outgoing(_, crate::message::generic::Message::Status(_)) => true,
@@ -706,6 +725,7 @@ pub trait TestNetFactory: Sized {
 		let mut to_disconnect = HashSet::new();
 		let peers = self.peers();
 		for peer in peers {
+			debug!(target: "test_network", "checking peer");
 			if let Some(message) = peer.pending_message(message_filter) {
 				match message {
 					NetworkMsg::Outgoing(recipient_id, packet) => {
@@ -744,10 +764,13 @@ pub trait TestNetFactory: Sized {
 				}
 			}
 		}
+		debug!(target: "test_network", "syncing queues");
 
 		// make sure that the protocol(s) has processed all messages that have been queued
 		self.peers().iter().for_each(|peer| peer.import_queue_sync());
 
+		debug!(target: "test_network", "queues synced");
+
 		had_messages
 	}