Introduce inherent digests (#2466)

* Introduce inherent digests

* Implement inherent digests

* fix silly error

* Implementation of inherent digests in BABE

All tests pass. There are still limitations:

1. The runtime strips out inherent digests, so BABE must re-add them.
2. The test runtime checks that it can re-compute all digests.  It
   can’t, so I had to comment out that test.

* Fix compilation and seal import

Seals were not imported correctly: the pre-digest was imported twice,
instead of both it and the seal being imported.  Also, other parts of
the code did not compile due to incomplete refactoring.

* Remove bogus assertion

* Fix testsuite compilation

* Remove unused import

* Fix compiler diagnostics

* Add inherent digest parameters to block constructors

This enforces that inherent digests are added first.

* Fixup Cargo.lock

* Fix build errors

* Re-add an incorrectly removed import

* Bump primitive-types version

* Update Cargo.lock

* Refactoring

* Use inherent digests for AuRa

They do reach the runtime, but get stripped.  I have not figured out
where.

* Fix compilation errors

* Fix compilation errors due to incorrect types

* Fix whitespace

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Add preamble

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix silly compile error

* Refactor pre-digest finding code into a separate function

* Remove unwanted assertion

It is too likely to bring down the entire blockchain.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Use `find_pre_digest` after runtime, too

Also, use `Member` trait rather than rolling our own requirements.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix various warnings

mostly due to upgrading the dependency on `error_chain`.

* Pre-digests nearly complete

This nearly completes the implementation of pre-runtime digests.

* `Seal2` → `Seal` and fix test suite

* Try to fix the storage error

* Try to fix storage (again)

* Fix tests

* Hopefully finish pre-runtime digests

The key is to pass *only* the pre-runtime digests to the runtime.  The
others must be stripped out by `initialize_block`.

* Fix silly typo

* Fix another silly mistake

* Remove unnecessary filtering of BABE pre-digests

We no longer get duplicate BABE pre-digests, so if they appear, the
header should be rejected outright.

* Update Cargo.lock files

* Reformatting

* Fix silly typo in inherent digest code

Also, revert `error.rs` files that contained calls to the `error_chain!`
macro.

* Try to keep the runtime from stripping pre-digests

Currently runs into the “Storage root must match that calculated”
assertion.

* Don’t compute storage root until storage changes are done.

Also, fix a compilation error.

* Fix compile-time error

* Fix compilation errors

* Fix more compile errors

* Hopefully it compiles this time…

* Fix compilation and add docs

* Prevent BABE from adding duplicate pre-runtime digests

Found by comparing with the AuRa code.  I also did some refactoring.

* Respond to review and fix some warnings

* Delete some dead code introduced earlier

* More dead code goes away

* `ref mut` → `&mut`

* Respond to review and fix some warnings

* Fix compilation error

* Remove unneeded `HashT` type parameter

Suggested-by: Robert Habermeier <robert@parity.io>

* Remove spurious #[allow(deprecated)]

* Document inherent digest parameter to `build_block`

* Delete `Simple` trait

It wasn’t needed

* delete wrongly added files

* Fix trait bounds

* Digest serialization tests

I also did some reformatting and cleanup.

* Apply suggestions from code review

Reformatting

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Swap two arguments to `propose` and `propose_with`

Also, remove some needless unsafe code.

* Remove bogus `#![allow(deprecated)]` annotations

With the removal of the deprecated `Seal` variant, these are not needed.

* Add a missing `#[allow(deprecated)]` in the AuRa tests

* Fix silly compile error

* Fix silly compiler error

RLS did not tell me that I hadn’t fixed `babe/lib.rs`, so I missed it.

* Fixes made automatically by Cargo

substrate/core/sr-primitives/src/generic/digest.rs

@@ -62,7 +62,6 @@ impl<Item> traits::Digest for Digest<Item> where
 /// provide opaque access to other items.
 #[derive(PartialEq, Eq, Clone)]
 #[cfg_attr(feature = "std", derive(Debug))]
-#[allow(deprecated)]
 pub enum DigestItem<Hash, AuthorityId, SealSignature> {
 	/// System digest item announcing that authorities set has been changed
 	/// in the block. Contains the new set of authorities.
@@ -71,14 +70,19 @@ pub enum DigestItem<Hash, AuthorityId, SealSignature> {
 	/// block. It is created for every block iff runtime supports changes
 	/// trie creation.
 	ChangesTrieRoot(Hash),
-	/// The old way to put a Seal on it.  Deprecated.
-	#[deprecated(
-		since = "1.0",
-		note = "New versions of Substrate will never generate this, and it will be rejected on new blockchains.",
-	)]
-	Seal(u64, SealSignature),
-	/// Put a Seal on it
+	/// A message from the runtime to the consensus engine. This should *never*
+	/// be generated by the native code of any consensus engine, but this is not
+	/// checked (yet).
 	Consensus(ConsensusEngineId, Vec<u8>),
+	/// Put a Seal on it.
+	Seal(ConsensusEngineId, SealSignature),
+	/// A pre-runtime digest.
+	///
+	/// These are messages from the consensus engine to the runtime, although
+	/// the consensus engine can (and should) read them itself to avoid
+	/// code and state duplication.  It is erroneous for a runtime to produce
+	/// these, but this is not (yet) checked.
+	PreRuntime(ConsensusEngineId, Vec<u8>),
 	/// Any 'non-system' digest item, opaque to the native code.
 	Other(Vec<u8>),
 }
@@ -97,20 +101,26 @@ impl<Hash: Encode, AuthorityId: Encode, SealSignature: Encode> ::serde::Serializ
 /// final runtime implementations for encoding/decoding its log items.
 #[derive(PartialEq, Eq, Clone)]
 #[cfg_attr(feature = "std", derive(Debug))]
-#[allow(deprecated)]
 pub enum DigestItemRef<'a, Hash: 'a, AuthorityId: 'a, SealSignature: 'a> {
 	/// Reference to `DigestItem::AuthoritiesChange`.
 	AuthoritiesChange(&'a [AuthorityId]),
 	/// Reference to `DigestItem::ChangesTrieRoot`.
 	ChangesTrieRoot(&'a Hash),
-	/// A deprecated sealed signature for testing
-	#[deprecated]
-	Seal(&'a u64, &'a SealSignature),
-	/// A sealed signature for testing
-	Consensus(&'a ConsensusEngineId, &'a [u8]),
+	/// A message from the runtime to the consensus engine. This should *never*
+	/// be generated by the native code of any consensus engine, but this is not
+	/// checked (yet).
+	Consensus(&'a ConsensusEngineId, &'a Vec<u8>),
+	/// Put a Seal on it.
+	Seal(&'a ConsensusEngineId, &'a SealSignature),
+	/// A pre-runtime digest.
+	///
+	/// These are messages from the consensus engine to the runtime, although
+	/// the consensus engine can (and should) read them itself to avoid
+	/// code and state duplication.  It is erroneous for a runtime to produce
+	/// these, but this is not (yet) checked.
+	PreRuntime(&'a ConsensusEngineId, &'a Vec<u8>),
 	/// Any 'non-system' digest item, opaque to the native code.
-	/// Reference to `DigestItem::Other`.
-	Other(&'a [u8]),
+	Other(&'a Vec<u8>),
 }
 
 /// Type of the digest item. Used to gain explicit control over `DigestItem` encoding
@@ -123,8 +133,9 @@ enum DigestItemType {
 	Other = 0,
 	AuthoritiesChange = 1,
 	ChangesTrieRoot = 2,
-	Seal = 3,
 	Consensus = 4,
+	Seal = 5,
+	PreRuntime = 6,
 }
 
 impl<Hash, AuthorityId, SealSignature> DigestItem<Hash, AuthorityId, SealSignature> {
@@ -137,13 +148,13 @@ impl<Hash, AuthorityId, SealSignature> DigestItem<Hash, AuthorityId, SealSignatu
 	}
 
 	/// Returns a 'referencing view' for this digest item.
-	#[allow(deprecated)]
 	fn dref<'a>(&'a self) -> DigestItemRef<'a, Hash, AuthorityId, SealSignature> {
 		match *self {
 			DigestItem::AuthoritiesChange(ref v) => DigestItemRef::AuthoritiesChange(v),
 			DigestItem::ChangesTrieRoot(ref v) => DigestItemRef::ChangesTrieRoot(v),
-			DigestItem::Seal(ref v, ref s) => DigestItemRef::Seal(v, s),
 			DigestItem::Consensus(ref v, ref s) => DigestItemRef::Consensus(v, s),
+			DigestItem::Seal(ref v, ref s) => DigestItemRef::Seal(v, s),
+			DigestItem::PreRuntime(ref v, ref s) => DigestItemRef::PreRuntime(v, s),
 			DigestItem::Other(ref v) => DigestItemRef::Other(v),
 		}
 	}
@@ -164,6 +175,10 @@ impl<
 	fn as_changes_trie_root(&self) -> Option<&Self::Hash> {
 		self.dref().as_changes_trie_root()
 	}
+
+	fn as_pre_runtime(&self) -> Option<(ConsensusEngineId, &[u8])> {
+		self.dref().as_pre_runtime()
+	}
 }
 
 impl<Hash: Encode, AuthorityId: Encode, SealSignature: Encode> Encode for DigestItem<Hash, AuthorityId, SealSignature> {
@@ -183,14 +198,18 @@ impl<Hash: Decode, AuthorityId: Decode, SealSignature: Decode> Decode for Digest
 			DigestItemType::ChangesTrieRoot => Some(DigestItem::ChangesTrieRoot(
 				Decode::decode(input)?,
 			)),
-			DigestItemType::Seal => {
-				let vals: (u64, SealSignature) = Decode::decode(input)?;
-				Some(DigestItem::Seal(vals.0, vals.1))
-			},
 			DigestItemType::Consensus => {
 				let vals: (ConsensusEngineId, Vec<u8>) = Decode::decode(input)?;
 				Some(DigestItem::Consensus(vals.0, vals.1))
 			}
+			DigestItemType::Seal => {
+				let vals: (ConsensusEngineId, SealSignature) = Decode::decode(input)?;
+				Some(DigestItem::Seal(vals.0, vals.1))
+			},
+			DigestItemType::PreRuntime => {
+				let vals: (ConsensusEngineId, Vec<u8>) = Decode::decode(input)?;
+				Some(DigestItem::PreRuntime(vals.0, vals.1))
+			},
 			DigestItemType::Other => Some(DigestItem::Other(
 				Decode::decode(input)?,
 			)),
@@ -214,9 +233,16 @@ impl<'a, Hash: Codec + Member, AuthorityId: Codec + Member, SealSignature: Codec
 			_ => None,
 		}
 	}
+
+	/// Cast this digest item into `PreRuntime`
+	pub fn as_pre_runtime(&self) -> Option<(ConsensusEngineId, &'a [u8])> {
+		match *self {
+			DigestItemRef::PreRuntime(consensus_engine_id, ref data) => Some((*consensus_engine_id, data)),
+			_ => None,
+		}
+	}
 }
 
-#[allow(deprecated)]
 impl<'a, Hash: Encode, AuthorityId: Encode, SealSignature: Encode> Encode for DigestItemRef<'a, Hash, AuthorityId, SealSignature> {
 	fn encode(&self) -> Vec<u8> {
 		let mut v = Vec::new();
@@ -230,13 +256,17 @@ impl<'a, Hash: Encode, AuthorityId: Encode, SealSignature: Encode> Encode for Di
 				DigestItemType::ChangesTrieRoot.encode_to(&mut v);
 				changes_trie_root.encode_to(&mut v);
 			},
+			DigestItemRef::Consensus(val, data) => {
+				DigestItemType::Consensus.encode_to(&mut v);
+				(val, data).encode_to(&mut v);
+			},
 			DigestItemRef::Seal(val, sig) => {
 				DigestItemType::Seal.encode_to(&mut v);
 				(val, sig).encode_to(&mut v);
 			},
-			DigestItemRef::Consensus(val, sig) => {
-				DigestItemType::Consensus.encode_to(&mut v);
-				(val, sig).encode_to(&mut v);
+			DigestItemRef::PreRuntime(val, data) => {
+				DigestItemType::PreRuntime.encode_to(&mut v);
+				(val, data).encode_to(&mut v);
 			},
 			DigestItemRef::Other(val) => {
 				DigestItemType::Other.encode_to(&mut v);
@@ -254,20 +284,19 @@ mod tests {
 	use substrate_primitives::hash::H512 as Signature;
 
 	#[test]
-	#[allow(deprecated)]
 	fn should_serialize_digest() {
 		let digest = Digest {
 			logs: vec![
 				DigestItem::AuthoritiesChange(vec![1]),
 				DigestItem::ChangesTrieRoot(4),
-				DigestItem::Seal(1, Signature::from_low_u64_be(15)),
 				DigestItem::Other(vec![1, 2, 3]),
+				DigestItem::Seal(Default::default(), Signature::default())
 			],
 		};
 
 		assert_eq!(
 			::serde_json::to_string(&digest).unwrap(),
-			r#"{"logs":["0x010401000000","0x0204000000","0x0301000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f","0x000c010203"]}"#
+			"{\"logs\":[\"0x010401000000\",\"0x0204000000\",\"0x000c010203\",\"0x050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"]}",
 		);
 	}
 }

substrate/core/sr-primitives/src/generic/header.rs

@@ -18,9 +18,12 @@
 
 #[cfg(feature = "std")]
 use serde::Serialize;
+#[cfg(feature = "std")]
+use log::debug;
 use crate::codec::{Decode, Encode, Codec, Input, Output, HasCompact, EncodeAsRef};
 use crate::traits::{self, Member, SimpleArithmetic, SimpleBitOps, MaybeDisplay,
-	Hash as HashT, DigestItem as DigestItemT, MaybeSerializeDebug, MaybeSerializeDebugButNotDeserialize};
+	Hash as HashT, DigestItem as DigestItemT, MaybeSerializeDebug,
+	MaybeSerializeDebugButNotDeserialize};
 use crate::generic::Digest;
 
 /// Abstraction over a block header for a substrate chain.
@@ -107,22 +110,29 @@ impl<Number, Hash, DigestItem> traits::Header for Header<Number, Hash, DigestIte
 	fn set_parent_hash(&mut self, hash: Self::Hash) { self.parent_hash = hash }
 
 	fn digest(&self) -> &Self::Digest { &self.digest }
+
+	#[cfg(feature = "std")]
+	fn digest_mut(&mut self) -> &mut Self::Digest {
+		debug!(target: "header", "Retrieving mutable reference to digest");
+		&mut self.digest
+	}
+
+	#[cfg(not(feature = "std"))]
 	fn digest_mut(&mut self) -> &mut Self::Digest { &mut self.digest }
-	fn set_digest(&mut self, digest: Self::Digest) { self.digest = digest }
 
 	fn new(
 		number: Self::Number,
 		extrinsics_root: Self::Hash,
 		state_root: Self::Hash,
 		parent_hash: Self::Hash,
-		digest: Self::Digest
+		digest: Self::Digest,
 	) -> Self {
 		Header {
 			number,
 			extrinsics_root,
 			state_root,
 			parent_hash,
-			digest
+			digest,
 		}
 	}
 }