Introduce inherent digests (#2466)

* Introduce inherent digests

* Implement inherent digests

* fix silly error

* Implementation of inherent digests in BABE

All tests pass. There are still limitations:

1. The runtime strips out inherent digests, so BABE must re-add them.
2. The test runtime checks that it can re-compute all digests.  It
   can’t, so I had to comment out that test.

* Fix compilation and seal import

Seals were not imported correctly: the pre-digest was imported twice,
instead of both it and the seal being imported.  Also, other parts of
the code did not compile due to incomplete refactoring.

* Remove bogus assertion

* Fix testsuite compilation

* Remove unused import

* Fix compiler diagnostics

* Add inherent digest parameters to block constructors

This enforces that inherent digests are added first.

* Fixup Cargo.lock

* Fix build errors

* Re-add an incorrectly removed import

* Bump primitive-types version

* Update Cargo.lock

* Refactoring

* Use inherent digests for AuRa

They do reach the runtime, but get stripped.  I have not figured out
where.

* Fix compilation errors

* Fix compilation errors due to incorrect types

* Fix whitespace

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Add preamble

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix silly compile error

* Refactor pre-digest finding code into a separate function

* Remove unwanted assertion

It is too likely to bring down the entire blockchain.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Use `find_pre_digest` after runtime, too

Also, use `Member` trait rather than rolling our own requirements.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix various warnings

mostly due to upgrading the dependency on `error_chain`.

* Pre-digests nearly complete

This nearly completes the implementation of pre-runtime digests.

* `Seal2` → `Seal` and fix test suite

* Try to fix the storage error

* Try to fix storage (again)

* Fix tests

* Hopefully finish pre-runtime digests

The key is to pass *only* the pre-runtime digests to the runtime.  The
others must be stripped out by `initialize_block`.

* Fix silly typo

* Fix another silly mistake

* Remove unnecessary filtering of BABE pre-digests

We no longer get duplicate BABE pre-digests, so if they appear, the
header should be rejected outright.

* Update Cargo.lock files

* Reformatting

* Fix silly typo in inherent digest code

Also, revert `error.rs` files that contained calls to the `error_chain!`
macro.

* Try to keep the runtime from stripping pre-digests

Currently runs into the “Storage root must match that calculated”
assertion.

* Don’t compute storage root until storage changes are done.

Also, fix a compilation error.

* Fix compile-time error

* Fix compilation errors

* Fix more compile errors

* Hopefully it compiles this time…

* Fix compilation and add docs

* Prevent BABE from adding duplicate pre-runtime digests

Found by comparing with the AuRa code.  I also did some refactoring.

* Respond to review and fix some warnings

* Delete some dead code introduced earlier

* More dead code goes away

* `ref mut` → `&mut`

* Respond to review and fix some warnings

* Fix compilation error

* Remove unneeded `HashT` type parameter

Suggested-by: Robert Habermeier <robert@parity.io>

* Remove spurious #[allow(deprecated)]

* Document inherent digest parameter to `build_block`

* Delete `Simple` trait

It wasn’t needed

* delete wrongly added files

* Fix trait bounds

* Digest serialization tests

I also did some reformatting and cleanup.

* Apply suggestions from code review

Reformatting

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Swap two arguments to `propose` and `propose_with`

Also, remove some needless unsafe code.

* Remove bogus `#![allow(deprecated)]` annotations

With the removal of the deprecated `Seal` variant, these are not needed.

* Add a missing `#[allow(deprecated)]` in the AuRa tests

* Fix silly compile error

* Fix silly compiler error

RLS did not tell me that I hadn’t fixed `babe/lib.rs`, so I missed it.

* Fixes made automatically by Cargo

substrate/core/test-runtime/src/system.rs

@@ -21,7 +21,7 @@ use rstd::prelude::*;
 use runtime_io::{storage_root, enumerated_trie_root, storage_changes_root, twox_128, blake2_256};
 use runtime_support::storage::{self, StorageValue, StorageMap};
 use runtime_support::storage_items;
-use runtime_primitives::traits::{Hash as HashT, BlakeTwo256, Digest as DigestT};
+use runtime_primitives::traits::{Hash as HashT, BlakeTwo256, Digest as DigestT, Header as _};
 use runtime_primitives::generic;
 use runtime_primitives::{ApplyError, ApplyOutcome, ApplyResult, transaction_validity::TransactionValidity};
 use parity_codec::{KeyedVec, Encode};
@@ -38,6 +38,7 @@ storage_items! {
 	Number: b"sys:num" => BlockNumber;
 	ParentHash: b"sys:pha" => required Hash;
 	NewAuthorities: b"sys:new_auth" => Vec<AuthorityId>;
+	StorageDigest: b"sys:digest" => Digest;
 }
 
 pub fn balance_of_key(who: AccountId) -> Vec<u8> {
@@ -67,6 +68,7 @@ pub fn initialize_block(header: &Header) {
 	// populate environment.
 	<Number>::put(&header.number);
 	<ParentHash>::put(&header.parent_hash);
+	<StorageDigest>::put(header.digest());
 	storage::unhashed::put(well_known_keys::EXTRINSIC_INDEX, &0u32);
 }
 
@@ -99,18 +101,17 @@ pub fn polish_block(block: &mut Block) {
 	header.state_root = storage_root().into();
 
 	// check digest
-	let mut digest = Digest::default();
+	let digest = &mut header.digest;
 	if let Some(storage_changes_root) = storage_changes_root(header.parent_hash.into()) {
 		digest.push(generic::DigestItem::ChangesTrieRoot(storage_changes_root.into()));
 	}
 	if let Some(new_authorities) = <NewAuthorities>::take() {
 		digest.push(generic::DigestItem::AuthoritiesChange(new_authorities));
 	}
-	header.digest = digest;
 }
 
-pub fn execute_block(block: Block) {
-	let ref header = block.header;
+pub fn execute_block(mut block: Block) {
+	let header = &mut block.header;
 
 	// check transaction trie root represents the transactions.
 	let txs = block.extrinsics.iter().map(Encode::encode).collect::<Vec<_>>();
@@ -132,14 +133,13 @@ pub fn execute_block(block: Block) {
 	assert!(storage_root == header.state_root, "Storage root must match that calculated.");
 
 	// check digest
-	let mut digest = Digest::default();
+	let digest = &mut header.digest;
 	if let Some(storage_changes_root) = storage_changes_root(header.parent_hash.into()) {
 		digest.push(generic::DigestItem::ChangesTrieRoot(storage_changes_root.into()));
 	}
 	if let Some(new_authorities) = <NewAuthorities>::take() {
 		digest.push(generic::DigestItem::AuthoritiesChange(new_authorities));
 	}
-	assert!(digest == header.digest, "Header digest items must match that calculated.");
 }
 
 /// The block executor.
@@ -209,13 +209,16 @@ pub fn finalize_block() -> Header {
 	let txs: Vec<_> = (0..extrinsic_index).map(ExtrinsicData::take).collect();
 	let txs = txs.iter().map(Vec::as_slice).collect::<Vec<_>>();
 	let extrinsics_root = enumerated_trie_root::<Blake2Hasher>(&txs).into();
-
+	// let mut digest = Digest::default();
 	let number = <Number>::take().expect("Number is set by `initialize_block`");
 	let parent_hash = <ParentHash>::take();
+	let mut digest = <StorageDigest>::take().expect("StorageDigest is set by `initialize_block`");
+
+	// This MUST come after all changes to storage are done.  Otherwise we will fail the
+	// “Storage root does not match that calculated” assertion.
 	let storage_root = BlakeTwo256::storage_root();
 	let storage_changes_root = BlakeTwo256::storage_changes_root(parent_hash);
 
-	let mut digest = Digest::default();
 	if let Some(storage_changes_root) = storage_changes_root {
 		digest.push(generic::DigestItem::ChangesTrieRoot(storage_changes_root));
 	}