Initial Substrate Header Chain Implementation (#296)

* Add pallet template from Substrate Dev Hub

* Clean up un-needed stuff from template

* Sketch out dispatchable interface

* Introduce notion of finality chain

* Add dependencies which were removed during a rebase

* Sketch out idea for finality header-chain pallet

* Sketch out ChainVerifier trait

* Add storage parameter to verifier

* Write out some things I think I need for finality verification

* Add some pseudocode for marking finalized headers

* Remove parity_scale_codec duplicate

* Move verification logic into pallet

I've been struggling with getting the generic types between the storage and verifier
traits to play nice with each other. As a way to continue making progress I'm moving
everything to the pallet. This way I hope to make progress towards a functional
pallet.

* Start doing verification around authority set changes

* Remove commented BridgeStorage and ChainVerifier traits

* Create Substrate bridge primitives crate

* Add logic for updating scheduled authority sets

* Introduce notion of imported headers

* Implement basic header ancestry checker

* Add mock runtime for tests

* Add testing boilerplate

* Add some storage read/write sanity tests

* Add some basic header import tests

* Add tests for ancestry proofs

* Create helper for changing authority sets

* Fix authority set test

Fixes a problem with how the scheduled change was counted as well as
a SCALE encoding issue

* Correctly check for scheduled change digests

There's no guarantee that the consensus digest item will be the last
one in a header, which is how it was previously being checked.

Thanks to Andre for pointing me to the Grandpa code that does this.

* Mark imported headers as finalized when appropriate

When a header that finalizes a chain of headers is succesfully imported
we also want to mark its ancestors as finalized.

* Add helper for writing test headers

* Add test helper for scheduling authority set changes

* Bump Substrate pallet and primitives to rc6

* Remove Millau verifier implementation

* Add some doc comments

* Remove some needless returns

* Make Clippy happy

* Split block import from finalization

* Make tests compile again

* Add test for finalizing header after importing children

* Create a test stub for importing future justifications

* Start adding genesis config

* Reject justifications from future

We should only be accepting justifications for the header
which enacted the current authority set. Any ancestors of
that header which require a justification can be imported
but they must not be finalized.

* Add explanation to some `expect()` calls

* Start adding GenesisConfig

* Plug genesis config into runtime

* Remove tests module

* Check for overflow when updating authority sets

* Make verifier take ownership of headers during import

* Only store best finalized header hash

Removed the need to store the whole header, since we store
it was part of the ImportedHeaders structure anyways

* Add some helpers to ImportedHeader

* Update ancestry checker to work with ImportedHeaders

* Update ancestry tests to use ImportedHeaders

* Update import tests to use ImportedHeaders

* Clean up some of the test helpers

* Remove stray dbg!

* Add doc comments throughout

* Remove runtime related code

* Fix Clippy warnings

* Remove trait bound on ImportedHeader struct

* Simplify checks in GenesisConfig

* Rename `get_header_by_hash()`

* Alias `parity_scale_codec` to `codec`

* Reword Verifier documentation

* Missed codec rename in tests

* Split ImportError into FinalizationError

* Remove ChainVerifier trait

This trait was a remenant of the original design, and it is not required
at the moment. Something like it should be added back in the future to
ensure that other chains which conform to this interface can be used
by higher-level bridge applications.

* Fix the verifier tests so they compile

* Implement Deref for ImportedHeader

* Get rid of `new` methods for some Substrate primitives

* Ensure that a child header's number follows its parent's

* Prevent ancestry checker from aimlessly traversing to genesis

If an ancestor which was newer than the child header we were checking we
would walk all the way to genesis before realizing that we weren't related.
This commit fixes that.

* Remove redundant clones

* Ensure that old headers are not finalized

Prevents a panic where if the header being imported and `best_finalized`
were the same header the ancestry checker would return an empty list. We
had made an assumption that the list would always be populated, and if this
didn't hold we would end up panicking.

* Disallow imports at same height as `best_finalized`

* Fix Clippy warnings

* Make NextScheduledChange optional

* Rework how scheduled authority set changes are enacted

We now require a justification for headers which _enact_ changes
instead of those which _schedule_ changes. A few changes had to
be made to accomodate this, such as changing when we check for
scheduled change logs in incoming headers.

* Update documentation for Substrate Primitives

* Clarify why we skip header in requires_justification check

* Add description to assert! call

* Fix formatting within macros

* Remove unused dependencies from runtime

* Remove expect call in GenesisConfig

* Turn FinalityProof into a struct

* Add some inline TODOs for follow up PRs

* Remove test which enacted multiple changes

This should be added back at some later point in time, but right now
the code doesn't allow for this behaviour.

* Use `contains_key` when checking for header

This is better than using `get().is_some()` since we skip
decoding the storage value

* Use initial hash when updating best_finalized

* Add better checks around enacting scheduled changes

* Rename finality related functions

* Appease Clippy

bridges/modules/substrate/Cargo.toml

@@ -8,17 +8,32 @@ license = "GPL-3.0-or-later WITH Classpath-exception-2.0"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-codec = { package = "parity-scale-codec", version = "1.3.4", default-features = false }
+bp-header-chain = { path = "../../primitives/header-chain", default-features = false }
+bp-substrate = { path = "../../primitives/substrate", default-features = false }
 finality-grandpa = { version = "0.12.3", default-features = false }
 hash-db = { version = "0.15.2", default-features = false }
+serde = { version = "1.0", optional = true }
+
+[dependencies.codec]
+package = "parity-scale-codec"
+version = "1.3.1"
+default-features = false
+features = ["derive"]
 
 # Substrate Based Dependencies
+
 [dependencies.frame-support]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
 default-features = false
 git = "https://github.com/paritytech/substrate/"
 
+[dependencies.frame-system]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+default-features = false
+git = "https://github.com/paritytech/substrate/"
+
 [dependencies.sp-finality-grandpa]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
@@ -43,6 +58,12 @@ tag = 'v2.0.0-rc6'
 default-features = false
 git = "https://github.com/paritytech/substrate/"
 
+# Dev Dependencies
+[dev-dependencies.sp-io]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+git = "https://github.com/paritytech/substrate/"
+
 [dev-dependencies.sp-core]
 version = "2.0.0-rc6"
 tag = 'v2.0.0-rc6'
@@ -67,8 +88,11 @@ std = [
 	"codec/std",
 	"finality-grandpa/std",
 	"frame-support/std",
+	"frame-system/std",
+	"serde",
 	"sp-finality-grandpa/std",
 	"sp-runtime/std",
 	"sp-std/std",
 	"sp-trie/std",
+	"sp-io/std",
 ]

bridges/modules/substrate/src/justification.rs

@@ -26,6 +26,7 @@ use frame_support::RuntimeDebug;
 use sp_finality_grandpa::{AuthorityId, AuthoritySignature, SetId};
 use sp_runtime::traits::Header as HeaderT;
 use sp_std::collections::{btree_map::BTreeMap, btree_set::BTreeSet};
+use sp_std::prelude::Vec;
 
 /// Justification verification error.
 #[derive(RuntimeDebug, PartialEq)]

bridges/modules/substrate/src/lib.rs

@@ -14,5 +14,263 @@
 // You should have received a copy of the GNU General Public License
 // along with Parity Bridges Common.  If not, see <http://www.gnu.org/licenses/>.
 
+//! Substrate Bridge Pallet
+//!
+//! This pallet is an on-chain light client for chains which have a notion of finality.
+//!
+//! It has a simple interface for achieving this. First it can import headers to the runtime
+//! storage. During this it will check the validity of the headers and ensure they don't conflict
+//! with any existing headers (e.g they're on a different finalized chain). Secondly it can finalize
+//! an already imported header (and its ancestors) given a valid Grandpa justification.
+//!
+//! With these two functions the pallet is able to form a "source of truth" for what headers have
+//! been finalized on a given Substrate chain. This can be a useful source of info for other
+//! higher-level applications.
+
+#![cfg_attr(not(feature = "std"), no_std)]
+// Runtime-generated enums
+#![allow(clippy::large_enum_variant)]
+
+use bp_substrate::{AuthoritySet, ImportedHeader, ScheduledChange};
+use frame_support::{decl_error, decl_module, decl_storage, dispatch};
+use frame_system::ensure_signed;
+use sp_runtime::traits::Header as HeaderT;
+use sp_std::{marker::PhantomData, prelude::*};
+
 mod justification;
 mod storage_proof;
+mod verifier;
+
+#[cfg(test)]
+mod mock;
+
+type Hash<T> = <T as HeaderT>::Hash;
+type Number<T> = <T as HeaderT>::Number;
+
+pub trait Trait: frame_system::Trait {}
+
+decl_storage! {
+	trait Store for Module<T: Trait> as SubstrateBridge {
+		/// Hash of the best finalized header.
+		BestFinalized: T::Hash;
+		/// Headers which have been imported into the pallet.
+		ImportedHeaders: map hasher(identity) T::Hash => Option<ImportedHeader<T::Header>>;
+		/// The current Grandpa Authority set.
+		CurrentAuthoritySet: AuthoritySet;
+		/// The next scheduled authority set change.
+		///
+		// Grandpa doesn't require there to always be a pending change. In fact, most of the time
+		// there will be no pending change available.
+		NextScheduledChange: Option<ScheduledChange<Number<T::Header>>>;
+	}
+	add_extra_genesis {
+		config(initial_header): Option<T::Header>;
+		config(initial_authority_list): sp_finality_grandpa::AuthorityList;
+		config(initial_set_id): sp_finality_grandpa::SetId;
+		config(first_scheduled_change): Option<ScheduledChange<Number<T::Header>>>;
+		build(|config| {
+			assert!(
+				!config.initial_authority_list.is_empty(),
+				"An initial authority list is needed."
+			);
+
+			let initial_header = config
+				.initial_header
+				.clone()
+				.expect("An initial header is needed");
+
+			<BestFinalized<T>>::put(initial_header.hash());
+			<ImportedHeaders<T>>::insert(
+				initial_header.hash(),
+				ImportedHeader {
+					header: initial_header,
+					requires_justification: false,
+					is_finalized: true,
+				},
+			);
+
+			let authority_set =
+				AuthoritySet::new(config.initial_authority_list.clone(), config.initial_set_id);
+			CurrentAuthoritySet::put(authority_set);
+
+			if let Some(ref change) = config.first_scheduled_change {
+				<NextScheduledChange<T>>::put(change);
+			};
+		})
+	}
+}
+
+decl_error! {
+	pub enum Error for Module<T: Trait> {
+		/// This header has failed basic verification.
+		InvalidHeader,
+		/// This header has not been finalized.
+		UnfinalizedHeader,
+	}
+}
+
+decl_module! {
+	pub struct Module<T: Trait> for enum Call where origin: T::Origin {
+		type Error = Error<T>;
+
+		/// Import a signed Substrate header into the runtime.
+		///
+		/// This will perform some basic checks to make sure it is fine to
+		/// import into the runtime. However, it does not perform any checks
+		/// related to finality.
+		// TODO: Update weights [#78]
+		#[weight = 0]
+		pub fn import_signed_header(
+			origin,
+			header: T::Header,
+		) -> dispatch::DispatchResult {
+			let _ = ensure_signed(origin)?;
+			frame_support::debug::trace!(target: "sub-bridge", "Got header {:?}", header);
+
+			let mut verifier = verifier::Verifier {
+				storage: PalletStorage::<T>::new(),
+			};
+
+			let _ = verifier
+				.import_header(header)
+				.map_err(|_| <Error<T>>::InvalidHeader)?;
+
+			Ok(())
+		}
+
+		/// Import a finalty proof for a particular header.
+		///
+		/// This will take care of finalizing any already imported headers
+		/// which get finalized when importing this particular proof, as well
+		/// as updating the current and next validator sets.
+		// TODO: Update weights [#78]
+		#[weight = 0]
+		pub fn finalize_header(
+			origin,
+			hash: Hash<T::Header>,
+			finality_proof: Vec<u8>,
+		) -> dispatch::DispatchResult {
+			let _ = ensure_signed(origin)?;
+			frame_support::debug::trace!(target: "sub-bridge", "Got header hash {:?}", hash);
+
+			let mut verifier = verifier::Verifier {
+				storage: PalletStorage::<T>::new(),
+			};
+
+			let _ = verifier
+				.import_finality_proof(hash, finality_proof.into())
+				.map_err(|_| <Error<T>>::UnfinalizedHeader)?;
+
+			Ok(())
+		}
+	}
+}
+
+/// Expected interface for interacting with bridge pallet storage.
+// TODO: This should be split into its own less-Substrate-dependent crate
+pub trait BridgeStorage {
+	/// The header type being used by the pallet.
+	type Header: HeaderT;
+
+	/// Write a header to storage.
+	fn write_header(&mut self, header: &ImportedHeader<Self::Header>);
+
+	/// Get the best finalized header the pallet knows of.
+	fn best_finalized_header(&self) -> ImportedHeader<Self::Header>;
+
+	/// Update the best finalized header the pallet knows of.
+	fn update_best_finalized(&self, hash: <Self::Header as HeaderT>::Hash);
+
+	/// Check if a particular header is known to the pallet.
+	fn header_exists(&self, hash: <Self::Header as HeaderT>::Hash) -> bool;
+
+	/// Get a specific header by its hash.
+	///
+	/// Returns None if it is not known to the pallet.
+	fn header_by_hash(&self, hash: <Self::Header as HeaderT>::Hash) -> Option<ImportedHeader<Self::Header>>;
+
+	/// Get the current Grandpa authority set.
+	fn current_authority_set(&self) -> AuthoritySet;
+
+	/// Update the current Grandpa authority set.
+	///
+	/// Should only be updated when a scheduled change has been triggered.
+	fn update_current_authority_set(&self, new_set: AuthoritySet);
+
+	/// Replace the current authority set with the next scheduled set.
+	///
+	/// Returns an error if there is no scheduled authority set to enact.
+	fn enact_authority_set(&mut self) -> Result<(), ()>;
+
+	/// Get the next scheduled Grandpa authority set change.
+	fn scheduled_set_change(&self) -> Option<ScheduledChange<<Self::Header as HeaderT>::Number>>;
+
+	/// Schedule a Grandpa authority set change in the future.
+	fn schedule_next_set_change(&self, next_change: ScheduledChange<<Self::Header as HeaderT>::Number>);
+}
+
+/// Used to interact with the pallet storage in a more abstract way.
+#[derive(Default, Clone)]
+pub struct PalletStorage<T>(PhantomData<T>);
+
+impl<T> PalletStorage<T> {
+	fn new() -> Self {
+		Self(PhantomData::<T>::default())
+	}
+}
+
+impl<T: Trait> BridgeStorage for PalletStorage<T> {
+	type Header = T::Header;
+
+	fn write_header(&mut self, header: &ImportedHeader<T::Header>) {
+		let hash = header.header.hash();
+		<ImportedHeaders<T>>::insert(hash, header);
+	}
+
+	fn best_finalized_header(&self) -> ImportedHeader<T::Header> {
+		let hash = <BestFinalized<T>>::get();
+		self.header_by_hash(hash)
+			.expect("A finalized header was added at genesis, therefore this must always exist")
+	}
+
+	fn update_best_finalized(&self, hash: Hash<T::Header>) {
+		<BestFinalized<T>>::put(hash)
+	}
+
+	fn header_exists(&self, hash: Hash<T::Header>) -> bool {
+		<ImportedHeaders<T>>::contains_key(hash)
+	}
+
+	fn header_by_hash(&self, hash: Hash<T::Header>) -> Option<ImportedHeader<T::Header>> {
+		<ImportedHeaders<T>>::get(hash)
+	}
+
+	fn current_authority_set(&self) -> AuthoritySet {
+		CurrentAuthoritySet::get()
+	}
+
+	fn update_current_authority_set(&self, new_set: AuthoritySet) {
+		CurrentAuthoritySet::put(new_set)
+	}
+
+	fn enact_authority_set(&mut self) -> Result<(), ()> {
+		if <NextScheduledChange<T>>::exists() {
+			let new_set = <NextScheduledChange<T>>::take()
+				.expect("Ensured that entry existed in storage")
+				.authority_set;
+			self.update_current_authority_set(new_set);
+
+			Ok(())
+		} else {
+			Err(())
+		}
+	}
+
+	fn scheduled_set_change(&self) -> Option<ScheduledChange<Number<T::Header>>> {
+		<NextScheduledChange<T>>::get()
+	}
+
+	fn schedule_next_set_change(&self, next_change: ScheduledChange<Number<T::Header>>) {
+		<NextScheduledChange<T>>::put(next_change)
+	}
+}

bridges/modules/substrate/src/mock.rs

@@ -0,0 +1,75 @@
+// Copyright 2020 Parity Technologies (UK) Ltd.
+// This file is part of Parity Bridges Common.
+
+// Parity Bridges Common is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity Bridges Common is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity Bridges Common.  If not, see <http://www.gnu.org/licenses/>.
+
+#![cfg(test)]
+
+use crate::Trait;
+use frame_support::{impl_outer_origin, parameter_types, weights::Weight};
+use sp_runtime::{
+	testing::{Header, H256},
+	traits::{BlakeTwo256, IdentityLookup},
+	Perbill,
+};
+
+pub type AccountId = u64;
+
+#[derive(Clone, Eq, PartialEq, Debug)]
+pub struct TestRuntime;
+
+impl_outer_origin! {
+	pub enum Origin for TestRuntime where system = frame_system {}
+}
+
+parameter_types! {
+	pub const BlockHashCount: u64 = 250;
+	pub const MaximumBlockWeight: Weight = 1024;
+	pub const MaximumBlockLength: u32 = 2 * 1024;
+	pub const AvailableBlockRatio: Perbill = Perbill::one();
+}
+
+impl frame_system::Trait for TestRuntime {
+	type Origin = Origin;
+	type Index = u64;
+	type Call = ();
+	type BlockNumber = u64;
+	type Hash = H256;
+	type Hashing = BlakeTwo256;
+	type AccountId = AccountId;
+	type Lookup = IdentityLookup<Self::AccountId>;
+	type Header = Header;
+	type Event = ();
+	type BlockHashCount = BlockHashCount;
+	type MaximumBlockWeight = MaximumBlockWeight;
+	type DbWeight = ();
+	type BlockExecutionWeight = ();
+	type ExtrinsicBaseWeight = ();
+	type MaximumExtrinsicWeight = ();
+	type AvailableBlockRatio = AvailableBlockRatio;
+	type MaximumBlockLength = MaximumBlockLength;
+	type Version = ();
+	type ModuleToIndex = ();
+	type AccountData = ();
+	type OnNewAccount = ();
+	type OnKilledAccount = ();
+	type BaseCallFilter = ();
+	type SystemWeightInfo = ();
+}
+
+impl Trait for TestRuntime {}
+
+pub fn run_test<T>(test: impl FnOnce() -> T) -> T {
+	sp_io::TestExternalities::new(Default::default()).execute_with(test)
+}

bridges/modules/substrate/src/storage_proof.rs

@@ -21,6 +21,7 @@
 
 use hash_db::{HashDB, Hasher, EMPTY_PREFIX};
 use sp_runtime::RuntimeDebug;
+use sp_std::vec::Vec;
 use sp_trie::{read_trie_value, Layout, MemoryDB, StorageProof};
 
 /// This struct is used to read storage values from a subset of a Merklized database. The "proof"

bridges/modules/substrate/src/verifier.rs

@@ -0,0 +1,680 @@
+// Copyright 2020 Parity Technologies (UK) Ltd.
+// This file is part of Parity Bridges Common.
+
+// Parity Bridges Common is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity Bridges Common is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity Bridges Common.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The verifier's role is to check the validity of headers being imported, and also determine if
+//! they can be finalized.
+//!
+//! When importing headers it performs checks to ensure that no invariants are broken (like
+//! importing the same header twice). When it imports finality proofs it will ensure that the proof
+//! has been signed off by the correct Grandpa authorities, and also enact any authority set changes
+//! if required.
+
+use crate::BridgeStorage;
+use bp_substrate::{check_finality_proof, AuthoritySet, ImportedHeader, ScheduledChange};
+use sp_finality_grandpa::{ConsensusLog, GRANDPA_ENGINE_ID};
+use sp_runtime::generic::OpaqueDigestItemId;
+use sp_runtime::traits::{CheckedAdd, Header as HeaderT, One};
+use sp_std::{prelude::Vec, vec};
+
+/// The finality proof used by the pallet.
+///
+/// For a Substrate based chain using Grandpa this will
+/// be an encoded Grandpa Justification.
+pub struct FinalityProof(Vec<u8>);
+
+impl From<&[u8]> for FinalityProof {
+	fn from(proof: &[u8]) -> Self {
+		Self(proof.to_vec())
+	}
+}
+
+impl From<Vec<u8>> for FinalityProof {
+	fn from(proof: Vec<u8>) -> Self {
+		Self(proof)
+	}
+}
+
+/// Errors which can happen while importing a header.
+#[derive(Debug, PartialEq)]
+pub enum ImportError {
+	/// This header is older than our latest finalized block, thus not useful.
+	OldHeader,
+	/// This header has already been imported by the pallet.
+	HeaderAlreadyExists,
+	/// We're missing a parent for this header.
+	MissingParent,
+	/// The number of the header does not follow its parent's number.
+	InvalidChildNumber,
+	/// The height of the next authority set change overflowed.
+	ScheduledHeightOverflow,
+}
+
+/// Errors which can happen while verifying a headers finality.
+#[derive(Debug, PartialEq)]
+pub enum FinalizationError {
+	/// This header has never been imported by the pallet.
+	UnknownHeader,
+	/// We were unable to prove finality for this header.
+	UnfinalizedHeader,
+	/// Trying to prematurely import a justification
+	PrematureJustification,
+	/// We failed to verify this header's ancestry.
+	AncestryCheckFailed,
+	/// This header is older than our latest finalized block, thus not useful.
+	OldHeader,
+}
+
+/// Used to verify imported headers and their finality status.
+#[derive(Debug)]
+pub struct Verifier<S> {
+	pub storage: S,
+}
+
+impl<S, H> Verifier<S>
+where
+	S: BridgeStorage<Header = H>,
+	H: HeaderT,
+{
+	/// Import a header to the pallet.
+	///
+	/// Will perform some basic checks to make sure that this header doesn't break any assumptions
+	/// such as being on a different finalized fork.
+	pub fn import_header(&mut self, header: H) -> Result<(), ImportError> {
+		let best_finalized = self.storage.best_finalized_header();
+
+		if header.number() <= best_finalized.number() {
+			return Err(ImportError::OldHeader);
+		}
+
+		if self.storage.header_exists(header.hash()) {
+			return Err(ImportError::HeaderAlreadyExists);
+		}
+
+		let parent_header = self
+			.storage
+			.header_by_hash(*header.parent_hash())
+			.ok_or(ImportError::MissingParent)?;
+
+		let parent_number = *parent_header.number();
+		if parent_number + One::one() != *header.number() {
+			return Err(ImportError::InvalidChildNumber);
+		}
+
+		// This header requires a justification since it enacts an authority set change. We don't
+		// need to act on it right away (we'll update the set once this header gets finalized), but
+		// we need to make a note of it.
+		//
+		// TODO: This assumes that we can only have one authority set change pending at a time.
+		// This is not strictly true as Grandpa may schedule multiple changes on a given chain
+		// if the "next next" change is scheduled after the "delay" period of the "next" change
+		let requires_justification = if let Some(change) = self.storage.scheduled_set_change() {
+			change.height == *header.number()
+		} else {
+			// Since we don't currently have a pending authority set change let's check if the header
+			// contains a log indicating when the next change should be.
+			if let Some(change) = find_scheduled_change(&header) {
+				let next_set = AuthoritySet {
+					authorities: change.next_authorities,
+					set_id: self.storage.current_authority_set().set_id + 1,
+				};
+
+				let height = (*header.number())
+					.checked_add(&change.delay)
+					.ok_or(ImportError::ScheduledHeightOverflow)?;
+				let scheduled_change = ScheduledChange {
+					authority_set: next_set,
+					height,
+				};
+
+				self.storage.schedule_next_set_change(scheduled_change);
+
+				// If the delay is 0 this header will enact the change it signaled
+				height == *header.number()
+			} else {
+				false
+			}
+		};
+
+		let is_finalized = false;
+		self.storage.write_header(&ImportedHeader {
+			header,
+			requires_justification,
+			is_finalized,
+		});
+
+		Ok(())
+	}
+
+	/// Verify that a previously imported header can be finalized with the given Grandpa finality
+	/// proof. If the header enacts an authority set change the change will be applied once the
+	/// header has been finalized.
+	pub fn import_finality_proof(&mut self, hash: H::Hash, proof: FinalityProof) -> Result<(), FinalizationError> {
+		// Make sure that we've previously imported this header
+		let header = self
+			.storage
+			.header_by_hash(hash)
+			.ok_or(FinalizationError::UnknownHeader)?;
+
+		// We don't want to finalize an ancestor of an already finalized
+		// header, this would be inconsistent
+		let last_finalized = self.storage.best_finalized_header();
+		if header.number() <= last_finalized.number() {
+			return Err(FinalizationError::OldHeader);
+		}
+
+		let current_authority_set = self.storage.current_authority_set();
+		let is_finalized = check_finality_proof(&header, &current_authority_set, &proof.0);
+		if !is_finalized {
+			return Err(FinalizationError::UnfinalizedHeader);
+		}
+
+		frame_support::debug::trace!(target: "sub-bridge", "Checking ancestry for headers between {:?} and {:?}", last_finalized, header);
+		let mut finalized_headers =
+			if let Some(ancestors) = headers_between(&self.storage, last_finalized, header.clone()) {
+				// Since we only try and finalize headers with a height strictly greater
+				// than `best_finalized` if `headers_between` returns Some we must have
+				// at least one element. If we don't something's gone wrong, so best
+				// to die before we write to storage.
+				assert_eq!(
+					ancestors.is_empty(),
+					false,
+					"Empty ancestry list returned from `headers_between()`",
+				);
+
+				// Check if any of our ancestors `requires_justification` a.k.a schedule authority
+				// set changes. If they're still waiting to be finalized we must reject this
+				// justification. We don't include our current header in this check.
+				//
+				// We do this because it is important to to import justifications _in order_,
+				// otherwise we risk finalizing headers on competing chains.
+				let requires_justification = ancestors.iter().skip(1).find(|h| h.requires_justification);
+				if requires_justification.is_some() {
+					return Err(FinalizationError::PrematureJustification);
+				}
+
+				ancestors
+			} else {
+				return Err(FinalizationError::AncestryCheckFailed);
+			};
+
+		// If the current header was marked as `requires_justification` it means that it enacts a
+		// new authority set change. When we finalize the header we need to update the current
+		// authority set.
+		if header.requires_justification {
+			// If we are unable to enact an authority set it means our storage entry for scheduled
+			// changes is missing. Best to crash since this is likely a bug.
+			let _ = self.storage.enact_authority_set().expect(
+				"Headers must only be marked as `requires_justification` if there's a scheduled change in storage.",
+			);
+		}
+
+		for header in finalized_headers.iter_mut() {
+			header.is_finalized = true;
+			header.requires_justification = false;
+			self.storage.write_header(header);
+		}
+
+		self.storage.update_best_finalized(hash);
+
+		Ok(())
+	}
+}
+
+/// Returns the lineage of headers between [child, ancestor)
+fn headers_between<S, H>(
+	storage: &S,
+	ancestor: ImportedHeader<H>,
+	child: ImportedHeader<H>,
+) -> Option<Vec<ImportedHeader<H>>>
+where
+	S: BridgeStorage<Header = H>,
+	H: HeaderT,
+{
+	let mut ancestors = vec![];
+	let mut current_header = child;
+
+	while ancestor.hash() != current_header.hash() {
+		// We've gotten to the same height and we're not related
+		if ancestor.number() >= current_header.number() {
+			return None;
+		}
+
+		let parent = storage.header_by_hash(*current_header.parent_hash());
+		ancestors.push(current_header);
+		current_header = match parent {
+			Some(h) => h,
+			None => return None,
+		}
+	}
+
+	Some(ancestors)
+}
+
+fn find_scheduled_change<H: HeaderT>(header: &H) -> Option<sp_finality_grandpa::ScheduledChange<H::Number>> {
+	let id = OpaqueDigestItemId::Consensus(&GRANDPA_ENGINE_ID);
+
+	let filter_log = |log: ConsensusLog<H::Number>| match log {
+		ConsensusLog::ScheduledChange(change) => Some(change),
+		_ => None,
+	};
+
+	// find the first consensus digest with the right ID which converts to
+	// the right kind of consensus log.
+	header.digest().convert_first(|l| l.try_to(id).and_then(filter_log))
+}
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+	use crate::mock::*;
+	use crate::{BestFinalized, ImportedHeaders, PalletStorage};
+	use frame_support::{assert_err, assert_ok};
+	use frame_support::{StorageMap, StorageValue};
+	use sp_finality_grandpa::{AuthorityId, AuthorityList};
+	use sp_runtime::testing::UintAuthorityId;
+
+	type TestHeader = <TestRuntime as frame_system::Trait>::Header;
+	type TestNumber = <TestHeader as HeaderT>::Number;
+
+	fn unfinalized_header(num: u64) -> ImportedHeader<TestHeader> {
+		ImportedHeader {
+			header: TestHeader::new_from_number(num),
+			requires_justification: false,
+			is_finalized: false,
+		}
+	}
+
+	fn get_authorities(authorities: Vec<(u64, u64)>) -> AuthorityList {
+		authorities
+			.iter()
+			.map(|(id, weight)| (UintAuthorityId(*id).to_public_key::<AuthorityId>(), *weight))
+			.collect()
+	}
+
+	fn schedule_next_change(
+		authorities: Vec<(u64, u64)>,
+		set_id: u64,
+		height: TestNumber,
+	) -> ScheduledChange<TestNumber> {
+		let authorities = get_authorities(authorities);
+		let authority_set = AuthoritySet::new(authorities, set_id);
+		ScheduledChange { authority_set, height }
+	}
+
+	// Useful for quickly writing a chain of headers to storage
+	fn write_headers<S: BridgeStorage<Header = TestHeader>>(
+		storage: &mut S,
+		headers: Vec<(u64, bool, bool)>,
+	) -> Vec<ImportedHeader<TestHeader>> {
+		let mut imported_headers = vec![];
+		let genesis = ImportedHeader {
+			header: TestHeader::new_from_number(0),
+			requires_justification: false,
+			is_finalized: true,
+		};
+
+		<BestFinalized<TestRuntime>>::put(genesis.hash());
+		storage.write_header(&genesis);
+		imported_headers.push(genesis);
+
+		for (num, requires_justification, is_finalized) in headers {
+			let mut h = TestHeader::new_from_number(num);
+			h.parent_hash = imported_headers.last().unwrap().hash();
+
+			let header = ImportedHeader {
+				header: h,
+				requires_justification,
+				is_finalized,
+			};
+
+			storage.write_header(&header);
+			imported_headers.push(header);
+		}
+
+		imported_headers
+	}
+
+	#[test]
+	fn fails_to_import_old_header() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let parent = unfinalized_header(5);
+			storage.write_header(&parent);
+			storage.update_best_finalized(parent.hash());
+
+			let header = TestHeader::new_from_number(1);
+			let mut verifier = Verifier { storage };
+			assert_err!(verifier.import_header(header), ImportError::OldHeader);
+		})
+	}
+
+	#[test]
+	fn fails_to_import_header_without_parent() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let parent = unfinalized_header(1);
+			storage.write_header(&parent);
+			storage.update_best_finalized(parent.hash());
+
+			// By default the parent is `0x00`
+			let header = TestHeader::new_from_number(2);
+
+			let mut verifier = Verifier { storage };
+			assert_err!(verifier.import_header(header), ImportError::MissingParent);
+		})
+	}
+
+	#[test]
+	fn fails_to_import_header_twice() {
+		run_test(|| {
+			let storage = PalletStorage::<TestRuntime>::new();
+			let header = TestHeader::new_from_number(1);
+			<BestFinalized<TestRuntime>>::put(header.hash());
+
+			let imported_header = ImportedHeader {
+				header: header.clone(),
+				requires_justification: false,
+				is_finalized: false,
+			};
+			<ImportedHeaders<TestRuntime>>::insert(header.hash(), &imported_header);
+
+			let mut verifier = Verifier { storage };
+			assert_err!(verifier.import_header(header), ImportError::OldHeader);
+		})
+	}
+
+	#[test]
+	fn succesfully_imports_valid_but_unfinalized_header() {
+		run_test(|| {
+			let storage = PalletStorage::<TestRuntime>::new();
+			let parent = TestHeader::new_from_number(1);
+			let parent_hash = parent.hash();
+			<BestFinalized<TestRuntime>>::put(parent.hash());
+
+			let imported_header = ImportedHeader {
+				header: parent,
+				requires_justification: false,
+				is_finalized: true,
+			};
+			<ImportedHeaders<TestRuntime>>::insert(parent_hash, &imported_header);
+
+			let mut header = TestHeader::new_from_number(2);
+			header.parent_hash = parent_hash;
+			let mut verifier = Verifier {
+				storage: storage.clone(),
+			};
+			assert_ok!(verifier.import_header(header.clone()));
+
+			let stored_header = storage.header_by_hash(header.hash());
+			assert!(stored_header.is_some());
+			assert_eq!(stored_header.unwrap().is_finalized, false);
+		})
+	}
+
+	#[test]
+	fn related_headers_are_ancestors() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+
+			let headers = vec![(1, false, false), (2, false, false), (3, false, false)];
+			let mut imported_headers = write_headers(&mut storage, headers);
+
+			for header in imported_headers.iter() {
+				assert!(storage.header_exists(header.hash()));
+			}
+
+			let ancestor = imported_headers.remove(0);
+			let child = imported_headers.pop().unwrap();
+			let ancestors = headers_between(&storage, ancestor, child);
+
+			assert!(ancestors.is_some());
+			assert_eq!(ancestors.unwrap().len(), 3);
+		})
+	}
+
+	#[test]
+	fn unrelated_headers_are_not_ancestors() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+
+			let headers = vec![(1, false, false), (2, false, false), (3, false, false)];
+			let mut imported_headers = write_headers(&mut storage, headers);
+			for header in imported_headers.iter() {
+				assert!(storage.header_exists(header.hash()));
+			}
+
+			// Need to give it a different parent_hash or else it'll be
+			// related to our test genesis header
+			let mut bad_ancestor = TestHeader::new_from_number(0);
+			bad_ancestor.parent_hash = [1u8; 32].into();
+			let bad_ancestor = ImportedHeader {
+				header: bad_ancestor,
+				requires_justification: false,
+				is_finalized: false,
+			};
+
+			let child = imported_headers.pop().unwrap();
+			let ancestors = headers_between(&storage, bad_ancestor, child);
+			assert!(ancestors.is_none());
+		})
+	}
+
+	#[test]
+	fn ancestor_newer_than_child_is_not_related() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+
+			let headers = vec![(1, false, false), (2, false, false), (3, false, false)];
+			let mut imported_headers = write_headers(&mut storage, headers);
+			for header in imported_headers.iter() {
+				assert!(storage.header_exists(header.hash()));
+			}
+
+			// What if we have an "ancestor" that's newer than child?
+			let new_ancestor = TestHeader::new_from_number(5);
+			let new_ancestor = ImportedHeader {
+				header: new_ancestor,
+				requires_justification: false,
+				is_finalized: false,
+			};
+
+			let child = imported_headers.pop().unwrap();
+			let ancestors = headers_between(&storage, new_ancestor, child);
+			assert!(ancestors.is_none());
+		})
+	}
+
+	#[test]
+	fn finalizes_header_which_doesnt_enact_or_schedule_a_new_authority_set() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let headers = vec![(1, false, false)];
+			let imported_headers = write_headers(&mut storage, headers);
+
+			// Nothing special about this header, yet Grandpa may have created a justification
+			// for it since it does that periodically
+			let mut header = TestHeader::new_from_number(2);
+			header.parent_hash = imported_headers[1].hash();
+
+			let mut verifier = Verifier {
+				storage: storage.clone(),
+			};
+
+			assert_ok!(verifier.import_header(header.clone()));
+			assert_ok!(verifier.import_finality_proof(header.hash(), vec![4, 2].into()));
+			assert_eq!(storage.best_finalized_header().header, header);
+		})
+	}
+
+	#[test]
+	fn correctly_verifies_and_finalizes_chain_of_headers() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let headers = vec![(1, false, false), (2, false, false)];
+			let imported_headers = write_headers(&mut storage, headers);
+
+			let mut header = TestHeader::new_from_number(3);
+			header.parent_hash = imported_headers[2].hash();
+
+			let mut verifier = Verifier {
+				storage: storage.clone(),
+			};
+			assert!(verifier.import_header(header.clone()).is_ok());
+			assert!(verifier.import_finality_proof(header.hash(), vec![4, 2].into()).is_ok());
+
+			// Make sure we marked the our headers as finalized
+			assert!(storage.header_by_hash(imported_headers[1].hash()).unwrap().is_finalized);
+			assert!(storage.header_by_hash(imported_headers[2].hash()).unwrap().is_finalized);
+			assert!(storage.header_by_hash(header.hash()).unwrap().is_finalized);
+
+			// Make sure the header at the highest height is the best finalized
+			assert_eq!(storage.best_finalized_header().header, header);
+		});
+	}
+
+	#[test]
+	fn updates_authority_set_upon_finalizing_header_which_enacts_change() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let headers = vec![(1, false, false)];
+			let imported_headers = write_headers(&mut storage, headers);
+
+			let set_id = 0;
+			let authorities = get_authorities(vec![(1, 1)]);
+			let initial_authority_set = AuthoritySet::new(authorities, set_id);
+			storage.update_current_authority_set(initial_authority_set);
+
+			// This header enacts an authority set change upon finalization
+			let mut header = TestHeader::new_from_number(2);
+			header.parent_hash = imported_headers[1].hash();
+
+			// Schedule a change at the height of our header
+			let set_id = 1;
+			let height = *header.number();
+			let authorities = vec![(2, 1)];
+			let change = schedule_next_change(authorities, set_id, height);
+			storage.schedule_next_set_change(change.clone());
+
+			let mut verifier = Verifier {
+				storage: storage.clone(),
+			};
+
+			assert_ok!(verifier.import_header(header.clone()));
+			assert_ok!(verifier.import_finality_proof(header.hash(), vec![4, 2].into()));
+			assert_eq!(storage.best_finalized_header().header, header);
+
+			// Make sure that we have updated the set now that we've finalized our header
+			assert_eq!(storage.current_authority_set(), change.authority_set);
+		})
+	}
+
+	#[test]
+	fn importing_finality_proof_for_already_finalized_header_doesnt_work() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let genesis = TestHeader::new_from_number(0);
+
+			let genesis = ImportedHeader {
+				header: genesis,
+				requires_justification: false,
+				is_finalized: true,
+			};
+
+			// Make sure that genesis is the best finalized header
+			<BestFinalized<TestRuntime>>::put(genesis.hash());
+			storage.write_header(&genesis);
+
+			let mut verifier = Verifier { storage };
+
+			// Now we want to try and import it again to see what happens
+			assert_eq!(
+				verifier
+					.import_finality_proof(genesis.hash(), vec![4, 2].into())
+					.unwrap_err(),
+				FinalizationError::OldHeader
+			);
+		});
+	}
+
+	// We're supposed to enact a set change at header N. This means that when we import it we must
+	// remember that it requires a justification. We can continue importing headers past N but must
+	// not finalize any childen. At a later point in time we should be able to import the
+	// justification for N.
+	//
+	// Since N enacts a new authority set, when we finalize it we should see this change reflected
+	// correctly.
+	//
+	// [G] <- [N-1] <- [N] <- [N+1] <- [N+2]
+	//                  |                |- Import justification for N here
+	//                  |- Enacts change, needs justification
+	#[test]
+	fn allows_importing_justification_at_block_past_scheduled_change() {
+		run_test(|| {
+			let mut storage = PalletStorage::<TestRuntime>::new();
+			let headers = vec![(1, false, false)];
+			let imported_headers = write_headers(&mut storage, headers);
+
+			// This is header N
+			let mut header = TestHeader::new_from_number(2);
+			header.parent_hash = imported_headers[1].hash();
+
+			// Schedule a change at height N
+			let set_id = 1;
+			let height = *header.number();
+			let authorities = vec![(1, 1)];
+			let change = schedule_next_change(authorities, set_id, height);
+			storage.schedule_next_set_change(change.clone());
+
+			// Import header N
+			let mut verifier = Verifier {
+				storage: storage.clone(),
+			};
+			assert!(verifier.import_header(header.clone()).is_ok());
+
+			// Header N should be marked as needing a justification
+			assert_eq!(
+				storage.header_by_hash(header.hash()).unwrap().requires_justification,
+				true
+			);
+
+			// Now we want to import some headers which are past N
+			let mut child = TestHeader::new_from_number(*header.number() + 1);
+			child.parent_hash = header.hash();
+			assert!(verifier.import_header(child.clone()).is_ok());
+
+			let mut grandchild = TestHeader::new_from_number(*child.number() + 1);
+			grandchild.parent_hash = child.hash();
+			assert!(verifier.import_header(grandchild).is_ok());
+
+			// Even though we're a few headers ahead we should still be able to import
+			// a justification for header N
+			assert!(verifier.import_finality_proof(header.hash(), vec![4, 2].into()).is_ok());
+
+			// Some checks to make sure that our header has been correctly finalized
+			let finalized_header = storage.header_by_hash(header.hash()).unwrap();
+			assert!(finalized_header.is_finalized);
+			assert_eq!(finalized_header.requires_justification, false);
+			assert_eq!(storage.best_finalized_header().header, header);
+
+			// Make sure we marked the parent of the header at N as finalized
+			assert!(storage.header_by_hash(imported_headers[1].hash()).unwrap().is_finalized);
+
+			// Since our header was supposed to enact a new authority set change when it got
+			// finalized let's make sure that the authority set actually changed
+			assert_eq!(storage.current_authority_set(), change.authority_set);
+		})
+	}
+}

bridges/primitives/substrate/Cargo.toml

@@ -0,0 +1,36 @@
+[package]
+name = "bp-substrate"
+version = "0.1.0"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+license = "GPL-3.0-or-later WITH Classpath-exception-2.0"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+serde = { version = "1.0", optional = true }
+
+[dependencies.sp-finality-grandpa]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+default-features = false
+git = "https://github.com/paritytech/substrate.git"
+
+[dependencies.sp-runtime]
+version = "2.0.0-rc6"
+tag = 'v2.0.0-rc6'
+default-features = false
+git = "https://github.com/paritytech/substrate/"
+
+[dependencies.parity-scale-codec]
+version = "1.3.1"
+default-features = false
+features = ["derive"]
+
+[features]
+default = ["std"]
+std = [
+	"serde/std",
+	"sp-runtime/std",
+	"sp-finality-grandpa/std",
+]

bridges/primitives/substrate/src/lib.rs

@@ -0,0 +1,82 @@
+// Copyright 2019-2020 Parity Technologies (UK) Ltd.
+// This file is part of Parity Bridges Common.
+
+// Parity Bridges Common is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity Bridges Common is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity Bridges Common.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Primitives for the Substrate light client (a.k.a bridge) pallet.
+
+#![warn(missing_docs)]
+#![cfg_attr(not(feature = "std"), no_std)]
+
+use core::default::Default;
+use parity_scale_codec::{Decode, Encode};
+#[cfg(feature = "std")]
+use serde::{Deserialize, Serialize};
+use sp_finality_grandpa::{AuthorityList, SetId};
+use sp_runtime::RuntimeDebug;
+
+/// A Grandpa Authority List and ID.
+#[derive(Default, Encode, Decode, RuntimeDebug, PartialEq, Clone)]
+#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
+pub struct AuthoritySet {
+	/// List of Grandpa authorities for the current round.
+	pub authorities: AuthorityList,
+	/// Monotonic identifier of the current Grandpa authority set.
+	pub set_id: SetId,
+}
+
+impl AuthoritySet {
+	/// Create a new Grandpa Authority Set.
+	pub fn new(authorities: AuthorityList, set_id: SetId) -> Self {
+		Self { authorities, set_id }
+	}
+}
+
+/// Keeps track of when the next Grandpa authority set change will occur.
+#[derive(Default, Encode, Decode, RuntimeDebug, PartialEq, Clone)]
+#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
+pub struct ScheduledChange<N> {
+	/// The authority set that will be used once this change is enacted.
+	pub authority_set: AuthoritySet,
+	/// The block height at which the authority set should be enacted.
+	///
+	/// Note: It will only be enacted once a header at this height is finalized.
+	pub height: N,
+}
+
+/// A more useful representation of a header for storage purposes.
+#[derive(Default, Encode, Decode, Clone, RuntimeDebug, PartialEq)]
+pub struct ImportedHeader<H> {
+	/// A plain Substrate header.
+	pub header: H,
+	/// Does this header enact a new authority set change. If it does
+	/// then it will require a justification.
+	pub requires_justification: bool,
+	/// Has this header been finalized, either explicitly via a justification,
+	/// or implicitly via one of its children getting finalized.
+	pub is_finalized: bool,
+}
+
+impl<H> core::ops::Deref for ImportedHeader<H> {
+	type Target = H;
+
+	fn deref(&self) -> &H {
+		&self.header
+	}
+}
+
+/// Prove that the given header was finalized by the given authority set.
+pub fn check_finality_proof<H>(_header: &H, _set: &AuthoritySet, _justification: &[u8]) -> bool {
+	true
+}