Squashed commit of slava-async-bridge:

commit 657deb4cf4b90f24b9c5bfd62764b197776c262c
Author: Hernando Castano <castano.ha@gmail.com>
Date:   Wed Jan 29 20:14:20 2020 -0500

    Move Slava's bridge code into relays folder

commit 4868c42c7da959dde7252766996b3ed4e408e439
Author: Hernando Castano <castano.ha@gmail.com>
Date:   Wed Jan 29 20:01:06 2020 -0500

    Move files into `modules/ethereum`

commit d1093f3e4238acb1a1a020011452cb928d3f8d7a
Merge: 29dc6f9 bfd30ef
Author: Hernando Castano <castano.ha@gmail.com>
Date:   Wed Jan 29 19:59:27 2020 -0500

    Merge branch 'master' of slava-async-bridge

commit 29dc6f97b1b7d1db99086d35a5336f43d2f0f8af
Author: Hernando Castano <castano.ha@gmail.com>
Date:   Wed Jan 29 19:51:31 2020 -0500

    Squashed commit of the following:

    commit 265365920836bb1d286c9b48b1902a2de278fdd9
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Wed Jan 29 19:51:15 2020 -0500

        Move hc-jp-bridge repo to different folder

    commit 8271991e95320baba70bd1cb9c4234d0ffd5b638
    Merge: 57d0811 304cbc5
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Wed Jan 29 19:36:41 2020 -0500

        Merge branch 'hc-jp-bridge-module' of hc-jp-bridge-module

    commit 304cbc5f02d003ffa5404c1c01e461e5b8539888
    Author: Hernando Castano <HCastano@users.noreply.github.com>
    Date:   Wed Jan 29 00:38:27 2020 -0500

        Update bridge pallet to work with the (almost) lastest master (#4672)

        * Update decl_error usage

        * WIP: Update error handling to use DispatchResult

        * Get module compiling with new error handling

        * Make tests compile again

        Main change was updating the usage of InMemoryBackend

        * Move `sp-state-machine` into dev-dependencies

        * Bump dependencies to v2.0.0

        * Remove some stray comments

        * Appy code review suggestion

    commit 510cd6d96372688517496efa61773ea2839f8474
    Author: Hernando Castano <HCastano@users.noreply.github.com>
    Date:   Tue Dec 17 12:52:51 2019 -0500

        Move Bridge Pallet into FRAME (#4373)

        * Move `bridge` crate into `frame` folder

        * Make `bridge` pallet compile after `the-big-reorg`

    commit ab54e838ef75e6a3f68fd0944bf22598c10c552f
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Mon Nov 11 21:56:40 2019 +0100

        Use new StorageProof type from #3834

    commit 8fc8911fd1b4acc2274c6863fb3dba91b30c90af
    Author: Hernando Castano <HCastano@users.noreply.github.com>
    Date:   Tue Nov 5 00:50:34 2019 +0100

        Verify Ancestry between Headers (#3963)

        * Create module for checking ancestry proofs

        * Use Vec of Headers instead of a HashMap

        * Move the ancestry verification into the lib.rs file

        * Change the proof format to exclude `child` and `ancestor` headers

        * Add a testing function for building header chains

        * Rename AncestorNotFound error to InvalidAncestryProof

        * Use ancestor hash instead of header when verifying ancestry

        * Clean up some stuff missed in the merge

    commit dbe85738b68358b790cf927b34a804b965a88f96
    or: Hernando Castano <HCastano@users.noreply.github.com>
    Date:   Fri Nov 1 15:41:58 2019 +0100

        Check given Grandpa validator set against set found in storage (#3915)

        * Make StorageProofChecker happy

        * Update some tests

        * Check given validator set against set found in storage

        * Use Finality Grandpa's Authority Id and Weight

        * Add better error handling

        * Use error type from decl_error! macro

    commit 31b09216603d3e9c21144ce8c0b6bf59307a4f97
    or: Hernando Castano <HCastano@users.noreply.github.com>
    Date:   Wed Oct 23 14:55:37 2019 +0200

        Make tests work after the changes introduced in #3793 (#3874)

        * Make tests work after the changes introduced in #3793

        * Remove unneccessary import

    commit bce6d804aa86504599ff912387295c58f846cbf3
    Author: Jim Posen <jim.posen@gmail.com>
    Date:   Thu Oct 10 12:18:58 2019 +0200

        Logic for checking Substrate proofs from within runtime module. (#3783)

    commit a7013e94b6c772c1d45a7cacbb445f73f6554fca
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Fri Oct 4 15:21:00 2019 +0300

        Allow tracking of multiple bridges

    commit 3cf648242d631e32bd553a67df54bf5a48912839
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Tue Oct 1 14:55:04 2019 +0200

        Add BridgeId => Bridge mapping

    commit 001c74c45072213e01857d0a2454379b447c5a76
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Tue Oct 1 11:10:19 2019 +0200

        Get the mock runtime for tests set up

    commit 38443a1e8b424ed2f148eb95121d009f730e3b5a
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Fri Sep 27 14:52:53 2019 +0200

        Clean up some warnings

    commit bdc3b01401e89c7111f8bf71f84c50750d25089f
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Thu Sep 26 16:41:01 2019 +0200

        Add more skeleton code

    commit 26995efbf4bac2842eb2822322f7ad3c3e88feb8
    Author: Hernando Castano <castano.ha@gmail.com>
    Date:   Wed Sep 25 15:16:57 2019 +0200

        Create `bridge` module skeleton

commit bfd30ef8363b1483ef1107ae1eb958a4e944c93b
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Tue Dec 10 12:10:53 2019 +0300

    actually use signer from CLI to sign Substrate transactions

commit 504028eac60d9d14ba95b506cd355b0d2f405ce0
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Tue Dec 10 12:02:22 2019 +0300

    go offline for a bit on connection error

commit 446d0c8d20187dfd1beb173958ea28f2ad97887d
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Tue Dec 10 11:25:50 2019 +0300

    enable info logs by default

commit d039c60ec72bc91adfdad85442bc99a93b7f8e8d
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Tue Dec 10 11:12:51 2019 +0300

    support basic CLI arguments

commit 65c6d48e23576f36e8541878b920a03730226392
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 15:37:48 2019 +0300

    fix restart

commit 96e94c1c4b22d732078f8c401b872c5f8246c3fe
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 14:57:53 2019 +0300

    license

commit 68f4191e6cdd211ac8975e0b79f8a6f46a3ca953
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 14:56:05 2019 +0300

    restart sync when Substrate reorgs && we are unlucky

commit 29887c446167d580d73cc03a0b71c31890cafb51
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 13:49:31 2019 +0300

    only read genesis hash once

commit 832492b8393fe2063adf9c58c2b9e060dc3e4efb
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 13:23:26 2019 +0300

    changed TODO

commit 9dbc130e5fa036ae63d973819daf30f4ed6ffb5b
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 13:16:56 2019 +0300

    removed obsolete exit future

commit d03408cd8284eb0c61e7e96429b4f6199353e030
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 13:16:17 2019 +0300

    removed obsolete TODOs + moved a couple of TODOs to runtime module

commit ed8bec44b79f9a2ce829e59f10181368b2f42139
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 12:37:05 2019 +0300

    explained TODO fix

commit aa9c4c66ec2904eeb6072d654718b0ac0b7d8803
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 12:28:09 2019 +0300

    fix tx outcome serialization

commit 126f8f5484dac8c4af588ae86dc8855919d6c822
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Mon Dec 9 12:05:05 2019 +0300

    prune old ethereum headers when Substrate best header is too far in the future

commit c7bd301e631a44fe3263e188d0956081aa84f31e
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Fri Dec 6 12:51:50 2019 +0300

    fix trace

commit 549bb7acdb30cfdafe6c8600f0410212539ea63d
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Fri Dec 6 12:51:26 2019 +0300

    tx hashes are already a part of Block response

commit 7864017909f87ea36955d605a924c3c88bc88df3
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Thu Dec 5 12:29:37 2019 +0300

    submit bunch of headers at once + some fixes

commit 96485f85d38c144f0771f02ba692216a60356665
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Wed Dec 4 17:22:13 2019 +0300

    print status messages

commit ae0ec4c087136db653339537daab7f96a8c21b65
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Wed Dec 4 17:06:00 2019 +0300

    continue actual Substrate client implementation

commit 8146293740d70b88904568ff8e5acdfbadf06fd3
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Wed Dec 4 13:49:30 2019 +0300

    fix IncompleteHeader condition

commit 767c6201157dabcccf7f62e643681ca298224fb1
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Wed Dec 4 10:55:06 2019 +0300

    actual Substrate client implementation

commit 221fd4ccd2b1eea12c9dacf800d80e15ec115c1b
Author: Svyatoslav Nikolsky <svyatonik@gmail.com>
Date:   Wed Nov 20 17:28:13 2019 +0300

    initial commit

bridges/relays/ethereum/src/ethereum_sync.rs

@@ -0,0 +1,311 @@
+// Copyright 2019 Parity Technologies (UK) Ltd.
+// This file is part of Parity-Bridge.
+
+// Parity-Bridge is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity-Bridge is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity-Bridge.  If not, see <http://www.gnu.org/licenses/>.
+
+use codec::Encode;
+use crate::ethereum_headers::QueuedHeaders;
+use crate::ethereum_types::{HeaderId, HeaderStatus, QueuedHeader};
+use crate::ethereum_sync_loop::EthereumSyncParams;
+use crate::substrate_types::{into_substrate_ethereum_header, into_substrate_ethereum_receipts};
+
+/// Ethereum headers synchronization context.
+#[derive(Debug)]
+pub struct HeadersSync {
+	/// Synchronization parameters.
+	params: EthereumSyncParams,
+	/// Best header number known to Ethereum node.
+	target_header_number: Option<u64>,
+	/// Best header known to Substrate node.
+	best_header: Option<HeaderId>,
+	/// Headers queue.
+	headers: QueuedHeaders,
+}
+
+impl HeadersSync {
+	/// Creates new Ethereum headers synchronizer.
+	pub fn new(params: EthereumSyncParams) -> Self {
+		HeadersSync {
+			params,
+			target_header_number: None,
+			best_header: None,
+			headers: Default::default(),
+		}
+	}
+
+	/// Returns true if we have synced almost all known headers.
+	pub fn is_almost_synced(&self) -> bool {
+		match self.target_header_number {
+			Some(target_header_number) => self.best_header
+				.map(|best| target_header_number.saturating_sub(best.0) < 4)
+				.unwrap_or(false),
+			None => true,
+		}
+	}
+
+	/// Returns synchronization status.
+	pub fn status(&self) -> (&Option<HeaderId>, &Option<u64>) {
+		(&self.best_header, &self.target_header_number)
+	}
+
+	/// Returns reference to the headers queue.
+	pub fn headers(&self) -> &QueuedHeaders {
+		&self.headers
+	}
+
+	/// Returns mutable reference to the headers queue.
+	pub fn headers_mut(&mut self) -> &mut QueuedHeaders {
+		&mut self.headers
+	}
+
+	/// Select header that needs to be downloaded from the Ethereum node.
+	pub fn select_new_header_to_download(&self) -> Option<u64> {
+		// if we haven't received best header from Ethereum node yet, there's nothing we can download
+		let target_header_number = self.target_header_number.clone()?;
+
+		// if we haven't received known best header from Substrate node yet, there's nothing we can download
+		let best_header = self.best_header.as_ref()?;
+
+		// if there's too many headers in the queue, stop downloading
+		let in_memory_headers = self.headers.total_headers();
+		if in_memory_headers >= self.params.max_future_headers_to_download {
+			return None;
+		}
+
+		// we assume that there were no reorgs if we have already downloaded best header
+		let best_downloaded_number = std::cmp::max(
+			self.headers.best_queued_number(),
+			best_header.0,
+		);
+		if best_downloaded_number == target_header_number {
+			return None;
+		}
+
+		// download new header
+		Some(best_downloaded_number + 1)
+	}
+
+	/// Select headers that need to be submitted to the Substrate node.
+	pub fn select_headers_to_submit(&self) -> Option<Vec<&QueuedHeader>> {
+		let headers_in_submit_status = self.headers.headers_in_status(HeaderStatus::Submitted);
+		let headers_to_submit_count = self.params.max_headers_in_submitted_status
+			.checked_sub(headers_in_submit_status)?;
+
+		let mut total_size = 0;
+		let mut total_headers = 0;
+		self.headers.headers(HeaderStatus::Ready, |header| {
+			if total_headers == headers_to_submit_count {
+				return false;
+			}
+			if total_headers == self.params.max_headers_in_single_submit {
+				return false;
+			}
+
+			let encoded_size = into_substrate_ethereum_header(header.header()).encode().len()
+				+ into_substrate_ethereum_receipts(header.receipts()).map(|receipts| receipts.encode().len())
+					.unwrap_or(0);
+			if total_headers != 0 && total_size + encoded_size > self.params.max_headers_size_in_single_submit {
+				return false;
+			}
+
+			total_size += encoded_size;
+			total_headers += 1;
+
+			true
+		})
+	}
+
+	/// Receive new target header number from the Ethereum node.
+	pub fn ethereum_best_header_number_response(&mut self, best_header_number: u64) {
+		log::debug!(target: "bridge", "Received best header number from Ethereum: {}", best_header_number);
+		self.target_header_number = Some(best_header_number);
+	}
+
+	/// Receive new best header from the Substrate node.
+	/// Returns true if it is different from the previous block known to us.
+	pub fn substrate_best_header_response(&mut self, best_header: HeaderId) -> bool {
+		log::debug!(target: "bridge", "Received best known header from Substrate: {:?}", best_header);
+
+		// early return if it is still the same
+		if self.best_header == Some(best_header) {
+			return false;
+		}
+
+		// remember that this header is now known to the Substrate runtime
+		self.headers.substrate_best_header_response(&best_header);
+
+		// prune ancient headers
+		self.headers.prune(best_header.0.saturating_sub(self.params.prune_depth));
+
+		// finally remember the best header itself
+		self.best_header = Some(best_header);
+
+		true
+	}
+
+	/// Restart synchronization.
+	pub fn restart(&mut self) {
+		self.target_header_number = None;
+		self.best_header = None;
+		self.headers.clear();
+	}
+}
+
+#[cfg(test)]
+mod tests {
+	use crate::ethereum_headers::tests::{header, id};
+	use crate::ethereum_types::{H256, HeaderStatus};
+	use super::*;
+
+	fn side_hash(number: u64) -> H256 {
+		H256::from_low_u64_le(1000 + number)
+	}
+
+	#[test]
+	fn select_new_header_to_download_works() {
+		let mut eth_sync = HeadersSync::new(Default::default());
+
+		// both best && target headers are unknown
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+
+		// best header is known, target header is unknown
+		eth_sync.best_header = Some(HeaderId(0, Default::default()));
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+
+		// target header is known, best header is unknown
+		eth_sync.best_header = None;
+		eth_sync.target_header_number = Some(100);
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+
+		// when our best block has the same number as the target
+		eth_sync.best_header = Some(HeaderId(100, Default::default()));
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+
+		// when we actually need a new header
+		eth_sync.target_header_number = Some(101);
+		assert_eq!(eth_sync.select_new_header_to_download(), Some(101));
+
+		// when there are too many headers scheduled for submitting
+		for i in 1..1000 {
+			eth_sync.headers.header_response(header(i).header().clone());
+		}
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+	}
+
+	#[test]
+	fn sync_without_reorgs_works() {
+		let mut eth_sync = HeadersSync::new(Default::default());
+		eth_sync.params.max_headers_in_submitted_status = 1;
+
+		// ethereum reports best header #102
+		eth_sync.ethereum_best_header_number_response(102);
+
+		// substrate reports that it is at block #100
+		eth_sync.substrate_best_header_response(id(100));
+
+		// block #101 is downloaded first
+		assert_eq!(eth_sync.select_new_header_to_download(), Some(101));
+		eth_sync.headers.header_response(header(101).header().clone());
+
+		// now header #101 is ready to be submitted
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeReceipts), Some(&header(101)));
+		eth_sync.headers.maybe_receipts_response(&id(101), false);
+		assert_eq!(eth_sync.headers.header(HeaderStatus::Ready), Some(&header(101)));
+		assert_eq!(eth_sync.select_headers_to_submit(), Some(vec![&header(101)]));
+
+		// and header #102 is ready to be downloaded
+		assert_eq!(eth_sync.select_new_header_to_download(), Some(102));
+		eth_sync.headers.header_response(header(102).header().clone());
+
+		// receive submission confirmation
+		eth_sync.headers.headers_submitted(vec![id(101)]);
+
+		// we have nothing to submit because previous header hasn't been confirmed yet
+		// (and we allow max 1 submit transaction in the wild)
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeReceipts), Some(&header(102)));
+		eth_sync.headers.maybe_receipts_response(&id(102), false);
+		assert_eq!(eth_sync.headers.header(HeaderStatus::Ready), Some(&header(102)));
+		assert_eq!(eth_sync.select_headers_to_submit(), None);
+
+		// substrate reports that it has imported block #101
+		eth_sync.substrate_best_header_response(id(101));
+
+		// and we are ready to submit #102
+		assert_eq!(eth_sync.select_headers_to_submit(), Some(vec![&header(102)]));
+		eth_sync.headers.headers_submitted(vec![id(102)]);
+
+		// substrate reports that it has imported block #102
+		eth_sync.substrate_best_header_response(id(102));
+
+		// and we have nothing to download
+		assert_eq!(eth_sync.select_new_header_to_download(), None);
+	}
+
+	#[test]
+	fn sync_with_orphan_headers_work() {
+		let mut eth_sync = HeadersSync::new(Default::default());
+
+		// ethereum reports best header #102
+		eth_sync.ethereum_best_header_number_response(102);
+
+		// substrate reports that it is at block #100, but it isn't part of best chain
+		eth_sync.substrate_best_header_response(HeaderId(100, side_hash(100)));
+
+		// block #101 is downloaded first
+		assert_eq!(eth_sync.select_new_header_to_download(), Some(101));
+		eth_sync.headers.header_response(header(101).header().clone());
+
+		// we can't submit header #101, because its parent status is unknown
+		assert_eq!(eth_sync.select_headers_to_submit(), None);
+
+		// instead we are trying to determine status of its parent (#100)
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeOrphan), Some(&header(101)));
+
+		// and the status is still unknown
+		eth_sync.headers.maybe_orphan_response(&id(100), false);
+
+		// so we consider #101 orphaned now && will download its parent - #100
+		assert_eq!(eth_sync.headers.header(HeaderStatus::Orphan), Some(&header(101)));
+		eth_sync.headers.header_response(header(100).header().clone());
+
+		// we can't submit header #100, because its parent status is unknown
+		assert_eq!(eth_sync.select_headers_to_submit(), None);
+
+		// instead we are trying to determine status of its parent (#99)
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeOrphan), Some(&header(100)));
+
+		// and the status is known, so we move previously orphaned #100 and #101 to ready queue
+		eth_sync.headers.maybe_orphan_response(&id(99), true);
+
+		// and we are ready to submit #100
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeReceipts), Some(&header(100)));
+		eth_sync.headers.maybe_receipts_response(&id(100), false);
+		assert_eq!(eth_sync.select_headers_to_submit(), Some(vec![&header(100)]));
+		eth_sync.headers.headers_submitted(vec![id(100)]);
+
+		// and we are ready to submit #101
+		assert_eq!(eth_sync.headers.header(HeaderStatus::MaybeReceipts), Some(&header(101)));
+		eth_sync.headers.maybe_receipts_response(&id(101), false);
+		assert_eq!(eth_sync.select_headers_to_submit(), Some(vec![&header(101)]));
+		eth_sync.headers.headers_submitted(vec![id(101)]);
+	}
+
+	#[test]
+	fn pruning_happens_on_substrate_best_header_response() {
+		let mut eth_sync = HeadersSync::new(Default::default());
+		eth_sync.params.prune_depth = 50;
+		eth_sync.substrate_best_header_response(id(100));
+		assert_eq!(eth_sync.headers.prune_border(), 50);
+	}
+}