Adds test parachain adder collator (#1864)

* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

* Adds test parachain adder collator

* Add sudo to Rococo, change session length to 30 seconds and some renaming

* Update to the latest changes on master

* Some fixes

* Fix compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Review comments

* Downgrade transaction version

* Fixes

* MOARE

* Register notification protocols

* utils: remove unused error

* av-store: more resilient to some errors

* address review nits

* address more review nits

Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Sergey Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>

polkadot/parachain/test-parachains/adder/collator/src/lib.rs

@@ -0,0 +1,157 @@
+// Copyright 2020 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Collator for the adder test parachain.
+
+use std::{pin::Pin, sync::{Arc, Mutex}};
+use test_parachain_adder::{hash_state, BlockData, HeadData, execute};
+use futures::{Future, FutureExt};
+use polkadot_primitives::v1::{ValidationData, PoV, Hash};
+use polkadot_node_primitives::Collation;
+use codec::Encode;
+
+/// The amount we add when producing a new block.
+///
+/// This is a constant to make tests easily reproducible.
+const ADD: u64 = 2;
+
+/// The state of the adder parachain.
+struct State {
+	genesis_state: HeadData,
+	last_head: HeadData,
+	state: u64,
+}
+
+impl State {
+	/// Init the genesis state.
+	fn genesis() -> Self {
+		let genesis_state = HeadData {
+			number: 0,
+			parent_hash: Default::default(),
+			post_state: hash_state(0),
+		};
+		let last_head = genesis_state.clone();
+
+		Self {
+			genesis_state,
+			last_head,
+			state: 0,
+		}
+	}
+
+	/// Advance the state and produce a new block.
+	///
+	/// Returns the new [`BlockData`] and the new [`HeadData`].
+	fn advance(&mut self) -> (BlockData, HeadData) {
+		let block = BlockData {
+			state: self.state,
+			add: ADD,
+		};
+		let new_head = execute(self.last_head.hash(), self.last_head.clone(), &block)
+			.expect("Produces valid block");
+
+		self.last_head = new_head.clone();
+		self.state = self.state.wrapping_add(ADD);
+
+		(block, new_head)
+	}
+}
+
+/// The collator of the adder parachain.
+pub struct Collator {
+	state: Arc<Mutex<State>>,
+}
+
+impl Collator {
+	/// Create a new collator instance with the state initialized as genesis.
+	pub fn new() -> Self {
+		Self {
+			state: Arc::new(Mutex::new(State::genesis())),
+		}
+	}
+
+	/// Get the SCALE encoded genesis head of the adder parachain.
+	pub fn genesis_head(&self) -> Vec<u8> {
+		self.state.lock().unwrap().genesis_state.encode()
+	}
+
+	/// Get the validation code of the adder parachain.
+	pub fn validation_code(&self) -> &[u8] {
+		test_parachain_adder::wasm_binary_unwrap()
+	}
+
+	/// Create the collation function.
+	///
+	/// This collation function can be plugged into the overseer to generate collations for the adder parachain.
+	pub fn create_collation_function(
+		&self,
+	) -> Box<dyn Fn(Hash, &ValidationData) -> Pin<Box<dyn Future<Output = Option<Collation>> + Send>> + Send + Sync> {
+		let state = self.state.clone();
+
+		Box::new(move |_, _| {
+			let (block_data, head_data) = state.lock().unwrap().advance();
+
+			let collation = Collation {
+				upward_messages: Vec::new(),
+				new_validation_code: None,
+				head_data: head_data.encode().into(),
+				proof_of_validity: PoV { block_data: block_data.encode().into() },
+				processed_downward_messages: 0,
+			};
+
+			async move { Some(collation) }.boxed()
+		})
+	}
+}
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+
+	use futures::executor::block_on;
+	use polkadot_parachain::{primitives::ValidationParams, wasm_executor::ExecutionMode};
+	use codec::Decode;
+
+	#[test]
+	fn collator_works() {
+		let collator = Collator::new();
+		let collation_function = collator.create_collation_function();
+
+		for _ in 0..5 {
+			let parent_head = collator.state.lock().unwrap().last_head.clone();
+			let collation = block_on(collation_function(Default::default(), &Default::default())).unwrap();
+			validate_collation(&collator, parent_head, collation);
+		}
+	}
+
+	fn validate_collation(collator: &Collator, parent_head: HeadData, collation: Collation) {
+		let ret = polkadot_parachain::wasm_executor::validate_candidate(
+			collator.validation_code(),
+			ValidationParams {
+				parent_head: parent_head.encode().into(),
+				block_data: collation.proof_of_validity.block_data,
+				relay_chain_height: 1,
+				hrmp_mqc_heads: Vec::new(),
+				dmq_mqc_head: Default::default(),
+			},
+			&ExecutionMode::InProcess,
+			sp_core::testing::TaskExecutor::new(),
+		).unwrap();
+
+		let new_head = HeadData::decode(&mut &ret.head_data.0[..]).unwrap();
+		assert_eq!(collator.state.lock().unwrap().last_head, new_head);
+	}
+}