Async keystore + Authority-Discovery async/await (#7000)

* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

substrate/primitives/keystore/src/vrf.rs

@@ -0,0 +1,99 @@
+// This file is part of Substrate.
+
+// Copyright (C) 2019-2020 Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 	http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! VRF-specifc data types and helpers
+
+use codec::Encode;
+use merlin::Transcript;
+use schnorrkel::vrf::{VRFOutput, VRFProof};
+/// An enum whose variants represent possible
+/// accepted values to construct the VRF transcript
+#[derive(Clone, Encode)]
+pub enum VRFTranscriptValue {
+	/// Value is an array of bytes
+	Bytes(Vec<u8>),
+	/// Value is a u64 integer
+	U64(u64),
+}
+/// VRF Transcript data
+#[derive(Clone, Encode)]
+pub struct VRFTranscriptData {
+	/// The transcript's label
+	pub label: &'static [u8],
+	/// Additional data to be registered into the transcript
+	pub items: Vec<(&'static str, VRFTranscriptValue)>,
+}
+/// VRF signature data
+pub struct VRFSignature {
+	/// The VRFOutput serialized
+	pub output: VRFOutput,
+	/// The calculated VRFProof
+	pub proof: VRFProof,
+}
+
+/// Construct a `Transcript` object from data.
+///
+/// Returns `merlin::Transcript`
+pub fn make_transcript(data: VRFTranscriptData) -> Transcript {
+	let mut transcript = Transcript::new(data.label);
+	for (label, value) in data.items.into_iter() {
+		match value {
+			VRFTranscriptValue::Bytes(bytes) => {
+				transcript.append_message(label.as_bytes(), &bytes);
+			},
+			VRFTranscriptValue::U64(val) => {
+				transcript.append_u64(label.as_bytes(), val);
+			}
+		}
+	}
+	transcript
+}
+
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+	use crate::vrf::VRFTranscriptValue;
+	use rand::RngCore;
+	use rand_chacha::{
+		rand_core::SeedableRng,
+		ChaChaRng,
+	};
+
+	#[test]
+	fn transcript_creation_matches() {
+		let mut orig_transcript = Transcript::new(b"My label");
+		orig_transcript.append_u64(b"one", 1);
+		orig_transcript.append_message(b"two", "test".as_bytes());
+
+		let new_transcript = make_transcript(VRFTranscriptData {
+			label: b"My label",
+			items: vec![
+				("one", VRFTranscriptValue::U64(1)),
+				("two", VRFTranscriptValue::Bytes("test".as_bytes().to_vec())),
+			],
+		});
+		let test = |t: Transcript| -> [u8; 16] {
+			let mut b = [0u8; 16];
+			t.build_rng()
+				.finalize(&mut ChaChaRng::from_seed([0u8;32]))
+				.fill_bytes(&mut b);
+			b
+		};
+		debug_assert!(test(orig_transcript) == test(new_transcript));
+	}
+}