Async keystore + Authority-Discovery async/await (#7000)

* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

substrate/client/consensus/babe/Cargo.toml

@@ -18,6 +18,7 @@ codec = { package = "parity-scale-codec", version = "1.3.4", features = ["derive
 sp-consensus-babe = { version = "0.8.0", path = "../../../primitives/consensus/babe" }
 sp-core = { version = "2.0.0", path = "../../../primitives/core" }
 sp-application-crypto = { version = "2.0.0", path = "../../../primitives/application-crypto" }
+sp-keystore = { version = "0.8.0", path = "../../../primitives/keystore" }
 num-bigint = "0.2.3"
 num-rational = "0.2.2"
 num-traits = "0.2.8"

substrate/client/consensus/babe/rpc/Cargo.toml

@@ -29,11 +29,12 @@ sp-api = { version = "2.0.0", path = "../../../../primitives/api" }
 sp-consensus = { version = "0.8.0", path = "../../../../primitives/consensus/common" }
 sp-core = { version = "2.0.0", path = "../../../../primitives/core" }
 sp-application-crypto = { version = "2.0.0", path = "../../../../primitives/application-crypto" }
-sc-keystore = { version = "2.0.0", path = "../../../keystore" }
+sp-keystore = { version = "0.8.0", path = "../../../../primitives/keystore" }
 
 [dev-dependencies]
 sc-consensus = { version = "0.8.0", path = "../../../consensus/common" }
 serde_json = "1.0.50"
 sp-keyring = { version = "2.0.0", path = "../../../../primitives/keyring" }
+sc-keystore = { version = "2.0.0", path = "../../../keystore" }
 substrate-test-runtime-client = { version = "2.0.0", path = "../../../../test-utils/runtime/client" }
 tempfile = "3.1.0"

substrate/client/consensus/babe/rpc/src/lib.rs

@@ -34,10 +34,9 @@ use sp_consensus_babe::{
 use serde::{Deserialize, Serialize};
 use sp_core::{
 	crypto::Public,
-	traits::BareCryptoStore,
 };
 use sp_application_crypto::AppKey;
-use sc_keystore::KeyStorePtr;
+use sp_keystore::{SyncCryptoStorePtr, SyncCryptoStore};
 use sc_rpc_api::DenyUnsafe;
 use sp_api::{ProvideRuntimeApi, BlockId};
 use sp_runtime::traits::{Block as BlockT, Header as _};
@@ -63,7 +62,7 @@ pub struct BabeRpcHandler<B: BlockT, C, SC> {
 	/// shared reference to EpochChanges
 	shared_epoch_changes: SharedEpochChanges<B, Epoch>,
 	/// shared reference to the Keystore
-	keystore: KeyStorePtr,
+	keystore: SyncCryptoStorePtr,
 	/// config (actually holds the slot duration)
 	babe_config: Config,
 	/// The SelectChain strategy
@@ -77,7 +76,7 @@ impl<B: BlockT, C, SC> BabeRpcHandler<B, C, SC> {
 	pub fn new(
 		client: Arc<C>,
 		shared_epoch_changes: SharedEpochChanges<B, Epoch>,
-		keystore: KeyStorePtr,
+		keystore: SyncCryptoStorePtr,
 		babe_config: Config,
 		select_chain: SC,
 		deny_unsafe: DenyUnsafe,
@@ -131,11 +130,10 @@ impl<B, C, SC> BabeApi for BabeRpcHandler<B, C, SC>
 			let mut claims: HashMap<AuthorityId, EpochAuthorship> = HashMap::new();
 
 			let keys = {
-				let ks = keystore.read();
 				epoch.authorities.iter()
 					.enumerate()
 					.filter_map(|(i, a)| {
-						if ks.has_keys(&[(a.0.to_raw_vec(), AuthorityId::ID)]) {
+						if SyncCryptoStore::has_keys(&*keystore, &[(a.0.to_raw_vec(), AuthorityId::ID)]) {
 							Some((a.0.clone(), i))
 						} else {
 							None
@@ -236,18 +234,23 @@ mod tests {
 		TestClientBuilder,
 	};
 	use sp_application_crypto::AppPair;
-	use sp_keyring::Ed25519Keyring;
-	use sc_keystore::Store;
+	use sp_keyring::Sr25519Keyring;
+	use sp_core::{crypto::key_types::BABE};
+	use sp_keystore::{SyncCryptoStorePtr, SyncCryptoStore};
+	use sc_keystore::LocalKeystore;
 
 	use std::sync::Arc;
 	use sc_consensus_babe::{Config, block_import, AuthorityPair};
 	use jsonrpc_core::IoHandler;
 
 	/// creates keystore backed by a temp file
-	fn create_temp_keystore<P: AppPair>(authority: Ed25519Keyring) -> (KeyStorePtr, tempfile::TempDir) {
+	fn create_temp_keystore<P: AppPair>(
+		authority: Sr25519Keyring,
+	) -> (SyncCryptoStorePtr, tempfile::TempDir) {
 		let keystore_path = tempfile::tempdir().expect("Creates keystore path");
-		let keystore = Store::open(keystore_path.path(), None).expect("Creates keystore");
-		keystore.write().insert_ephemeral_from_seed::<P>(&authority.to_seed())
+		let keystore = Arc::new(LocalKeystore::open(keystore_path.path(), None)
+			.expect("Creates keystore"));
+		SyncCryptoStore::sr25519_generate_new(&*keystore, BABE, Some(&authority.to_seed()))
 			.expect("Creates authority key");
 
 		(keystore, keystore_path)
@@ -267,7 +270,7 @@ mod tests {
 		).expect("can initialize block-import");
 
 		let epoch_changes = link.epoch_changes().clone();
-		let keystore = create_temp_keystore::<AuthorityPair>(Ed25519Keyring::Alice).0;
+		let keystore = create_temp_keystore::<AuthorityPair>(Sr25519Keyring::Alice).0;
 
 		BabeRpcHandler::new(
 			client.clone(),

substrate/client/consensus/babe/src/authorship.rs

@@ -28,13 +28,13 @@ use sp_consensus_babe::digests::{
 	PreDigest, PrimaryPreDigest, SecondaryPlainPreDigest, SecondaryVRFPreDigest,
 };
 use sp_consensus_vrf::schnorrkel::{VRFOutput, VRFProof};
-use sp_core::{U256, blake2_256, crypto::Public, traits::BareCryptoStore};
+use sp_core::{U256, blake2_256, crypto::Public};
+use sp_keystore::{SyncCryptoStorePtr, SyncCryptoStore};
 use codec::Encode;
 use schnorrkel::{
 	keys::PublicKey,
 	vrf::VRFInOut,
 };
-use sc_keystore::KeyStorePtr;
 use super::Epoch;
 
 /// Calculates the primary selection threshold for a given authority, taking
@@ -131,7 +131,7 @@ fn claim_secondary_slot(
 	slot_number: SlotNumber,
 	epoch: &Epoch,
 	keys: &[(AuthorityId, usize)],
-	keystore: &KeyStorePtr,
+	keystore: &SyncCryptoStorePtr,
 	author_secondary_vrf: bool,
 ) -> Option<(PreDigest, AuthorityId)> {
 	let Epoch { authorities, randomness, epoch_index, .. } = epoch;
@@ -154,7 +154,8 @@ fn claim_secondary_slot(
 					slot_number,
 					*epoch_index,
 				);
-				let result = keystore.read().sr25519_vrf_sign(
+				let result = SyncCryptoStore::sr25519_vrf_sign(
+					&**keystore,
 					AuthorityId::ID,
 					authority_id.as_ref(),
 					transcript_data,
@@ -169,7 +170,7 @@ fn claim_secondary_slot(
 				} else {
 					None
 				}
-			} else if keystore.read().has_keys(&[(authority_id.to_raw_vec(), AuthorityId::ID)]) {
+			} else if SyncCryptoStore::has_keys(&**keystore, &[(authority_id.to_raw_vec(), AuthorityId::ID)]) {
 				Some(PreDigest::SecondaryPlain(SecondaryPlainPreDigest {
 					slot_number,
 					authority_index: *authority_index as u32,
@@ -194,7 +195,7 @@ fn claim_secondary_slot(
 pub fn claim_slot(
 	slot_number: SlotNumber,
 	epoch: &Epoch,
-	keystore: &KeyStorePtr,
+	keystore: &SyncCryptoStorePtr,
 ) -> Option<(PreDigest, AuthorityId)> {
 	let authorities = epoch.authorities.iter()
 		.enumerate()
@@ -208,7 +209,7 @@ pub fn claim_slot(
 pub fn claim_slot_using_keys(
 	slot_number: SlotNumber,
 	epoch: &Epoch,
-	keystore: &KeyStorePtr,
+	keystore: &SyncCryptoStorePtr,
 	keys: &[(AuthorityId, usize)],
 ) -> Option<(PreDigest, AuthorityId)> {
 	claim_primary_slot(slot_number, epoch, epoch.config.c, keystore, &keys)
@@ -220,7 +221,7 @@ pub fn claim_slot_using_keys(
 					slot_number,
 					&epoch,
 					keys,
-					keystore,
+					&keystore,
 					epoch.config.allowed_slots.is_secondary_vrf_slots_allowed(),
 				)
 			} else {
@@ -237,7 +238,7 @@ fn claim_primary_slot(
 	slot_number: SlotNumber,
 	epoch: &Epoch,
 	c: (u64, u64),
-	keystore: &KeyStorePtr,
+	keystore: &SyncCryptoStorePtr,
 	keys: &[(AuthorityId, usize)],
 ) -> Option<(PreDigest, AuthorityId)> {
 	let Epoch { authorities, randomness, epoch_index, .. } = epoch;
@@ -259,7 +260,8 @@ fn claim_primary_slot(
 		// be empty.  Therefore, this division in `calculate_threshold` is safe.
 		let threshold = super::authorship::calculate_primary_threshold(c, authorities, *authority_index);
 
-		let result = keystore.read().sr25519_vrf_sign(
+		let result = SyncCryptoStore::sr25519_vrf_sign(
+			&**keystore,
 			AuthorityId::ID,
 			authority_id.as_ref(),
 			transcript_data,
@@ -289,13 +291,16 @@ fn claim_primary_slot(
 #[cfg(test)]
 mod tests {
 	use super::*;
+	use std::sync::Arc;
 	use sp_core::{sr25519::Pair, crypto::Pair as _};
 	use sp_consensus_babe::{AuthorityId, BabeEpochConfiguration, AllowedSlots};
+	use sc_keystore::LocalKeystore;
 
 	#[test]
 	fn claim_secondary_plain_slot_works() {
-		let keystore = sc_keystore::Store::new_in_memory();
-		let valid_public_key = keystore.write().sr25519_generate_new(
+		let keystore: SyncCryptoStorePtr = Arc::new(LocalKeystore::in_memory());
+		let valid_public_key = SyncCryptoStore::sr25519_generate_new(
+			&*keystore,
 			AuthorityId::ID,
 			Some(sp_core::crypto::DEV_PHRASE),
 		).unwrap();

substrate/client/consensus/babe/src/lib.rs

@@ -82,14 +82,14 @@ use sp_consensus::{ImportResult, CanAuthorWith};
 use sp_consensus::import_queue::{
 	BoxJustificationImport, BoxFinalityProofImport,
 };
-use sp_core::{crypto::Public, traits::BareCryptoStore};
+use sp_core::crypto::Public;
 use sp_application_crypto::AppKey;
+use sp_keystore::{SyncCryptoStorePtr, SyncCryptoStore};
 use sp_runtime::{
 	generic::{BlockId, OpaqueDigestItemId}, Justification,
 	traits::{Block as BlockT, Header, DigestItemFor, Zero},
 };
 use sp_api::{ProvideRuntimeApi, NumberFor};
-use sc_keystore::KeyStorePtr;
 use parking_lot::Mutex;
 use sp_inherents::{InherentDataProviders, InherentData};
 use sc_telemetry::{telemetry, CONSENSUS_TRACE, CONSENSUS_DEBUG};
@@ -328,7 +328,7 @@ impl std::ops::Deref for Config {
 /// Parameters for BABE.
 pub struct BabeParams<B: BlockT, C, E, I, SO, SC, CAW> {
 	/// The keystore that manages the keys of the node.
-	pub keystore: KeyStorePtr,
+	pub keystore: SyncCryptoStorePtr,
 
 	/// The client to use
 	pub client: Arc<C>,
@@ -468,7 +468,7 @@ struct BabeSlotWorker<B: BlockT, C, E, I, SO> {
 	env: E,
 	sync_oracle: SO,
 	force_authoring: bool,
-	keystore: KeyStorePtr,
+	keystore: SyncCryptoStorePtr,
 	epoch_changes: SharedEpochChanges<B, Epoch>,
 	slot_notification_sinks: SlotNotificationSinks<B>,
 	config: Config,
@@ -597,15 +597,15 @@ impl<B, C, E, I, Error, SO> sc_consensus_slots::SimpleSlotWorker<B> for BabeSlot
 			// add it to a digest item.
 			let public_type_pair = public.clone().into();
 			let public = public.to_raw_vec();
-			let signature = keystore.read()
-				.sign_with(
-					<AuthorityId as AppKey>::ID,
-					&public_type_pair,
-					header_hash.as_ref()
-				)
-				.map_err(|e| sp_consensus::Error::CannotSign(
-					public.clone(), e.to_string(),
-				))?;
+			let signature = SyncCryptoStore::sign_with(
+				&*keystore,
+				<AuthorityId as AppKey>::ID,
+				&public_type_pair,
+				header_hash.as_ref()
+			)
+			.map_err(|e| sp_consensus::Error::CannotSign(
+				public.clone(), e.to_string(),
+			))?;
 			let signature: AuthoritySignature = signature.clone().try_into()
 				.map_err(|_| sp_consensus::Error::InvalidSignature(
 					signature, public
@@ -1492,7 +1492,7 @@ pub mod test_helpers {
 		slot_number: u64,
 		parent: &B::Header,
 		client: &C,
-		keystore: &KeyStorePtr,
+		keystore: SyncCryptoStorePtr,
 		link: &BabeLink<B>,
 	) -> Option<PreDigest> where
 		B: BlockT,
@@ -1514,7 +1514,7 @@ pub mod test_helpers {
 		authorship::claim_slot(
 			slot_number,
 			&epoch,
-			keystore,
+			&keystore,
 		).map(|(digest, _)| digest)
 	}
 }

substrate/client/consensus/babe/src/tests.rs

@@ -21,7 +21,11 @@
 #![allow(deprecated)]
 use super::*;
 use authorship::claim_slot;
-use sp_core::{crypto::Pair, vrf::make_transcript as transcript_from_data};
+use sp_core::crypto::Pair;
+use sp_keystore::{
+	SyncCryptoStore,
+	vrf::make_transcript as transcript_from_data,
+};
 use sp_consensus_babe::{
 	AuthorityPair,
 	SlotNumber,
@@ -46,6 +50,8 @@ use rand_chacha::{
 	rand_core::SeedableRng,
 	ChaChaRng,
 };
+use sc_keystore::LocalKeystore;
+use sp_application_crypto::key_types::BABE;
 
 type Item = DigestItem<Hash>;
 
@@ -384,8 +390,9 @@ fn run_one_test(
 		let select_chain = peer.select_chain().expect("Full client has select_chain");
 
 		let keystore_path = tempfile::tempdir().expect("Creates keystore path");
-		let keystore = sc_keystore::Store::open(keystore_path.path(), None).expect("Creates keystore");
-		keystore.write().insert_ephemeral_from_seed::<AuthorityPair>(seed).expect("Generates authority key");
+		let keystore: SyncCryptoStorePtr = Arc::new(LocalKeystore::open(keystore_path.path(), None)
+			.expect("Creates keystore"));
+		SyncCryptoStore::sr25519_generate_new(&*keystore, BABE, Some(seed)).expect("Generates authority key");
 		keystore_paths.push(keystore_path);
 
 		let mut got_own = false;
@@ -432,7 +439,6 @@ fn run_one_test(
 			can_author_with: sp_consensus::AlwaysCanAuthor,
 		}).expect("Starts babe"));
 	}
-
 	futures::executor::block_on(future::select(
 		futures::future::poll_fn(move |cx| {
 			let mut net = net.lock();
@@ -516,14 +522,15 @@ fn sig_is_not_pre_digest() {
 fn can_author_block() {
 	sp_tracing::try_init_simple();
 	let keystore_path = tempfile::tempdir().expect("Creates keystore path");
-	let keystore = sc_keystore::Store::open(keystore_path.path(), None).expect("Creates keystore");
-	let pair = keystore.write().insert_ephemeral_from_seed::<AuthorityPair>("//Alice")
+	let keystore: SyncCryptoStorePtr = Arc::new(LocalKeystore::open(keystore_path.path(), None)
+		.expect("Creates keystore"));
+	let public = SyncCryptoStore::sr25519_generate_new(&*keystore, BABE, Some("//Alice"))
 		.expect("Generates authority pair");
 
 	let mut i = 0;
 	let epoch = Epoch {
 		start_slot: 0,
-		authorities: vec![(pair.public(), 1)],
+		authorities: vec![(public.into(), 1)],
 		randomness: [0; 32],
 		epoch_index: 1,
 		duration: 100,
@@ -823,13 +830,14 @@ fn verify_slots_are_strictly_increasing() {
 fn babe_transcript_generation_match() {
 	sp_tracing::try_init_simple();
 	let keystore_path = tempfile::tempdir().expect("Creates keystore path");
-	let keystore = sc_keystore::Store::open(keystore_path.path(), None).expect("Creates keystore");
-	let pair = keystore.write().insert_ephemeral_from_seed::<AuthorityPair>("//Alice")
+	let keystore: SyncCryptoStorePtr = Arc::new(LocalKeystore::open(keystore_path.path(), None)
+		.expect("Creates keystore"));
+	let public = SyncCryptoStore::sr25519_generate_new(&*keystore, BABE, Some("//Alice"))
 		.expect("Generates authority pair");
 
 	let epoch = Epoch {
 		start_slot: 0,
-		authorities: vec![(pair.public(), 1)],
+		authorities: vec![(public.into(), 1)],
 		randomness: [0; 32],
 		epoch_index: 1,
 		duration: 100,