Support for multiple signature scheme for BEEFY primitves (#14373)

* Merged BEEFY primitives with generic signature and keyset commitment support from old pull to current code

* - Add bls-experimental feature to application-crypto and beefy primitives
- Fix remaining crypto -> ecdsa_crypto
- code build but not tests

* Make beefy primitive tests compile

* move bls related beefy primitives code and test behind bls-experimental flag

* Make BEEFY clients complies with BEEFY API depending on AuthorityId

* - Rename `BeefyAuthoritySet.root` → `BeefyAuthoritySet.keyset_commitment`.
- Remove apk proof keyset_commitment from `BeefyAuthoritySet`.
- Fix failing signed commitment and signature to witness test.
- Make client compatible with BeefyAPI generic on AuthorityId.
- `crypto` → `ecdsa_crypto` in BEEFY client and frame.

* Commit Cargo lock remove ark-serialize from BEEFY primitives

* Use Codec instead of Encode + Decode in primitives/consensus/beefy/src/lib.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* - Make `BeefyApi` generic over Signature type.
- Make new `BeeyApi` functinos also generic over AuthorityId and Signature

* Unmake BeefyAPI generic over Signature. Recover Signature type from AuthId.

* - dont use hex or hex-literal use array-bytes instead in beefy primitives and bls crypto.
- CamelCase ECDSA and BLS everywhere.

* Move the definition of BEEFY key type from `primitives/beefy` to `crypto.rs` according to new convention.

* - Add bls377_generate_new to `sp-io` and `application_crypto::bls`.
- Add `bls-experimental` to `sp-io`

Does not compile because PassByCodec can not derive PassBy using customly implemented PassByIner.

* Implement PassBy for `bls::Public` manually

* fix Beefy `KEY_TYPE` in `frame/beefy` tests to come from `sp-core::key_types` enum

* specify both generic for `hex2array_unchecked` in `sp-core/bls.rs`

* Rename `crypto`→`ecdsa_crypto` in `primitives/consensus/beefy/src/test_utils.rs` docs

* remove commented-out code in `primitives/consensus/beefy/src/commitment.rs`

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Fix inconsistency in panic message in  `primitives/io/src/lib.rs`

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Remove redundant feature activation in `primitives/io/Cargo.toml`

Co-authored-by: Davide Galassi <davxy@datawok.net>

* - make `w3f-bls` a dev-dependancy only for beefy primitives.

- clean up comments.

Co-authored-by: Davide Galassi <davxy@datawok.net>

* export BEEFY KEY_TYPE from primitives/consensus/beefy
make `frame/consensus/beefy` in dependent of sp_crypto_app
use consistent naming in the beefy primitive tests.

* - implement `BeefyAuthorityId` for `bls_crypto::AuthorityId`.
- implement `bls_verify_works` test for BEEFY `bls_crypto`.

* Remove BEEFY `ecdsa_n_bls_crypto` for now for later re-introduction

* Make commitment and witness BEEFY tests not use Keystore.

* put `bls_beefy_verify_works` test under `bls-experimental` flag.

* bump up Runtime `BeefyAPI` to version 3 due to introducing generic AuthorityId.

* reuse code and encapsulate w3f-bls backend in sp-core as most as possible

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Make comments in primities BEEFY `commitment.rs` and `witness.rs``tests convention conforming

* Use master dep versions

* Trivial change. Mostly to trigger CI

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fix Cargo.toml

* Trigger CI with cumulus companion

* Trigger CI after polkadot companion change

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

substrate/client/consensus/beefy/src/lib.rs

@@ -46,7 +46,8 @@ use sp_blockchain::{
 };
 use sp_consensus::{Error as ConsensusError, SyncOracle};
 use sp_consensus_beefy::{
-	crypto::AuthorityId, BeefyApi, MmrRootHash, PayloadProvider, ValidatorSet, BEEFY_ENGINE_ID,
+	ecdsa_crypto::AuthorityId, BeefyApi, MmrRootHash, PayloadProvider, ValidatorSet,
+	BEEFY_ENGINE_ID,
 };
 use sp_keystore::KeystorePtr;
 use sp_mmr_primitives::MmrApi;
@@ -142,7 +143,7 @@ where
 		+ Send
 		+ Sync,
 	RuntimeApi: ProvideRuntimeApi<B> + Send + Sync,
-	RuntimeApi::Api: BeefyApi<B>,
+	RuntimeApi::Api: BeefyApi<B, AuthorityId>,
 {
 	// Voter -> RPC links
 	let (to_rpc_justif_sender, from_voter_justif_stream) =
@@ -224,7 +225,7 @@ pub async fn start_beefy_gadget<B, BE, C, N, P, R, S>(
 	C: Client<B, BE> + BlockBackend<B>,
 	P: PayloadProvider<B>,
 	R: ProvideRuntimeApi<B>,
-	R::Api: BeefyApi<B> + MmrApi<B, MmrRootHash, NumberFor<B>>,
+	R::Api: BeefyApi<B, AuthorityId> + MmrApi<B, MmrRootHash, NumberFor<B>>,
 	N: GossipNetwork<B> + NetworkRequest + Send + Sync + 'static,
 	S: GossipSyncing<B> + SyncOracle + 'static,
 {
@@ -339,7 +340,7 @@ where
 	B: Block,
 	BE: Backend<B>,
 	R: ProvideRuntimeApi<B>,
-	R::Api: BeefyApi<B>,
+	R::Api: BeefyApi<B, AuthorityId>,
 {
 	// Initialize voter state from AUX DB if compatible.
 	crate::aux_schema::load_persistent(backend)?
@@ -374,7 +375,7 @@ where
 	B: Block,
 	BE: Backend<B>,
 	R: ProvideRuntimeApi<B>,
-	R::Api: BeefyApi<B>,
+	R::Api: BeefyApi<B, AuthorityId>,
 {
 	let beefy_genesis = runtime
 		.runtime_api()
@@ -478,7 +479,7 @@ async fn wait_for_runtime_pallet<B, R>(
 where
 	B: Block,
 	R: ProvideRuntimeApi<B>,
-	R::Api: BeefyApi<B>,
+	R::Api: BeefyApi<B, AuthorityId>,
 {
 	info!(target: LOG_TARGET, "🥩 BEEFY gadget waiting for BEEFY pallet to become available...");
 	loop {
@@ -518,7 +519,7 @@ fn expect_validator_set<B, R>(
 where
 	B: Block,
 	R: ProvideRuntimeApi<B>,
-	R::Api: BeefyApi<B>,
+	R::Api: BeefyApi<B, AuthorityId>,
 {
 	runtime
 		.runtime_api()