Utilize Metadata V15 (#1041)

* Update frame-metadata to the latest branch

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Add outer enum types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Extend the extrinsic with address,call,sign,extra types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Codegen test Event, Error and Call for outer enums

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Revert "Codegen test Event, Error and Call for outer enums"

This reverts commit db542dca0369eedd257a7ec031d5b5549bc46a88.

* Update frame-metadata from the latest release

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update scale-info

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen/error: Support v15 message

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Convert v14 to v15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/retain: Adjust to extrinsic type for V15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/validation: Adjust hashing for extrinsic types V15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* scripts: Fetch V15 and output codegen for full_client only

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt/blocks: Use extrinsic types directly

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Fetch V15 for build script

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Generate from latest polkadot version

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Fetch legacy with old API for v14 only

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc: Fetch metadata versions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client: Fetch latest unstable then V15 then V14

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Adjust testing API to latest interface

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Generate the `RuntimeError` type for V14

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove testing files

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing/staking: Remove controller account from bond

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/validation: Use specific variants for hashing RuntimeCall

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* XXX: Custom Substrate binary: must revert with next release

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* XXX: To revert: CI use hardcoded substrate

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Use v15 outer enum types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Retain outer enum types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Use outer enum types instead of generating them

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update artifacts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Revert "XXX: Custom Substrate binary: must revert with next release"

This reverts commit e9705298661919f5769720b35030759fb8a7b01d.

Revert "XXX: To revert: CI use hardcoded substrate"

This reverts commit b18a5a0985a56ee4ad99bc9a1c0f9cd733cf4271.

* testing: Include env for dummy wat contracts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjsut clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Use new link for fetching latest substrate binary

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* tests: Include dummy RuntimeEvent into test metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Bump light-client timeout tests to 25min

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/validation: Use specific pallets as provided

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Rename metadata constant

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Use call_ty instead of signature_ty

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Rename retaining variant function

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Use Option<&[&str]>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* online_client: Fetch V15 metadata explicitely

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/validation: Include the hash of the outer enum types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Fix sign typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Update the artifacts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Remove RootError RootEvent and RootExtrinsic traits

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update polkadot.rs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/tests: Ensure outer enum variants are retained

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* scripts: Include multiple pallets for our decoding purposes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Apply clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Update small metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* error: Keep raw bytes for the ModuleError representation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* error: Modify docs to not include links

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt/tests: Propagate `RuntimeCall` to outer enums

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Provide proper byte slice for decoding

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update artifacts

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli/tests: Adjust expected pallets message

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Test conversion from v14 to v15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Fix typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update subxt/src/blocks/extrinsic_types.rs

Co-authored-by: James Wilson <james@jsdw.me>

* metadata: Simplify type path for RuntimeError

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/validation: Use visited ids per outer enum

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* error: Remove RawModuleError

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Fix new clippy error from updated rust version

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>
This commit is contained in:
Alexandru Vasile
2023-07-17 12:52:02 +03:00
committed by GitHub
parent 4fb051f233
commit 78a106f059
33 changed files with 3283 additions and 9056 deletions
+66 -26
View File
@@ -5,7 +5,8 @@
//! Utility functions to generate a subset of the metadata.
use crate::{
ExtrinsicMetadata, Metadata, PalletMetadataInner, RuntimeApiMetadataInner, StorageEntryType,
ExtrinsicMetadata, Metadata, OuterEnumsMetadata, PalletMetadataInner, RuntimeApiMetadataInner,
StorageEntryType,
};
use scale_info::TypeDef;
use std::collections::{BTreeMap, HashSet};
@@ -82,7 +83,10 @@ fn update_pallet_types(pallet: &mut PalletMetadataInner, map_ids: &BTreeMap<u32,
/// Collect all type IDs needed to represent the extrinsic metadata.
fn collect_extrinsic_types(extrinsic: &ExtrinsicMetadata, type_ids: &mut HashSet<u32>) {
type_ids.insert(extrinsic.ty);
type_ids.insert(extrinsic.address_ty);
type_ids.insert(extrinsic.call_ty);
type_ids.insert(extrinsic.signature_ty);
type_ids.insert(extrinsic.extra_ty);
for signed in &extrinsic.signed_extensions {
type_ids.insert(signed.extra_ty);
@@ -92,7 +96,10 @@ fn collect_extrinsic_types(extrinsic: &ExtrinsicMetadata, type_ids: &mut HashSet
/// Update all type IDs of the provided extrinsic metadata using the new type IDs from the portable registry.
fn update_extrinsic_types(extrinsic: &mut ExtrinsicMetadata, map_ids: &BTreeMap<u32, u32>) {
update_type(&mut extrinsic.ty, map_ids);
update_type(&mut extrinsic.address_ty, map_ids);
update_type(&mut extrinsic.call_ty, map_ids);
update_type(&mut extrinsic.signature_ty, map_ids);
update_type(&mut extrinsic.extra_ty, map_ids);
for signed in &mut extrinsic.signed_extensions {
update_type(&mut signed.extra_ty, map_ids);
@@ -122,6 +129,20 @@ fn update_runtime_api_types(apis: &mut [RuntimeApiMetadataInner], map_ids: &BTre
}
}
/// Collect the outer enums type IDs.
fn collect_outer_enums(enums: &OuterEnumsMetadata, type_ids: &mut HashSet<u32>) {
type_ids.insert(enums.call_enum_ty);
type_ids.insert(enums.event_enum_ty);
type_ids.insert(enums.error_enum_ty);
}
/// Update all the type IDs for outer enums.
fn update_outer_enums(enums: &mut OuterEnumsMetadata, map_ids: &BTreeMap<u32, u32>) {
update_type(&mut enums.call_enum_ty, map_ids);
update_type(&mut enums.event_enum_ty, map_ids);
update_type(&mut enums.error_enum_ty, map_ids);
}
/// Update the given type using the new type ID from the portable registry.
///
/// # Panics
@@ -136,38 +157,36 @@ fn update_type(ty: &mut u32, map_ids: &BTreeMap<u32, u32>) {
*ty = new_id;
}
/// Strip any pallets out of the RuntimeCall type that aren't the ones we want to keep.
/// The RuntimeCall type is referenced in a bunch of places, so doing this prevents us from
/// holding on to stuff in pallets we've asked not to keep.
fn retain_pallets_in_runtime_call_type<F>(metadata: &mut Metadata, mut filter: F)
/// Retain the enum type identified by ID and keep only the variants that
/// match the provided filter.
fn retain_variants_in_enum_type<F>(metadata: &mut Metadata, id: u32, mut filter: F)
where
F: FnMut(&str) -> bool,
{
let extrinsic_ty = metadata
let ty = metadata
.types
.types
.get_mut(metadata.extrinsic.ty as usize)
.expect("Metadata should contain extrinsic type in registry");
.get_mut(id as usize)
.expect("Metadata should contain enum type in registry");
let Some(call_ty) = extrinsic_ty.ty.type_params
.iter_mut()
.find(|ty| ty.name == "Call")
.and_then(|ty| ty.ty) else { return; };
let call_ty = metadata
.types
.types
.get_mut(call_ty.id as usize)
.expect("Metadata should contain Call type information");
let TypeDef::Variant(variant) = &mut call_ty.ty.type_def else {
panic!("Metadata Call type is expected to be a variant type");
let TypeDef::Variant(variant) = &mut ty.ty.type_def else {
panic!("Metadata type is expected to be a variant type");
};
// Remove all variants from the call type that aren't the pallet(s) we want to keep.
// Remove all variants from the type that aren't the pallet(s) we want to keep.
variant.variants.retain(|v| filter(&v.name));
}
/// Strip any pallets out of the outer enum types that aren't the ones we want to keep.
fn retain_pallets_in_runtime_outer_types<F>(metadata: &mut Metadata, mut filter: F)
where
F: FnMut(&str) -> bool,
{
retain_variants_in_enum_type(metadata, metadata.outer_enums.call_enum_ty, &mut filter);
retain_variants_in_enum_type(metadata, metadata.outer_enums.event_enum_ty, &mut filter);
retain_variants_in_enum_type(metadata, metadata.outer_enums.error_enum_ty, &mut filter);
}
/// Generate a subset of the metadata that contains only the
/// types needed to represent the provided pallets and runtime APIs.
///
@@ -190,10 +209,13 @@ pub fn retain_metadata<F, G>(
{
let mut type_ids = HashSet::new();
// There is a special RuntimeCall type which points to all pallets and call types by default.
// There are special outer enum types that point to all pallets types (call, error, event) by default.
// This brings in a significant chunk of types. We trim this down to only include variants
// for the pallets we're retaining, to avoid this.
retain_pallets_in_runtime_call_type(metadata, &mut pallets_filter);
retain_pallets_in_runtime_outer_types(metadata, &mut pallets_filter);
// Collect the stripped outer enums.
collect_outer_enums(&metadata.outer_enums, &mut type_ids);
// Filter our pallet list to only those pallets we want to keep. Keep hold of all
// type IDs in the pallets we're keeping. Retain all, if no filter specified.
@@ -245,6 +267,7 @@ pub fn retain_metadata<F, G>(
let map_ids = metadata.types.retain(|id| type_ids.contains(&id));
// And finally, we can go and update all of our type IDs in the metadata as a result of this:
update_outer_enums(&mut metadata.outer_enums, &map_ids);
for pallets in metadata.pallets.values_mut() {
update_pallet_types(pallets, &map_ids);
}
@@ -257,6 +280,7 @@ pub fn retain_metadata<F, G>(
mod tests {
use super::*;
use crate::Metadata;
use assert_matches::assert_matches;
use codec::Decode;
use frame_metadata::{RuntimeMetadata, RuntimeMetadataPrefixed};
use std::{fs, path::Path};
@@ -281,6 +305,7 @@ mod tests {
// Retain one pallet at a time ensuring the test does not panic.
for pallet in metadata_cache.pallets() {
let mut metadata = metadata_cache.clone();
retain_metadata(
&mut metadata,
|pallet_name| pallet_name == pallet.name(),
@@ -292,6 +317,21 @@ mod tests {
&*metadata.pallets.get_by_index(0).unwrap().name,
pallet.name()
);
let id = metadata.outer_enums().call_enum_ty;
let ty = metadata.types.resolve(id).unwrap();
let num_variants = if pallet.call_ty_id().is_some() { 1 } else { 0 };
assert_matches!(&ty.type_def, TypeDef::Variant(variant) if variant.variants.len() == num_variants);
let id = metadata.outer_enums().error_enum_ty;
let ty = metadata.types.resolve(id).unwrap();
let num_variants = if pallet.error_ty_id().is_some() { 1 } else { 0 };
assert_matches!(&ty.type_def, TypeDef::Variant(variant) if variant.variants.len() == num_variants);
let id = metadata.outer_enums().event_enum_ty;
let ty = metadata.types.resolve(id).unwrap();
let num_variants = if pallet.event_ty_id().is_some() { 1 } else { 0 };
assert_matches!(&ty.type_def, TypeDef::Variant(variant) if variant.variants.len() == num_variants);
}
}
+97 -18
View File
@@ -5,10 +5,10 @@
//! Utility functions for metadata validation.
use crate::{
ExtrinsicMetadata, Metadata, PalletMetadata, RuntimeApiMetadata, RuntimeApiMethodMetadata,
StorageEntryMetadata, StorageEntryType,
ExtrinsicMetadata, Metadata, OuterEnumsMetadata, PalletMetadata, RuntimeApiMetadata,
RuntimeApiMethodMetadata, StorageEntryMetadata, StorageEntryType,
};
use scale_info::{form::PortableForm, Field, PortableRegistry, TypeDef, Variant};
use scale_info::{form::PortableForm, Field, PortableRegistry, TypeDef, TypeDefVariant, Variant};
use std::collections::HashSet;
/// Predefined value to be returned when we already visited a type.
@@ -104,6 +104,30 @@ fn get_variant_hash(
concat_and_hash2(&variant_name_bytes, &variant_field_bytes)
}
fn get_type_def_variant_hash(
registry: &PortableRegistry,
variant: &TypeDefVariant<PortableForm>,
only_these_variants: Option<&[&str]>,
visited_ids: &mut HashSet<u32>,
) -> [u8; HASH_LEN] {
let variant_id_bytes = [TypeBeingHashed::Variant as u8; HASH_LEN];
let variant_field_bytes = variant.variants.iter().fold([0u8; HASH_LEN], |bytes, var| {
// With EncodeAsType and DecodeAsType we no longer care which order the variants are in,
// as long as all of the names+types are there. XOR to not care about ordering.
let should_hash = only_these_variants
.as_ref()
.map(|only_these_variants| only_these_variants.contains(&var.name.as_str()))
.unwrap_or(true);
if should_hash {
xor(bytes, get_variant_hash(registry, var, visited_ids))
} else {
bytes
}
});
concat_and_hash2(&variant_id_bytes, &variant_field_bytes)
}
/// Obtain the hash representation of a `scale_info::TypeDef`.
fn get_type_def_hash(
registry: &PortableRegistry,
@@ -125,14 +149,7 @@ fn get_type_def_hash(
concat_and_hash2(&composite_id_bytes, &composite_field_bytes)
}
TypeDef::Variant(variant) => {
let variant_id_bytes = [TypeBeingHashed::Variant as u8; HASH_LEN];
let variant_field_bytes =
variant.variants.iter().fold([0u8; HASH_LEN], |bytes, var| {
// With EncodeAsType and DecodeAsType we no longer care which order the variants are in,
// as long as all of the names+types are there. XOR to not care about ordering.
xor(bytes, get_variant_hash(registry, var, visited_ids))
});
concat_and_hash2(&variant_id_bytes, &variant_field_bytes)
get_type_def_variant_hash(registry, variant, None, visited_ids)
}
TypeDef::Sequence(sequence) => concat_and_hash2(
&[TypeBeingHashed::Sequence as u8; HASH_LEN],
@@ -198,8 +215,16 @@ fn get_extrinsic_hash(
) -> [u8; HASH_LEN] {
let mut visited_ids = HashSet::<u32>::new();
let mut bytes = concat_and_hash2(
&get_type_hash(registry, extrinsic.ty, &mut visited_ids),
// Get the hashes of the extrinsic type.
let address_hash = get_type_hash(registry, extrinsic.address_ty, &mut visited_ids);
// The `RuntimeCall` type is intentionally omitted and hashed by the outer enums instead.
let signature_hash = get_type_hash(registry, extrinsic.signature_ty, &mut visited_ids);
let extra_hash = get_type_hash(registry, extrinsic.extra_ty, &mut visited_ids);
let mut bytes = concat_and_hash4(
&address_hash,
&signature_hash,
&extra_hash,
&[extrinsic.version; 32],
);
@@ -215,6 +240,39 @@ fn get_extrinsic_hash(
bytes
}
/// Obtain the hash representation of the `frame_metadata::v15::OuterEnums`.
fn get_outer_enums_hash(
registry: &PortableRegistry,
enums: &OuterEnumsMetadata,
only_these_variants: Option<&[&str]>,
) -> [u8; HASH_LEN] {
/// Hash the provided enum type.
fn get_enum_hash(
registry: &PortableRegistry,
id: u32,
only_these_variants: Option<&[&str]>,
) -> [u8; HASH_LEN] {
let ty = registry
.types
.get(id as usize)
.expect("Metadata should contain enum type in registry");
if let TypeDef::Variant(variant) = &ty.ty.type_def {
get_type_def_variant_hash(registry, variant, only_these_variants, &mut HashSet::new())
} else {
get_type_hash(registry, id, &mut HashSet::new())
}
}
let call_hash = get_enum_hash(registry, enums.call_enum_ty, only_these_variants);
let event_hash = get_enum_hash(registry, enums.event_enum_ty, only_these_variants);
let error_hash = get_enum_hash(registry, enums.error_enum_ty, only_these_variants);
concat_and_hash3(&call_hash, &event_hash, &error_hash)
}
/// Get the hash corresponding to a single storage entry.
fn get_storage_entry_hash(
registry: &PortableRegistry,
@@ -441,8 +499,6 @@ impl<'a> MetadataHasher<'a> {
/// Hash the given metadata.
pub fn hash(&self) -> [u8; HASH_LEN] {
let mut visited_ids = HashSet::<u32>::new();
let metadata = self.metadata;
let pallet_hash = metadata.pallets().fold([0u8; HASH_LEN], |bytes, pallet| {
@@ -480,9 +536,21 @@ impl<'a> MetadataHasher<'a> {
});
let extrinsic_hash = get_extrinsic_hash(&metadata.types, &metadata.extrinsic);
let runtime_hash = get_type_hash(&metadata.types, metadata.runtime_ty(), &mut visited_ids);
let runtime_hash =
get_type_hash(&metadata.types, metadata.runtime_ty(), &mut HashSet::new());
let outer_enums_hash = get_outer_enums_hash(
&metadata.types,
&metadata.outer_enums(),
self.specific_pallets.as_deref(),
);
concat_and_hash4(&pallet_hash, &apis_hash, &extrinsic_hash, &runtime_hash)
concat_and_hash5(
&pallet_hash,
&apis_hash,
&extrinsic_hash,
&runtime_hash,
&outer_enums_hash,
)
}
}
@@ -552,9 +620,12 @@ mod tests {
fn build_default_extrinsic() -> v15::ExtrinsicMetadata {
v15::ExtrinsicMetadata {
ty: meta_type::<()>(),
version: 0,
signed_extensions: vec![],
address_ty: meta_type::<()>(),
call_ty: meta_type::<()>(),
signature_ty: meta_type::<()>(),
extra_ty: meta_type::<()>(),
}
}
@@ -597,6 +668,14 @@ mod tests {
build_default_extrinsic(),
meta_type::<()>(),
vec![],
v15::OuterEnums {
call_enum_ty: meta_type::<()>(),
event_enum_ty: meta_type::<()>(),
error_enum_ty: meta_type::<()>(),
},
v15::CustomMetadata {
map: Default::default(),
},
)
.try_into()
.expect("can build valid metadata")