Metadata V15: Generate Runtime APIs (#918)

* Update frame-metadata to v15.1.0 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Enable V15 unstable metadata in frame-metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Move validation hashing to dedicated file Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Use sp-metadata-ir from substrate to work with metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert using sp-metadata-ir in favor of conversion to v15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Convert v14 to v15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Use v15 for validation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Use v15 for codegen Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/bench: Use v15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Adjust to v15 metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Adjust testing Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Improve documentation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * force CI Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * rpc: Fetch metadata at version Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Update polkadot.scale from commit 6dc9e84dde2 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Fetch V15 using the new API Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Add runtime API interface Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Hash runtime API metadata for validation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Extract runtime API metadata wrapper from subxt::Metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Adjust hashing cache to reflect root+item keys Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * rpc: Add raw state_call API method Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * runtime_api: Add payload with static and dynamic variants Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Allow payloads to call into the runtime Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * examples: Add example to make a runtime API call both static and dynamic Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update polkadot.rs Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Simplify client fetching Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Address feedback and fallback to old API if needed Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * runtime_api: Make mutability conditional on input params Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Regenerate polkadot.rs Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Retain only pallets without runtime API info Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Retry via `Metadata_metadata` without conversion Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * payload: Remove `Decode` and change validation fn Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Retain runtime API types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Runtime APIs documentation based on flag Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update examples/examples/custom_metadata_url.rs Co-authored-by: James Wilson <james@jsdw.me> * Update artifacts from polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update polkadot.rs with polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Generate input structures for runtime API Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * runtime_api: Remove the static paylaod and use single impl Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * examples: Fetch account nonce Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Adjust build script to fetch latest metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Check account nonce from runtime API Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update cargo.lock Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Fix doc generation for runtime types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Rename `inputs` runtime calls module to `types` Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Generate Calls structs inside the types module Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Check Alice account nonce before submitting the tx Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Add metadata version option flag supporting v14 and unstable Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Specify version to fetch Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Fallback to fetching latest stable metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Add unstable-metadata feature to fetch the latest Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * RuntimeVersion with Latest and Version(u32) Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update polkadot.rs Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Adjust fetch_metadata to inspect version list Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Adjust metadata to metadata_legacy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * events: Adjust docs to use metadata_legacy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * have a pass over fetch_metadata * cargo fmt * Option<String> when fetch metadata via latest API * clippy * fmt * cli: Use the MetadataVersion from codegen Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Specify latest as default for MetadataVersion Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli: Remove version from metadata and use the one from file_or_url Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Fix clippy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Decode metadata independently for different RPC calls Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: James Wilson <james@jsdw.me>
2026-04-25 17:37:56 +00:00 · 2023-05-03 17:31:27 +03:00
parent f4eb80e78d
commit 432e856c37
33 changed files with 13959 additions and 8390 deletions
@@ -3,6 +3,7 @@
 // see LICENSE for license details.

 use crate::error::FetchMetadataError;
+use codec::{Decode, Encode};
 use jsonrpsee::{
    async_client::ClientBuilder,
    client_transport::ws::{Uri, WsTransportClientBuilder},
@@ -12,14 +13,51 @@ use jsonrpsee::{
 };
 use std::time::Duration;

+/// The metadata version that is fetched from the node.
+#[derive(Default, Debug, Clone, Copy)]
+pub enum MetadataVersion {
+    /// Latest stable version of the metadata.
+    #[default]
+    Latest,
+    /// Fetch a specified version of the metadata.
+    Version(u32),
+    /// Latest unstable version of the metadata.
+    Unstable,
+}
+
+// Note: Implementation needed for the CLI tool.
+impl std::str::FromStr for MetadataVersion {
+    type Err = String;
+
+    fn from_str(input: &str) -> Result<Self, Self::Err> {
+        match input {
+            "unstable" => Ok(MetadataVersion::Unstable),
+            "latest" => Ok(MetadataVersion::Latest),
+            version => {
+                let num: u32 = version
+                    .parse()
+                    .map_err(|_| format!("Invalid metadata version specified {:?}", version))?;
+
+                Ok(MetadataVersion::Version(num))
+            }
+        }
+    }
+}
+
 /// Returns the metadata bytes from the provided URL, blocking the current thread.
-pub fn fetch_metadata_bytes_blocking(url: &Uri) -> Result<Vec<u8>, FetchMetadataError> {
-    tokio_block_on(fetch_metadata_bytes(url))
+pub fn fetch_metadata_bytes_blocking(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<Vec<u8>, FetchMetadataError> {
+    tokio_block_on(fetch_metadata_bytes(url, version))
 }

 /// Returns the raw, 0x prefixed metadata hex from the provided URL, blocking the current thread.
-pub fn fetch_metadata_hex_blocking(url: &Uri) -> Result<String, FetchMetadataError> {
-    tokio_block_on(fetch_metadata_hex(url))
+pub fn fetch_metadata_hex_blocking(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<String, FetchMetadataError> {
+    tokio_block_on(fetch_metadata_hex(url, version))
 }

 // Block on some tokio runtime for sync contexts
@@ -32,26 +70,36 @@ fn tokio_block_on<T, Fut: std::future::Future<Output = T>>(fut: Fut) -> T {
 }

 /// Returns the metadata bytes from the provided URL.
-pub async fn fetch_metadata_bytes(url: &Uri) -> Result<Vec<u8>, FetchMetadataError> {
-    let hex = fetch_metadata_hex(url).await?;
-    let bytes = hex::decode(hex.trim_start_matches("0x"))?;
-    Ok(bytes)
-}
-
-/// Returns the raw, 0x prefixed metadata hex from the provided URL.
-pub async fn fetch_metadata_hex(url: &Uri) -> Result<String, FetchMetadataError> {
-    let hex_data = match url.scheme_str() {
-        Some("http") | Some("https") => fetch_metadata_http(url).await,
-        Some("ws") | Some("wss") => fetch_metadata_ws(url).await,
+pub async fn fetch_metadata_bytes(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<Vec<u8>, FetchMetadataError> {
+    let bytes = match url.scheme_str() {
+        Some("http") | Some("https") => fetch_metadata_http(url, version).await,
+        Some("ws") | Some("wss") => fetch_metadata_ws(url, version).await,
        invalid_scheme => {
            let scheme = invalid_scheme.unwrap_or("no scheme");
            Err(FetchMetadataError::InvalidScheme(scheme.to_owned()))
        }
    }?;
+
+    Ok(bytes)
+}
+
+/// Returns the raw, 0x prefixed metadata hex from the provided URL.
+pub async fn fetch_metadata_hex(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<String, FetchMetadataError> {
+    let bytes = fetch_metadata_bytes(url, version).await?;
+    let hex_data = format!("0x{}", hex::encode(bytes));
    Ok(hex_data)
 }

-async fn fetch_metadata_ws(url: &Uri) -> Result<String, FetchMetadataError> {
+async fn fetch_metadata_ws(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<Vec<u8>, FetchMetadataError> {
    let (sender, receiver) = WsTransportClientBuilder::default()
        .build(url.to_string().parse::<Uri>().unwrap())
        .await
@@ -62,13 +110,124 @@ async fn fetch_metadata_ws(url: &Uri) -> Result<String, FetchMetadataError> {
        .max_notifs_per_subscription(4096)
        .build_with_tokio(sender, receiver);

-    Ok(client.request("state_getMetadata", rpc_params![]).await?)
+    fetch_metadata(client, version).await
 }

-async fn fetch_metadata_http(url: &Uri) -> Result<String, FetchMetadataError> {
+async fn fetch_metadata_http(
+    url: &Uri,
+    version: MetadataVersion,
+) -> Result<Vec<u8>, FetchMetadataError> {
    let client = HttpClientBuilder::default()
        .request_timeout(Duration::from_secs(180))
        .build(url.to_string())?;

-    Ok(client.request("state_getMetadata", rpc_params![]).await?)
+    fetch_metadata(client, version).await
+}
+
+/// The innermost call to fetch metadata:
+async fn fetch_metadata(
+    client: impl ClientT,
+    version: MetadataVersion,
+) -> Result<Vec<u8>, FetchMetadataError> {
+    const UNSTABLE_METADATA_VERSION: u32 = u32::MAX;
+
+    // Fetch metadata using the "new" state_call interface
+    async fn fetch_inner(
+        client: &impl ClientT,
+        version: MetadataVersion,
+    ) -> Result<Vec<u8>, FetchMetadataError> {
+        // Look up supported versions:
+        let supported_versions: Vec<u32> = {
+            let res: String = client
+                .request(
+                    "state_call",
+                    rpc_params!["Metadata_metadata_versions", "0x"],
+                )
+                .await?;
+            let raw_bytes = hex::decode(res.trim_start_matches("0x"))?;
+            Decode::decode(&mut &raw_bytes[..])?
+        };
+
+        // Return the version the user wants if it's supported:
+        let version = match version {
+            MetadataVersion::Latest => *supported_versions
+                .iter()
+                .filter(|&&v| v != UNSTABLE_METADATA_VERSION)
+                .max()
+                .ok_or_else(|| {
+                    FetchMetadataError::Other("No valid metadata versions returned".to_string())
+                })?,
+            MetadataVersion::Unstable => {
+                if supported_versions.contains(&UNSTABLE_METADATA_VERSION) {
+                    UNSTABLE_METADATA_VERSION
+                } else {
+                    return Err(FetchMetadataError::Other(
+                        "The node does not have an unstable metadata version available".to_string(),
+                    ));
+                }
+            }
+            MetadataVersion::Version(version) => {
+                if supported_versions.contains(&version) {
+                    version
+                } else {
+                    return Err(FetchMetadataError::Other(format!(
+                        "The node does not have version {version} available"
+                    )));
+                }
+            }
+        };
+
+        let bytes = version.encode();
+        let version: String = format!("0x{}", hex::encode(&bytes));
+
+        // Fetch the metadata at that version:
+        let metadata_string: String = client
+            .request(
+                "state_call",
+                rpc_params!["Metadata_metadata_at_version", &version],
+            )
+            .await?;
+        // Decode the metadata.
+        let metadata_bytes = hex::decode(metadata_string.trim_start_matches("0x"))?;
+        let metadata: Option<frame_metadata::OpaqueMetadata> =
+            Decode::decode(&mut &metadata_bytes[..])?;
+        let Some(metadata) = metadata else {
+            return Err(FetchMetadataError::Other(format!(
+                "The node does not have version {version} available"
+            )));
+        };
+        Ok(metadata.0)
+    }
+
+    // Fetch metadata using the "old" state_call interface
+    async fn fetch_inner_legacy(
+        client: &impl ClientT,
+        version: MetadataVersion,
+    ) -> Result<Vec<u8>, FetchMetadataError> {
+        if !matches!(
+            version,
+            MetadataVersion::Latest | MetadataVersion::Version(14)
+        ) {
+            return Err(FetchMetadataError::Other(
+                "The node can only return version 14 metadata but you've asked for something else"
+                    .to_string(),
+            ));
+        }
+
+        // Fetch the metadata at that version:
+        let metadata_string: String = client
+            .request("state_call", rpc_params!["Metadata_metadata", "0x"])
+            .await?;
+
+        // Decode the metadata.
+        let metadata_bytes = hex::decode(metadata_string.trim_start_matches("0x"))?;
+        let metadata: frame_metadata::OpaqueMetadata = Decode::decode(&mut &metadata_bytes[..])?;
+        Ok(metadata.0)
+    }
+
+    // Fetch using the new interface, falling back to trying old one if there's an error.
+    match fetch_inner(&client, version).await {
+        Ok(s) => Ok(s),
+        Err(_) => fetch_inner_legacy(&client, version).await,
+    }
 }
@@ -11,5 +11,5 @@ pub use jsonrpsee::client_transport::ws::Uri;

 pub use fetch_metadata::{
    fetch_metadata_bytes, fetch_metadata_bytes_blocking, fetch_metadata_hex,
-    fetch_metadata_hex_blocking,
+    fetch_metadata_hex_blocking, MetadataVersion,
 };