Generate runtime API from metadata (#294)

* Remove test macro * Remove client crate * Create tests crate and move pallet specific tests there * Extract client, remove metadata and extra, more demolition * Update substrate dependencies to git dependencies * Remove Store stuff for now * Comment out some Call usages * Add back Runtime trait coped from original System trait * Make subxt lib compile * Delete old proc macros and copy over type generation from chameleon * WIP make transfer balance test pass * Change to subxt attribute macro * WIP provide user defined type substitutes * User defined type substitutes compile * WIP submitting transactions * WIP transfer balance test * Fix macro * Cargo fmt * WIP generating storage hashers * WIP add AccountData trait for fetching the nonce * Support single type storage map keys * WIP impl AccountInfo retrieval * Fix up storage struct generation * Implement AccountData triait directly on storage entry * Borrow storage map key and convert account id * Implement storage fetch client methods * Remove legacy metadata storage key construction * Rename CheckEra to CheckMortality * Substitute perthings types for compact impls * Fmt * Downgrade dyn-clone for cargo-contract compat * Scale-fo 1.0 * scale-info 1.0 * Remove special range handling * Restore wildcard type params * Frame metadata 14.0 * WIP decoding events * WIP more dynamically decoding events * Fmt * Decode events, handle errors * Uncomment some tests * Remove unused get_mod function * Fix some warnings * Fix some more warnings * Fix some more warnings * Add tests mod * Rename node-runtime tests mod to frame * Fix some warnings * Fmt * WIP generate storage client with getters * Storage client compiling * Generate storage client api * Fix up system account query account ids * WIP generating tx api fns * Only generate tx api fields when calls available * Fix tx api call fns * Fmt * WIP generate event structs * call functions not async * Derive Eq for comparison on generated types * Generate event structs * Fix call name * Fmt * Update node runtime metadata to substrate c000780db * Download latest substrate release for integration testing * Fix event decoding * Remove unused imports * Fix plain storage access, total_issuance pass * Fmt * Restore contracts tests * Backoff connecting to substrate node * Add required TypeInfo impls for local SignedExtension impls * Remove unnecessary assert formatting * Fix handling of DispatchError * Refactor contracts tests * Troubleshooting contract not found * Remove more client feature stuff * Fix dynamic event variant decoding, write consumed index to output * Fmt * Use substrate branch with heavy dependency removed * Remove sp-rcp dependency, define types locally * Ignore cargo timeing files * Use my branch for substrate test deps * Fix storage key type gen * Comment out fetching contract info * Add key iteration, extract storage client from main client * Debugging key generation * Use substrate master branch * Fix call test * Remove TypeSegmenter and dynclone dependency * Publicly expose Rpc mod * Unused import warnings * Add getter for runtime metadata * Add pallet and event indices for raw events * Add is_call and is_event convenience trait functions * Add missing docs * Refactor tests crate * Restore remaining client tests * Fmt * Fix warnings * Restore get_mod as test helper and fmt * Use client references for api calls * Fix api usages with methods * Use Bytes for RawEvent debug * Update metadata * Restoring some Balances tests * Populate runtime storage metadata * Restore balances lock test * Restore Balances error test * Fmt * Restore transfer subscription API * Staking test * Restore another staking test * Restore another staking test * Restore another staking test * Partially restore chill_works_for_controller_only staking test * Fix fetching Optional storage entries * Restore staking bond test * Restore remaining staking tests * Fmt * Restore sudo tests * Add some system tests * Fmt * Resolve some todos * Remove pass through rpc methods on Client, expose via rpc() getter * Remove more rpc pass through methods * Remove submit tx pass through rpc methods * Add some comments to SubmittableExtrinsic methods * Construct the runtime api from the client * Fmt * Use From trait instead of new for AccountData query * Rename subxt_proc_macro crate to subxt_macro * Fix AccountData From impl * Extract codegen crate from macro crate * Fmt * Replace chameleon hidden field name * Extract StructDef for generating structs * More refactoring of StructDef, moving towards sharing with typegen * Replace explicit tests crate with single implicit integration tests crate * Rename from substrate-subxt to subxt * Fix runtime path relative to root Cargo.toml * Move RpcClient creation to RpcClient * WIP get examples to compile * Rename Runtime to Config trait * WIP implementing default Config * WIP implementing default extrinsic extras * fix metadata constants (#299) * Move DefaultConfig definition and impl to macro * Extract type substitute parsing to ir mod * Extract calls, events and storage from api generation * Add some hardcoded type substitute defaults * Fmt * Add utility pallet tests (#300) * add batch call test example * add pallet utility tests * add utility module * fix warnings * Add polkadot runtime metadata for example * Fix system errors and fmt * Add subxt-cli crate * Add metadata and codegen subcommands * Make subxt-cli codegen command work * Fmt * Add polkadot codegen test * Comment about how to run codegen * Derive AsCompact for structs with single concrete unsigned int field * Fix bitvec codegen, adds as non optional dependency * Regenerate polkadot api with bitvec fix * Edition 2021 * Fix polkadot codegen with bitvec * Polkadot balance transfer is working * Fix fetch remote * Fix transfer_subscribe example * Fix submit_and_watch example * Fmt * Generate storage iter method for iterating over keys * Fmt * Fix existential deposit test * Fix staking tests * Add option for custom generated type derives * Add generated type derives for test runtime api * Fmt * Copy WrapperTypeOpaque from substrate, add Encode/Decode * Fmt * Extract type generator to module, separate & fix tests * Fully qualified primitive and prelude types * Fix up remaining type gen tests * Skip formatting of generated polkadot example code * Remove empty utility test file. * Newline * Update cli/src/main.rs Co-authored-by: David <dvdplm@gmail.com> * Rename subxt-cli executable to subxt * Update src/client.rs Co-authored-by: David <dvdplm@gmail.com> * Add some code docs to TypeGenerator. * Extract TypePath to own file * Extract type def generation to separate file * Renamed ModuleType to TypeDefGen * Fmt * Factor out type parameter from final_key * Fix some type paths * Resolve some todos * Resolve some panic todos in events * Add EventsDecodingError * Decode compact composite types with a single primitive field * Decode compact composite types with a single primitive field * Update src/metadata.rs Co-authored-by: Andrew Plaza <aplaza@liquidthink.net> * Remove Perbill compact substitute types * Remove todos regarding maintaining Rust code items, promoted to follow up issue. * Remove todo regarding overridding default config impl * Remove todo regarding overridding default Extra * Remove todo regarding AccountData storage type defintion * Remove todo regarding borrowing storage key arguments * Remove type substitution tests todo * Remove `Box` field name type hack todo * Remove Compact todo * Remove sudo todos * Remove BitVec implementation todo * Fmt * Add health warning to README * Fix up health warning Co-authored-by: Paulo Martins <paulormart@users.noreply.github.com> Co-authored-by: David <dvdplm@gmail.com> Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>
2026-04-22 23:08:00 +00:00 · 2021-11-03 11:28:59 +00:00
parent 8f0641e660
commit 793c945fbd
80 changed files with 5929 additions and 6475 deletions
@@ -0,0 +1,157 @@
+// Copyright 2019-2021 Parity Technologies (UK) Ltd.
+// This file is part of subxt.
+//
+// subxt is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// subxt is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with subxt.  If not, see <http://www.gnu.org/licenses/>.
+
+use color_eyre::eyre::{
+    self,
+    WrapErr,
+};
+use frame_metadata::RuntimeMetadataPrefixed;
+use scale::{
+    Decode,
+    Input,
+};
+use std::{
+    fs,
+    io::{
+        self,
+        Read,
+        Write,
+    },
+    path::PathBuf,
+};
+use structopt::StructOpt;
+
+/// Utilities for working with substrate metadata for subxt.
+#[derive(Debug, StructOpt)]
+struct Opts {
+    #[structopt(subcommand)]
+    command: Command,
+}
+
+#[derive(Debug, StructOpt)]
+enum Command {
+    /// Download metadata from a substrate node, for use with `subxt` codegen.
+    #[structopt(name = "metadata")]
+    Metadata {
+        /// the url of the substrate node to query for metadata
+        #[structopt(
+            name = "url",
+            long,
+            parse(try_from_str),
+            default_value = "http://localhost:9933"
+        )]
+        url: url::Url,
+        /// the format of the metadata to display: `json`, `hex` or `bytes`
+        #[structopt(long, short, default_value = "json")]
+        format: String,
+    },
+    /// Generate runtime API client code from metadata.
+    ///
+    /// # Example (with code formatting)
+    ///
+    /// `subxt codegen | rustfmt --edition=2018 --emit=stdout`
+    Codegen {
+        /// the url of the substrate node to query for metadata for codegen.
+        #[structopt(name = "url", long, parse(try_from_str))]
+        url: Option<url::Url>,
+        /// the path to the encoded metadata file.
+        #[structopt(short, long, parse(from_os_str))]
+        file: Option<PathBuf>,
+    },
+}
+
+fn main() -> color_eyre::Result<()> {
+    color_eyre::install()?;
+    let args = Opts::from_args();
+
+    match args.command {
+        Command::Metadata { url, format } => {
+            let (hex_data, bytes) = fetch_metadata(&url)?;
+
+            match format.as_str() {
+                "json" => {
+                    let metadata =
+                        <RuntimeMetadataPrefixed as Decode>::decode(&mut &bytes[..])?;
+                    let json = serde_json::to_string_pretty(&metadata)?;
+                    println!("{}", json);
+                    Ok(())
+                }
+                "hex" => {
+                    println!("{}", hex_data);
+                    Ok(())
+                }
+                "bytes" => Ok(io::stdout().write_all(&bytes)?),
+                _ => {
+                    Err(eyre::eyre!(
+                        "Unsupported format `{}`, expected `json`, `hex` or `bytes`",
+                        format
+                    ))
+                }
+            }
+        }
+        Command::Codegen { url, file } => {
+            if let Some(file) = file.as_ref() {
+                if url.is_some() {
+                    eyre::bail!("specify one of `--url` or `--file` but not both")
+                };
+
+                let mut file = fs::File::open(file)?;
+                let mut bytes = Vec::new();
+                file.read_to_end(&mut bytes)?;
+                codegen(&mut &bytes[..])?;
+                return Ok(())
+            }
+
+            let url = url.unwrap_or_else(|| {
+                url::Url::parse("http://localhost:9933").expect("default url is valid")
+            });
+            let (_, bytes) = fetch_metadata(&url)?;
+            codegen(&mut &bytes[..])?;
+            return Ok(())
+        }
+    }
+}
+
+fn fetch_metadata(url: &url::Url) -> color_eyre::Result<(String, Vec<u8>)> {
+    let resp = ureq::post(url.as_str())
+        .set("Content-Type", "application/json")
+        .send_json(ureq::json!({
+            "jsonrpc": "2.0",
+            "method": "state_getMetadata",
+            "id": 1
+        }))
+        .context("error fetching metadata from the substrate node")?;
+    let json: serde_json::Value = resp.into_json()?;
+
+    let hex_data = json["result"]
+        .as_str()
+        .map(ToString::to_string)
+        .ok_or(eyre::eyre!("metadata result field should be a string"))?;
+    let bytes = hex::decode(hex_data.trim_start_matches("0x"))?;
+
+    Ok((hex_data, bytes))
+}
+
+fn codegen<I: Input>(encoded: &mut I) -> color_eyre::Result<()> {
+    let metadata = <RuntimeMetadataPrefixed as Decode>::decode(encoded)?;
+    let generator = subxt_codegen::RuntimeGenerator::new(metadata);
+    let item_mod = syn::parse_quote!(
+        pub mod api {}
+    );
+    let runtime_api = generator.generate_runtime(item_mod, Default::default());
+    println!("{}", runtime_api);
+    Ok(())
+}