Generate runtime API from metadata (#294)

* Remove test macro

* Remove client crate

* Create tests crate and move pallet specific tests there

* Extract client, remove metadata and extra, more demolition

* Update substrate dependencies to git dependencies

* Remove Store stuff for now

* Comment out some Call usages

* Add back Runtime trait coped from original System trait

* Make subxt lib compile

* Delete old proc macros and copy over type generation from chameleon

* WIP make transfer balance test pass

* Change to subxt attribute macro

* WIP provide user defined type substitutes

* User defined type substitutes compile

* WIP submitting transactions

* WIP transfer balance test

* Fix macro

* Cargo fmt

* WIP generating storage hashers

* WIP add AccountData trait for fetching the nonce

* Support single type storage map keys

* WIP impl AccountInfo retrieval

* Fix up storage struct generation

* Implement AccountData triait directly on storage entry

* Borrow storage map key and convert account id

* Implement storage fetch client methods

* Remove legacy metadata storage key construction

* Rename CheckEra to CheckMortality

* Substitute perthings types for compact impls

* Fmt

* Downgrade dyn-clone for cargo-contract compat

* Scale-fo 1.0

* scale-info 1.0

* Remove special range handling

* Restore wildcard type params

* Frame metadata 14.0

* WIP decoding events

* WIP more dynamically decoding events

* Fmt

* Decode events, handle errors

* Uncomment some tests

* Remove unused get_mod function

* Fix some warnings

* Fix some more warnings

* Fix some more warnings

* Add tests mod

* Rename node-runtime tests mod to frame

* Fix some warnings

* Fmt

* WIP generate storage client with getters

* Storage client compiling

* Generate storage client api

* Fix up system account query account ids

* WIP generating tx api fns

* Only generate tx api fields when calls available

* Fix tx api call fns

* Fmt

* WIP generate event structs

* call functions not async

* Derive Eq for comparison on generated types

* Generate event structs

* Fix call name

* Fmt

* Update node runtime metadata to substrate c000780db

* Download latest substrate release for integration testing

* Fix event decoding

* Remove unused imports

* Fix plain storage access, total_issuance pass

* Fmt

* Restore contracts tests

* Backoff connecting to substrate node

* Add required TypeInfo impls for local SignedExtension impls

* Remove unnecessary assert formatting

* Fix handling of DispatchError

* Refactor contracts tests

* Troubleshooting contract not found

* Remove more client feature stuff

* Fix dynamic event variant decoding, write consumed index to output

* Fmt

* Use substrate branch with heavy dependency removed

* Remove sp-rcp dependency, define types locally

* Ignore cargo timeing files

* Use my branch for substrate test deps

* Fix storage key type gen

* Comment out fetching contract info

* Add key iteration, extract storage client from main client

* Debugging key generation

* Use substrate master branch

* Fix call test

* Remove TypeSegmenter and dynclone dependency

* Publicly expose Rpc mod

* Unused import warnings

* Add getter for runtime metadata

* Add pallet and event indices for raw events

* Add is_call and is_event convenience trait functions

* Add missing docs

* Refactor tests crate

* Restore remaining client tests

* Fmt

* Fix warnings

* Restore get_mod as test helper and fmt

* Use client references for api calls

* Fix api usages with methods

* Use Bytes for RawEvent debug

* Update metadata

* Restoring some Balances tests

* Populate runtime storage metadata

* Restore balances lock test

* Restore Balances error test

* Fmt

* Restore transfer subscription API

* Staking test

* Restore another staking test

* Restore another staking test

* Restore another staking test

* Partially restore chill_works_for_controller_only staking test

* Fix fetching Optional storage entries

* Restore staking bond test

* Restore remaining staking tests

* Fmt

* Restore sudo tests

* Add some system tests

* Fmt

* Resolve some todos

* Remove pass through rpc methods on Client, expose via rpc() getter

* Remove more rpc pass through methods

* Remove submit tx pass through rpc methods

* Add some comments to SubmittableExtrinsic methods

* Construct the runtime api from the client

* Fmt

* Use From trait instead of new for AccountData query

* Rename subxt_proc_macro crate to subxt_macro

* Fix AccountData From impl

* Extract codegen crate from macro crate

* Fmt

* Replace chameleon hidden field name

* Extract StructDef for generating structs

* More refactoring of StructDef, moving towards sharing with typegen

* Replace explicit tests crate with single implicit integration tests crate

* Rename from substrate-subxt to subxt

* Fix runtime path relative to root Cargo.toml

* Move RpcClient creation to RpcClient

* WIP get examples to compile

* Rename Runtime to Config trait

* WIP implementing default Config

* WIP implementing default extrinsic extras

* fix metadata constants (#299)

* Move DefaultConfig definition and impl to macro

* Extract type substitute parsing to ir mod

* Extract calls, events and storage from api generation

* Add some hardcoded type substitute defaults

* Fmt

* Add utility pallet tests (#300)

* add batch call test example

* add pallet utility tests

* add utility module

* fix warnings

* Add polkadot runtime metadata for example

* Fix system errors and fmt

* Add subxt-cli crate

* Add metadata and codegen subcommands

* Make subxt-cli codegen command work

* Fmt

* Add polkadot codegen test

* Comment about how to run codegen

* Derive AsCompact for structs with single concrete unsigned int field

* Fix bitvec codegen, adds as non optional dependency

* Regenerate polkadot api with bitvec fix

* Edition 2021

* Fix polkadot codegen with bitvec

* Polkadot balance transfer is working

* Fix fetch remote

* Fix transfer_subscribe example

* Fix submit_and_watch example

* Fmt

* Generate storage iter method for iterating over keys

* Fmt

* Fix existential deposit test

* Fix staking tests

* Add option for custom generated type derives

* Add generated type derives for test runtime api

* Fmt

* Copy WrapperTypeOpaque from substrate, add Encode/Decode

* Fmt

* Extract type generator to module, separate & fix tests

* Fully qualified primitive and prelude types

* Fix up remaining type gen tests

* Skip formatting of generated polkadot example code

* Remove empty utility test file.

* Newline

* Update cli/src/main.rs

Co-authored-by: David <dvdplm@gmail.com>

* Rename subxt-cli executable to subxt

* Update src/client.rs

Co-authored-by: David <dvdplm@gmail.com>

* Add some code docs to TypeGenerator.

* Extract TypePath to own file

* Extract type def generation to separate file

* Renamed ModuleType to TypeDefGen

* Fmt

* Factor out type parameter from final_key

* Fix some type paths

* Resolve some todos

* Resolve some panic todos in events

* Add EventsDecodingError

* Decode compact composite types with a single primitive field

* Decode compact composite types with a single primitive field

* Update src/metadata.rs

Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

* Remove Perbill compact substitute types

* Remove todos regarding maintaining Rust code items, promoted to follow up issue.

* Remove todo regarding overridding default config impl

* Remove todo regarding overridding default Extra

* Remove todo regarding AccountData storage type defintion

* Remove todo regarding borrowing storage key arguments

* Remove type substitution tests todo

* Remove `Box` field name type hack todo

* Remove Compact todo

* Remove sudo todos

* Remove BitVec implementation todo

* Fmt

* Add health warning to README

* Fix up health warning

Co-authored-by: Paulo Martins <paulormart@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

src/extrinsic/extra.rs

@@ -1,5 +1,5 @@
 // Copyright 2019-2021 Parity Technologies (UK) Ltd.
-// This file is part of substrate-subxt.
+// This file is part of subxt.
 //
 // subxt is free software: you can redistribute it and/or modify
 // it under the terms of the GNU General Public License as published by
@@ -12,7 +12,7 @@
 // GNU General Public License for more details.
 //
 // You should have received a copy of the GNU General Public License
-// along with substrate-subxt.  If not, see <http://www.gnu.org/licenses/>.
+// along with subxt.  If not, see <http://www.gnu.org/licenses/>.
 
 use codec::{
     Decode,
@@ -22,22 +22,17 @@ use core::{
     fmt::Debug,
     marker::PhantomData,
 };
+use scale_info::TypeInfo;
 use sp_runtime::{
     generic::Era,
     traits::SignedExtension,
     transaction_validity::TransactionValidityError,
 };
 
-use crate::{
-    frame::{
-        balances::Balances,
-        system::System,
-    },
-    runtimes::Runtime,
-};
+use crate::Config;
 
 /// Extra type.
-pub type Extra<T> = <<T as Runtime>::Extra as SignedExtra<T>>::Extra;
+// pub type Extra<T> = <<T as Config>::Extra as SignedExtra<T>>::Extra;
 
 /// SignedExtra checks copied from substrate, in order to remove requirement to implement
 /// substrate's `frame_system::Trait`
@@ -50,8 +45,9 @@ pub type Extra<T> = <<T as Runtime>::Extra as SignedExtra<T>>::Extra;
 /// returned via `additional_signed()`.
 
 /// Ensure the runtime version registered in the transaction is the same as at present.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckSpecVersion<T: System>(
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckSpecVersion<T: Config>(
     pub PhantomData<T>,
     /// Local version to be used for `AdditionalSigned`
     #[codec(skip)]
@@ -60,7 +56,7 @@ pub struct CheckSpecVersion<T: System>(
 
 impl<T> SignedExtension for CheckSpecVersion<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
     const IDENTIFIER: &'static str = "CheckSpecVersion";
     type AccountId = u64;
@@ -80,8 +76,9 @@ where
 ///
 /// This is modified from the substrate version to allow passing in of the version, which is
 /// returned via `additional_signed()`.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckTxVersion<T: System>(
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckTxVersion<T: Config>(
     pub PhantomData<T>,
     /// Local version to be used for `AdditionalSigned`
     #[codec(skip)]
@@ -90,7 +87,7 @@ pub struct CheckTxVersion<T: System>(
 
 impl<T> SignedExtension for CheckTxVersion<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
     const IDENTIFIER: &'static str = "CheckTxVersion";
     type AccountId = u64;
@@ -110,8 +107,9 @@ where
 ///
 /// This is modified from the substrate version to allow passing in of the genesis hash, which is
 /// returned via `additional_signed()`.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckGenesis<T: System>(
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckGenesis<T: Config>(
     pub PhantomData<T>,
     /// Local genesis hash to be used for `AdditionalSigned`
     #[codec(skip)]
@@ -120,7 +118,7 @@ pub struct CheckGenesis<T: System>(
 
 impl<T> SignedExtension for CheckGenesis<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
     const IDENTIFIER: &'static str = "CheckGenesis";
     type AccountId = u64;
@@ -141,8 +139,9 @@ where
 /// This is modified from the substrate version to allow passing in of the genesis hash, which is
 /// returned via `additional_signed()`. It assumes therefore `Era::Immortal` (The transaction is
 /// valid forever)
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckEra<T: System>(
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckMortality<T: Config>(
     /// The default structure for the Extra encoding
     pub (Era, PhantomData<T>),
     /// Local genesis hash to be used for `AdditionalSigned`
@@ -150,11 +149,11 @@ pub struct CheckEra<T: System>(
     pub T::Hash,
 );
 
-impl<T> SignedExtension for CheckEra<T>
+impl<T> SignedExtension for CheckMortality<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
-    const IDENTIFIER: &'static str = "CheckEra";
+    const IDENTIFIER: &'static str = "CheckMortality";
     type AccountId = u64;
     type Call = ();
     type AdditionalSigned = T::Hash;
@@ -167,12 +166,13 @@ where
 }
 
 /// Nonce check and increment to give replay protection for transactions.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckNonce<T: System>(#[codec(compact)] pub T::Index);
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckNonce<T: Config>(#[codec(compact)] pub T::Index);
 
 impl<T> SignedExtension for CheckNonce<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
     const IDENTIFIER: &'static str = "CheckNonce";
     type AccountId = u64;
@@ -187,12 +187,13 @@ where
 }
 
 /// Resource limit check.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct CheckWeight<T: System>(pub PhantomData<T>);
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct CheckWeight<T: Config>(pub PhantomData<T>);
 
 impl<T> SignedExtension for CheckWeight<T>
 where
-    T: System + Clone + Debug + Eq + Send + Sync,
+    T: Config + Clone + Debug + Eq + Send + Sync,
 {
     const IDENTIFIER: &'static str = "CheckWeight";
     type AccountId = u64;
@@ -208,13 +209,11 @@ where
 
 /// Require the transactor pay for themselves and maybe include a tip to gain additional priority
 /// in the queue.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct ChargeTransactionPayment<T: Balances>(#[codec(compact)] pub T::Balance);
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct ChargeTransactionPayment(#[codec(compact)] pub u128);
 
-impl<T> SignedExtension for ChargeTransactionPayment<T>
-where
-    T: Balances + Clone + Debug + Eq + Send + Sync,
-{
+impl SignedExtension for ChargeTransactionPayment {
     const IDENTIFIER: &'static str = "ChargeTransactionPayment";
     type AccountId = u64;
     type Call = ();
@@ -228,7 +227,7 @@ where
 }
 
 /// Trait for implementing transaction extras for a runtime.
-pub trait SignedExtra<T: System>: SignedExtension {
+pub trait SignedExtra<T: Config>: SignedExtension {
     /// The type the extras.
     type Extra: SignedExtension + Send + Sync;
 
@@ -245,25 +244,24 @@ pub trait SignedExtra<T: System>: SignedExtension {
 }
 
 /// Default `SignedExtra` for substrate runtimes.
-#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug)]
-pub struct DefaultExtra<T: System> {
+#[derive(Encode, Decode, Clone, Eq, PartialEq, Debug, TypeInfo)]
+#[scale_info(skip_type_params(T))]
+pub struct DefaultExtra<T: Config> {
     spec_version: u32,
     tx_version: u32,
     nonce: T::Index,
     genesis_hash: T::Hash,
 }
 
-impl<T: System + Balances + Clone + Debug + Eq + Send + Sync> SignedExtra<T>
-    for DefaultExtra<T>
-{
+impl<T: Config + Clone + Debug + Eq + Send + Sync> SignedExtra<T> for DefaultExtra<T> {
     type Extra = (
         CheckSpecVersion<T>,
         CheckTxVersion<T>,
         CheckGenesis<T>,
-        CheckEra<T>,
+        CheckMortality<T>,
         CheckNonce<T>,
         CheckWeight<T>,
-        ChargeTransactionPayment<T>,
+        ChargeTransactionPayment,
     );
 
     fn new(
@@ -285,17 +283,15 @@ impl<T: System + Balances + Clone + Debug + Eq + Send + Sync> SignedExtra<T>
             CheckSpecVersion(PhantomData, self.spec_version),
             CheckTxVersion(PhantomData, self.tx_version),
             CheckGenesis(PhantomData, self.genesis_hash),
-            CheckEra((Era::Immortal, PhantomData), self.genesis_hash),
+            CheckMortality((Era::Immortal, PhantomData), self.genesis_hash),
             CheckNonce(self.nonce),
             CheckWeight(PhantomData),
-            ChargeTransactionPayment(<T as Balances>::Balance::default()),
+            ChargeTransactionPayment(u128::default()),
         )
     }
 }
 
-impl<T: System + Balances + Clone + Debug + Eq + Send + Sync> SignedExtension
-    for DefaultExtra<T>
-{
+impl<T: Config + Clone + Debug + Eq + Send + Sync> SignedExtension for DefaultExtra<T> {
     const IDENTIFIER: &'static str = "DefaultExtra";
     type AccountId = T::AccountId;
     type Call = ();