Generate runtime API from metadata (#294)

* Remove test macro

* Remove client crate

* Create tests crate and move pallet specific tests there

* Extract client, remove metadata and extra, more demolition

* Update substrate dependencies to git dependencies

* Remove Store stuff for now

* Comment out some Call usages

* Add back Runtime trait coped from original System trait

* Make subxt lib compile

* Delete old proc macros and copy over type generation from chameleon

* WIP make transfer balance test pass

* Change to subxt attribute macro

* WIP provide user defined type substitutes

* User defined type substitutes compile

* WIP submitting transactions

* WIP transfer balance test

* Fix macro

* Cargo fmt

* WIP generating storage hashers

* WIP add AccountData trait for fetching the nonce

* Support single type storage map keys

* WIP impl AccountInfo retrieval

* Fix up storage struct generation

* Implement AccountData triait directly on storage entry

* Borrow storage map key and convert account id

* Implement storage fetch client methods

* Remove legacy metadata storage key construction

* Rename CheckEra to CheckMortality

* Substitute perthings types for compact impls

* Fmt

* Downgrade dyn-clone for cargo-contract compat

* Scale-fo 1.0

* scale-info 1.0

* Remove special range handling

* Restore wildcard type params

* Frame metadata 14.0

* WIP decoding events

* WIP more dynamically decoding events

* Fmt

* Decode events, handle errors

* Uncomment some tests

* Remove unused get_mod function

* Fix some warnings

* Fix some more warnings

* Fix some more warnings

* Add tests mod

* Rename node-runtime tests mod to frame

* Fix some warnings

* Fmt

* WIP generate storage client with getters

* Storage client compiling

* Generate storage client api

* Fix up system account query account ids

* WIP generating tx api fns

* Only generate tx api fields when calls available

* Fix tx api call fns

* Fmt

* WIP generate event structs

* call functions not async

* Derive Eq for comparison on generated types

* Generate event structs

* Fix call name

* Fmt

* Update node runtime metadata to substrate c000780db

* Download latest substrate release for integration testing

* Fix event decoding

* Remove unused imports

* Fix plain storage access, total_issuance pass

* Fmt

* Restore contracts tests

* Backoff connecting to substrate node

* Add required TypeInfo impls for local SignedExtension impls

* Remove unnecessary assert formatting

* Fix handling of DispatchError

* Refactor contracts tests

* Troubleshooting contract not found

* Remove more client feature stuff

* Fix dynamic event variant decoding, write consumed index to output

* Fmt

* Use substrate branch with heavy dependency removed

* Remove sp-rcp dependency, define types locally

* Ignore cargo timeing files

* Use my branch for substrate test deps

* Fix storage key type gen

* Comment out fetching contract info

* Add key iteration, extract storage client from main client

* Debugging key generation

* Use substrate master branch

* Fix call test

* Remove TypeSegmenter and dynclone dependency

* Publicly expose Rpc mod

* Unused import warnings

* Add getter for runtime metadata

* Add pallet and event indices for raw events

* Add is_call and is_event convenience trait functions

* Add missing docs

* Refactor tests crate

* Restore remaining client tests

* Fmt

* Fix warnings

* Restore get_mod as test helper and fmt

* Use client references for api calls

* Fix api usages with methods

* Use Bytes for RawEvent debug

* Update metadata

* Restoring some Balances tests

* Populate runtime storage metadata

* Restore balances lock test

* Restore Balances error test

* Fmt

* Restore transfer subscription API

* Staking test

* Restore another staking test

* Restore another staking test

* Restore another staking test

* Partially restore chill_works_for_controller_only staking test

* Fix fetching Optional storage entries

* Restore staking bond test

* Restore remaining staking tests

* Fmt

* Restore sudo tests

* Add some system tests

* Fmt

* Resolve some todos

* Remove pass through rpc methods on Client, expose via rpc() getter

* Remove more rpc pass through methods

* Remove submit tx pass through rpc methods

* Add some comments to SubmittableExtrinsic methods

* Construct the runtime api from the client

* Fmt

* Use From trait instead of new for AccountData query

* Rename subxt_proc_macro crate to subxt_macro

* Fix AccountData From impl

* Extract codegen crate from macro crate

* Fmt

* Replace chameleon hidden field name

* Extract StructDef for generating structs

* More refactoring of StructDef, moving towards sharing with typegen

* Replace explicit tests crate with single implicit integration tests crate

* Rename from substrate-subxt to subxt

* Fix runtime path relative to root Cargo.toml

* Move RpcClient creation to RpcClient

* WIP get examples to compile

* Rename Runtime to Config trait

* WIP implementing default Config

* WIP implementing default extrinsic extras

* fix metadata constants (#299)

* Move DefaultConfig definition and impl to macro

* Extract type substitute parsing to ir mod

* Extract calls, events and storage from api generation

* Add some hardcoded type substitute defaults

* Fmt

* Add utility pallet tests (#300)

* add batch call test example

* add pallet utility tests

* add utility module

* fix warnings

* Add polkadot runtime metadata for example

* Fix system errors and fmt

* Add subxt-cli crate

* Add metadata and codegen subcommands

* Make subxt-cli codegen command work

* Fmt

* Add polkadot codegen test

* Comment about how to run codegen

* Derive AsCompact for structs with single concrete unsigned int field

* Fix bitvec codegen, adds as non optional dependency

* Regenerate polkadot api with bitvec fix

* Edition 2021

* Fix polkadot codegen with bitvec

* Polkadot balance transfer is working

* Fix fetch remote

* Fix transfer_subscribe example

* Fix submit_and_watch example

* Fmt

* Generate storage iter method for iterating over keys

* Fmt

* Fix existential deposit test

* Fix staking tests

* Add option for custom generated type derives

* Add generated type derives for test runtime api

* Fmt

* Copy WrapperTypeOpaque from substrate, add Encode/Decode

* Fmt

* Extract type generator to module, separate & fix tests

* Fully qualified primitive and prelude types

* Fix up remaining type gen tests

* Skip formatting of generated polkadot example code

* Remove empty utility test file.

* Newline

* Update cli/src/main.rs

Co-authored-by: David <dvdplm@gmail.com>

* Rename subxt-cli executable to subxt

* Update src/client.rs

Co-authored-by: David <dvdplm@gmail.com>

* Add some code docs to TypeGenerator.

* Extract TypePath to own file

* Extract type def generation to separate file

* Renamed ModuleType to TypeDefGen

* Fmt

* Factor out type parameter from final_key

* Fix some type paths

* Resolve some todos

* Resolve some panic todos in events

* Add EventsDecodingError

* Decode compact composite types with a single primitive field

* Decode compact composite types with a single primitive field

* Update src/metadata.rs

Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

* Remove Perbill compact substitute types

* Remove todos regarding maintaining Rust code items, promoted to follow up issue.

* Remove todo regarding overridding default config impl

* Remove todo regarding overridding default Extra

* Remove todo regarding AccountData storage type defintion

* Remove todo regarding borrowing storage key arguments

* Remove type substitution tests todo

* Remove `Box` field name type hack todo

* Remove Compact todo

* Remove sudo todos

* Remove BitVec implementation todo

* Fmt

* Add health warning to README

* Fix up health warning

Co-authored-by: Paulo Martins <paulormart@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

src/rpc.rs

@@ -1,5 +1,5 @@
 // Copyright 2019-2021 Parity Technologies (UK) Ltd.
-// This file is part of substrate-subxt.
+// This file is part of subxt.
 //
 // subxt is free software: you can redistribute it and/or modify
 // it under the terms of the GNU General Public License as published by
@@ -12,11 +12,13 @@
 // GNU General Public License for more details.
 //
 // You should have received a copy of the GNU General Public License
-// along with substrate-subxt.  If not, see <http://www.gnu.org/licenses/>.
+// along with subxt.  If not, see <http://www.gnu.org/licenses/>.
+
+//! RPC types and client for interacting with a substrate node.
 
 // jsonrpsee subscriptions are interminable.
 // Allows `while let status = subscription.next().await {}`
-// Related: https://github.com/paritytech/substrate-subxt/issues/66
+// Related: https://github.com/paritytech/subxt/issues/66
 #![allow(irrefutable_let_patterns)]
 
 use std::sync::Arc;
@@ -31,7 +33,10 @@ use core::{
     marker::PhantomData,
 };
 use frame_metadata::RuntimeMetadataPrefixed;
-use jsonrpsee_http_client::HttpClient;
+use jsonrpsee_http_client::{
+    HttpClient,
+    HttpClientBuilder,
+};
 use jsonrpsee_types::{
     to_json_value,
     traits::{
@@ -43,7 +48,10 @@ use jsonrpsee_types::{
     JsonValue,
     Subscription,
 };
-use jsonrpsee_ws_client::WsClient;
+use jsonrpsee_ws_client::{
+    WsClient,
+    WsClientBuilder,
+};
 use serde::{
     Deserialize,
     Serialize,
@@ -55,10 +63,7 @@ use sp_core::{
         StorageKey,
     },
     Bytes,
-};
-use sp_rpc::{
-    list::ListOrValue,
-    number::NumberOrHex,
+    U256,
 };
 use sp_runtime::{
     generic::{
@@ -75,22 +80,48 @@ use crate::{
         EventsDecoder,
         RawEvent,
     },
-    frame::{
-        system::System,
-        Event,
-    },
-    metadata::Metadata,
-    runtimes::Runtime,
+    storage::StorageKeyPrefix,
     subscription::{
         EventStorageSubscription,
         EventSubscription,
         FinalizedEventStorageSubscription,
         SystemEvents,
     },
+    Config,
+    Event,
+    Metadata,
 };
 
+/// A number type that can be serialized both as a number or a string that encodes a number in a
+/// string.
+///
+/// We allow two representations of the block number as input. Either we deserialize to the type
+/// that is specified in the block type or we attempt to parse given hex value.
+///
+/// The primary motivation for having this type is to avoid overflows when using big integers in
+/// JavaScript (which we consider as an important RPC API consumer).
+#[derive(Copy, Clone, Serialize, Deserialize, Debug, PartialEq)]
+#[serde(untagged)]
+pub enum NumberOrHex {
+    /// The number represented directly.
+    Number(u64),
+    /// Hex representation of the number.
+    Hex(U256),
+}
+
+/// RPC list or value wrapper.
+#[derive(Serialize, Deserialize, Debug, PartialEq)]
+#[serde(untagged)]
+pub enum ListOrValue<T> {
+    /// A list of values of given type.
+    List(Vec<T>),
+    /// A single value of given type.
+    Value(T),
+}
+
+/// Alias for the type of a block returned by `chain_getBlock`
 pub type ChainBlock<T> =
-    SignedBlock<Block<<T as System>::Header, <T as System>::Extrinsic>>;
+    SignedBlock<Block<<T as Config>::Header, <T as Config>::Extrinsic>>;
 
 /// Wrapper for NumberOrHex to allow custom From impls
 #[derive(Serialize)]
@@ -153,9 +184,6 @@ pub enum TransactionStatus<Hash, BlockHash> {
     Invalid,
 }
 
-#[cfg(feature = "client")]
-use substrate_subxt_client::SubxtClient;
-
 /// Rpc client wrapper.
 /// This is workaround because adding generic types causes the macros to fail.
 #[derive(Clone)]
@@ -165,12 +193,27 @@ pub enum RpcClient {
     /// JSONRPC client HTTP transport.
     // NOTE: Arc because `HttpClient` is not clone.
     Http(Arc<HttpClient>),
-    #[cfg(feature = "client")]
-    /// Embedded substrate node.
-    Subxt(SubxtClient),
 }
 
 impl RpcClient {
+    /// Create a new [`RpcClient`] from the given URL.
+    ///
+    /// Infers the protocol from the URL, supports:
+    ///     - Websockets (`ws://`, `wss://`)
+    ///     - Http (`http://`, `https://`)
+    pub async fn try_from_url(url: &str) -> Result<Self, Error> {
+        if url.starts_with("ws://") || url.starts_with("wss://") {
+            let client = WsClientBuilder::default()
+                .max_notifs_per_subscription(4096)
+                .build(url)
+                .await?;
+            Ok(RpcClient::WebSocket(Arc::new(client)))
+        } else {
+            let client = HttpClientBuilder::default().build(&url)?;
+            Ok(RpcClient::Http(Arc::new(client)))
+        }
+    }
+
     /// Start a JSON-RPC request.
     pub async fn request<'a, T: DeserializeOwned + std::fmt::Debug>(
         &self,
@@ -178,15 +221,13 @@ impl RpcClient {
         params: &[JsonValue],
     ) -> Result<T, Error> {
         let params = params.into();
+        log::debug!("request {}: {:?}", method, params);
         let data = match self {
             Self::WebSocket(inner) => {
                 inner.request(method, params).await.map_err(Into::into)
             }
             Self::Http(inner) => inner.request(method, params).await.map_err(Into::into),
-            #[cfg(feature = "client")]
-            Self::Subxt(inner) => inner.request(method, params).await.map_err(Into::into),
         };
-        log::debug!("{}: {:?}", method, data);
         data
     }
 
@@ -211,13 +252,6 @@ impl RpcClient {
                 )
                 .into())
             }
-            #[cfg(feature = "client")]
-            Self::Subxt(inner) => {
-                inner
-                    .subscribe(subscribe_method, params, unsubscribe_method)
-                    .await
-                    .map_err(Into::into)
-            }
         }
     }
 }
@@ -246,13 +280,6 @@ impl From<Arc<HttpClient>> for RpcClient {
     }
 }
 
-#[cfg(feature = "client")]
-impl From<SubxtClient> for RpcClient {
-    fn from(client: SubxtClient) -> Self {
-        RpcClient::Subxt(client)
-    }
-}
-
 /// ReadProof struct returned by the RPC
 ///
 /// # Note
@@ -269,14 +296,14 @@ pub struct ReadProof<Hash> {
 }
 
 /// Client for substrate rpc interfaces
-pub struct Rpc<T: Runtime> {
+pub struct Rpc<T: Config> {
     /// Rpc client for sending requests.
     pub client: RpcClient,
     marker: PhantomData<T>,
     accept_weak_inclusion: bool,
 }
 
-impl<T: Runtime> Clone for Rpc<T> {
+impl<T: Config> Clone for Rpc<T> {
     fn clone(&self) -> Self {
         Self {
             client: self.client.clone(),
@@ -286,7 +313,8 @@ impl<T: Runtime> Clone for Rpc<T> {
     }
 }
 
-impl<T: Runtime> Rpc<T> {
+impl<T: Config> Rpc<T> {
+    /// Create a new [`Rpc`]
     pub fn new(client: RpcClient) -> Self {
         Self {
             client,
@@ -317,11 +345,12 @@ impl<T: Runtime> Rpc<T> {
     /// If `start_key` is passed, return next keys in storage in lexicographic order.
     pub async fn storage_keys_paged(
         &self,
-        prefix: Option<StorageKey>,
+        prefix: Option<StorageKeyPrefix>,
         count: u32,
         start_key: Option<StorageKey>,
         hash: Option<T::Hash>,
     ) -> Result<Vec<StorageKey>, Error> {
+        let prefix = prefix.map(|p| p.to_storage_key());
         let params = &[
             to_json_value(prefix)?,
             to_json_value(count)?,
@@ -338,7 +367,7 @@ impl<T: Runtime> Rpc<T> {
         keys: Vec<StorageKey>,
         from: T::Hash,
         to: Option<T::Hash>,
-    ) -> Result<Vec<StorageChangeSet<<T as System>::Hash>>, Error> {
+    ) -> Result<Vec<StorageChangeSet<T::Hash>>, Error> {
         let params = &[
             to_json_value(keys)?,
             to_json_value(from)?,
@@ -355,7 +384,7 @@ impl<T: Runtime> Rpc<T> {
         &self,
         keys: &[StorageKey],
         at: Option<T::Hash>,
-    ) -> Result<Vec<StorageChangeSet<<T as System>::Hash>>, Error> {
+    ) -> Result<Vec<StorageChangeSet<T::Hash>>, Error> {
         let params = &[to_json_value(keys)?, to_json_value(at)?];
         self.client
             .request("state_queryStorageAt", params)
@@ -520,6 +549,7 @@ impl<T: Runtime> Rpc<T> {
         Ok(xt_hash)
     }
 
+    /// Create and submit an extrinsic and return a subscription to the events triggered.
     pub async fn watch_extrinsic<E: Encode>(
         &self,
         extrinsic: E,
@@ -678,7 +708,7 @@ impl<T: Runtime> Rpc<T> {
 
 /// Captures data for when an extrinsic is successfully included in a block
 #[derive(Debug)]
-pub struct ExtrinsicSuccess<T: System> {
+pub struct ExtrinsicSuccess<T: Config> {
     /// Block hash.
     pub block: T::Hash,
     /// Extrinsic hash.
@@ -687,20 +717,20 @@ pub struct ExtrinsicSuccess<T: System> {
     pub events: Vec<RawEvent>,
 }
 
-impl<T: System> ExtrinsicSuccess<T> {
+impl<T: Config> ExtrinsicSuccess<T> {
     /// Find the Event for the given module/variant, with raw encoded event data.
     /// Returns `None` if the Event is not found.
     pub fn find_event_raw(&self, module: &str, variant: &str) -> Option<&RawEvent> {
         self.events
             .iter()
-            .find(|raw| raw.module == module && raw.variant == variant)
+            .find(|raw| raw.pallet == module && raw.variant == variant)
     }
 
     /// Find the Event for the given module/variant, attempting to decode the event data.
     /// Returns `None` if the Event is not found.
     /// Returns `Err` if the data fails to decode into the supplied type.
-    pub fn find_event<E: Event<T>>(&self) -> Result<Option<E>, CodecError> {
-        if let Some(event) = self.find_event_raw(E::MODULE, E::EVENT) {
+    pub fn find_event<E: Event>(&self) -> Result<Option<E>, CodecError> {
+        if let Some(event) = self.find_event_raw(E::PALLET, E::EVENT) {
             Ok(Some(E::decode(&mut &event.data[..])?))
         } else {
             Ok(None)