Generate runtime API from metadata (#294)

* Remove test macro

* Remove client crate

* Create tests crate and move pallet specific tests there

* Extract client, remove metadata and extra, more demolition

* Update substrate dependencies to git dependencies

* Remove Store stuff for now

* Comment out some Call usages

* Add back Runtime trait coped from original System trait

* Make subxt lib compile

* Delete old proc macros and copy over type generation from chameleon

* WIP make transfer balance test pass

* Change to subxt attribute macro

* WIP provide user defined type substitutes

* User defined type substitutes compile

* WIP submitting transactions

* WIP transfer balance test

* Fix macro

* Cargo fmt

* WIP generating storage hashers

* WIP add AccountData trait for fetching the nonce

* Support single type storage map keys

* WIP impl AccountInfo retrieval

* Fix up storage struct generation

* Implement AccountData triait directly on storage entry

* Borrow storage map key and convert account id

* Implement storage fetch client methods

* Remove legacy metadata storage key construction

* Rename CheckEra to CheckMortality

* Substitute perthings types for compact impls

* Fmt

* Downgrade dyn-clone for cargo-contract compat

* Scale-fo 1.0

* scale-info 1.0

* Remove special range handling

* Restore wildcard type params

* Frame metadata 14.0

* WIP decoding events

* WIP more dynamically decoding events

* Fmt

* Decode events, handle errors

* Uncomment some tests

* Remove unused get_mod function

* Fix some warnings

* Fix some more warnings

* Fix some more warnings

* Add tests mod

* Rename node-runtime tests mod to frame

* Fix some warnings

* Fmt

* WIP generate storage client with getters

* Storage client compiling

* Generate storage client api

* Fix up system account query account ids

* WIP generating tx api fns

* Only generate tx api fields when calls available

* Fix tx api call fns

* Fmt

* WIP generate event structs

* call functions not async

* Derive Eq for comparison on generated types

* Generate event structs

* Fix call name

* Fmt

* Update node runtime metadata to substrate c000780db

* Download latest substrate release for integration testing

* Fix event decoding

* Remove unused imports

* Fix plain storage access, total_issuance pass

* Fmt

* Restore contracts tests

* Backoff connecting to substrate node

* Add required TypeInfo impls for local SignedExtension impls

* Remove unnecessary assert formatting

* Fix handling of DispatchError

* Refactor contracts tests

* Troubleshooting contract not found

* Remove more client feature stuff

* Fix dynamic event variant decoding, write consumed index to output

* Fmt

* Use substrate branch with heavy dependency removed

* Remove sp-rcp dependency, define types locally

* Ignore cargo timeing files

* Use my branch for substrate test deps

* Fix storage key type gen

* Comment out fetching contract info

* Add key iteration, extract storage client from main client

* Debugging key generation

* Use substrate master branch

* Fix call test

* Remove TypeSegmenter and dynclone dependency

* Publicly expose Rpc mod

* Unused import warnings

* Add getter for runtime metadata

* Add pallet and event indices for raw events

* Add is_call and is_event convenience trait functions

* Add missing docs

* Refactor tests crate

* Restore remaining client tests

* Fmt

* Fix warnings

* Restore get_mod as test helper and fmt

* Use client references for api calls

* Fix api usages with methods

* Use Bytes for RawEvent debug

* Update metadata

* Restoring some Balances tests

* Populate runtime storage metadata

* Restore balances lock test

* Restore Balances error test

* Fmt

* Restore transfer subscription API

* Staking test

* Restore another staking test

* Restore another staking test

* Restore another staking test

* Partially restore chill_works_for_controller_only staking test

* Fix fetching Optional storage entries

* Restore staking bond test

* Restore remaining staking tests

* Fmt

* Restore sudo tests

* Add some system tests

* Fmt

* Resolve some todos

* Remove pass through rpc methods on Client, expose via rpc() getter

* Remove more rpc pass through methods

* Remove submit tx pass through rpc methods

* Add some comments to SubmittableExtrinsic methods

* Construct the runtime api from the client

* Fmt

* Use From trait instead of new for AccountData query

* Rename subxt_proc_macro crate to subxt_macro

* Fix AccountData From impl

* Extract codegen crate from macro crate

* Fmt

* Replace chameleon hidden field name

* Extract StructDef for generating structs

* More refactoring of StructDef, moving towards sharing with typegen

* Replace explicit tests crate with single implicit integration tests crate

* Rename from substrate-subxt to subxt

* Fix runtime path relative to root Cargo.toml

* Move RpcClient creation to RpcClient

* WIP get examples to compile

* Rename Runtime to Config trait

* WIP implementing default Config

* WIP implementing default extrinsic extras

* fix metadata constants (#299)

* Move DefaultConfig definition and impl to macro

* Extract type substitute parsing to ir mod

* Extract calls, events and storage from api generation

* Add some hardcoded type substitute defaults

* Fmt

* Add utility pallet tests (#300)

* add batch call test example

* add pallet utility tests

* add utility module

* fix warnings

* Add polkadot runtime metadata for example

* Fix system errors and fmt

* Add subxt-cli crate

* Add metadata and codegen subcommands

* Make subxt-cli codegen command work

* Fmt

* Add polkadot codegen test

* Comment about how to run codegen

* Derive AsCompact for structs with single concrete unsigned int field

* Fix bitvec codegen, adds as non optional dependency

* Regenerate polkadot api with bitvec fix

* Edition 2021

* Fix polkadot codegen with bitvec

* Polkadot balance transfer is working

* Fix fetch remote

* Fix transfer_subscribe example

* Fix submit_and_watch example

* Fmt

* Generate storage iter method for iterating over keys

* Fmt

* Fix existential deposit test

* Fix staking tests

* Add option for custom generated type derives

* Add generated type derives for test runtime api

* Fmt

* Copy WrapperTypeOpaque from substrate, add Encode/Decode

* Fmt

* Extract type generator to module, separate & fix tests

* Fully qualified primitive and prelude types

* Fix up remaining type gen tests

* Skip formatting of generated polkadot example code

* Remove empty utility test file.

* Newline

* Update cli/src/main.rs

Co-authored-by: David <dvdplm@gmail.com>

* Rename subxt-cli executable to subxt

* Update src/client.rs

Co-authored-by: David <dvdplm@gmail.com>

* Add some code docs to TypeGenerator.

* Extract TypePath to own file

* Extract type def generation to separate file

* Renamed ModuleType to TypeDefGen

* Fmt

* Factor out type parameter from final_key

* Fix some type paths

* Resolve some todos

* Resolve some panic todos in events

* Add EventsDecodingError

* Decode compact composite types with a single primitive field

* Decode compact composite types with a single primitive field

* Update src/metadata.rs

Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>

* Remove Perbill compact substitute types

* Remove todos regarding maintaining Rust code items, promoted to follow up issue.

* Remove todo regarding overridding default config impl

* Remove todo regarding overridding default Extra

* Remove todo regarding AccountData storage type defintion

* Remove todo regarding borrowing storage key arguments

* Remove type substitution tests todo

* Remove `Box` field name type hack todo

* Remove Compact todo

* Remove sudo todos

* Remove BitVec implementation todo

* Fmt

* Add health warning to README

* Fix up health warning

Co-authored-by: Paulo Martins <paulormart@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Andrew Plaza <aplaza@liquidthink.net>
This commit is contained in:
Andrew Jones
2021-11-03 11:28:59 +00:00
committed by GitHub
parent 8f0641e660
commit 793c945fbd
80 changed files with 5929 additions and 6475 deletions
-535
View File
@@ -1,535 +0,0 @@
// Copyright 2019-2021 Parity Technologies (UK) Ltd.
// This file is part of substrate-subxt.
//
// subxt is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// subxt is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with substrate-subxt. If not, see <http://www.gnu.org/licenses/>.
//! Client for embedding substrate nodes.
#![deny(missing_docs)]
use async_std::{
sync::{
Arc,
RwLock,
},
task,
};
use futures::{
channel::{
mpsc,
oneshot,
},
compat::Stream01CompatExt,
future::{
select,
FutureExt,
},
sink::SinkExt,
stream::StreamExt,
};
use futures01::sync::mpsc as mpsc01;
use jsonrpsee_types::{
v2::{
error::{
JsonRpcError,
JsonRpcErrorCode,
},
params::{
Id,
JsonRpcParams,
JsonRpcSubscriptionParams,
SubscriptionId,
TwoPointZero,
},
request::{
JsonRpcCallSer,
JsonRpcInvalidRequest,
JsonRpcNotification,
JsonRpcNotificationSer,
},
response::JsonRpcResponse,
},
DeserializeOwned,
Error as JsonRpseeError,
FrontToBack,
JsonValue,
RequestMessage,
Subscription,
SubscriptionKind,
SubscriptionMessage,
};
use sc_network::config::TransportConfig;
pub use sc_service::{
config::{
DatabaseConfig,
KeystoreConfig,
WasmExecutionMethod,
},
Error as ServiceError,
};
use sc_service::{
config::{
NetworkConfiguration,
TaskType,
TelemetryEndpoints,
},
ChainSpec,
Configuration,
KeepBlocks,
RpcHandlers,
RpcSession,
TaskManager,
};
use std::{
collections::HashMap,
sync::atomic::{
AtomicU64,
Ordering,
},
};
use thiserror::Error;
const DEFAULT_CHANNEL_SIZE: usize = 16;
/// Error thrown by the client.
#[derive(Debug, Error)]
pub enum SubxtClientError {
/// Failed to parse json rpc message.
#[error("{0}")]
Json(#[from] serde_json::Error),
/// Channel closed.
#[error("{0}")]
Mpsc(#[from] mpsc::SendError),
}
/// Client for an embedded substrate node.
#[derive(Clone)]
pub struct SubxtClient {
to_back: mpsc::Sender<FrontToBack>,
next_id: Arc<AtomicU64>,
}
impl SubxtClient {
/// Create a new client.
pub fn new(mut task_manager: TaskManager, rpc: RpcHandlers) -> Self {
let (to_back, from_front) = mpsc::channel(DEFAULT_CHANNEL_SIZE);
let subscriptions =
Arc::new(RwLock::new(HashMap::<SubscriptionId, (String, Id)>::new()));
task::spawn(
select(
Box::pin(from_front.for_each(move |message: FrontToBack| {
let rpc = rpc.clone();
let (to_front, from_back) = mpsc01::channel(DEFAULT_CHANNEL_SIZE);
let session = RpcSession::new(to_front.clone());
let subscriptions = subscriptions.clone();
async move {
match message {
FrontToBack::Notification(raw) => {
let _ = rpc.rpc_query(&session, &raw).await;
}
FrontToBack::Request(RequestMessage {
raw,
id,
send_back,
}) => {
let raw_response = rpc.rpc_query(&session, &raw).await;
let to_front = match read_jsonrpc_response(
raw_response,
Id::Number(id),
) {
Some(Err(e)) => Err(e),
Some(Ok(rp)) => Ok(rp),
None => return,
};
send_back
.expect("request should have send_back")
.send(to_front)
.expect("failed to send request response");
}
FrontToBack::Subscribe(SubscriptionMessage {
raw,
subscribe_id,
unsubscribe_id,
unsubscribe_method,
send_back,
}) => {
let raw_response = rpc.rpc_query(&session, &raw).await;
let sub_id: SubscriptionId = match read_jsonrpc_response(
raw_response,
Id::Number(subscribe_id),
) {
Some(Ok(rp)) => {
serde_json::from_value(rp)
.expect("infalliable; qed")
}
Some(Err(e)) => {
send_back
.send(Err(e))
.expect("failed to send request response");
return
}
None => return,
};
let (mut send_front_sub, send_back_sub) =
mpsc::channel(DEFAULT_CHANNEL_SIZE);
send_back
.send(Ok((send_back_sub, sub_id.clone())))
.expect("failed to send request response");
{
let mut subscriptions = subscriptions.write().await;
subscriptions.insert(
sub_id.clone(),
(unsubscribe_method, Id::Number(unsubscribe_id)),
);
}
task::spawn(async move {
let mut from_back = from_back.compat();
let _session = session.clone();
while let Some(Ok(response)) = from_back.next().await
{
let notif = serde_json::from_str::<
JsonRpcNotification<
JsonRpcSubscriptionParams<_>,
>,
>(
&response
)
.expect("failed to decode subscription notif");
// ignore send error since the channel is probably closed
let _ = send_front_sub
.send(notif.params.result)
.await;
}
});
}
FrontToBack::SubscriptionClosed(sub_id) => {
let params: &[JsonValue] = &[sub_id.clone().into()];
let subscriptions = subscriptions.read().await;
if let Some((unsub_method, unsub_id)) =
subscriptions.get(&sub_id)
{
let message =
serde_json::to_string(&JsonRpcCallSer::new(
unsub_id.clone(),
unsub_method,
params.into(),
))
.unwrap();
let _ = rpc.rpc_query(&session, &message).await;
}
}
_ => (),
}
}
})),
Box::pin(async move {
task_manager.future().await.ok();
}),
)
.map(drop),
);
Self {
to_back,
next_id: Arc::new(AtomicU64::new(0)),
}
}
/// Creates a new client from a config.
pub fn from_config<C: ChainSpec + 'static>(
config: SubxtClientConfig<C>,
builder: impl Fn(Configuration) -> Result<(TaskManager, RpcHandlers), ServiceError>,
) -> Result<Self, ServiceError> {
let config = config.into_service_config();
let (task_manager, rpc_handlers) = (builder)(config)?;
Ok(Self::new(task_manager, rpc_handlers))
}
/// Send a JSONRPC notification.
pub async fn notification<'a>(
&self,
method: &'a str,
params: JsonRpcParams<'a>,
) -> Result<(), JsonRpseeError> {
let msg = serde_json::to_string(&JsonRpcNotificationSer::new(method, params))
.map_err(JsonRpseeError::ParseError)?;
self.to_back
.clone()
.send(FrontToBack::Notification(msg))
.await
.map_err(|e| JsonRpseeError::Transport(Box::new(e)))
}
/// Send a JSONRPC request.
pub async fn request<'a, T>(
&self,
method: &'a str,
params: JsonRpcParams<'a>,
) -> Result<T, JsonRpseeError>
where
T: DeserializeOwned,
{
let (send_back_tx, send_back_rx) = oneshot::channel();
let id = self.next_id.fetch_add(1, Ordering::Relaxed);
let msg =
serde_json::to_string(&JsonRpcCallSer::new(Id::Number(id), method, params))
.map_err(JsonRpseeError::ParseError)?;
self.to_back
.clone()
.send(FrontToBack::Request(RequestMessage {
raw: msg,
id,
send_back: Some(send_back_tx),
}))
.await
.map_err(|e| JsonRpseeError::Transport(Box::new(e)))?;
let json_value = match send_back_rx.await {
Ok(Ok(v)) => v,
Ok(Err(err)) => return Err(err),
Err(err) => return Err(JsonRpseeError::Transport(Box::new(err))),
};
serde_json::from_value(json_value).map_err(JsonRpseeError::ParseError)
}
/// Send a subscription request to the server.
pub async fn subscribe<'a, N>(
&self,
subscribe_method: &'a str,
params: JsonRpcParams<'a>,
unsubscribe_method: &'a str,
) -> Result<Subscription<N>, JsonRpseeError>
where
N: DeserializeOwned,
{
let sub_req_id = self.next_id.fetch_add(1, Ordering::Relaxed);
let unsub_req_id = self.next_id.fetch_add(1, Ordering::Relaxed);
let msg = serde_json::to_string(&JsonRpcCallSer::new(
Id::Number(sub_req_id),
subscribe_method,
params,
))
.map_err(JsonRpseeError::ParseError)?;
let (send_back_tx, send_back_rx) = oneshot::channel();
self.to_back
.clone()
.send(FrontToBack::Subscribe(SubscriptionMessage {
raw: msg,
subscribe_id: sub_req_id,
unsubscribe_id: unsub_req_id,
unsubscribe_method: unsubscribe_method.to_owned(),
send_back: send_back_tx,
}))
.await
.map_err(JsonRpseeError::Internal)?;
let (notifs_rx, id) = match send_back_rx.await {
Ok(Ok(val)) => val,
Ok(Err(err)) => return Err(err),
Err(err) => return Err(JsonRpseeError::Transport(Box::new(err))),
};
Ok(Subscription::new(
self.to_back.clone(),
notifs_rx,
SubscriptionKind::Subscription(id),
))
}
}
/// Role of the node.
#[derive(Clone, Copy, Debug)]
pub enum Role {
/// Light client.
Light,
/// A full node (mainly used for testing purposes).
Authority(sp_keyring::AccountKeyring),
}
impl From<Role> for sc_service::Role {
fn from(role: Role) -> Self {
match role {
Role::Light => Self::Light,
Role::Authority(_) => {
Self::Authority {
sentry_nodes: Default::default(),
}
}
}
}
}
impl From<Role> for Option<String> {
fn from(role: Role) -> Self {
match role {
Role::Light => None,
Role::Authority(key) => Some(key.to_seed()),
}
}
}
/// Client configuration.
#[derive(Clone)]
pub struct SubxtClientConfig<C: ChainSpec + 'static> {
/// Name of the implementation.
pub impl_name: &'static str,
/// Version of the implementation.
pub impl_version: &'static str,
/// Author of the implementation.
pub author: &'static str,
/// Copyright start year.
pub copyright_start_year: i32,
/// Database configuration.
pub db: DatabaseConfig,
/// Keystore configuration.
pub keystore: KeystoreConfig,
/// Chain specification.
pub chain_spec: C,
/// Role of the node.
pub role: Role,
/// Enable telemetry on the given port.
pub telemetry: Option<u16>,
/// Wasm execution method
pub wasm_method: WasmExecutionMethod,
}
impl<C: ChainSpec + 'static> SubxtClientConfig<C> {
/// Creates a service configuration.
pub fn into_service_config(self) -> Configuration {
let mut network = NetworkConfiguration::new(
format!("{} (subxt client)", self.chain_spec.name()),
"unknown",
Default::default(),
None,
);
network.boot_nodes = self.chain_spec.boot_nodes().to_vec();
network.transport = TransportConfig::Normal {
enable_mdns: true,
allow_private_ipv4: true,
wasm_external_transport: None,
};
let telemetry_endpoints = if let Some(port) = self.telemetry {
let endpoints = TelemetryEndpoints::new(vec![(
format!("/ip4/127.0.0.1/tcp/{}/ws", port),
0,
)])
.expect("valid config; qed");
Some(endpoints)
} else {
None
};
let service_config = Configuration {
network,
impl_name: self.impl_name.to_string(),
impl_version: self.impl_version.to_string(),
chain_spec: Box::new(self.chain_spec),
role: self.role.into(),
task_executor: (move |fut, ty| {
match ty {
TaskType::Async => task::spawn(fut),
TaskType::Blocking => task::spawn_blocking(|| task::block_on(fut)),
}
})
.into(),
database: self.db,
keystore: self.keystore,
max_runtime_instances: 8,
announce_block: true,
dev_key_seed: self.role.into(),
telemetry_endpoints,
telemetry_external_transport: Default::default(),
telemetry_handle: Default::default(),
telemetry_span: Default::default(),
default_heap_pages: Default::default(),
disable_grandpa: Default::default(),
disable_log_reloading: Default::default(),
execution_strategies: Default::default(),
force_authoring: Default::default(),
keep_blocks: KeepBlocks::All,
keystore_remote: Default::default(),
offchain_worker: Default::default(),
prometheus_config: Default::default(),
rpc_cors: Default::default(),
rpc_http: Default::default(),
rpc_ipc: Default::default(),
rpc_ws: Default::default(),
rpc_ws_max_connections: Default::default(),
rpc_methods: Default::default(),
state_cache_child_ratio: Default::default(),
state_cache_size: Default::default(),
tracing_receiver: Default::default(),
tracing_targets: Default::default(),
transaction_pool: Default::default(),
wasm_method: self.wasm_method,
base_path: Default::default(),
informant_output_format: Default::default(),
state_pruning: Default::default(),
transaction_storage: sc_client_db::TransactionStorageMode::BlockBody,
wasm_runtime_overrides: Default::default(),
};
log::info!("{}", service_config.impl_name);
log::info!("✌️ version {}", service_config.impl_version);
log::info!("❤️ by {}, {}", self.author, self.copyright_start_year);
log::info!(
"📋 Chain specification: {}",
service_config.chain_spec.name()
);
log::info!("🏷 Node name: {}", service_config.network.node_name);
log::info!("👤 Role: {:?}", self.role);
service_config
}
}
fn read_jsonrpc_response(
maybe_msg: Option<String>,
id: Id,
) -> Option<Result<JsonValue, JsonRpseeError>> {
let msg: String = maybe_msg?;
// NOTE: `let res` is a workaround because rustc otherwise doesn't compile
// `msg` doesn't live long enough.
let res = match serde_json::from_str::<JsonRpcResponse<JsonValue>>(&msg) {
Ok(rp) if rp.id == id => Some(Ok(rp.result)),
Ok(_) => Some(Err(JsonRpseeError::InvalidRequestId)),
Err(_) => {
match serde_json::from_str::<JsonRpcInvalidRequest<'_>>(&msg) {
Ok(err) => {
let err = JsonRpcError {
jsonrpc: TwoPointZero,
error: JsonRpcErrorCode::InvalidRequest.into(),
id: err.id,
};
Some(Err(JsonRpseeError::Request(err.to_string())))
}
Err(_) => None,
}
}
};
res
}