light-client: Add experimental light-client support (#965)

* rpc/types: Decode `SubstrateTxStatus` for substrate and smoldot

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Add light client Error

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Add background task to manage RPC responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Implement the light client RPC in subxt

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Expose light client under experimental feature-flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Add development chain spec for local nodes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update cargo lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Add light client example

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update sp-* crates and smoldot to use git with branch / rev

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply cargo fmt

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Import hashmap entry

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Fetch spec only if jsonrpsee feature is enabled

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update subxt/src/rpc/lightclient/background.rs

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Fix typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Update dev chain spec

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* types: Handle storage replies from chainHead_storage

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Add polkadot spec

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Handle RPC error responses

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Tx basic with light client for local nodes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* example: Light client coprehensive example for live chains

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Remove prior light client example

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* feature: Rename experimental to unstable

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* book: Add light client section

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* lightclient: Ignore validated events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust tests for light-clients and normal clients

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Keep lightclient variant

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove support for chainHead_storage for light client

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update light client to point to crates.io

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update sp-crates from crates.io

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Replace Atomic with u64

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Add LightClientBuilder

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust chainspec with provided bootnodes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Add potential_relay_chains to light client builder

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Move the light-client to the background task

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust tracing logs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update book and example

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply cargo fmt

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove dev_spec.json artifact

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Examples fix duplicate Cargo.toml

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Use tracing_subscriber crate

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Fix clippy for different features

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Add comment about bootNodes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Add comment about tracing-sub dependency

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Run integration-tests with light-client

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Feature guard some incompatible tests

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Enable light-client tests under feature flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Fix git step name

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust flags for testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust warnings

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Rename feature flag jsonrpsee-ws to jsonrpsee

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Fix cargo check

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Run tests on just 2 threads

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Move light-client to subxt/src/client

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust LightClientBuilder

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Use ws_url to construct light client for testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Refactor background

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Address feedback

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove polkadot.spec and trim sub_id

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Wait for substrate to produce block before connecting light client

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust builder and tests

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply fmt

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* ci: Use release for light client testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Add single test for light-client

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Wait for more blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Use polkadot endpoint for testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust cargo check

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Remove light client chain connection example

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust cargo.toml section for the old example

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust background task to use usize for subscription Id

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Build bootnodes with serde_json::Value directly

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Make channel between subxt user and subxt background unbounded

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update subxt/src/client/lightclient/builder.rs

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Switch to smoldot 0.6.0 from 0.5.0

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Move testing to `full_client` and `light_client` higher modules

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove subscriptionID type

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove subxt/integration-testing feature flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust wait_for_blocks documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust utils import for testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Remove into_iter from builder construction

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
This commit is contained in:
Alexandru Vasile
2023-06-26 12:10:57 +03:00
committed by GitHub
parent 8413c4d2dd
commit ef89752904
42 changed files with 2352 additions and 147 deletions
@@ -0,0 +1,219 @@
// Copyright 2019-2023 Parity Technologies (UK) Ltd.
// This file is dual-licensed as Apache-2.0 or GPL-3.0.
// see LICENSE for license details.
use frame_metadata::{
v15::{ExtrinsicMetadata, RuntimeMetadataV15},
RuntimeMetadataPrefixed,
};
use scale_info::{meta_type, IntoPortable, PortableRegistry, Registry, TypeInfo};
use subxt_codegen::{CratePath, DerivesRegistry, RuntimeGenerator, TypeSubstitutes};
use syn::__private::quote;
fn generate_runtime_interface_from_metadata(metadata: RuntimeMetadataPrefixed) -> String {
// Generate a runtime interface from the provided metadata.
let metadata = metadata
.try_into()
.expect("frame_metadata should be convertible into Metadata");
let generator = RuntimeGenerator::new(metadata);
let item_mod = syn::parse_quote!(
pub mod api {}
);
let crate_path = CratePath::default();
let derives = DerivesRegistry::with_default_derives(&crate_path);
let type_substitutes = TypeSubstitutes::with_default_substitutes(&crate_path);
generator
.generate_runtime(item_mod, derives, type_substitutes, crate_path, false)
.expect("API generation must be valid")
.to_string()
}
fn generate_runtime_interface_with_type_registry<F>(f: F) -> String
where
F: Fn(&mut scale_info::Registry),
{
#[derive(TypeInfo)]
struct Runtime;
#[derive(TypeInfo)]
enum RuntimeCall {}
#[derive(TypeInfo)]
enum RuntimeEvent {}
#[derive(TypeInfo)]
pub enum DispatchError {}
// We need these types for codegen to work:
let mut registry = scale_info::Registry::new();
let ty = registry.register_type(&meta_type::<Runtime>());
registry.register_type(&meta_type::<RuntimeCall>());
registry.register_type(&meta_type::<RuntimeEvent>());
registry.register_type(&meta_type::<DispatchError>());
// Allow custom types to be added for testing:
f(&mut registry);
let extrinsic = ExtrinsicMetadata {
ty: meta_type::<()>(),
version: 0,
signed_extensions: vec![],
}
.into_portable(&mut registry);
let metadata = RuntimeMetadataV15 {
types: registry.into(),
pallets: Vec::new(),
extrinsic,
ty,
apis: vec![],
};
let metadata = RuntimeMetadataPrefixed::from(metadata);
generate_runtime_interface_from_metadata(metadata)
}
#[test]
fn dupe_types_do_not_overwrite_each_other() {
let interface = generate_runtime_interface_with_type_registry(|registry| {
// Now we duplicate some types with same type info. We need two unique types here,
// and can't just add one type to the registry twice, because the registry knows if
// type IDs are the same.
enum Foo {}
impl TypeInfo for Foo {
type Identity = Self;
fn type_info() -> scale_info::Type {
scale_info::Type::builder()
.path(scale_info::Path::new("DuplicateType", "dupe_mod"))
.variant(
scale_info::build::Variants::new()
.variant("FirstDupeTypeVariant", |builder| builder.index(0)),
)
}
}
enum Bar {}
impl TypeInfo for Bar {
type Identity = Self;
fn type_info() -> scale_info::Type {
scale_info::Type::builder()
.path(scale_info::Path::new("DuplicateType", "dupe_mod"))
.variant(
scale_info::build::Variants::new()
.variant("SecondDupeTypeVariant", |builder| builder.index(0)),
)
}
}
registry.register_type(&meta_type::<Foo>());
registry.register_type(&meta_type::<Bar>());
});
assert!(interface.contains("DuplicateType"));
assert!(interface.contains("FirstDupeTypeVariant"));
assert!(interface.contains("DuplicateType2"));
assert!(interface.contains("SecondDupeTypeVariant"));
}
#[test]
fn generic_types_overwrite_each_other() {
let interface = generate_runtime_interface_with_type_registry(|registry| {
// If we have two types mentioned in the registry that have generic params,
// only one type will be output (the codegen assumes that the generic param will disambiguate)
enum Foo {}
impl TypeInfo for Foo {
type Identity = Self;
fn type_info() -> scale_info::Type {
scale_info::Type::builder()
.path(scale_info::Path::new("DuplicateType", "dupe_mod"))
.type_params([scale_info::TypeParameter::new("T", Some(meta_type::<u8>()))])
.variant(scale_info::build::Variants::new())
}
}
enum Bar {}
impl TypeInfo for Bar {
type Identity = Self;
fn type_info() -> scale_info::Type {
scale_info::Type::builder()
.path(scale_info::Path::new("DuplicateType", "dupe_mod"))
.type_params([scale_info::TypeParameter::new("T", Some(meta_type::<u8>()))])
.variant(scale_info::build::Variants::new())
}
}
registry.register_type(&meta_type::<Foo>());
registry.register_type(&meta_type::<Bar>());
});
assert!(interface.contains("DuplicateType"));
// We do _not_ expect this to exist, since a generic is present on the type:
assert!(!interface.contains("DuplicateType2"));
}
#[test]
fn more_than_1_generic_parameters_work() {
#[allow(unused)]
#[derive(TypeInfo)]
struct Foo<T, U, V, W> {
a: T,
b: U,
c: V,
d: W,
}
#[allow(unused)]
#[derive(TypeInfo)]
struct Bar {
p: Foo<u32, u32, u64, u128>,
q: Foo<u8, u8, u8, u8>,
}
let mut registry = Registry::new();
registry.register_type(&meta_type::<Bar>());
let portable_types: PortableRegistry = registry.into();
let type_gen = subxt_codegen::TypeGenerator::new(
&portable_types,
"root",
Default::default(),
Default::default(),
CratePath::default(),
false,
);
let types = type_gen.generate_types_mod().unwrap();
let generated_mod = quote::quote!( #types);
let expected_mod = quote::quote! {
pub mod root {
use super::root;
pub mod integration_tests {
use super::root;
pub mod codegen {
use super::root;
pub mod codegen_tests {
use super::root;
pub struct Bar {
pub p: root::integration_tests::codegen::codegen_tests::Foo<
::core::primitive::u32,
::core::primitive::u32,
::core::primitive::u64,
::core::primitive::u128
>,
pub q: root::integration_tests::codegen::codegen_tests::Foo<
::core::primitive::u8,
::core::primitive::u8,
::core::primitive::u8,
::core::primitive::u8
>,
}
pub struct Foo<_0, _1, _2, _3> {
pub a: _0,
pub b: _1,
pub c: _2,
pub d: _3,
}
}
}
}
}
};
assert_eq!(generated_mod.to_string(), expected_mod.to_string());
}
@@ -0,0 +1,176 @@
// Copyright 2019-2023 Parity Technologies (UK) Ltd.
// This file is dual-licensed as Apache-2.0 or GPL-3.0.
// see LICENSE for license details.
use codec::Decode;
use regex::Regex;
use subxt_codegen::{CratePath, DerivesRegistry, RuntimeGenerator, TypeSubstitutes};
use subxt_metadata::Metadata;
fn load_test_metadata() -> Metadata {
let bytes = test_runtime::METADATA;
Metadata::decode(&mut &*bytes).expect("Cannot decode scale metadata")
}
fn metadata_docs() -> Vec<String> {
// Load the runtime metadata downloaded from a node via `test-runtime`.
let metadata = load_test_metadata();
// Inspect the metadata types and collect the documentation.
let mut docs = Vec::new();
for ty in &metadata.types().types {
docs.extend_from_slice(&ty.ty.docs);
}
for pallet in metadata.pallets() {
if let Some(storage) = pallet.storage() {
for entry in storage.entries() {
docs.extend_from_slice(entry.docs());
}
}
// Note: Calls, Events and Errors are deduced directly to
// PortableTypes which are handled above.
for constant in pallet.constants() {
docs.extend_from_slice(constant.docs());
}
}
// Note: Extrinsics do not have associated documentation, but is implied by
// associated Type.
// Inspect the runtime API types and collect the documentation.
for api in metadata.runtime_api_traits() {
docs.extend_from_slice(api.docs());
for method in api.methods() {
docs.extend_from_slice(method.docs());
}
}
docs
}
fn generate_runtime_interface(crate_path: CratePath, should_gen_docs: bool) -> String {
// Load the runtime metadata downloaded from a node via `test-runtime`.
let metadata = load_test_metadata();
// Generate a runtime interface from the provided metadata.
let generator = RuntimeGenerator::new(metadata);
let item_mod = syn::parse_quote!(
pub mod api {}
);
let derives = DerivesRegistry::with_default_derives(&crate_path);
let type_substitutes = TypeSubstitutes::with_default_substitutes(&crate_path);
generator
.generate_runtime(
item_mod,
derives,
type_substitutes,
crate_path,
should_gen_docs,
)
.expect("API generation must be valid")
.to_string()
}
fn interface_docs(should_gen_docs: bool) -> Vec<String> {
// Generate the runtime interface from the node's metadata.
// Note: the API is generated on a single line.
let runtime_api = generate_runtime_interface(CratePath::default(), should_gen_docs);
// Documentation lines have the following format:
// # [ doc = "Upward message is invalid XCM."]
// Given the API is generated on a single line, the regex matching
// must be lazy hence the `?` in the matched group `(.*?)`.
//
// The greedy `non-?` matching would lead to one single match
// from the beginning of the first documentation tag, containing everything up to
// the last documentation tag
// `# [ doc = "msg"] # [ doc = "msg2"] ... api ... # [ doc = "msgN" ]`
//
// The `(.*?)` stands for match any character zero or more times lazily.
let re = Regex::new(r#"\# \[doc = "(.*?)"\]"#).unwrap();
re.captures_iter(&runtime_api)
.filter_map(|capture| {
// Get the matched group (ie index 1).
capture.get(1).as_ref().map(|doc| {
// Generated documentation will escape special characters.
// Replace escaped characters with unescaped variants for
// exact matching on the raw metadata documentation.
doc.as_str()
.replace("\\n", "\n")
.replace("\\t", "\t")
.replace("\\\"", "\"")
})
})
.collect()
}
#[test]
fn check_documentation() {
// Inspect metadata recursively and obtain all associated documentation.
let raw_docs = metadata_docs();
// Obtain documentation from the generated API.
let runtime_docs = interface_docs(true);
for raw in raw_docs.iter() {
assert!(
runtime_docs.contains(raw),
"Documentation not present in runtime API: {raw}"
);
}
}
#[test]
fn check_no_documentation() {
// Inspect metadata recursively and obtain all associated documentation.
let raw_docs = metadata_docs();
// Obtain documentation from the generated API.
let runtime_docs = interface_docs(false);
for raw in raw_docs.iter() {
assert!(
!runtime_docs.contains(raw),
"Documentation should not be present in runtime API: {raw}"
);
}
}
#[test]
fn check_root_attrs_preserved() {
let metadata = load_test_metadata();
// Test that the root docs/attr are preserved.
let item_mod = syn::parse_quote!(
/// Some root level documentation
#[some_root_attribute]
pub mod api {}
);
// Generate a runtime interface from the provided metadata.
let generator = RuntimeGenerator::new(metadata);
let derives = DerivesRegistry::with_default_derives(&CratePath::default());
let type_substitutes = TypeSubstitutes::with_default_substitutes(&CratePath::default());
let generated_code = generator
.generate_runtime(
item_mod,
derives,
type_substitutes,
CratePath::default(),
true,
)
.expect("API generation must be valid")
.to_string();
let doc_str_loc = generated_code
.find("Some root level documentation")
.expect("root docs should be preserved");
let attr_loc = generated_code
.find("some_root_attribute") // '#' is space separated in generated output.
.expect("root attr should be preserved");
let mod_start = generated_code
.find("pub mod api")
.expect("'pub mod api' expected");
// These things should be before the mod start
assert!(doc_str_loc < mod_start);
assert!(attr_loc < mod_start);
}
@@ -0,0 +1,18 @@
// Copyright 2019-2023 Parity Technologies (UK) Ltd.
// This file is dual-licensed as Apache-2.0 or GPL-3.0.
// see LICENSE for license details.
/// Checks that code generated by `subxt-cli codegen` compiles. Allows inspection of compiler errors
/// directly, more accurately than via the macro and `cargo expand`.
///
/// Generate by running this at the root of the repository:
///
/// ```
/// cargo run --bin subxt -- codegen --file artifacts/polkadot_metadata_full.scale | rustfmt > testing/integration-tests/src/codegen/polkadot.rs
/// ```
#[rustfmt::skip]
#[allow(clippy::all)]
mod polkadot;
mod codegen_tests;
mod documentation;
File diff suppressed because one or more lines are too long