Allow integrated relay chain light client (#2270)

* Add embedded light client to cli

* Prepare for light-client-worker

* First working version

* Clean up

* Remove unwanted logs

* Simplify subscription code

* Let jsonrpsee handle rpc management

* Simplify implementation

* Reorganize crate structure

* Use relay chain arg chainspec for light-client

* Clean up command line

* Add light client worker file

* Use smoldot master to avoid wasmtime conflict

* Remove sleep

* Improve naming of cli option

* Remove conflict with `validator`

* Improve docs

* Update smoldot, remove unwanted change

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Disable collation

* Reviewer comments

* Update smoldot and tokio-platform

* Update smoldot

* Update smoldot

* Adjust to new version

* Patch substrate

* Use constants

* Add readme entry, improve zombienet tests

* Apply suggestions from code review

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Make execution mode an enum

* Update smoldot, remove substrate patch

* Update client/relay-chain-rpc-interface/src/rpc_client.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Reduce duplicate code

* Update smoldot

* Update smoldot

* Fix build

* Update smoldot

* Make platform compile

* Clean up dependencies

* Use crates.io instead of github for smoldot

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Docs

* Improve docs

* Remove `RpcFrontend`

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
This commit is contained in:
Sebastian Kunert
2023-08-22 15:53:32 +02:00
committed by GitHub
parent 5eea3939de
commit 817c4503db
21 changed files with 1405 additions and 384 deletions
+481 -83
View File
File diff suppressed because it is too large Load Diff
+24 -3
View File
@@ -43,9 +43,10 @@ You may run `polkadot-parachain` locally after building it or using one of the c
### Relay Chain Interaction
To operate a parachain node, a connection to the corresponding relay chain is necessary. This can be
achieved in one of two ways:
achieved in one of three ways:
1. Run a full relay chain node within the parachain node (default)
2. Connect to an external relay chain node via WebSocket RPC
3. Run a light client for the relay chain
#### In-process Relay Chain Node
If an external relay chain node is not specified (default behavior), then a full relay chain node is
@@ -55,7 +56,7 @@ This node has all of the typical components of a regular Polkadot node and will
with the relay chain to work.
##### Example command
```shell=
```bash
polkadot-parachain \
--chain parachain-chainspec.json \
--tmp \
@@ -76,7 +77,7 @@ they will use fewer system resources.
relay chain node in-process. Even though they lack the majority of normal Polkadot subsystems, they
will still need to connect directly to the relay chain network.
##### Example command
```shell=
```bash
polkadot-parachain \
--chain parachain-chainspec.json \
--tmp \
@@ -87,6 +88,26 @@ polkadot-parachain \
--chain relaychain-chainspec.json
```
#### Relay Chain Light Client
An internal relay chain light client provides a fast and lightweight approach for connecting to the relay chain network.
It provides relay chain notifications and facilitates runtime calls.
To specify which chain the light client should connect to, users need to supply a relay chain chain-spec as part of the relay chain arguments.
**Note:** At this time, any parachain nodes using this feature will still spawn a significantly cut-down
relay chain node in-process. Even though they lack the majority of normal Polkadot subsystems, they
will still need to connect directly to the relay chain network.
##### Example command
```bash
polkadot-parachain \
--chain parachain-chainspec.json \
--tmp \
--relay-chain-light-client \
-- \
--chain relaychain-chainspec.json
```
## Installation and Setup
Before building Cumulus SDK based nodes / runtimes prepare your environment by following Substrate
[installation instructions](https://docs.substrate.io/main-docs/install/).
+26 -3
View File
@@ -304,6 +304,11 @@ pub struct RunCmd {
alias = "relay-chain-rpc-url"
)]
pub relay_chain_rpc_urls: Vec<Url>,
/// EXPERIMENTAL: Embed a light client for the relay chain. Only supported for full-nodes.
/// Will use the specified relay chain chainspec.
#[arg(long, conflicts_with_all = ["relay_chain_rpc_urls", "collator"])]
pub relay_chain_light_client: bool,
}
impl RunCmd {
@@ -319,15 +324,33 @@ impl RunCmd {
/// Create [`CollatorOptions`] representing options only relevant to parachain collator nodes
pub fn collator_options(&self) -> CollatorOptions {
CollatorOptions { relay_chain_rpc_urls: self.relay_chain_rpc_urls.clone() }
let relay_chain_mode =
match (self.relay_chain_light_client, !self.relay_chain_rpc_urls.is_empty()) {
(true, _) => RelayChainMode::LightClient,
(_, true) => RelayChainMode::ExternalRpc(self.relay_chain_rpc_urls.clone()),
_ => RelayChainMode::Embedded,
};
CollatorOptions { relay_chain_mode }
}
}
/// Possible modes for the relay chain to operate in.
#[derive(Clone, Debug)]
pub enum RelayChainMode {
/// Spawn embedded relay chain node
Embedded,
/// Connect to remote relay chain node via websocket RPC
ExternalRpc(Vec<Url>),
/// Spawn embedded relay chain light client
LightClient,
}
/// Options only relevant for collator nodes
#[derive(Clone, Debug)]
pub struct CollatorOptions {
/// Location of relay chain full node
pub relay_chain_rpc_urls: Vec<Url>,
/// How this collator retrieves relay chain information
pub relay_chain_mode: RelayChainMode,
}
/// A non-redundant version of the `RunCmd` that sets the `validator` field when the
@@ -20,14 +20,12 @@ polkadot-node-core-runtime-api = { git = "https://github.com/paritytech/polkadot
# substrate deps
sc-authority-discovery = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-client-api = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-network = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-network-common = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-service = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-tracing = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-utils = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-api = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-blockchain = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-consensus-babe = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-consensus = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-runtime = { git = "https://github.com/paritytech/substrate", branch = "master" }
@@ -42,7 +40,6 @@ lru = "0.11.0"
tracing = "0.1.37"
async-trait = "0.1.73"
futures = "0.3.28"
tokio = { version = "1.32.0", features = ["macros"] }
[features]
network-protocol-staging = ["polkadot-node-network-protocol/network-protocol-staging"]
@@ -17,7 +17,7 @@
use collator_overseer::{CollatorOverseerGenArgs, NewMinimalNode};
use cumulus_relay_chain_interface::{RelayChainError, RelayChainInterface, RelayChainResult};
use cumulus_relay_chain_rpc_interface::{RelayChainRpcInterface, Url};
use cumulus_relay_chain_rpc_interface::{RelayChainRpcClient, RelayChainRpcInterface, Url};
use network::build_collator_network;
use polkadot_network_bridge::{peer_sets_info, IsAuthority};
use polkadot_node_network_protocol::{
@@ -38,13 +38,14 @@ use sp_runtime::{app_crypto::Pair, traits::Block as BlockT};
use futures::StreamExt;
use std::sync::Arc;
mod blockchain_rpc_client;
mod collator_overseer;
mod network;
mod blockchain_rpc_client;
pub use blockchain_rpc_client::BlockChainRpcClient;
const LOG_TARGET: &str = "minimal-relaychain-node";
fn build_authority_discovery_service<Block: BlockT>(
task_manager: &TaskManager,
client: Arc<BlockChainRpcClient>,
@@ -82,16 +83,11 @@ fn build_authority_discovery_service<Block: BlockT>(
service
}
pub async fn build_minimal_relay_chain_node(
async fn build_interface(
polkadot_config: Configuration,
task_manager: &mut TaskManager,
relay_chain_url: Vec<Url>,
client: RelayChainRpcClient,
) -> RelayChainResult<(Arc<(dyn RelayChainInterface + 'static)>, Option<CollatorPair>)> {
let client = cumulus_relay_chain_rpc_interface::create_client_and_start_worker(
relay_chain_url,
task_manager,
)
.await?;
let collator_pair = CollatorPair::generate().0;
let collator_node = new_minimal_relay_chain(
polkadot_config,
@@ -106,6 +102,44 @@ pub async fn build_minimal_relay_chain_node(
))
}
pub async fn build_minimal_relay_chain_node_with_rpc(
polkadot_config: Configuration,
task_manager: &mut TaskManager,
relay_chain_url: Vec<Url>,
) -> RelayChainResult<(Arc<(dyn RelayChainInterface + 'static)>, Option<CollatorPair>)> {
let client = cumulus_relay_chain_rpc_interface::create_client_and_start_worker(
relay_chain_url,
task_manager,
)
.await?;
build_interface(polkadot_config, task_manager, client).await
}
pub async fn build_minimal_relay_chain_node_light_client(
polkadot_config: Configuration,
task_manager: &mut TaskManager,
) -> RelayChainResult<(Arc<(dyn RelayChainInterface + 'static)>, Option<CollatorPair>)> {
tracing::info!(
target: LOG_TARGET,
chain_name = polkadot_config.chain_spec.name(),
chain_id = polkadot_config.chain_spec.id(),
"Initializing embedded light client with chain spec."
);
let spec = polkadot_config
.chain_spec
.as_json(false)
.map_err(RelayChainError::GenericError)?;
let client = cumulus_relay_chain_rpc_interface::create_client_and_start_light_client_worker(
spec,
task_manager,
)
.await?;
build_interface(polkadot_config, task_manager, client).await
}
/// Builds a minimal relay chain node. Chain data is fetched
/// via [`BlockChainRpcClient`] and fed into the overseer and its subsystems.
///
@@ -117,8 +151,6 @@ pub async fn build_minimal_relay_chain_node(
/// - NetworkBridgeRx
/// - NetworkBridgeTx
/// - RuntimeApi
/// - ChainApi
/// - AvailabilityDistribution
#[sc_tracing::logging::prefix_logs_with("Relaychain")]
async fn new_minimal_relay_chain(
config: Configuration,
@@ -17,11 +17,13 @@ sp-consensus-babe = { git = "https://github.com/paritytech/substrate", branch =
sp-authority-discovery = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-state-machine = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-storage = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-runtime = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-client-api = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-rpc-api = { git = "https://github.com/paritytech/substrate", branch = "master" }
sc-service = { git = "https://github.com/paritytech/substrate", branch = "master" }
tokio = { version = "1.32.0", features = ["sync"] }
tokio-util = { version = "0.7.8", features = ["compat"] }
futures = "0.3.28"
futures-timer = "3.0.2"
@@ -33,3 +35,9 @@ url = "2.4.0"
serde_json = "1.0.105"
serde = "1.0.183"
lru = "0.11.0"
smoldot = { version = "0.11.0", default_features = false, features = ["std"]}
smoldot-light = { version = "0.9.0", default_features = false, features = ["std"] }
either = "1.8.1"
thiserror = "1.0.38"
rand = "0.8.5"
pin-project = "1.1.3"
@@ -38,9 +38,15 @@ use std::pin::Pin;
use cumulus_primitives_core::relay_chain::BlockId;
pub use url::Url;
mod light_client_worker;
mod reconnecting_ws_client;
mod rpc_client;
pub use rpc_client::{create_client_and_start_worker, RelayChainRpcClient};
mod tokio_platform;
pub use rpc_client::{
create_client_and_start_light_client_worker, create_client_and_start_worker,
RelayChainRpcClient,
};
const TIMEOUT_IN_SECONDS: u64 = 6;
@@ -0,0 +1,297 @@
// Copyright 2023 Parity Technologies (UK) Ltd.
// This file is part of Cumulus.
// Cumulus is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Cumulus is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Cumulus. If not, see <http://www.gnu.org/licenses/>.
//! This module contains a backend that sends RPC requests to an
//! embedded light client. Even though no networking is involved,
//! we treat the light-client as a normal JsonRPC target.
use futures::{channel::mpsc::Sender, prelude::*, stream::FuturesUnordered};
use jsonrpsee::core::{
client::{
Client as JsonRpseeClient, ClientBuilder, ClientT, ReceivedMessage, TransportReceiverT,
TransportSenderT,
},
Error,
};
use smoldot_light::{ChainId, Client as SmoldotClient, JsonRpcResponses};
use std::{num::NonZeroU32, sync::Arc};
use tokio::sync::mpsc::{channel as tokio_channel, Receiver, Sender as TokioSender};
use cumulus_primitives_core::relay_chain::{
Block as RelayBlock, BlockNumber as RelayNumber, Hash as RelayHash, Header as RelayHeader,
};
use cumulus_relay_chain_interface::{RelayChainError, RelayChainResult};
use sp_runtime::generic::SignedBlock;
use sc_rpc_api::chain::ChainApiClient;
use sc_service::SpawnTaskHandle;
use crate::{rpc_client::RpcDispatcherMessage, tokio_platform::TokioPlatform};
const LOG_TARGET: &str = "rpc-light-client-worker";
const MAX_PENDING_REQUESTS: u32 = 128;
const MAX_SUBSCRIPTIONS: u32 = 64;
#[derive(thiserror::Error, Debug)]
enum LightClientError {
#[error("Error occured while executing smoldot request: {0}")]
SmoldotError(String),
#[error("Nothing returned from json_rpc_responses")]
EmptyResult,
}
/// Sending adapter allowing JsonRpsee to send messages to smoldot
struct SimpleStringSender {
inner: SmoldotClient<TokioPlatform, ()>,
chain_id: ChainId,
}
#[async_trait::async_trait]
impl TransportSenderT for SimpleStringSender {
type Error = LightClientError;
async fn send(&mut self, msg: String) -> Result<(), Self::Error> {
self.inner
.json_rpc_request(msg, self.chain_id)
.map_err(|e| LightClientError::SmoldotError(e.to_string()))
}
}
/// Receiving adapter allowing JsonRpsee to receive messages from smoldot
struct SimpleStringReceiver {
inner: JsonRpcResponses,
}
#[async_trait::async_trait]
impl TransportReceiverT for SimpleStringReceiver {
type Error = LightClientError;
async fn receive(&mut self) -> Result<ReceivedMessage, Self::Error> {
self.inner
.next()
.await
.map(|message| jsonrpsee::core::client::ReceivedMessage::Text(message))
.ok_or(LightClientError::EmptyResult)
}
}
/// Build a smoldot client and add the specified chain spec to it.
pub async fn build_smoldot_client(
spawner: SpawnTaskHandle,
chain_spec: &str,
) -> RelayChainResult<(SmoldotClient<TokioPlatform, ()>, ChainId, JsonRpcResponses)> {
let platform = TokioPlatform::new(spawner);
let mut client = SmoldotClient::new(platform);
// Ask the client to connect to a chain.
let smoldot_light::AddChainSuccess { chain_id, json_rpc_responses } = client
.add_chain(smoldot_light::AddChainConfig {
specification: chain_spec,
json_rpc: smoldot_light::AddChainConfigJsonRpc::Enabled {
max_pending_requests: NonZeroU32::new(MAX_PENDING_REQUESTS)
.expect("Constant larger than 0; qed"),
max_subscriptions: MAX_SUBSCRIPTIONS,
},
potential_relay_chains: core::iter::empty(),
database_content: "",
user_data: (),
})
.map_err(|e| RelayChainError::GenericError(e.to_string()))?;
Ok((client, chain_id, json_rpc_responses.expect("JSON RPC is enabled; qed")))
}
/// Worker to process incoming [`RpcDispatcherMessage`] requests.
/// On startup, this worker opens subscriptions for imported, best and finalized
/// heads. Incoming notifications are distributed to registered listeners.
pub struct LightClientRpcWorker {
client_receiver: Receiver<RpcDispatcherMessage>,
imported_header_listeners: Vec<Sender<RelayHeader>>,
finalized_header_listeners: Vec<Sender<RelayHeader>>,
best_header_listeners: Vec<Sender<RelayHeader>>,
smoldot_client: Arc<JsonRpseeClient>,
}
fn handle_notification(
maybe_header: Option<Result<RelayHeader, Error>>,
senders: &mut Vec<Sender<RelayHeader>>,
) -> Result<(), ()> {
match maybe_header {
Some(Ok(header)) => {
crate::rpc_client::distribute_header(header, senders);
Ok(())
},
None => {
tracing::error!(target: LOG_TARGET, "Subscription closed.");
Err(())
},
Some(Err(error)) => {
tracing::error!(target: LOG_TARGET, ?error, "Error in RPC subscription.");
Err(())
},
}
}
impl LightClientRpcWorker {
/// Create new light-client worker.
///
/// Returns the worker itself and a channel to send messages.
pub fn new(
smoldot_client: smoldot_light::Client<TokioPlatform, ()>,
json_rpc_responses: JsonRpcResponses,
chain_id: ChainId,
) -> (LightClientRpcWorker, TokioSender<RpcDispatcherMessage>) {
let (tx, rx) = tokio_channel(100);
let smoldot_adapter_sender = SimpleStringSender { inner: smoldot_client, chain_id };
let smoldot_adapter_receiver = SimpleStringReceiver { inner: json_rpc_responses };
// Build jsonrpsee client that will talk to the inprocess smoldot node
let smoldot_jsonrpsee_client = Arc::new(
ClientBuilder::default()
.build_with_tokio(smoldot_adapter_sender, smoldot_adapter_receiver),
);
let worker = LightClientRpcWorker {
client_receiver: rx,
imported_header_listeners: Default::default(),
finalized_header_listeners: Default::default(),
best_header_listeners: Default::default(),
smoldot_client: smoldot_jsonrpsee_client,
};
(worker, tx)
}
// Main worker loop.
//
// Does the following:
// 1. Initialize notification streams
// 2. Enter main loop
// a. On listening request, register listener for respective notification stream
// b. On incoming notification, distribute notification to listeners
// c. On incoming request, forward request to smoldot
// d. Advance execution of pending requests
pub async fn run(mut self) {
let mut pending_requests = FuturesUnordered::new();
let Ok(mut new_head_subscription) = <JsonRpseeClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_new_heads(&self.smoldot_client)
.await
else {
tracing::error!(
target: LOG_TARGET,
"Unable to initialize new heads subscription"
);
return
};
let Ok(mut finalized_head_subscription) =
<JsonRpseeClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_finalized_heads(&self.smoldot_client)
.await
else {
tracing::error!(
target: LOG_TARGET,
"Unable to initialize finalized heads subscription"
);
return
};
let Ok(mut all_head_subscription) = <JsonRpseeClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_all_heads(&self.smoldot_client)
.await
else {
tracing::error!(
target: LOG_TARGET,
"Unable to initialize all heads subscription"
);
return
};
loop {
tokio::select! {
evt = self.client_receiver.recv() => match evt {
Some(RpcDispatcherMessage::RegisterBestHeadListener(tx)) => {
self.best_header_listeners.push(tx);
},
Some(RpcDispatcherMessage::RegisterImportListener(tx)) => {
self.imported_header_listeners.push(tx)
},
Some(RpcDispatcherMessage::RegisterFinalizationListener(tx)) => {
self.finalized_header_listeners.push(tx)
},
Some(RpcDispatcherMessage::Request(method, params, response_sender)) => {
let closure_client = self.smoldot_client.clone();
tracing::debug!(
target: LOG_TARGET,
len = pending_requests.len(),
method,
"Request"
);
pending_requests.push(async move {
let response = closure_client.request(&method, params).await;
tracing::debug!(
target: LOG_TARGET,
method,
?response,
"Response"
);
if let Err(err) = response_sender.send(response) {
tracing::debug!(
target: LOG_TARGET,
?err,
"Recipient no longer interested in request result"
);
};
});
},
None => {
tracing::error!(target: LOG_TARGET, "RPC client receiver closed. Stopping RPC Worker.");
return;
}
},
_ = pending_requests.next(), if !pending_requests.is_empty() => {},
import_event = all_head_subscription.next() => {
if handle_notification(import_event, &mut self.imported_header_listeners).is_err() {
return
}
},
best_header_event = new_head_subscription.next() => {
if handle_notification(best_header_event, &mut self.best_header_listeners).is_err() {
return
}
}
finalized_event = finalized_head_subscription.next() => {
if handle_notification(finalized_event, &mut self.finalized_header_listeners).is_err() {
return
}
}
}
}
}
}
@@ -15,53 +15,47 @@
// along with Cumulus. If not, see <http://www.gnu.org/licenses/>.
use cumulus_primitives_core::relay_chain::{
BlockNumber as RelayBlockNumber, Header as RelayHeader,
Block as RelayBlock, BlockNumber as RelayNumber, Hash as RelayHash, Header as RelayHeader,
};
use cumulus_relay_chain_interface::{RelayChainError, RelayChainResult};
use futures::{
channel::{
mpsc::{Receiver, Sender},
oneshot::Sender as OneshotSender,
},
channel::{mpsc::Sender, oneshot::Sender as OneshotSender},
future::BoxFuture,
stream::FuturesUnordered,
FutureExt, StreamExt,
};
use jsonrpsee::{
core::{
client::{Client as JsonRpcClient, ClientT, Subscription, SubscriptionClientT},
client::{Client as JsonRpcClient, ClientT, Subscription},
params::ArrayParams,
Error as JsonRpseeError, JsonValue,
},
rpc_params,
ws_client::WsClientBuilder,
};
use lru::LruCache;
use sc_service::TaskManager;
use sc_rpc_api::chain::ChainApiClient;
use sp_runtime::generic::SignedBlock;
use std::{num::NonZeroUsize, sync::Arc};
use tokio::sync::mpsc::{
channel as tokio_channel, Receiver as TokioReceiver, Sender as TokioSender,
};
use url::Url;
const NOTIFICATION_CHANNEL_SIZE_LIMIT: usize = 20;
use crate::rpc_client::{distribute_header, RpcDispatcherMessage};
const LOG_TARGET: &str = "reconnecting-websocket-client";
/// Messages for communication between [`ReconnectingWsClient`] and [`ReconnectingWebsocketWorker`].
#[derive(Debug)]
enum RpcDispatcherMessage {
RegisterBestHeadListener(Sender<RelayHeader>),
RegisterImportListener(Sender<RelayHeader>),
RegisterFinalizationListener(Sender<RelayHeader>),
Request(String, ArrayParams, OneshotSender<Result<JsonValue, JsonRpseeError>>),
}
/// Worker that should be used in combination with [`RelayChainRpcClient`].
///
/// Must be polled to distribute header notifications to listeners.
pub struct ReconnectingWebsocketWorker {
ws_urls: Vec<String>,
/// Communication channel with the RPC client
client_receiver: TokioReceiver<RpcDispatcherMessage>,
/// Frontend for performing websocket requests.
/// Requests and stream requests are forwarded to [`ReconnectingWebsocketWorker`].
#[derive(Debug, Clone)]
pub struct ReconnectingWsClient {
/// Channel to communicate with the RPC worker
to_worker_channel: TokioSender<RpcDispatcherMessage>,
/// Senders to distribute incoming header notifications to
imported_header_listeners: Vec<Sender<RelayHeader>>,
finalized_header_listeners: Vec<Sender<RelayHeader>>,
best_header_listeners: Vec<Sender<RelayHeader>>,
}
/// Format url and force addition of a port
@@ -83,115 +77,6 @@ fn url_to_string_with_port(url: Url) -> Option<String> {
))
}
impl ReconnectingWsClient {
/// Create a new websocket client frontend.
pub async fn new(urls: Vec<Url>, task_manager: &mut TaskManager) -> RelayChainResult<Self> {
tracing::debug!(target: LOG_TARGET, "Instantiating reconnecting websocket client");
let (worker, sender) = ReconnectingWebsocketWorker::new(urls).await;
task_manager
.spawn_essential_handle()
.spawn("relay-chain-rpc-worker", None, worker.run());
Ok(Self { to_worker_channel: sender })
}
}
impl ReconnectingWsClient {
/// Perform a request via websocket connection.
pub async fn request<R>(&self, method: &str, params: ArrayParams) -> Result<R, RelayChainError>
where
R: serde::de::DeserializeOwned,
{
let (tx, rx) = futures::channel::oneshot::channel();
let message = RpcDispatcherMessage::Request(method.into(), params, tx);
self.to_worker_channel.send(message).await.map_err(|err| {
RelayChainError::WorkerCommunicationError(format!(
"Unable to send message to RPC worker: {}",
err
))
})?;
let value = rx.await.map_err(|err| {
RelayChainError::WorkerCommunicationError(format!(
"Unexpected channel close on RPC worker side: {}",
err
))
})??;
serde_json::from_value(value)
.map_err(|_| RelayChainError::GenericError("Unable to deserialize value".to_string()))
}
/// Get a stream of new best relay chain headers
pub fn get_best_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterBestHeadListener(tx))?;
Ok(rx)
}
/// Get a stream of finalized relay chain headers
pub fn get_finalized_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterFinalizationListener(
tx,
))?;
Ok(rx)
}
/// Get a stream of all imported relay chain headers
pub fn get_imported_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterImportListener(tx))?;
Ok(rx)
}
fn send_register_message_to_worker(
&self,
message: RpcDispatcherMessage,
) -> Result<(), RelayChainError> {
self.to_worker_channel
.try_send(message)
.map_err(|e| RelayChainError::WorkerCommunicationError(e.to_string()))
}
}
/// Worker that should be used in combination with [`crate::RelayChainRpcClient`].
///
/// Must be polled to distribute header notifications to listeners.
struct ReconnectingWebsocketWorker {
ws_urls: Vec<String>,
/// Communication channel with the RPC client
client_receiver: TokioReceiver<RpcDispatcherMessage>,
/// Senders to distribute incoming header notifications to
imported_header_listeners: Vec<Sender<RelayHeader>>,
finalized_header_listeners: Vec<Sender<RelayHeader>>,
best_header_listeners: Vec<Sender<RelayHeader>>,
}
fn distribute_header(header: RelayHeader, senders: &mut Vec<Sender<RelayHeader>>) {
senders.retain_mut(|e| {
match e.try_send(header.clone()) {
// Receiver has been dropped, remove Sender from list.
Err(error) if error.is_disconnected() => false,
// Channel is full. This should not happen.
// TODO: Improve error handling here
// https://github.com/paritytech/cumulus/issues/1482
Err(error) => {
tracing::error!(target: LOG_TARGET, ?error, "Event distribution channel has reached its limit. This can lead to missed notifications.");
true
},
_ => true,
}
});
}
/// Manages the active websocket client.
/// Responsible for creating request futures, subscription streams
/// and reconnections.
@@ -248,54 +133,53 @@ impl ClientManager {
}
async fn get_subscriptions(&self) -> Result<RelayChainSubscriptions, JsonRpseeError> {
let import_subscription = self
.active_client
.subscribe::<RelayHeader, _>(
"chain_subscribeAllHeads",
rpc_params![],
"chain_unsubscribeAllHeads",
)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeAllHeads` subscription."
);
e
})?;
let best_subscription = self
.active_client
.subscribe::<RelayHeader, _>(
"chain_subscribeNewHeads",
rpc_params![],
"chain_unsubscribeNewHeads",
)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeNewHeads` subscription."
);
e
})?;
let finalized_subscription = self
.active_client
.subscribe::<RelayHeader, _>(
"chain_subscribeFinalizedHeads",
rpc_params![],
"chain_unsubscribeFinalizedHeads",
)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeFinalizedHeads` subscription."
);
e
})?;
let import_subscription = <JsonRpcClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_all_heads(&self.active_client)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeAllHeads` subscription."
);
e
})?;
let best_subscription = <JsonRpcClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_new_heads(&self.active_client)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeNewHeads` subscription."
);
e
})?;
let finalized_subscription = <JsonRpcClient as ChainApiClient<
RelayNumber,
RelayHash,
RelayHeader,
SignedBlock<RelayBlock>,
>>::subscribe_finalized_heads(&self.active_client)
.await
.map_err(|e| {
tracing::error!(
target: LOG_TARGET,
?e,
"Unable to open `chain_subscribeFinalizedHeads` subscription."
);
e
})?;
Ok(RelayChainSubscriptions {
import_subscription,
@@ -344,7 +228,7 @@ enum ConnectionStatus {
impl ReconnectingWebsocketWorker {
/// Create new worker. Returns the worker and a channel to register new listeners.
async fn new(
pub async fn new(
urls: Vec<Url>,
) -> (ReconnectingWebsocketWorker, TokioSender<RpcDispatcherMessage>) {
let urls = urls.into_iter().filter_map(url_to_string_with_port).collect();
@@ -410,7 +294,7 @@ impl ReconnectingWebsocketWorker {
/// the sender from the list.
/// - Find a new valid RPC server to connect to in case the websocket connection is terminated.
/// If the worker is not able to connec to an RPC server from the list, the worker shuts down.
async fn run(mut self) {
pub async fn run(mut self) {
let mut pending_requests = FuturesUnordered::new();
let urls = std::mem::take(&mut self.ws_urls);
@@ -426,7 +310,7 @@ impl ReconnectingWebsocketWorker {
let mut imported_blocks_cache =
LruCache::new(NonZeroUsize::new(40).expect("40 is nonzero; qed."));
let mut should_reconnect = ConnectionStatus::Connected;
let mut last_seen_finalized_num: RelayBlockNumber = 0;
let mut last_seen_finalized_num: RelayNumber = 0;
loop {
// This branch is taken if the websocket connection to the current RPC server is closed.
if let ConnectionStatus::ReconnectRequired(maybe_failed_request) = should_reconnect {
@@ -14,19 +14,19 @@
// You should have received a copy of the GNU General Public License
// along with Cumulus. If not, see <http://www.gnu.org/licenses/>.
use futures::channel::mpsc::Receiver;
use jsonrpsee::{core::params::ArrayParams, rpc_params};
use parity_scale_codec::{Decode, Encode};
use futures::channel::{
mpsc::{Receiver, Sender},
oneshot::Sender as OneshotSender,
};
use jsonrpsee::{
core::{params::ArrayParams, Error as JsonRpseeError},
rpc_params,
};
use serde::de::DeserializeOwned;
pub use url::Url;
use serde_json::Value as JsonValue;
use tokio::sync::mpsc::Sender as TokioSender;
use sc_client_api::StorageData;
use sc_rpc_api::{state::ReadProof, system::Health};
use sc_service::TaskManager;
use sp_api::RuntimeVersion;
use sp_consensus_babe::Epoch;
use sp_core::sp_std::collections::btree_map::BTreeMap;
use sp_storage::StorageKey;
use parity_scale_codec::{Decode, Encode};
use cumulus_primitives_core::{
relay_chain::{
@@ -42,35 +42,96 @@ use cumulus_primitives_core::{
};
use cumulus_relay_chain_interface::{RelayChainError, RelayChainResult};
use crate::reconnecting_ws_client::ReconnectingWsClient;
use sc_client_api::StorageData;
use sc_rpc_api::{state::ReadProof, system::Health};
use sc_service::TaskManager;
use sp_api::RuntimeVersion;
use sp_consensus_babe::Epoch;
use sp_core::sp_std::collections::btree_map::BTreeMap;
use sp_storage::StorageKey;
use crate::{
light_client_worker::{build_smoldot_client, LightClientRpcWorker},
reconnecting_ws_client::ReconnectingWebsocketWorker,
};
pub use url::Url;
const LOG_TARGET: &str = "relay-chain-rpc-client";
const NOTIFICATION_CHANNEL_SIZE_LIMIT: usize = 20;
/// Client that maps RPC methods and deserializes results
#[derive(Clone)]
pub struct RelayChainRpcClient {
/// Websocket client to make calls
ws_client: ReconnectingWsClient,
/// Messages for communication between [`RelayChainRpcClient`] and the RPC workers.
#[derive(Debug)]
pub enum RpcDispatcherMessage {
/// Register new listener for the best headers stream. Contains a sender which will be used
/// to send incoming headers.
RegisterBestHeadListener(Sender<RelayHeader>),
/// Register new listener for the import headers stream. Contains a sender which will be used
/// to send incoming headers.
RegisterImportListener(Sender<RelayHeader>),
/// Register new listener for the finalized headers stream. Contains a sender which will be
/// used to send incoming headers.
RegisterFinalizationListener(Sender<RelayHeader>),
/// Register new listener for the finalized headers stream.
/// Contains the following:
/// - [`String`] representing the RPC method to be called
/// - [`ArrayParams`] for the parameters to the RPC call
/// - [`OneshotSender`] for the return value of the request
Request(String, ArrayParams, OneshotSender<Result<JsonValue, JsonRpseeError>>),
}
/// Entry point to create [`RelayChainRpcClient`] and start a worker that distributes notifications.
/// Entry point to create [`RelayChainRpcClient`] and start a worker that communicates
/// to JsonRPC servers over the network.
pub async fn create_client_and_start_worker(
urls: Vec<Url>,
task_manager: &mut TaskManager,
) -> RelayChainResult<RelayChainRpcClient> {
let ws_client = ReconnectingWsClient::new(urls, task_manager).await?;
let (worker, sender) = ReconnectingWebsocketWorker::new(urls).await;
let client = RelayChainRpcClient::new(ws_client).await?;
task_manager
.spawn_essential_handle()
.spawn("relay-chain-rpc-worker", None, worker.run());
let client = RelayChainRpcClient::new(sender);
Ok(client)
}
/// Entry point to create [`RelayChainRpcClient`] and start a worker that communicates
/// with an embedded smoldot instance.
pub async fn create_client_and_start_light_client_worker(
chain_spec: String,
task_manager: &mut TaskManager,
) -> RelayChainResult<RelayChainRpcClient> {
let (client, chain_id, json_rpc_responses) =
build_smoldot_client(task_manager.spawn_handle(), &chain_spec).await?;
let (worker, sender) = LightClientRpcWorker::new(client, json_rpc_responses, chain_id);
task_manager
.spawn_essential_handle()
.spawn("relay-light-client-worker", None, worker.run());
let client = RelayChainRpcClient::new(sender);
Ok(client)
}
/// Client that maps RPC methods and deserializes results
#[derive(Clone)]
pub struct RelayChainRpcClient {
/// Sender to send messages to the worker.
worker_channel: TokioSender<RpcDispatcherMessage>,
}
impl RelayChainRpcClient {
/// Initialize new RPC Client.
async fn new(ws_client: ReconnectingWsClient) -> RelayChainResult<Self> {
let client = RelayChainRpcClient { ws_client };
Ok(client)
///
/// This client expects a channel connected to a worker that processes
/// requests sent via this channel.
pub(crate) fn new(worker_channel: TokioSender<RpcDispatcherMessage>) -> Self {
RelayChainRpcClient { worker_channel }
}
/// Call a call to `state_call` rpc method.
@@ -129,8 +190,25 @@ impl RelayChainRpcClient {
R: DeserializeOwned + std::fmt::Debug,
OR: Fn(&RelayChainError),
{
self.ws_client.request(method, params).await.map_err(|err| {
trace_error(&err);
let (tx, rx) = futures::channel::oneshot::channel();
let message = RpcDispatcherMessage::Request(method.into(), params, tx);
self.worker_channel.send(message).await.map_err(|err| {
RelayChainError::WorkerCommunicationError(format!(
"Unable to send message to RPC worker: {}",
err
))
})?;
let value = rx.await.map_err(|err| {
RelayChainError::WorkerCommunicationError(format!(
"Unexpected channel close on RPC worker side: {}",
err
))
})??;
serde_json::from_value(value).map_err(|_| {
trace_error(&RelayChainError::GenericError("Unable to deserialize value".to_string()));
RelayChainError::RpcCallError(method.to_string())
})
}
@@ -537,18 +615,57 @@ impl RelayChainRpcClient {
.await
}
fn send_register_message_to_worker(
&self,
message: RpcDispatcherMessage,
) -> Result<(), RelayChainError> {
self.worker_channel
.try_send(message)
.map_err(|e| RelayChainError::WorkerCommunicationError(e.to_string()))
}
/// Get a stream of all imported relay chain headers
pub fn get_imported_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
self.ws_client.get_imported_heads_stream()
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterImportListener(tx))?;
Ok(rx)
}
/// Get a stream of new best relay chain headers
pub fn get_best_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
self.ws_client.get_best_heads_stream()
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterBestHeadListener(tx))?;
Ok(rx)
}
/// Get a stream of finalized relay chain headers
pub fn get_finalized_heads_stream(&self) -> Result<Receiver<RelayHeader>, RelayChainError> {
self.ws_client.get_finalized_heads_stream()
let (tx, rx) =
futures::channel::mpsc::channel::<RelayHeader>(NOTIFICATION_CHANNEL_SIZE_LIMIT);
self.send_register_message_to_worker(RpcDispatcherMessage::RegisterFinalizationListener(
tx,
))?;
Ok(rx)
}
}
/// Send `header` through all channels contained in `senders`.
/// If no one is listening to the sender, it is removed from the vector.
pub fn distribute_header(header: RelayHeader, senders: &mut Vec<Sender<RelayHeader>>) {
senders.retain_mut(|e| {
match e.try_send(header.clone()) {
// Receiver has been dropped, remove Sender from list.
Err(error) if error.is_disconnected() => false,
// Channel is full. This should not happen.
// TODO: Improve error handling here
// https://github.com/paritytech/cumulus/issues/1482
Err(error) => {
tracing::error!(target: LOG_TARGET, ?error, "Event distribution channel has reached its limit. This can lead to missed notifications.");
true
},
_ => true,
}
});
}
@@ -0,0 +1,211 @@
// Copyright 2023 Parity Technologies (UK) Ltd.
// This file is part of Cumulus.
// Cumulus is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Cumulus is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Cumulus. If not, see <http://www.gnu.org/licenses/>.
use core::time::Duration;
use futures::prelude::*;
use sc_service::SpawnTaskHandle;
use smoldot::libp2p::{websocket, with_buffers};
use smoldot_light::platform::{
Address, ConnectError, ConnectionType, IpAddr, MultiStreamWebRtcConnection, PlatformRef,
SubstreamDirection,
};
use std::{net::SocketAddr, pin::Pin, time::Instant};
use tokio::net::TcpStream;
use tokio_util::compat::{Compat, TokioAsyncReadCompatExt};
type CompatTcpStream = Compat<TcpStream>;
/// Platform implementation for tokio
/// This implementation is a port of the implementation for smol:
/// https://github.com/smol-dot/smoldot/blob/8c577b4a753fe96190f813070564ecc742b91a16/light-base/src/platform/default.rs
#[derive(Clone)]
pub struct TokioPlatform {
spawner: SpawnTaskHandle,
}
impl TokioPlatform {
pub fn new(spawner: SpawnTaskHandle) -> Self {
TokioPlatform { spawner }
}
}
impl PlatformRef for TokioPlatform {
type Delay = future::BoxFuture<'static, ()>;
type Instant = Instant;
type MultiStream = std::convert::Infallible;
type Stream = Stream;
type StreamConnectFuture = future::BoxFuture<'static, Result<Self::Stream, ConnectError>>;
type MultiStreamConnectFuture = future::BoxFuture<
'static,
Result<MultiStreamWebRtcConnection<Self::MultiStream>, ConnectError>,
>;
type ReadWriteAccess<'a> = with_buffers::ReadWriteAccess<'a>;
type StreamUpdateFuture<'a> = future::BoxFuture<'a, ()>;
type StreamErrorRef<'a> = &'a std::io::Error;
type NextSubstreamFuture<'a> = future::Pending<Option<(Self::Stream, SubstreamDirection)>>;
fn now_from_unix_epoch(&self) -> Duration {
// Intentionally panic if the time is configured earlier than the UNIX EPOCH.
std::time::UNIX_EPOCH.elapsed().unwrap()
}
fn now(&self) -> Self::Instant {
Instant::now()
}
fn fill_random_bytes(&self, buffer: &mut [u8]) {
rand::RngCore::fill_bytes(&mut rand::thread_rng(), buffer);
}
fn sleep(&self, duration: Duration) -> Self::Delay {
tokio::time::sleep(duration).boxed()
}
fn sleep_until(&self, when: Self::Instant) -> Self::Delay {
let duration = when.saturating_duration_since(Instant::now());
self.sleep(duration)
}
fn supports_connection_type(&self, connection_type: ConnectionType) -> bool {
matches!(
connection_type,
ConnectionType::TcpIpv4 |
ConnectionType::TcpIpv6 |
ConnectionType::TcpDns |
ConnectionType::WebSocketIpv4 { .. } |
ConnectionType::WebSocketIpv6 { .. } |
ConnectionType::WebSocketDns { secure: false, .. }
)
}
fn connect_stream(&self, multiaddr: Address) -> Self::StreamConnectFuture {
let (tcp_socket_addr, host_if_websocket): (
either::Either<SocketAddr, (String, u16)>,
Option<String>,
) = match multiaddr {
Address::TcpDns { hostname, port } =>
(either::Right((hostname.to_string(), port)), None),
Address::TcpIp { ip: IpAddr::V4(ip), port } =>
(either::Left(SocketAddr::from((ip, port))), None),
Address::TcpIp { ip: IpAddr::V6(ip), port } =>
(either::Left(SocketAddr::from((ip, port))), None),
Address::WebSocketDns { hostname, port, secure: false } => (
either::Right((hostname.to_string(), port)),
Some(format!("{}:{}", hostname, port)),
),
Address::WebSocketIp { ip: IpAddr::V4(ip), port } => {
let addr = SocketAddr::from((ip, port));
(either::Left(addr), Some(addr.to_string()))
},
Address::WebSocketIp { ip: IpAddr::V6(ip), port } => {
let addr = SocketAddr::from((ip, port));
(either::Left(addr), Some(addr.to_string()))
},
// The API user of the `PlatformRef` trait is never supposed to open connections of
// a type that isn't supported.
_ => unreachable!(),
};
Box::pin(async move {
let tcp_socket = match tcp_socket_addr {
either::Left(socket_addr) => TcpStream::connect(socket_addr).await,
either::Right((dns, port)) => TcpStream::connect((&dns[..], port)).await,
};
if let Ok(tcp_socket) = &tcp_socket {
let _ = tcp_socket.set_nodelay(true);
}
let socket: TcpOrWs = match (tcp_socket, host_if_websocket) {
(Ok(tcp_socket), Some(host)) => future::Either::Right(
websocket::websocket_client_handshake(websocket::Config {
tcp_socket: tcp_socket.compat(),
host: &host,
url: "/",
})
.await
.map_err(|err| ConnectError {
message: format!("Failed to negotiate WebSocket: {err}"),
})?,
),
(Ok(tcp_socket), None) => future::Either::Left(tcp_socket.compat()),
(Err(err), _) =>
return Err(ConnectError { message: format!("Failed to reach peer: {err}") }),
};
Ok(Stream(with_buffers::WithBuffers::new(socket)))
})
}
fn open_out_substream(&self, _c: &mut Self::MultiStream) {
// This function can only be called with so-called "multi-stream" connections. We never
// open such connection.
}
fn next_substream<'a>(&self, c: &'a mut Self::MultiStream) -> Self::NextSubstreamFuture<'a> {
// This function can only be called with so-called "multi-stream" connections. We never
// open such connection.
match *c {}
}
fn spawn_task(
&self,
_: std::borrow::Cow<str>,
task: impl Future<Output = ()> + Send + 'static,
) {
self.spawner.spawn("cumulus-internal-light-client-task", None, task)
}
fn client_name(&self) -> std::borrow::Cow<str> {
"cumulus-relay-chain-light-client".into()
}
fn client_version(&self) -> std::borrow::Cow<str> {
env!("CARGO_PKG_VERSION").into()
}
fn connect_multistream(
&self,
_address: smoldot_light::platform::MultiStreamAddress,
) -> Self::MultiStreamConnectFuture {
unimplemented!("Multistream not supported!")
}
fn read_write_access<'a>(
&self,
stream: Pin<&'a mut Self::Stream>,
) -> Result<Self::ReadWriteAccess<'a>, &'a std::io::Error> {
let stream = stream.project();
stream.0.read_write_access(Instant::now())
}
fn wait_read_write_again<'a>(
&self,
stream: Pin<&'a mut Self::Stream>,
) -> Self::StreamUpdateFuture<'a> {
let stream = stream.project();
Box::pin(stream.0.wait_read_write_again(|when| async move {
tokio::time::sleep_until(when.into()).await;
}))
}
}
type TcpOrWs = future::Either<CompatTcpStream, websocket::Connection<CompatTcpStream>>;
/// Implementation detail of [`TokioPlatform`].
#[pin_project::pin_project]
pub struct Stream(#[pin] with_buffers::WithBuffers<TcpOrWs>);
+17 -13
View File
@@ -25,7 +25,9 @@ use cumulus_client_pov_recovery::{PoVRecovery, RecoveryDelayRange, RecoveryHandl
use cumulus_primitives_core::{CollectCollationInfo, ParaId};
use cumulus_relay_chain_inprocess_interface::build_inprocess_relay_chain;
use cumulus_relay_chain_interface::{RelayChainInterface, RelayChainResult};
use cumulus_relay_chain_minimal_node::build_minimal_relay_chain_node;
use cumulus_relay_chain_minimal_node::{
build_minimal_relay_chain_node_light_client, build_minimal_relay_chain_node_with_rpc,
};
use futures::{
channel::{mpsc, oneshot},
FutureExt, StreamExt,
@@ -342,28 +344,30 @@ pub fn prepare_node_config(mut parachain_config: Configuration) -> Configuration
/// Will return a minimal relay chain node with RPC
/// client or an inprocess node, based on the [`CollatorOptions`] passed in.
pub async fn build_relay_chain_interface(
polkadot_config: Configuration,
relay_chain_config: Configuration,
parachain_config: &Configuration,
telemetry_worker_handle: Option<TelemetryWorkerHandle>,
task_manager: &mut TaskManager,
collator_options: CollatorOptions,
hwbench: Option<sc_sysinfo::HwBench>,
) -> RelayChainResult<(Arc<(dyn RelayChainInterface + 'static)>, Option<CollatorPair>)> {
if !collator_options.relay_chain_rpc_urls.is_empty() {
build_minimal_relay_chain_node(
polkadot_config,
task_manager,
collator_options.relay_chain_rpc_urls,
)
.await
} else {
build_inprocess_relay_chain(
polkadot_config,
match collator_options.relay_chain_mode {
cumulus_client_cli::RelayChainMode::Embedded => build_inprocess_relay_chain(
relay_chain_config,
parachain_config,
telemetry_worker_handle,
task_manager,
hwbench,
)
),
cumulus_client_cli::RelayChainMode::ExternalRpc(rpc_target_urls) =>
build_minimal_relay_chain_node_with_rpc(
relay_chain_config,
task_manager,
rpc_target_urls,
)
.await,
cumulus_client_cli::RelayChainMode::LightClient =>
build_minimal_relay_chain_node_light_client(relay_chain_config, task_manager).await,
}
}
+1 -11
View File
@@ -2,7 +2,7 @@ use std::net::SocketAddr;
use cumulus_primitives_core::ParaId;
use frame_benchmarking_cli::{BenchmarkCmd, SUBSTRATE_REFERENCE_HARDWARE};
use log::{info, warn};
use log::info;
use parachain_template_runtime::Block;
use sc_cli::{
ChainSpec, CliConfiguration, DefaultConfigurationValues, ImportParams, KeystoreParams,
@@ -278,16 +278,6 @@ pub fn run() -> Result<()> {
info!("Parachain Account: {parachain_account}");
info!("Is collating: {}", if config.role.is_authority() { "yes" } else { "no" });
if !collator_options.relay_chain_rpc_urls.is_empty() &&
!cli.relay_chain_args.is_empty()
{
warn!(
"Detected relay chain node arguments together with --relay-chain-rpc-url. \
This command starts a minimal Polkadot node that only uses a \
network-related subset of all relay chain CLI options."
);
}
crate::service::start_parachain_node(
config,
polkadot_config,
+1 -9
View File
@@ -21,7 +21,7 @@ use crate::{
};
use cumulus_primitives_core::ParaId;
use frame_benchmarking_cli::{BenchmarkCmd, SUBSTRATE_REFERENCE_HARDWARE};
use log::{info, warn};
use log::info;
use parachains_common::{AssetHubPolkadotAuraId, AuraId};
use sc_cli::{
ChainSpec, CliConfiguration, DefaultConfigurationValues, ImportParams, KeystoreParams,
@@ -821,14 +821,6 @@ pub fn run() -> Result<()> {
info!("Parachain Account: {}", parachain_account);
info!("Is collating: {}", if config.role.is_authority() { "yes" } else { "no" });
if !collator_options.relay_chain_rpc_urls.is_empty() && !cli.relaychain_args.is_empty() {
warn!(
"Detected relay chain node arguments together with --relay-chain-rpc-url. \
This command starts a minimal Polkadot node that only uses a \
network-related subset of all relay chain CLI options."
);
}
match config.chain_spec.runtime() {
Runtime::AssetHubPolkadot => crate::service::start_generic_aura_node::<
asset_hub_polkadot_runtime::RuntimeApi,
+31 -26
View File
@@ -34,7 +34,7 @@ use std::{
use url::Url;
use crate::runtime::Weight;
use cumulus_client_cli::CollatorOptions;
use cumulus_client_cli::{CollatorOptions, RelayChainMode};
use cumulus_client_consensus_common::{
ParachainBlockImport as TParachainBlockImport, ParachainCandidate, ParachainConsensus,
};
@@ -48,7 +48,9 @@ use cumulus_client_service::{
use cumulus_primitives_core::ParaId;
use cumulus_relay_chain_inprocess_interface::RelayChainInProcessInterface;
use cumulus_relay_chain_interface::{RelayChainError, RelayChainInterface, RelayChainResult};
use cumulus_relay_chain_minimal_node::build_minimal_relay_chain_node;
use cumulus_relay_chain_minimal_node::{
build_minimal_relay_chain_node_light_client, build_minimal_relay_chain_node_with_rpc,
};
use cumulus_test_runtime::{Hash, Header, NodeBlock as Block, RuntimeApi};
@@ -249,26 +251,30 @@ async fn build_relay_chain_interface(
collator_options: CollatorOptions,
task_manager: &mut TaskManager,
) -> RelayChainResult<Arc<dyn RelayChainInterface + 'static>> {
if !collator_options.relay_chain_rpc_urls.is_empty() {
return build_minimal_relay_chain_node(
let relay_chain_full_node = match collator_options.relay_chain_mode {
cumulus_client_cli::RelayChainMode::Embedded => polkadot_test_service::new_full(
relay_chain_config,
task_manager,
collator_options.relay_chain_rpc_urls,
if let Some(ref key) = collator_key {
polkadot_service::IsParachainNode::Collator(key.clone())
} else {
polkadot_service::IsParachainNode::Collator(CollatorPair::generate().0)
},
None,
)
.await
.map(|r| r.0)
}
let relay_chain_full_node = polkadot_test_service::new_full(
relay_chain_config,
if let Some(ref key) = collator_key {
polkadot_service::IsParachainNode::Collator(key.clone())
} else {
polkadot_service::IsParachainNode::Collator(CollatorPair::generate().0)
},
None,
)
.map_err(|e| RelayChainError::Application(Box::new(e) as Box<_>))?;
.map_err(|e| RelayChainError::Application(Box::new(e) as Box<_>))?,
cumulus_client_cli::RelayChainMode::ExternalRpc(rpc_target_urls) =>
return build_minimal_relay_chain_node_with_rpc(
relay_chain_config,
task_manager,
rpc_target_urls,
)
.await
.map(|r| r.0),
cumulus_client_cli::RelayChainMode::LightClient =>
return build_minimal_relay_chain_node_light_client(relay_chain_config, task_manager)
.await
.map(|r| r.0),
};
task_manager.add_child(relay_chain_full_node.task_manager);
tracing::info!("Using inprocess node.");
@@ -505,7 +511,7 @@ pub struct TestNodeBuilder {
storage_update_func_parachain: Option<Box<dyn Fn()>>,
storage_update_func_relay_chain: Option<Box<dyn Fn()>>,
consensus: Consensus,
relay_chain_full_node_url: Vec<Url>,
relay_chain_mode: RelayChainMode,
endowed_accounts: Vec<AccountId>,
}
@@ -529,8 +535,8 @@ impl TestNodeBuilder {
storage_update_func_parachain: None,
storage_update_func_relay_chain: None,
consensus: Consensus::RelayChain,
relay_chain_full_node_url: vec![],
endowed_accounts: Default::default(),
relay_chain_mode: RelayChainMode::Embedded,
}
}
@@ -624,7 +630,7 @@ impl TestNodeBuilder {
/// Connect to full node via RPC.
pub fn use_external_relay_chain_node_at_url(mut self, network_address: Url) -> Self {
self.relay_chain_full_node_url = vec![network_address];
self.relay_chain_mode = RelayChainMode::ExternalRpc(vec![network_address]);
self
}
@@ -633,7 +639,7 @@ impl TestNodeBuilder {
let mut localhost_url =
Url::parse("ws://localhost").expect("Should be able to parse localhost Url");
localhost_url.set_port(Some(port)).expect("Should be able to set port");
self.relay_chain_full_node_url = vec![localhost_url];
self.relay_chain_mode = RelayChainMode::ExternalRpc(vec![localhost_url]);
self
}
@@ -665,8 +671,7 @@ impl TestNodeBuilder {
false,
);
let collator_options =
CollatorOptions { relay_chain_rpc_urls: self.relay_chain_full_node_url };
let collator_options = CollatorOptions { relay_chain_mode: self.relay_chain_mode };
relay_chain_config.network.node_name =
format!("{} (relay chain)", relay_chain_config.network.node_name);
+14 -6
View File
@@ -33,8 +33,8 @@ add_to_genesis = false
command = "test-parachain"
args = ["--disable-block-announcements"]
# run alice as a parachain collator who does not produce blocks
# alice is a bootnode for bob and charlie
# run 'alice' as a parachain collator who does not produce blocks
# 'alice' is a bootnode for 'bob' and 'charlie'
[[parachains.collators]]
name = "alice"
validator = true # collator
@@ -42,7 +42,7 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--use-null-consensus", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}", "--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
# run eve as a parachain full node
# run 'charlie' as a parachain full node
[[parachains.collators]]
name = "charlie"
validator = false # full node
@@ -50,7 +50,7 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}","--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
# we fail recovery for eve from time to time to test retries
# we fail recovery for 'eve' from time to time to test retries
[[parachains.collators]]
name = "eve"
validator = true # collator
@@ -58,7 +58,7 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--fail-pov-recovery", "--use-null-consensus", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}", "--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
# run one as a RPC collator who does not produce blocks
# run 'one' as a RPC collator who does not produce blocks
[[parachains.collators]]
name = "one"
validator = true # collator
@@ -66,10 +66,18 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--use-null-consensus", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}", "--relay-chain-rpc-url {{'ferdie'|zombie('wsUri')}}", "--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
# run two as a RPC parachain full node
# run 'two' as a RPC parachain full node
[[parachains.collators]]
name = "two"
validator = false # full node
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}", "--relay-chain-rpc-url {{'ferdie'|zombie('wsUri')}}", "--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
# run 'three' with light client
[[parachains.collators]]
name = "three"
validator = false # full node
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["-lparachain::availability=trace,sync=debug,parachain=debug,cumulus-pov-recovery=debug,cumulus-consensus=debug", "--disable-block-announcements", "--bootnodes {{'bob'|zombie('multiAddress')}}", "--relay-chain-light-client", "--", "--reserved-only", "--reserved-nodes {{'ferdie'|zombie('multiAddress')}}"]
@@ -13,4 +13,5 @@ alice: reports block height is at least 20 within 600 seconds
charlie: reports block height is at least 20 within 600 seconds
one: reports block height is at least 20 within 800 seconds
two: reports block height is at least 20 within 800 seconds
three: reports block height is at least 20 within 800 seconds
eve: reports block height is at least 20 within 800 seconds
@@ -40,3 +40,11 @@ cumulus_based = true
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["--reserved-only", "--reserved-nodes {{'charlie'|zombie('multiAddress')}}", "--relay-chain-rpc-url {{'alice'|zombie('wsUri')}}"]
# run cumulus node ferdie (with embedded light client) and wait for it to sync some blocks
[[parachains.collators]]
name = "ferdie"
validator = false
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["--relay-chain-light-client", "--reserved-only", "--reserved-nodes {{'charlie'|zombie('multiAddress')}}"]
@@ -5,3 +5,4 @@ Creds: config
alice: parachain 2000 is registered within 225 seconds
dave: reports block height is at least 7 within 250 seconds
eve: reports block height is at least 7 within 250 seconds
ferdie: reports block height is at least 7 within 250 seconds
@@ -31,6 +31,7 @@ cumulus_based = true
chain_spec_path = "zombienet/tests/0007-warp-sync-parachain-spec.json"
add_to_genesis = false
# Run 'dave' as parachain collator.
[[parachains.collators]]
name = "dave"
validator = true
@@ -39,6 +40,7 @@ add_to_genesis = false
args = ["-lparachain=debug"]
db_snapshot = "https://storage.googleapis.com/zombienet-db-snaps/cumulus/0007-full_node_warp_sync/parachain-587c1ed24ddd7de05c237cf7c158fff53b8f5b26.tgz"
# Run 'eve' as parachain collator.
[[parachains.collators]]
name = "eve"
validator = true
@@ -47,6 +49,7 @@ add_to_genesis = false
args = ["-lparachain=debug"]
db_snapshot = "https://storage.googleapis.com/zombienet-db-snaps/cumulus/0007-full_node_warp_sync/parachain-587c1ed24ddd7de05c237cf7c158fff53b8f5b26.tgz"
# Run 'ferdie' as parachain collator.
[[parachains.collators]]
name = "ferdie"
validator = true
@@ -55,6 +58,7 @@ add_to_genesis = false
args = ["-lparachain=debug"]
db_snapshot = "https://storage.googleapis.com/zombienet-db-snaps/cumulus/0007-full_node_warp_sync/parachain-587c1ed24ddd7de05c237cf7c158fff53b8f5b26.tgz"
# Run 'one' as parachain full node. Parachain and relay chain are warpsyncing.
[[parachains.collators]]
name = "one"
validator = false
@@ -62,6 +66,8 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lsync=debug","--sync warp","--","--sync warp"]
# Run 'two' as parachain full node. Parachain is warpsyncing and the node
# uses relay chain node 'alice' as external RPC node.
[[parachains.collators]]
name = "two"
validator = false
@@ -69,9 +75,20 @@ add_to_genesis = false
command = "test-parachain"
args = ["-lsync=debug","--sync warp","--relay-chain-rpc-urls {{'alice'|zombie('wsUri')}}"]
# Run 'three' as parachain full node. Parachain is warpsyncing and the node
# uses relay chain node 'dave' as external RPC node.
[[parachains.collators]]
name = "three"
validator = false
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["-lsync=debug","--sync warp","--relay-chain-rpc-urls {{'dave'|zombie('wsUri')}}"]
# Run 'four' as parachain full node. Parachain is warpsyncing and the node
# uses an internal relay chain light client.
[[parachains.collators]]
name = "four"
validator = false
image = "{{COL_IMAGE}}"
command = "test-parachain"
args = ["-lsync=debug","--sync warp","--relay-chain-light-client"]
@@ -6,3 +6,4 @@ alice: parachain 2000 is registered within 225 seconds
one: reports block height is at least 770 within 225 seconds
two: reports block height is at least 770 within 225 seconds
three: reports block height is at least 770 within 225 seconds
four: reports block height is at least 770 within 225 seconds