Extract syncing protocol from sc-network (#12828)

* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Add missing docs

* Refactor code

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
This commit is contained in:
Aaro Altonen
2023-03-06 18:33:38 +02:00
committed by GitHub
parent 8adde84330
commit 1a7f5be07f
57 changed files with 2904 additions and 2877 deletions
+372 -3
View File
@@ -18,11 +18,33 @@
//! Configuration of the networking layer.
use crate::protocol;
pub use crate::{
protocol::{self, role::Role},
request_responses::{
IncomingRequest, OutgoingResponse, ProtocolConfig as RequestResponseConfig,
},
sync::warp::WarpSyncProvider,
ExHashT,
};
pub use libp2p::{build_multiaddr, core::PublicKey, identity};
use codec::Encode;
use libp2p::{multiaddr, Multiaddr, PeerId};
use std::{fmt, str, str::FromStr};
use libp2p::{
identity::{ed25519, Keypair},
multiaddr, Multiaddr, PeerId,
};
use zeroize::Zeroize;
use std::{
error::Error,
fmt, fs,
io::{self, Write},
iter,
net::Ipv4Addr,
path::{Path, PathBuf},
str,
str::FromStr,
};
/// Protocol name prefix, transmitted on the wire for legacy protocol names.
/// I.e., `dot` in `/dot/sync/2`. Should be unique for each chain. Always UTF-8.
@@ -331,3 +353,350 @@ impl NonReservedPeerMode {
}
}
}
/// Sync operation mode.
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum SyncMode {
/// Full block download and verification.
Full,
/// Download blocks and the latest state.
Fast {
/// Skip state proof download and verification.
skip_proofs: bool,
/// Download indexed transactions for recent blocks.
storage_chain_mode: bool,
},
/// Warp sync - verify authority set transitions and the latest state.
Warp,
}
impl SyncMode {
/// Returns if `self` is [`Self::Warp`].
pub fn is_warp(&self) -> bool {
matches!(self, Self::Warp)
}
/// Returns if `self` is [`Self::Fast`].
pub fn is_fast(&self) -> bool {
matches!(self, Self::Fast { .. })
}
}
impl Default for SyncMode {
fn default() -> Self {
Self::Full
}
}
/// Network service configuration.
#[derive(Clone, Debug)]
pub struct NetworkConfiguration {
/// Directory path to store network-specific configuration. None means nothing will be saved.
pub net_config_path: Option<PathBuf>,
/// Multiaddresses to listen for incoming connections.
pub listen_addresses: Vec<Multiaddr>,
/// Multiaddresses to advertise. Detected automatically if empty.
pub public_addresses: Vec<Multiaddr>,
/// List of initial node addresses
pub boot_nodes: Vec<MultiaddrWithPeerId>,
/// The node key configuration, which determines the node's network identity keypair.
pub node_key: NodeKeyConfig,
/// List of request-response protocols that the node supports.
pub request_response_protocols: Vec<RequestResponseConfig>,
/// Configuration for the default set of nodes used for block syncing and transactions.
pub default_peers_set: SetConfig,
/// Number of substreams to reserve for full nodes for block syncing and transactions.
/// Any other slot will be dedicated to light nodes.
///
/// This value is implicitly capped to `default_set.out_peers + default_set.in_peers`.
pub default_peers_set_num_full: u32,
/// Configuration for extra sets of nodes.
pub extra_sets: Vec<NonDefaultSetConfig>,
/// Client identifier. Sent over the wire for debugging purposes.
pub client_version: String,
/// Name of the node. Sent over the wire for debugging purposes.
pub node_name: String,
/// Configuration for the transport layer.
pub transport: TransportConfig,
/// Maximum number of peers to ask the same blocks in parallel.
pub max_parallel_downloads: u32,
/// Initial syncing mode.
pub sync_mode: SyncMode,
/// True if Kademlia random discovery should be enabled.
///
/// If true, the node will automatically randomly walk the DHT in order to find new peers.
pub enable_dht_random_walk: bool,
/// Should we insert non-global addresses into the DHT?
pub allow_non_globals_in_dht: bool,
/// Require iterative Kademlia DHT queries to use disjoint paths for increased resiliency in
/// the presence of potentially adversarial nodes.
pub kademlia_disjoint_query_paths: bool,
/// Enable serving block data over IPFS bitswap.
pub ipfs_server: bool,
/// Size of Yamux receive window of all substreams. `None` for the default (256kiB).
/// Any value less than 256kiB is invalid.
///
/// # Context
///
/// By design, notifications substreams on top of Yamux connections only allow up to `N` bytes
/// to be transferred at a time, where `N` is the Yamux receive window size configurable here.
/// This means, in practice, that every `N` bytes must be acknowledged by the receiver before
/// the sender can send more data. The maximum bandwidth of each notifications substream is
/// therefore `N / round_trip_time`.
///
/// It is recommended to leave this to `None`, and use a request-response protocol instead if
/// a large amount of data must be transferred. The reason why the value is configurable is
/// that some Substrate users mis-use notification protocols to send large amounts of data.
/// As such, this option isn't designed to stay and will likely get removed in the future.
///
/// Note that configuring a value here isn't a modification of the Yamux protocol, but rather
/// a modification of the way the implementation works. Different nodes with different
/// configured values remain compatible with each other.
pub yamux_window_size: Option<u32>,
}
impl NetworkConfiguration {
/// Create new default configuration
pub fn new<SN: Into<String>, SV: Into<String>>(
node_name: SN,
client_version: SV,
node_key: NodeKeyConfig,
net_config_path: Option<PathBuf>,
) -> Self {
let default_peers_set = SetConfig::default();
Self {
net_config_path,
listen_addresses: Vec::new(),
public_addresses: Vec::new(),
boot_nodes: Vec::new(),
node_key,
request_response_protocols: Vec::new(),
default_peers_set_num_full: default_peers_set.in_peers + default_peers_set.out_peers,
default_peers_set,
extra_sets: Vec::new(),
client_version: client_version.into(),
node_name: node_name.into(),
transport: TransportConfig::Normal { enable_mdns: false, allow_private_ip: true },
max_parallel_downloads: 5,
sync_mode: SyncMode::Full,
enable_dht_random_walk: true,
allow_non_globals_in_dht: false,
kademlia_disjoint_query_paths: false,
yamux_window_size: None,
ipfs_server: false,
}
}
/// Create new default configuration for localhost-only connection with random port (useful for
/// testing)
pub fn new_local() -> NetworkConfiguration {
let mut config =
NetworkConfiguration::new("test-node", "test-client", Default::default(), None);
config.listen_addresses =
vec![iter::once(multiaddr::Protocol::Ip4(Ipv4Addr::new(127, 0, 0, 1)))
.chain(iter::once(multiaddr::Protocol::Tcp(0)))
.collect()];
config.allow_non_globals_in_dht = true;
config
}
/// Create new default configuration for localhost-only connection with random port (useful for
/// testing)
pub fn new_memory() -> NetworkConfiguration {
let mut config =
NetworkConfiguration::new("test-node", "test-client", Default::default(), None);
config.listen_addresses =
vec![iter::once(multiaddr::Protocol::Ip4(Ipv4Addr::new(127, 0, 0, 1)))
.chain(iter::once(multiaddr::Protocol::Tcp(0)))
.collect()];
config.allow_non_globals_in_dht = true;
config
}
}
/// The configuration of a node's secret key, describing the type of key
/// and how it is obtained. A node's identity keypair is the result of
/// the evaluation of the node key configuration.
#[derive(Clone, Debug)]
pub enum NodeKeyConfig {
/// A Ed25519 secret key configuration.
Ed25519(Secret<ed25519::SecretKey>),
}
impl Default for NodeKeyConfig {
fn default() -> NodeKeyConfig {
Self::Ed25519(Secret::New)
}
}
/// The options for obtaining a Ed25519 secret key.
pub type Ed25519Secret = Secret<ed25519::SecretKey>;
/// The configuration options for obtaining a secret key `K`.
#[derive(Clone)]
pub enum Secret<K> {
/// Use the given secret key `K`.
Input(K),
/// Read the secret key from a file. If the file does not exist,
/// it is created with a newly generated secret key `K`. The format
/// of the file is determined by `K`:
///
/// * `ed25519::SecretKey`: An unencoded 32 bytes Ed25519 secret key.
File(PathBuf),
/// Always generate a new secret key `K`.
New,
}
impl<K> fmt::Debug for Secret<K> {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
Self::Input(_) => f.debug_tuple("Secret::Input").finish(),
Self::File(path) => f.debug_tuple("Secret::File").field(path).finish(),
Self::New => f.debug_tuple("Secret::New").finish(),
}
}
}
impl NodeKeyConfig {
/// Evaluate a `NodeKeyConfig` to obtain an identity `Keypair`:
///
/// * If the secret is configured as input, the corresponding keypair is returned.
///
/// * If the secret is configured as a file, it is read from that file, if it exists. Otherwise
/// a new secret is generated and stored. In either case, the keypair obtained from the
/// secret is returned.
///
/// * If the secret is configured to be new, it is generated and the corresponding keypair is
/// returned.
pub fn into_keypair(self) -> io::Result<Keypair> {
use NodeKeyConfig::*;
match self {
Ed25519(Secret::New) => Ok(Keypair::generate_ed25519()),
Ed25519(Secret::Input(k)) => Ok(Keypair::Ed25519(k.into())),
Ed25519(Secret::File(f)) => get_secret(
f,
|mut b| match String::from_utf8(b.to_vec()).ok().and_then(|s| {
if s.len() == 64 {
array_bytes::hex2bytes(&s).ok()
} else {
None
}
}) {
Some(s) => ed25519::SecretKey::from_bytes(s),
_ => ed25519::SecretKey::from_bytes(&mut b),
},
ed25519::SecretKey::generate,
|b| b.as_ref().to_vec(),
)
.map(ed25519::Keypair::from)
.map(Keypair::Ed25519),
}
}
}
/// Load a secret key from a file, if it exists, or generate a
/// new secret key and write it to that file. In either case,
/// the secret key is returned.
fn get_secret<P, F, G, E, W, K>(file: P, parse: F, generate: G, serialize: W) -> io::Result<K>
where
P: AsRef<Path>,
F: for<'r> FnOnce(&'r mut [u8]) -> Result<K, E>,
G: FnOnce() -> K,
E: Error + Send + Sync + 'static,
W: Fn(&K) -> Vec<u8>,
{
std::fs::read(&file)
.and_then(|mut sk_bytes| {
parse(&mut sk_bytes).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
})
.or_else(|e| {
if e.kind() == io::ErrorKind::NotFound {
file.as_ref().parent().map_or(Ok(()), fs::create_dir_all)?;
let sk = generate();
let mut sk_vec = serialize(&sk);
write_secret_file(file, &sk_vec)?;
sk_vec.zeroize();
Ok(sk)
} else {
Err(e)
}
})
}
/// Write secret bytes to a file.
fn write_secret_file<P>(path: P, sk_bytes: &[u8]) -> io::Result<()>
where
P: AsRef<Path>,
{
let mut file = open_secret_file(&path)?;
file.write_all(sk_bytes)
}
/// Opens a file containing a secret key in write mode.
#[cfg(unix)]
fn open_secret_file<P>(path: P) -> io::Result<fs::File>
where
P: AsRef<Path>,
{
use std::os::unix::fs::OpenOptionsExt;
fs::OpenOptions::new().write(true).create_new(true).mode(0o600).open(path)
}
/// Opens a file containing a secret key in write mode.
#[cfg(not(unix))]
fn open_secret_file<P>(path: P) -> Result<fs::File, io::Error>
where
P: AsRef<Path>,
{
fs::OpenOptions::new().write(true).create_new(true).open(path)
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
fn tempdir_with_prefix(prefix: &str) -> TempDir {
tempfile::Builder::new().prefix(prefix).tempdir().unwrap()
}
fn secret_bytes(kp: &Keypair) -> Vec<u8> {
let Keypair::Ed25519(p) = kp;
p.secret().as_ref().iter().cloned().collect()
}
#[test]
fn test_secret_file() {
let tmp = tempdir_with_prefix("x");
std::fs::remove_dir(tmp.path()).unwrap(); // should be recreated
let file = tmp.path().join("x").to_path_buf();
let kp1 = NodeKeyConfig::Ed25519(Secret::File(file.clone())).into_keypair().unwrap();
let kp2 = NodeKeyConfig::Ed25519(Secret::File(file.clone())).into_keypair().unwrap();
assert!(file.is_file() && secret_bytes(&kp1) == secret_bytes(&kp2))
}
#[test]
fn test_secret_input() {
let sk = ed25519::SecretKey::generate();
let kp1 = NodeKeyConfig::Ed25519(Secret::Input(sk.clone())).into_keypair().unwrap();
let kp2 = NodeKeyConfig::Ed25519(Secret::Input(sk)).into_keypair().unwrap();
assert!(secret_bytes(&kp1) == secret_bytes(&kp2));
}
#[test]
fn test_secret_new() {
let kp1 = NodeKeyConfig::Ed25519(Secret::New).into_keypair().unwrap();
let kp2 = NodeKeyConfig::Ed25519(Secret::New).into_keypair().unwrap();
assert!(secret_bytes(&kp1) != secret_bytes(&kp2));
}
}
@@ -48,18 +48,6 @@ pub enum Event {
/// Event generated by a DHT.
Dht(DhtEvent),
/// Now connected to a new peer for syncing purposes.
SyncConnected {
/// Node we are now syncing from.
remote: PeerId,
},
/// Now disconnected from a peer for syncing purposes.
SyncDisconnected {
/// Node we are no longer syncing from.
remote: PeerId,
},
/// Opened a substream with the given node with the given notifications protocol.
///
/// The protocol is always one of the notification protocols that have been registered.
@@ -79,6 +67,8 @@ pub enum Event {
negotiated_fallback: Option<ProtocolName>,
/// Role of the remote.
role: ObservedRole,
/// Received handshake.
received_handshake: Vec<u8>,
},
/// Closed a substream with the given node. Always matches a corresponding previous
+14 -21
View File
@@ -22,14 +22,12 @@ use crate::{
config::MultiaddrWithPeerId,
protocol::{event::Event, ProtocolName},
request_responses::{IfDisconnected, RequestFailure},
sync::{warp::WarpSyncProgress, StateDownloadProgress, SyncState},
};
use futures::{channel::oneshot, Stream};
pub use libp2p::{identity::error::SigningError, kad::record::Key as KademliaKey};
use libp2p::{Multiaddr, PeerId};
use sc_peerset::ReputationChange;
pub use signature::Signature;
use sp_runtime::traits::{Block as BlockT, NumberFor};
use std::{collections::HashSet, future::Future, pin::Pin, sync::Arc};
mod signature;
@@ -96,45 +94,33 @@ where
/// Overview status of the network.
#[derive(Clone)]
pub struct NetworkStatus<B: BlockT> {
/// Current global sync state.
pub sync_state: SyncState<NumberFor<B>>,
/// Target sync block number.
pub best_seen_block: Option<NumberFor<B>>,
/// Number of peers participating in syncing.
pub num_sync_peers: u32,
/// Total number of connected peers
pub struct NetworkStatus {
/// Total number of connected peers.
pub num_connected_peers: usize,
/// Total number of active peers.
pub num_active_peers: usize,
/// The total number of bytes received.
pub total_bytes_inbound: u64,
/// The total number of bytes sent.
pub total_bytes_outbound: u64,
/// State sync in progress.
pub state_sync: Option<StateDownloadProgress>,
/// Warp sync in progress.
pub warp_sync: Option<WarpSyncProgress<B>>,
}
/// Provides high-level status information about network.
#[async_trait::async_trait]
pub trait NetworkStatusProvider<Block: BlockT> {
pub trait NetworkStatusProvider {
/// High-level network status information.
///
/// Returns an error if the `NetworkWorker` is no longer running.
async fn status(&self) -> Result<NetworkStatus<Block>, ()>;
async fn status(&self) -> Result<NetworkStatus, ()>;
}
// Manual implementation to avoid extra boxing here
impl<T, Block: BlockT> NetworkStatusProvider<Block> for Arc<T>
impl<T> NetworkStatusProvider for Arc<T>
where
T: ?Sized,
T: NetworkStatusProvider<Block>,
T: NetworkStatusProvider,
{
fn status<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Result<NetworkStatus<Block>, ()>> + Send + 'async_trait>>
) -> Pin<Box<dyn Future<Output = Result<NetworkStatus, ()>> + Send + 'async_trait>>
where
'life0: 'async_trait,
Self: 'async_trait,
@@ -511,6 +497,9 @@ pub trait NetworkNotification {
target: PeerId,
protocol: ProtocolName,
) -> Result<Box<dyn NotificationSender>, NotificationSenderError>;
/// Set handshake for the notification protocol.
fn set_notification_handshake(&self, protocol: ProtocolName, handshake: Vec<u8>);
}
impl<T> NetworkNotification for Arc<T>
@@ -529,6 +518,10 @@ where
) -> Result<Box<dyn NotificationSender>, NotificationSenderError> {
T::notification_sender(self, target, protocol)
}
fn set_notification_handshake(&self, protocol: ProtocolName, handshake: Vec<u8>) {
T::set_notification_handshake(self, protocol, handshake)
}
}
/// Provides ability to send network requests.
+60 -1
View File
@@ -22,7 +22,11 @@ pub mod message;
pub mod metrics;
pub mod warp;
use crate::protocol::role::Roles;
use futures::Stream;
use libp2p::PeerId;
use message::{BlockAnnounce, BlockData, BlockRequest, BlockResponse};
use sc_consensus::{import_queue::RuntimeOrigin, IncomingBlock};
use sp_consensus::BlockOrigin;
@@ -30,9 +34,10 @@ use sp_runtime::{
traits::{Block as BlockT, NumberFor},
Justifications,
};
use std::{any::Any, fmt, fmt::Formatter, task::Poll};
use warp::WarpSyncProgress;
use std::{any::Any, fmt, fmt::Formatter, pin::Pin, sync::Arc, task::Poll};
/// The sync status of a peer we are trying to sync with
#[derive(Debug)]
pub struct PeerInfo<Block: BlockT> {
@@ -42,6 +47,17 @@ pub struct PeerInfo<Block: BlockT> {
pub best_number: NumberFor<Block>,
}
/// Info about a peer's known state (both full and light).
#[derive(Clone, Debug)]
pub struct ExtendedPeerInfo<B: BlockT> {
/// Roles
pub roles: Roles,
/// Peer best block hash
pub best_hash: B::Hash,
/// Peer best block number
pub best_number: NumberFor<B>,
}
/// Reported sync state.
#[derive(Clone, Eq, PartialEq, Debug)]
pub enum SyncState<BlockNumber> {
@@ -251,6 +267,49 @@ impl fmt::Debug for OpaqueBlockResponse {
}
}
/// Provides high-level status of syncing.
#[async_trait::async_trait]
pub trait SyncStatusProvider<Block: BlockT>: Send + Sync {
/// Get high-level view of the syncing status.
async fn status(&self) -> Result<SyncStatus<Block>, ()>;
}
#[async_trait::async_trait]
impl<T, Block> SyncStatusProvider<Block> for Arc<T>
where
T: ?Sized,
T: SyncStatusProvider<Block>,
Block: BlockT,
{
async fn status(&self) -> Result<SyncStatus<Block>, ()> {
T::status(self).await
}
}
/// Syncing-related events that other protocols can subscribe to.
pub enum SyncEvent {
/// Peer that the syncing implementation is tracking connected.
PeerConnected(PeerId),
/// Peer that the syncing implementation was tracking disconnected.
PeerDisconnected(PeerId),
}
pub trait SyncEventStream: Send + Sync {
/// Subscribe to syncing-related events.
fn event_stream(&self, name: &'static str) -> Pin<Box<dyn Stream<Item = SyncEvent> + Send>>;
}
impl<T> SyncEventStream for Arc<T>
where
T: ?Sized,
T: SyncEventStream,
{
fn event_stream(&self, name: &'static str) -> Pin<Box<dyn Stream<Item = SyncEvent> + Send>> {
T::event_stream(self, name)
}
}
/// Something that represents the syncing strategy to download past and future blocks of the chain.
pub trait ChainSync<Block: BlockT>: Send {
/// Returns the state of the sync of the given peer.