[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/network: upgrade to libp2p 0.51.0
* make discovery.rs compile
* make peer_info.rs compile
* changes to notifications and request-response proto
* make service.rs compile
* towards making request_responses.rs compile
* make request_responses.rs compile
* make request_responses.rs compile
* fix notifications/behaviour.rs tests
* fix warnings
* remove old code
* allow deprecated code (temporary)
* upgrade to libp2p 0.51.1
* add TODO for behaviour tests
* return empty vec if peer_id is absent
https://github.com/paritytech/substrate/pull/13587#discussion_r1141695167
fyi: I don't really know what the old behaviour was.
* update comment to reflect new defaults
Closes#13338
* Revert "update comment to reflect new defaults"
This reverts commit 7a981abd69308e9d522ec94905f181439a1b1dba.
* remove config.rs (from wrong merge)
* upgrade to libp2p 0.51.2
* fix formatting
* use handle_pending_outbound_connection in networt_state RPC
* update deps
* use re-exports when we use other libp2p packages
* Apply suggestions from code review
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* format code
* handle potential errors in network_state RPC
* only update libp2p crate
* update libp2p-core
* fix docs
* use libp2p-identity instead of libp2p
where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.
* Update client/network/src/discovery.rs
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
* update Cargo.lock
* add comment for per_connection_event_buffer_size
current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.
* fix link format
* update Cargo.lock
* upgrade to libp2p 0.51.3
* deprecate mplex
* Revert "deprecate mplex"
This reverts commit 9e25820e706e464a0e962a8604861fcb2a7641eb.
* Revert "upgrade to libp2p 0.51.3"
This reverts commit 6544dd4138e2f89517bd7c7281fc78a638ec7040.
* use new libp2p version in `statement` crate
* pin version temporarily
* libp2p 0.51.3
* deprecate mplex
* deprecate legacy noise handshake
* fix build error
* update libp2p-identity
* enable libp2p-identity:ed25519 feature in sc-consensus
* enable ed25519 for peerset as well
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency
* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`
* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls
* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Minimize `sc-authority-discovery` dependency on `sc-network`
* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`
* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`
* Move more methods from `NetworkService` into `NetworkPeers` trait
* Move `NetworkStateInfo` trait into `sc-network-common`
* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`
* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl
* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`
* Remove dependencies on `NetworkService` from most of the methods of `sc-service`
* Address simple review comments
* Upgrade to libp2p 0.45.1
* Limit max_negotiating_inbound_streams to 512
* Upgrade prost-build to 0.10
* Set max_negotiating_inbound_streams to 2048
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Fix authority discovery protobuf
* Fix comments in authority-discovery schema
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Add a comment about transport initialization
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Consolidating test and production code
* Signing/verifying authority discovery records with PeerId
Unsigned records cannot be rejected yet, they just produce
a warning in the log.
* Upgrading to libp2p 0.40
* libp2p::identity and sp_core::crypto Ed25519 are compatible
* Rejecting authority records unsigned by peer id can be configured
* Fixes based on review comments
* No command-line argument needed
* info was still too much spam in the logs
* Added tests for both strict and loose validation
* Fixing based on review comments
* Pierre preferred a signing method
* Ooops, I need to slow down
* Update bin/node/cli/src/service.rs
* Reexport libp2p crypto used in sc-network
* Added proto3 compatibility tests. And import noise.
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* client/authority-discovery: Rename AuthorityDiscovery to XXXWorker
* client/authority-discovery: Introduce AuthorityDiscoveryService
Add a basic `AuthorityDiscoveryService` implementation which enables
callers to get the addresses for a given `AuthorityId` from the local
cache.
* client/authority-discovery: Split into worker and service mod
Move `Service` and `Worker` to their own Rust modules resulting in the
following file structure.
├── build.rs
├── Cargo.toml
└── src
├── error.rs
├── lib.rs
├── service.rs
├── tests.rs
├── worker
│ ├── addr_cache.rs
│ ├── schema
│ │ └── dht.proto
│ └── tests.rs
└── worker.rs
* client/authority-discovery: Cache PeerId -> AuthorityId mapping
* client/authority-discovery: Update priority group on interval
Instead of updating the authority discovery peerset priority group each
time a new DHT value is found, update it regularly on an interval.
This removes the need for deterministic random selection. Instead of
trying to return a random stable set of `Multiaddr`s, the `AddrCache`
now returns a random set on each call.
* client/authority-discovery: Implement Service::get_authority_id
* client/authority-discovery: Use HashMap instead of BTreeMap
* client/authority-discovery: Rework priority group interval
* client/authority-discovery: Fix comment
* bin/node/cli: Update authority discovery constructor
* client/authority-discovery: Fuse from_service receiver
* client/authority-discovery: Remove Rng import
* client/authority-discovery: Ignore Multiaddr without PeerId
* client/authority-discovery/service: Add note on returned None
* client/authority-discovery/addr_cache: Replace double clone with deref
Aaro Altonen