Commit Graph

230 Commits

Author SHA1 Message Date
Aaro Altonen 80616f6d03 Integrate litep2p into Polkadot SDK (#2944)
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-08 16:44:13 +00:00
Alin Dima f2a750ee86 add fallback request for req-response protocols (#2771)
Previously, it was only possible to retry the same request on a
different protocol name that had the exact same binary payloads.

Introduce a way of trying a different request on a different protocol if
the first one fails with Unsupported protocol.

This helps with adding new req-response versions in polkadot while
preserving compatibility with unupgraded nodes.

The way req-response protocols were bumped previously was that they were
bundled with some other notifications protocol upgrade, like for async
backing (but that is more complicated, especially if the feature does
not require any changes to a notifications protocol). Will be needed for
implementing https://github.com/polkadot-fellows/RFCs/pull/47

TODO:
- [x]  add tests
- [x] add guidance docs in polkadot about req-response protocol
versioning
2024-01-10 15:19:50 +02:00
Nazar Mokrynskyi 9f3c67b4ff Support querying peer reputation (#2392)
# Description

Trivial change that resolves
https://github.com/paritytech/polkadot-sdk/issues/2185.

Since there was a mix of `who` and `peer_id` argument names nearby I
changed them all to `peer_id`.

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [x] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [x] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-12-07 13:07:12 +02:00
Aaro Altonen e71c484d5b Rework the event system of sc-network (#1370)
This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.

The `NotificationService` concept introduces two additional benefits:
  * allow protocols to start using custom handshakes
  * allow protocols to accept/reject inbound peers

Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.

With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.

This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556

---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197 but there are no
functional changes compared to that PR

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
2023-11-28 20:18:52 +02:00
Gavin Wood bc6fca3c44 Ensure cumulus/bridges is ignored by formatter and run it (#1369) 2023-09-03 15:11:26 +02:00
Aaro Altonen dd1b29c2f8 Revert "chore: update libp2p to 0.52.1 (#14429)" (#14722)
* Revert "chore: update libp2p to 0.52.1 (#14429)"

This reverts commit 59d8b86450.

* Fix dependencies

* Update dependencies

* Update Cargo.lock
2023-08-16 10:06:13 +00:00
Dmitry Markin 8dc3bd729b Get rid of Peerset compatibility layer (#14337)
* Move bootnodes from individual `SetConfig`s to `PeersetConfig`

* Move `SetId` & `SetConfig` from `peerset` to `protocol_controller`

* Remove unused `DropReason`

* Move `Message` & `IncomingIndex` from `peerset` to `protocol_controller`

* Restore running fuzz test

* Get rid of `Peerset` in `fuzz` test

* Spawn runners instead of manual polling in `fuzz` test

* Migrate `Protocol` from `Peerset` to `PeerStore` & `ProtocolController`

* Migrate `NetworkService` from `Peerset` to `PeerStore` & `ProtocolController`

* Migrate `Notifications` from `Peerset` to `ProtocolController`s

* Migrate `Notifications` tests from `Peerset` to `ProtocolController`

* Fix compilation of `NetworkService` & `Protocol`

* Fix borrowing issues in `Notifications`

* Migrate `RequestResponse`from `Peerset` to `PeerStore`

* rustfmt

* Migrate request-response tests from `Peerset` to `PeerStore`

* Migrate `reconnect_after_disconnect` test to `PeerStore` & `ProtocolController`

* Fix `Notifications` tests

* Remove `Peerset` completely

* Fix bug with counting sync peers in `Protocol`

* Eliminate indirect calls to `PeerStore` via `Protocol`

* Eliminate indirect calls to `ProtocolController` via `Protocol`

* Handle `Err` outcome from `remove_peers_from_reserved_set`

* Add note about disconnecting sync peers in `Protocol`

* minor: remove unneeded `clone()`

* minor: extra comma removed

* minor: use `Stream` API of `from_protocol_controllers` channel

* minor: remove TODO

* minor: replace `.map().flatten()` with `.flat_map()`

* minor: update `ProtocolController` docs

* rustfmt

* Apply suggestions from code review

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* Extract `MockPeerStore` to `mock.rs`

* Move `PeerStore` initialization to `build_network`

* minor: remove unused import

* minor: clarify error message

* Convert `syncs_header_only_forks` test into single-threaded

---------

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
2023-08-02 13:01:35 +00:00
Alin Dima addd8628ad network: optimize update procedure for listen_addrs and external_addrs (#14689)
* network: optimize listen_address update procedure

* network: optimize external_addr update procedure

* replace on_swarm_event with add/remove
2023-08-01 13:39:15 +03:00
Anton 59d8b86450 chore: update libp2p to 0.52.1 (#14429)
* update libp2p to 0.52.0

* proto name now must implement `AsRef<str>`

* update libp2p version everywhere

* ToSwarm, FromBehaviour, ToBehaviour

also LocalProtocolsChange and RemoteProtocolsChange

* new NetworkBehaviour invariants

* replace `Vec<u8>` with `StreamProtocol`

* rename ConnectionHandlerEvent::Custom to NotifyBehaviour

* remove DialError & ListenError invariants

also fix pending_events

* use connection_limits::Behaviour

See https://github.com/libp2p/rust-libp2p/pull/3885

* impl `void::Void` for `BehaviourOut`

also use `Behaviour::with_codec`

* KademliaHandler no longer public

* fix StreamProtocol construction

* update libp2p-identify to 0.2.0

* remove non-existing methods from PollParameters

rename ConnectionHandlerUpgrErr to StreamUpgradeError

* `P2p` now contains `PeerId`, not `Multihash`

* use multihash-codetable crate

* update Cargo.lock

* reformat text

* comment out tests for now

* remove `.into()` from P2p

* confirm observed addr manually

See https://github.com/libp2p/rust-libp2p/blob/master/protocols/identify/CHANGELOG.md#0430

* remove SwarmEvent::Banned

since we're not using `ban_peer_id`, this can be safely removed.
we may want to introduce `libp2p::allow_block_list` module in the future.

* fix imports

* replace `libp2p` with smaller deps in network-gossip

* bring back tests

* finish rewriting tests

* uncomment handler tests

* Revert "uncomment handler tests"

This reverts commit 720a06815887f4e10767c62b58864a7ec3a48e50.

* add a fixme

* update Cargo.lock

* remove extra From

* make void uninhabited

* fix discovery test

* use autonat protocols

confirming external addresses manually is unsafe in open networks

* fix SyncNotificationsClogged invariant

* only set server mode manually in tests

doubt that we need to set it on node since we're adding public addresses

* address @dmitry-markin comments

* remove autonat

* removed unused var

* fix EOL

* update smallvec and sha2

in attempt to compile polkadot

* bump k256

in attempt to build cumulus

---------

Co-authored-by: parity-processbot <>
2023-07-25 11:12:24 +00:00
Bastian Köcher b4e863c472 sc-network: Improve invalid boot node reporting (#14455)
This improves the reporting of invalid boot nodes. First, it will only report each boot node once
as invalid and not every time we try to connect to the node. Second, the node will only report for
addresses that we added as startup and not for addresses of the boot node that the node learned from
other nodes.

Closes: https://github.com/paritytech/substrate/issues/13584
Closes: https://github.com/paritytech/polkadot/issues/7385
2023-06-28 13:21:05 +02:00
klbrvik 1d2f16ab1f expose setting kademlia replication factor through node CLI (#14391)
* expose kademlia replication factor through node CLI

* set default CLI flag value for kademlia_replication_factor

Co-authored-by: Bastian Köcher <git@kchr.de>

* wrap CLI value as Option

* make kademlia replication factor non-optional

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-06-22 00:21:54 +02:00
Dmitry Markin d4b2bf7394 Incorporate sc-peerset into sc-network (#14236) 2023-05-29 20:56:57 +03:00
Dmitry Markin 01107e9ca5 Split Peerset into PeerStore & ProtocolControllers (#13611) 2023-05-23 14:49:02 +03:00
Anton e4b1aa1811 Upgrade to libp2p 0.51.3 (#13587)
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

https://github.com/paritytech/substrate/pull/13587#discussion_r1141695167

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes #13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981abd69308e9d522ec94905f181439a1b1dba.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820e706e464a0e962a8604861fcb2a7641eb.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4138e2f89517bd7c7281fc78a638ec7040.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
2023-05-12 07:12:51 +00:00
Aaro Altonen f36749b99e Prepare sc-network for ProtocolController/NotificationService (#14080)
* Prepare `sc-network` for `ProtocolController`/`NotificationService`

The upcoming notification protocol refactoring requires that protocols
are able to communicate with `sc-network` over unique and direct links.
This means that `sc-network` side of the link has to be created before
`sc-network` is initialized and that it is allowed to consume the object
as the receiver half of the link may not implement `Clone`.

Remove request-response and notification protocols from `NetworkConfiguration`
and create a new object that contains the configurations of these protocols
and which is consumable by `sc-network`. This is needed needed because, e.g.,
the receiver half of `NotificationService` is not clonable so `sc-network`
must consume it when it's initializing the protocols in `Notifications`.

Similar principe applies to `PeerStore`/`ProtocolController`: as per current
design, protocols are created before the network so `Protocol` cannot be
the one creating the `PeerStore` object. `FullNetworkConfiguration` will be
used to store the objects that `sc-network` will use to communicate with
protocols and it will also allow protocols to allocate handles so they
can directly communicate with `sc-network`.

* Fixes

* Update client/service/src/builder.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Updates

* Doc updates + cargo-fmt

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-05-11 10:27:21 +00:00
Aaro Altonen 4240490d1d Attempt to relieve pressure on mpsc_network_worker (#13725)
* Attempt to relieve pressure on `mpsc_network_worker`

`SyncingEngine` interacting with `NetworkWorker` can put a lot of strain
on the channel if the number of inbound connections is high. This is
because `SyncingEngine` is notified of each inbound substream which it
then can either accept or reject and this causes a lot of message
exchange on the already busy channel.

Use a direct channel pair between `Protocol` and `SyncingEngine`
to exchange notification events. It is a temporary change to alleviate
the problems caused by syncing being an independent protocol and the
fix will be removed once `NotificationService` is implemented.

* Apply review comments

* fixes

* trigger ci

* Fix tests

Verify that both peers have a connection now that the validation goes
through `SyncingEngine`. Depending on how the tasks are scheduled,
one of them might not have the peer registered in `SyncingEngine` at which
point the test won't make any progress because block announcement received
from an unknown peer is discarded.

Move polling of `ChainSync` at the end of the function so that if a block
announcement causes a block request to be sent, that can be sent in the
same call to `SyncingEngine::poll()`.

---------

Co-authored-by: parity-processbot <>
2023-03-30 11:59:58 +00:00
Sebastian Kunert c41eadc090 Remove HeaderBackend requirement from AuthorityDiscovery and NetworkWorker (#13730)
* Remove `HeaderBackend` requirement from `NetworkWorker`

* Remove HeaderBackend from authority-discovery
2023-03-28 07:10:50 +00:00
Aaro Altonen 9ced14e2de Move code from sc-network-common back to sc-network (#13592)
* Move service tests to `client/network/tests`

These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.

* Move some configs from `sc-network-common` to `sc-network`

* Move `NetworkService` traits to `sc-network`

* Move request-responses to `sc-network`

* Remove more stuff

* Remove rest of configs from `sc-network-common` to `sc-network`

* Remove more stuff

* Fix warnings

* Update client/network/src/request_responses.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Fix cargo doc

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-03-14 12:06:40 +00:00
Aaro Altonen 1a7f5be07f Extract syncing protocol from sc-network (#12828)
* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Add missing docs

* Refactor code

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-03-06 16:33:38 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Dmitry Markin 8d033b6dfb Use async/await instead of manual polling of NetworkWorker (#13219)
* Convert `NetworkWorker::poll()` into async `next_action()`

* Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`

* Revert "Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`"

This reverts commit 4b5d851ec864f78f9d083a18a618fbe117c896d2.

* Fix `sc-network-test` to poll `NetworkWorker::next_action`

* Fix `sc_network::service` tests to poll `NetworkWorker::next_action`

* Fix docs

* kick CI

* Factor out `next_worker_message()` & `next_swarm_event()`

* Error handling: replace `futures::pending!()` with `expect()`

* Simplify stream polling in `select!`

* Replace `NetworkWorker::next_action()` with `run()`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* minor: comment

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Print debug log when network future is shut down

* Evaluate `NetworkWorker::run()` future once before the loop

* Fix client code to match new `NetworkService` interfaces

* Make clippy happy

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Revert "Apply suggestions from code review"

This reverts commit 9fa646d0ed613e5f8623d3d37d1d59ec0a535850.

* Make `NetworkWorker::run()` consume `self`

* Terminate system RPC future if RPC rx stream has terminated.

* Rewrite with let-else

* Fix comments

* Get `best_seen_block` and call `on_block_finalized` via `ChainSync` instead of `NetworkService`

* rustfmt

* make clippy happy

* Tests: schedule wake if `next_action()` returned true

* minor: comment

* minor: fix `NetworkWorker` rustdoc

* minor: amend the rustdoc

* Fix bug that caused `on_demand_beefy_justification_sync` test to hang

* rustfmt

* Apply review suggestions

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-20 12:08:02 +00:00
Anton 7539d4d8d6 [client/network] remove peer entry from ephemeral_addresses (#13084)
* [client/network] remove peer entry from `ephemeral_addresses`

if there are no addresses associated with that peer

* refactor as per @bkchr suggestion

https://github.com/paritytech/substrate/pull/13084#discussion_r1068028534

* add missing import

* fix error
2023-02-06 06:21:51 +00:00
Bastian Köcher d20a9bdc21 Rename *-private-ipv4 to *-private-ip CLI args (#13208)
* Rename `*-private-ipv4` to `*-private-ip` CLI args

Renames the `*-private-ipv4` to `*-private-ip` in the CLI interface. The old names are staying as
alias, thus it will not break for anyone. Besides that it also fixes the naming in the rest of the code.

* FMT
2023-01-23 11:35:04 +00:00
Anton f2dcd9520c upgrade libp2p to 0.50.0 (#12734)
* upgrade libp2p to 0.50.0

* on_swarm_event and on_connection_handler_event

* replace `Swarm::new` with `Swarm::with_threadpool_executor`

* on_swarm_event and on_connection_handler_event part 2

* on_swarm_event and on_connection_handler_event part 3

* on_swarm_event and on_connection_handler_event part 4

* update libp2p

* libp2p 0.50.0

* rename OutboundQueryCompleted to OutboundQueryProgressed

refs https://github.com/libp2p/rust-libp2p/pull/2712

* remove unused var

* accumulate outbound_query_records until query is finished

* format code

* use p_handler instead of new_handler

https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610

* pass ListenFailure to kademlia

https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664

* use tokio executor in tests

https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776

* use chrono Local::now

instead of deprecated Local::today

* remove unused vars from request_responses tests

* attempt to fix pallet UI tests

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI
2023-01-05 12:03:41 +00:00
Dmitry Markin 34eb463d99 Runtime diagnostics for leaked messages in unbounded channels (#12971) 2022-12-23 16:03:08 +03:00
Aaro Altonen d6827185c3 Move import queue out of sc-network (#12764)
* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Fix tests

* Apply review comments

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-12-09 19:50:57 +00:00
Dmitry Markin 5eb84f9cc6 Upgrade tokio to 1.22.0 and replace async-std with tokio (#12646)
* Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests
2022-12-05 08:18:46 +00:00
Aaro Altonen 1b5d52deb2 Move block/state/warpc sync requests/responses to ChainSync (#12739)
* Move block/state/warpc sync requests/responses to `ChainSync`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply review suggestions

* cargo-fmt + doc fix

* Fix tests

Co-authored-by: Bastian Köcher <git@kchr.de>
2022-11-22 08:19:17 +00:00
Dmitrii Markin d0dcf008ec Base Kademlia protocol name on genesis hash and fork ID (#12545) 2022-10-24 14:47:58 +03:00
Nazar Mokrynskyi bf57a2e92d Actually fix major sync detection (#12114)
* Actually fix major sync detection

* Introduce `SyncState::Importing` state

* Add target to SyncState enum variants and add `is_major_syncing` method on it

* Remove unnecessary duplicated `best_seen_block` from `SyncState` struct

* Revert "Remove unnecessary duplicated `best_seen_block` from `SyncState` struct"

This reverts commit bb8abd458c939881c049f69d59f3acba47c97c5c.

* Add missing `websocket` feature to `libp2p`

Co-authored-by: parity-processbot <>
2022-10-21 13:38:53 +00:00
Aaro Altonen 7f8aab84b1 Refactor service tests in sc-network (#12517)
* Refactor service tests in `sc-network`

Create a separate directory for the tests and move common network
instantion related code to `mod.rs` from where it can be used by both
service and chainsync tests.

Use the builder pattern when creating the `TestNetwork` object to reduce
code duplication between the test files.

* Update client/network/src/service/tests/mod.rs

Co-authored-by: Dmitrii Markin <dmitry@markin.tech>

Co-authored-by: Dmitrii Markin <dmitry@markin.tech>
Co-authored-by: parity-processbot <>
2022-10-21 06:39:23 +00:00
Dmitrii Markin 3a10019a10 Remove multiple DHTs support from Discovery (#12524) 2022-10-20 19:30:01 +03:00
Dmitrii Markin bb175f0373 Upgrade libp2p to 0.49.0 (#12256)
* cargo upgrade libp2p

* Get rid of `NetworkBehaviourEventProcess` in handling of `CustomMessageOutcome`

* Get rid of `NetworkBehaviourEventProcess` in handling of `request_responses::Event`

* Get rid of `NetworkBehaviourEventProcess` in handling of `peer_info::PeerInfoEvent`

* Get rid of `NetworkBehaviourEventProcess` in handling of `DiscoveryOut`

* Get rid of `poll()` method in `Bahaviour`

* minor: comments

* Upgrade libp2p to 0.49.0 (unreleased)

* Support multiple Kad protocol names

* Make borrow checker happy

* minor: wording

* Make substrate build with libp2p-0.49.0

* rustfmt

* Get rid of MdnsWrapper

* Resolve deprecation warnings

* Fix documentation

* Apply suggestions from code review: fix typos

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* Apply suggestion: simplify kad protocol name matching

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
2022-10-17 11:02:37 +00:00
Aaro Altonen ce4cad8b8b Introduce ChainSyncInterface (#12489)
* Introduce `ChainSyncInterface`

`ChainSyncInterface` provides an asynchronous interface for other
subsystems to submit calls to `ChainSync`. This allows `NetworkService`
to delegate calls to `ChainSync` while still providing the same API
for other subsystems (for now). This makes it possible to move the
syncing code in piecemeal fashion out of `protocol.rs` as the calls
are just forwarded to `ChainSync`.

* Apply review comments

* Fix tests
2022-10-17 07:25:25 +00:00
Aaro Altonen 09164dbced Introduce mockable ChainSync object for testing (#12480)
* Introduce mockable `ChainSync` object for testing

`mockall` allows to mock `ChainSync` and to verify that the calls made
to `ChaiSync` are firstly executed at all, that they're executed in
correct order and with correct parameters.

This allows to verify, e.g., that delegating calls directly to
`ChainSync` from `NetworkService` still calls the correct functions with
correct arguments even if `Protocol` middleman is removed.

* Add Cargo.lock

* Fix tests

* Update client/network/Cargo.toml

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update Cargo.lock

* Fix clippy and documentation

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
2022-10-13 09:37:09 +00:00
Aaro Altonen ce9ce49bc6 Move block announcement protocol config out of Protocol (#12441)
* Move Role(s) to `sc-network-common`

* Introduce `NotificationHandshake` type

* Move block announce protocol config creation to `ChainSync`

* Include block announcement into `notification_protocols`

* Apply review comments

* Remove unneeded include

* Add missing include

* Apply review comments
2022-10-10 07:10:53 +00:00
Aaro Altonen 4c19c13d05 Move transactions protocol to its own crate (#12264)
* Move transaction protocol to its own crate

* Update Cargo.lock

* Fix binaries

* Update client/network/transactions/src/lib.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <info@kchr.de>

* Apply review comments

* Revert one change and apply cargo-fmt

* Remove Transaction from Message

* Add array-bytes

* trigger CI

* Add comment about codec index

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Bastian Köcher <info@kchr.de>
2022-09-26 12:10:09 +00:00
Aaro Altonen 103ac1bc84 Include Bitswap into request response protocols (#12262) 2022-09-15 07:27:56 +00:00
Aaro Altonen b494167d16 Move bitswap to its own crate (#12242)
* Introduce `sc-network-bitswap`

Move the bitswap protocol out of `sc-network` to its own crate.

* Improve test coverage

* Remove mention of bitswap from `sc-network`

* Fix documentation

* Fix clippy and remove the void dependency

* Remove unneeded trait bound and bump prost to 0.11
2022-09-13 08:02:31 +00:00
Dmitry Markin 24d09fe8c7 Use custom type for ProtocolName (#12172)
* Add ProtocolName custom type

* Use new ProtocolName in sc_network_common

* Use new ProtocolName in sc_network

* Use new ProtocolName for BEEFY and GRANDPA

* Use new ProtocolName for notifications

* Use new ProtocolName in sc_network (part 2)

* Use new ProtocolName in sc_network_gossip

* Use new ProtocolName in sc_offchain

* Remove unused imports

* Some more fixes

* Add tests

* Fix minor import issues

* Re-export ProtocolName in sc_network

* Revert "Re-export ProtocolName in sc_network"

This reverts commit 8d8ff71927e7750757f29c9bbd88dc0ba181d214.

* Re-export ProtocolName in sc_network

* Remove dependency on sc-network-common from beefy-gadget
2022-09-03 20:34:47 +00:00
Sebastian Kunert 800bc5cd8c Simplify trait bounds in network to prepare for collator-rpc (#12082)
* Hack towards PoC

* Abstract away runtime requirement

* blockchainevents

* Remove bitswap

* Remove unused sync more

* Remove unused features in network

* Re-enable bitswap change

* Remove `Chain` trait bound

* Reimplement blockchain-rpc-events

* Move network to cumulus

* Make AuthorityDiscovery async

* Remove `ProofProvider` requirement from network behaviour

* Extract bitswap

* Adjustments after merge

* Remove HeaderMetadata trait from network

* Introduce NetworkHeaderBackend

* Add comments

* Improve comments

* Move NetworkHeaderBackend to new module

* Improve naming, remove redundand send + sync

* Clean up generics

* Fix CI

* Improve comment and readability

* Remove NetworkHeaderBackend

* Fix Cargo.lock

Co-authored-by: Sebastian Kunert <skunert@Sebastians-MacBook-Pro.fritz.box>
2022-08-31 14:55:46 +02:00
Dmitry Markin 224562729d Make NetworkService::add_reserved_peer() accept MultiaddrWithPeerId (#12102)
* Make `add_reserved_peer()` accept `MultiaddrWithPeerId`

* minor: cargo fmt

* minor: error to string conversion

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-08-29 11:54:57 +00:00
Bastian Köcher e62e670631 Print bootnode mismatch message as warning (#12075)
* Print bootnode mismatch message as `warning`

* Fix test :thinking_face:

* Remove wrong warning
2022-08-23 22:22:58 +02:00
Nazar Mokrynskyi b20f83c18f Network sync refactoring (part 7) (#12006)
* Move `MultiaddrWithPeerId` and related parsing functions into `sc-network-common`, remove dependency on `sc-network` from `sc-chain-spec`

* Remove dependency on `sc-network` from `sc-offchain`

* Remove dependency on `sc-network` from `sc-network-gossip`
2022-08-22 15:53:14 +00:00
Nazar Mokrynskyi a685582bfd Network sync refactoring (part 6) (#11940)
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency

* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`

* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls

* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Minimize `sc-authority-discovery` dependency on `sc-network`

* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`

* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`

* Move more methods from `NetworkService` into `NetworkPeers` trait

* Move `NetworkStateInfo` trait into `sc-network-common`

* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`

* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl

* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`

* Remove dependencies on `NetworkService` from most of the methods of `sc-service`

* Address simple review comments
2022-08-09 18:28:32 +00:00
Dmitry Markin 6eda842cf0 Change on-the-wire protocol names to include genesis hash & fork id (#11938)
* Rename transactions protocol to include genesis hash

* Add protocol name generation to sc_network::utils

* Use utils functions for transactions protocol name generation

* Extract protocol name generation into public module

* Use sc_network::protocol_name::standard_protocol_name() for BEEFY and GRANDPA

* minor: add missing newline at EOF

* Change block-announces protocol name to include genesis_hash & fork_id

* Change protocol names to include genesis hash and fork id

Protocols changed:
    - sync
    - state
    - light
    - sync/warp

* Revert "Use sc_network::protocol_name::standard_protocol_name() for BEEFY and GRANDPA"

This reverts commit cd60a95a3face397e1b67f4bc95dd0f2b581bfae.

* Get rid of `protocol_name` module
2022-08-05 06:50:57 +00:00
Nazar Mokrynskyi 56f16f5500 Network sync refactoring (part 5) (#11825)
* Make `chain_sync` an explicit networking parameter instead of offering factory method

* Derive `Copy` on `SyncMode` and remove cloning
2022-07-29 20:17:49 +00:00
Dmitry Markin afc723cc30 Clarify wrong bootnode peer ID error message (#11872) 2022-07-22 15:08:52 +03:00
Dmitry Markin b7ecd1af85 Cleanup light client leftovers (#11865)
* Remove --light cli option

* Cleanup light client leftovers

* Remove commented-out code and clean-up more light client leftovers

* Fix formatting with `cargo +nightly fmt`

* Remove FIXME regarding db directory structure

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-07-21 08:36:00 +00:00
Nazar Mokrynskyi f514bc3892 Ignore boot and reserved nodes with peer ID the same as in local node itself (#11851) 2022-07-16 23:24:46 +02:00