* Evict inactive peers from `SyncingEngine`
If both halves of the block announce notification stream have been
inactive for 2 minutes, report the peer and disconnect it, allowing
`SyncingEngine` to free up a slot for some other peer that hopefully
is more active.
This needs to be done because the node may falsely believe it has open
connections to peers because the inbound substream can be closed without
any notification and closed outbound substream is noticed only when node
attempts to write to it which may not happen if the node has nothing to
send.
* zzz
* wip
* Evict peers only when timeout expires
* Use `debug!()`
---------
Co-authored-by: parity-processbot <>
* Keep track of the pending response for each peer individually
When peer disconnects or the syncing is restarted, remove the pending
response so syncing won't start sending duplicate requests/receive stale
responses from disconnected peers.
Before this commit pending responses where stored in `FuturesUnordered`
which made it hard to keep track of pending responses for each individual
peer.
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* ".git/.scripts/commands/fmt/fmt.sh"
* Apply suggestions from code review
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Update client/network/sync/src/lib.rs
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Poll the substream validation before polling `Notifications`
In tests, it can happen that `Notifications` doesn't produce any events
which causes `poll()` to return `Poll::Pending` and the substream
validation futures won't get polled.
Poll the futures before calling `Notifications` so results for substream
validations are received even if `Notifications` is not producing any
events.
* Remove `pending_messages`
* Remove unused import
* Attempt to relieve pressure on `mpsc_network_worker`
`SyncingEngine` interacting with `NetworkWorker` can put a lot of strain
on the channel if the number of inbound connections is high. This is
because `SyncingEngine` is notified of each inbound substream which it
then can either accept or reject and this causes a lot of message
exchange on the already busy channel.
Use a direct channel pair between `Protocol` and `SyncingEngine`
to exchange notification events. It is a temporary change to alleviate
the problems caused by syncing being an independent protocol and the
fix will be removed once `NotificationService` is implemented.
* Apply review comments
* fixes
* trigger ci
* Fix tests
Verify that both peers have a connection now that the validation goes
through `SyncingEngine`. Depending on how the tasks are scheduled,
one of them might not have the peer registered in `SyncingEngine` at which
point the test won't make any progress because block announcement received
from an unknown peer is discarded.
Move polling of `ChainSync` at the end of the function so that if a block
announcement causes a block request to be sent, that can be sent in the
same call to `SyncingEngine::poll()`.
---------
Co-authored-by: parity-processbot <>
`Protocol` is not a reliable source for the information of connected
peers because it doesn't have real-time information of the actual
connectivity state because it's not resposible for accepting/rejecting
connections and gets that information with delay from `SyncinEngine`.
* Move service tests to `client/network/tests`
These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.
* Move some configs from `sc-network-common` to `sc-network`
* Move `NetworkService` traits to `sc-network`
* Move request-responses to `sc-network`
* Remove more stuff
* Remove rest of configs from `sc-network-common` to `sc-network`
* Remove more stuff
* Fix warnings
* Update client/network/src/request_responses.rs
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Fix cargo doc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Replace `futures-channel` with `async-channel` in `out_events`
* Apply suggestions from code review
Co-authored-by: Koute <koute@users.noreply.github.com>
* Also print the backtrace of `send()` call
* Switch from `backtrace` crate to `std::backtrace`
* Remove outdated `backtrace` dependency
* Remove `backtrace` from `Cargo.lock`
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
* improve error message
* removed unused argument
* docs: disconnect_peer_inner no longer accepts `ban`
* remove redundant trace message
```
sync: Too many full nodes, rejecting 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ
sync: 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ disconnected
```
is enough to understand that we've refused to connect to the given peer
* Revert "removed unused argument"
This reverts commit c87f755b1fd03494fb446b604fe25c2418da7c87.
* ban peer for 10s after disconnect
* do not accept incoming conns if peer was banned
* Revert "do not accept incoming conns if peer was banned"
This reverts commit 7e59d05975765f2547468e9dcfd1361516c41e06.
* Revert "ban peer for 10s after disconnect"
This reverts commit 3859201ced42a5b2d18c0600e29efd20962a7289.
* Revert "Revert "removed unused argument""
This reverts commit f1dc623646dc5a69e1822c35f428e90dffe34d95.
* format code
* Revert "remove redundant trace message"
This reverts commit a87e65f08553dbe69027e9aa4f7ca4779ccaa7f2.
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* `BlockId` removal: `BlockBuilderProvider::new_block_at`
It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`
* fmt
* fix
* more fixes
* Convert `NetworkWorker::poll()` into async `next_action()`
* Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`
* Revert "Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`"
This reverts commit 4b5d851ec864f78f9d083a18a618fbe117c896d2.
* Fix `sc-network-test` to poll `NetworkWorker::next_action`
* Fix `sc_network::service` tests to poll `NetworkWorker::next_action`
* Fix docs
* kick CI
* Factor out `next_worker_message()` & `next_swarm_event()`
* Error handling: replace `futures::pending!()` with `expect()`
* Simplify stream polling in `select!`
* Replace `NetworkWorker::next_action()` with `run()`
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* minor: comment
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Print debug log when network future is shut down
* Evaluate `NetworkWorker::run()` future once before the loop
* Fix client code to match new `NetworkService` interfaces
* Make clippy happy
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Revert "Apply suggestions from code review"
This reverts commit 9fa646d0ed613e5f8623d3d37d1d59ec0a535850.
* Make `NetworkWorker::run()` consume `self`
* Terminate system RPC future if RPC rx stream has terminated.
* Rewrite with let-else
* Fix comments
* Get `best_seen_block` and call `on_block_finalized` via `ChainSync` instead of `NetworkService`
* rustfmt
* make clippy happy
* Tests: schedule wake if `next_action()` returned true
* minor: comment
* minor: fix `NetworkWorker` rustdoc
* minor: amend the rustdoc
* Fix bug that caused `on_demand_beefy_justification_sync` test to hang
* rustfmt
* Apply review suggestions
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* Rename `*-private-ipv4` to `*-private-ip` CLI args
Renames the `*-private-ipv4` to `*-private-ip` in the CLI interface. The old names are staying as
alias, thus it will not break for anyone. Besides that it also fixes the naming in the rest of the code.
* FMT
When running with `--no-private-ipv4` the node should not trying to connect to any private ip
addresses. With the switch to libp2p this behavior was broken. Part of this version upgrade was the
following pr: https://github.com/libp2p/rust-libp2p/pull/2995. This pr changed the default cache
size of `libp2p-identity` from `0` aka disabled to `100`. Together with our implementation that was
calling into `identity` to request addresses for a given peer. Before the switch to libp2p 0.50.0
this was returning zero addresses, but now with the cache enabled it started to return addresses.
This pr fixes this by only letting discovery return addresses for a peer. It also ensures that we
filter private addresses if requested. The cache is also disabled to restore the previous caching
behavior, but it will actually not be called anymore.
* join dns with another instance of WS transport
Secure Websocket transport needs unresolved addresses, so we join DNS transport with
yet another instance of Websocket transport.
Closes#12024
* WSS transport itself need to wrap DNS transport
in order to resolve addresses before passing them down to TCP transport
Refs https://github.com/libp2p/rust-libp2p/issues/3330
* reverse order
* simplify code: remove WS from WSS inner DNS transport
* remove the 2nd instance of WS transport
Before libp2p 0.50.0 we used a quorum of one to fetch records from the DHT. In the pr that upgraded
to libp2p 0.50.0 we accidentally changed this behavior. This pr brings back the old behavior of
using a qorum of one and thus, a faster discovery. After finding the first value, we directly finish
the query. There was also another behavior change in libp2p, they stopped automatic caching on
remote nodes. This pr also brings back the remote caching on nodes that are nearest to the key from
our point of view of the network.
The pr that changed the behavior in libp2p: https://github.com/libp2p/rust-libp2p/pull/2712
* upgrade libp2p to 0.50.0
* on_swarm_event and on_connection_handler_event
* replace `Swarm::new` with `Swarm::with_threadpool_executor`
* on_swarm_event and on_connection_handler_event part 2
* on_swarm_event and on_connection_handler_event part 3
* on_swarm_event and on_connection_handler_event part 4
* update libp2p
* libp2p 0.50.0
* rename OutboundQueryCompleted to OutboundQueryProgressed
refs https://github.com/libp2p/rust-libp2p/pull/2712
* remove unused var
* accumulate outbound_query_records until query is finished
* format code
* use p_handler instead of new_handler
https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610
* pass ListenFailure to kademlia
https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664
* use tokio executor in tests
https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776
* use chrono Local::now
instead of deprecated Local::today
* remove unused vars from request_responses tests
* attempt to fix pallet UI tests
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* sc-network-test::Peer: block push methods return hashes vec
This commit reworks the block generation/push methods in
sc-network-test::Peer.
Now methods are providing the vector of hashes that were built.
This allows to get rid of redundant `block_hash_from_id` call, as all
hashes are known just after being built.
Similar approach was taken in BeefyTestNet::generate_blocks_and_sync
method.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* fix
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Fix tests
* Apply review comments
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* implement crate publishing from CI
* fix indentation
* use resource_group for job exclusivity
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
* correct publish = false
* Remove YAML anchors as GitLab's `extends:` doesn't need it
* Temporarily force cache upload for the new jobs
* Revert `RUSTY_CACHIER_FORCE_UPLOAD`
* pin libp2p-tcp=0.37.0 for sc-telemetry
* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"
This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.
* always collect generated crates
* increase timeout for publish-crates-template
* Force upload the new job cache again
* Revert "Force upload the new job cache again"
This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.
* reformat
* improve timeout explanation
* s/usual/average
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
Aaro Altonen