* Move service tests to `client/network/tests`
These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.
* Move some configs from `sc-network-common` to `sc-network`
* Move `NetworkService` traits to `sc-network`
* Move request-responses to `sc-network`
* Remove more stuff
* Remove rest of configs from `sc-network-common` to `sc-network`
* Remove more stuff
* Fix warnings
* Update client/network/src/request_responses.rs
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Fix cargo doc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* improve error message
* removed unused argument
* docs: disconnect_peer_inner no longer accepts `ban`
* remove redundant trace message
```
sync: Too many full nodes, rejecting 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ
sync: 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ disconnected
```
is enough to understand that we've refused to connect to the given peer
* Revert "removed unused argument"
This reverts commit c87f755b1fd03494fb446b604fe25c2418da7c87.
* ban peer for 10s after disconnect
* do not accept incoming conns if peer was banned
* Revert "do not accept incoming conns if peer was banned"
This reverts commit 7e59d05975765f2547468e9dcfd1361516c41e06.
* Revert "ban peer for 10s after disconnect"
This reverts commit 3859201ced42a5b2d18c0600e29efd20962a7289.
* Revert "Revert "removed unused argument""
This reverts commit f1dc623646dc5a69e1822c35f428e90dffe34d95.
* format code
* Revert "remove redundant trace message"
This reverts commit a87e65f08553dbe69027e9aa4f7ca4779ccaa7f2.
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* `BlockId` removal: `BlockBuilderProvider::new_block_at`
It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`
* fmt
* fix
* more fixes
* Convert `NetworkWorker::poll()` into async `next_action()`
* Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`
* Revert "Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`"
This reverts commit 4b5d851ec864f78f9d083a18a618fbe117c896d2.
* Fix `sc-network-test` to poll `NetworkWorker::next_action`
* Fix `sc_network::service` tests to poll `NetworkWorker::next_action`
* Fix docs
* kick CI
* Factor out `next_worker_message()` & `next_swarm_event()`
* Error handling: replace `futures::pending!()` with `expect()`
* Simplify stream polling in `select!`
* Replace `NetworkWorker::next_action()` with `run()`
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* minor: comment
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Print debug log when network future is shut down
* Evaluate `NetworkWorker::run()` future once before the loop
* Fix client code to match new `NetworkService` interfaces
* Make clippy happy
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Revert "Apply suggestions from code review"
This reverts commit 9fa646d0ed613e5f8623d3d37d1d59ec0a535850.
* Make `NetworkWorker::run()` consume `self`
* Terminate system RPC future if RPC rx stream has terminated.
* Rewrite with let-else
* Fix comments
* Get `best_seen_block` and call `on_block_finalized` via `ChainSync` instead of `NetworkService`
* rustfmt
* make clippy happy
* Tests: schedule wake if `next_action()` returned true
* minor: comment
* minor: fix `NetworkWorker` rustdoc
* minor: amend the rustdoc
* Fix bug that caused `on_demand_beefy_justification_sync` test to hang
* rustfmt
* Apply review suggestions
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* upgrade libp2p to 0.50.0
* on_swarm_event and on_connection_handler_event
* replace `Swarm::new` with `Swarm::with_threadpool_executor`
* on_swarm_event and on_connection_handler_event part 2
* on_swarm_event and on_connection_handler_event part 3
* on_swarm_event and on_connection_handler_event part 4
* update libp2p
* libp2p 0.50.0
* rename OutboundQueryCompleted to OutboundQueryProgressed
refs https://github.com/libp2p/rust-libp2p/pull/2712
* remove unused var
* accumulate outbound_query_records until query is finished
* format code
* use p_handler instead of new_handler
https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610
* pass ListenFailure to kademlia
https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664
* use tokio executor in tests
https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776
* use chrono Local::now
instead of deprecated Local::today
* remove unused vars from request_responses tests
* attempt to fix pallet UI tests
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Fix tests
* Apply review comments
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* implement crate publishing from CI
* fix indentation
* use resource_group for job exclusivity
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
* correct publish = false
* Remove YAML anchors as GitLab's `extends:` doesn't need it
* Temporarily force cache upload for the new jobs
* Revert `RUSTY_CACHIER_FORCE_UPLOAD`
* pin libp2p-tcp=0.37.0 for sc-telemetry
* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"
This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.
* always collect generated crates
* increase timeout for publish-crates-template
* Force upload the new job cache again
* Revert "Force upload the new job cache again"
This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.
* reformat
* improve timeout explanation
* s/usual/average
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* Do not update peer information if ancestor search is in progress
If block announcement is received from a peer while ancestor search
for that same peer is still in progress, do not update the peer's best
hash and best number as that causes the ancestor search to yield
different information from what was expected and can cause, for example,
a fork of lower height not be be downloaded.
* Block until peers are in sync
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
Introduce a middleware called `NetworkServiceProvider` which the
`ChainSync` can use to communicate with `NetworkService`. `ChainSync` is
given a `NetworkServiceHandle` which it uses to call `NetworkServiceProvider`
which then dispatches the calls to `NetworkService` on behalf of `ChainSync`.
This change will allow `ChainSync` to disconnect and report peers and
in the future it'll be possible to send requests and notifications
through the `NetworkServiceProvider`.
`NetworkServiceProvider` is needed only until the `ChainSync` object
has been removed from `Protocol`. After that, a normal `NetworkService`
handle can be passed onto `ChainSync` and these changes can be
deprecated.
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Finalizer
It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* minor corrections
* failing test corrected
* minor rework
* BlockId removal: refactor: ProofProvider
It changes the arguments of methods of `ProofProvider` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* LightClientRequestHandler: excessive BlockIdTo bound removed
* imports cleanup
* formatting
* args tyeps cleanup
* cargo upgrade libp2p
* Get rid of `NetworkBehaviourEventProcess` in handling of `CustomMessageOutcome`
* Get rid of `NetworkBehaviourEventProcess` in handling of `request_responses::Event`
* Get rid of `NetworkBehaviourEventProcess` in handling of `peer_info::PeerInfoEvent`
* Get rid of `NetworkBehaviourEventProcess` in handling of `DiscoveryOut`
* Get rid of `poll()` method in `Bahaviour`
* minor: comments
* Upgrade libp2p to 0.49.0 (unreleased)
* Support multiple Kad protocol names
* Make borrow checker happy
* minor: wording
* Make substrate build with libp2p-0.49.0
* rustfmt
* Get rid of MdnsWrapper
* Resolve deprecation warnings
* Fix documentation
* Apply suggestions from code review: fix typos
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
* Apply suggestion: simplify kad protocol name matching
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
* Introduce `ChainSyncInterface`
`ChainSyncInterface` provides an asynchronous interface for other
subsystems to submit calls to `ChainSync`. This allows `NetworkService`
to delegate calls to `ChainSync` while still providing the same API
for other subsystems (for now). This makes it possible to move the
syncing code in piecemeal fashion out of `protocol.rs` as the calls
are just forwarded to `ChainSync`.
* Apply review comments
* Fix tests
* Introduce mockable `ChainSync` object for testing
`mockall` allows to mock `ChainSync` and to verify that the calls made
to `ChaiSync` are firstly executed at all, that they're executed in
correct order and with correct parameters.
This allows to verify, e.g., that delegating calls directly to
`ChainSync` from `NetworkService` still calls the correct functions with
correct arguments even if `Protocol` middleman is removed.
* Add Cargo.lock
* Fix tests
* Update client/network/Cargo.toml
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update Cargo.lock
* Fix clippy and documentation
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
* Use `array-bytes` for All Array/Bytes/Hex Operations
Signed-off-by: Xavier Lau <xavier@inv.cafe>
* Reorder
* Self Review
* Format
* Fix Tests
* Bump `array-bytes`
* Optimize large test res
Signed-off-by: Xavier Lau <xavier@inv.cafe>
Co-authored-by: parity-processbot <>
* Receive and import target block body
* Request target block
* minor: wording
* Check for block body in the test
* Import target block justifications
* Fix: do not fail block validation if no justifications received
* Fix: import target blocks without justifications
Co-authored-by: arkpar <arkady.paronyan@gmail.com>
* Rename transactions protocol to include genesis hash
* Add protocol name generation to sc_network::utils
* Use utils functions for transactions protocol name generation
* Extract protocol name generation into public module
* Use sc_network::protocol_name::standard_protocol_name() for BEEFY and GRANDPA
* minor: add missing newline at EOF
* Change block-announces protocol name to include genesis_hash & fork_id
* Change protocol names to include genesis hash and fork id
Protocols changed:
- sync
- state
- light
- sync/warp
* Revert "Use sc_network::protocol_name::standard_protocol_name() for BEEFY and GRANDPA"
This reverts commit cd60a95a3face397e1b67f4bc95dd0f2b581bfae.
* Get rid of `protocol_name` module
* Remove direct dependency of `sc-network` on `sc-network-light`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Create `sync` module in `sc-network-common`, create `ChainSync` trait there (not used yet), move a bunch of associated data structures from `sc-network-sync`
* Switch from concrete implementation to `ChainSync` trait from `sc-network-common`
* Introduce `OpaqueStateRequest`/`OpaqueStateResponse` to remove generics from `StateSync` trait
* Introduce `OpaqueBlockRequest`/`OpaqueBlockResponse`, make `scheme` module of `sc-network-sync` private
* Surface `sc-network-sync` into `sc-service` and make `sc-network` not depend on it anymore
* Remove now unnecessary dependency from `sc-network`
* Replace crate links with just text since dependencies are gone now
* Remove `warp_sync` re-export from `sc-network-common`
* Update copyright in network-related files
* Address review comments about documentation
* Apply review suggestion
* Rename `extra_requests` module to `metrics`
Co-authored-by: Bastian Köcher <info@kchr.de>
Sebastian Kunert