This PR brings the fix
https://github.com/paritytech/substrate/pull/13396 to polkadot-sdk.
In the past, due to insufficient inbound slot count on polkadot &
kusama, this fix led to low peer count. The situation has improved since
then after changing the default ratio between `--in-peers` &
`--out-peers`.
Nevertheless, it's expected that the reported total peer count with this
fix is going to be lower than without it. This should be seen as the
correct number of working connections reported, as opposed to also
reporting already closed connections, and not as lower count of working
connections with peers.
This PR also removes the peer eviction mechanism, as closed substream
detection is a more granular way of detecting peers that stopped syncing
with us.
The burn-in has been already performed as part of testing these changes
in https://github.com/paritytech/polkadot-sdk/pull/3426.
---------
Co-authored-by: Aaro Altonen <a.altonen@hotmail.com>
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
Currently, all protocols use the same metric name for
`mpsc-notification-to-protocol` this is bad because we can't actually
tell which protocol might cause problems.
This patch proposes we derive the name of the metric from the protocol
name, so that we have separate metrics for each protocol and properly
detect which one is having problem processing its messages.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.
The `NotificationService` concept introduces two additional benefits:
* allow protocols to start using custom handshakes
* allow protocols to accept/reject inbound peers
Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.
With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.
This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.
Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556
---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197 but there are no
functional changes compared to that PR
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
* update libp2p to 0.52.0
* proto name now must implement `AsRef<str>`
* update libp2p version everywhere
* ToSwarm, FromBehaviour, ToBehaviour
also LocalProtocolsChange and RemoteProtocolsChange
* new NetworkBehaviour invariants
* replace `Vec<u8>` with `StreamProtocol`
* rename ConnectionHandlerEvent::Custom to NotifyBehaviour
* remove DialError & ListenError invariants
also fix pending_events
* use connection_limits::Behaviour
See https://github.com/libp2p/rust-libp2p/pull/3885
* impl `void::Void` for `BehaviourOut`
also use `Behaviour::with_codec`
* KademliaHandler no longer public
* fix StreamProtocol construction
* update libp2p-identify to 0.2.0
* remove non-existing methods from PollParameters
rename ConnectionHandlerUpgrErr to StreamUpgradeError
* `P2p` now contains `PeerId`, not `Multihash`
* use multihash-codetable crate
* update Cargo.lock
* reformat text
* comment out tests for now
* remove `.into()` from P2p
* confirm observed addr manually
See https://github.com/libp2p/rust-libp2p/blob/master/protocols/identify/CHANGELOG.md#0430
* remove SwarmEvent::Banned
since we're not using `ban_peer_id`, this can be safely removed.
we may want to introduce `libp2p::allow_block_list` module in the future.
* fix imports
* replace `libp2p` with smaller deps in network-gossip
* bring back tests
* finish rewriting tests
* uncomment handler tests
* Revert "uncomment handler tests"
This reverts commit 720a06815887f4e10767c62b58864a7ec3a48e50.
* add a fixme
* update Cargo.lock
* remove extra From
* make void uninhabited
* fix discovery test
* use autonat protocols
confirming external addresses manually is unsafe in open networks
* fix SyncNotificationsClogged invariant
* only set server mode manually in tests
doubt that we need to set it on node since we're adding public addresses
* address @dmitry-markin comments
* remove autonat
* removed unused var
* fix EOL
* update smallvec and sha2
in attempt to compile polkadot
* bump k256
in attempt to build cumulus
---------
Co-authored-by: parity-processbot <>
* Accept only `--in-peers` many inbound full nodes in `SyncingEngine`
Due to full and light nodes being stored in the same set, it's possible
that `SyncingEngine` accepts more than `--in-peers` many inbound full
nodes which leaves some of its outbound slots unoccupied.
`ProtocolController` still tries to occupy these slots by opening
outbound substreams. As these substreams are accepted by the remote peer,
the connection is relayed to `SyncingEngine` which rejects the node
because it's already full. This in turn results in the substream being
inactive and the peer getting evicted.
Fixing this properly would require relocating the light peer slot
allocation away from `ProtocolController` or alternatively moving entire
the substream validation there, both of which are epic refactorings and
not necessarily in line with other goals. As a temporary measure, verify
in `SyncingEngine` that it doesn't accept more than the specified amount
of inbound full peers.
* Fix tests
* Apply review comments
* frame-benchmarking-cli: Remove native dispatch requirement
No need for this, we can just use the `WasmExecutor` directly.
* Fixes
* Pass benchmarking host functions
* Ensure we can pass custom host functions
* client/network: upgrade to libp2p 0.51.0
* make discovery.rs compile
* make peer_info.rs compile
* changes to notifications and request-response proto
* make service.rs compile
* towards making request_responses.rs compile
* make request_responses.rs compile
* make request_responses.rs compile
* fix notifications/behaviour.rs tests
* fix warnings
* remove old code
* allow deprecated code (temporary)
* upgrade to libp2p 0.51.1
* add TODO for behaviour tests
* return empty vec if peer_id is absent
https://github.com/paritytech/substrate/pull/13587#discussion_r1141695167
fyi: I don't really know what the old behaviour was.
* update comment to reflect new defaults
Closes#13338
* Revert "update comment to reflect new defaults"
This reverts commit 7a981abd69308e9d522ec94905f181439a1b1dba.
* remove config.rs (from wrong merge)
* upgrade to libp2p 0.51.2
* fix formatting
* use handle_pending_outbound_connection in networt_state RPC
* update deps
* use re-exports when we use other libp2p packages
* Apply suggestions from code review
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* format code
* handle potential errors in network_state RPC
* only update libp2p crate
* update libp2p-core
* fix docs
* use libp2p-identity instead of libp2p
where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.
* Update client/network/src/discovery.rs
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
* update Cargo.lock
* add comment for per_connection_event_buffer_size
current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.
* fix link format
* update Cargo.lock
* upgrade to libp2p 0.51.3
* deprecate mplex
* Revert "deprecate mplex"
This reverts commit 9e25820e706e464a0e962a8604861fcb2a7641eb.
* Revert "upgrade to libp2p 0.51.3"
This reverts commit 6544dd4138e2f89517bd7c7281fc78a638ec7040.
* use new libp2p version in `statement` crate
* pin version temporarily
* libp2p 0.51.3
* deprecate mplex
* deprecate legacy noise handshake
* fix build error
* update libp2p-identity
* enable libp2p-identity:ed25519 feature in sc-consensus
* enable ed25519 for peerset as well
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
* Move service tests to `client/network/tests`
These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.
* Move some configs from `sc-network-common` to `sc-network`
* Move `NetworkService` traits to `sc-network`
* Move request-responses to `sc-network`
* Remove more stuff
* Remove rest of configs from `sc-network-common` to `sc-network`
* Remove more stuff
* Fix warnings
* Update client/network/src/request_responses.rs
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Fix cargo doc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* improve error message
* removed unused argument
* docs: disconnect_peer_inner no longer accepts `ban`
* remove redundant trace message
```
sync: Too many full nodes, rejecting 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ
sync: 12D3KooWSQAP2fh4qBkLXBW4mvCtbAiK8sqMnExWHHTZtVAxZ8bQ disconnected
```
is enough to understand that we've refused to connect to the given peer
* Revert "removed unused argument"
This reverts commit c87f755b1fd03494fb446b604fe25c2418da7c87.
* ban peer for 10s after disconnect
* do not accept incoming conns if peer was banned
* Revert "do not accept incoming conns if peer was banned"
This reverts commit 7e59d05975765f2547468e9dcfd1361516c41e06.
* Revert "ban peer for 10s after disconnect"
This reverts commit 3859201ced42a5b2d18c0600e29efd20962a7289.
* Revert "Revert "removed unused argument""
This reverts commit f1dc623646dc5a69e1822c35f428e90dffe34d95.
* format code
* Revert "remove redundant trace message"
This reverts commit a87e65f08553dbe69027e9aa4f7ca4779ccaa7f2.
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* upgrade libp2p to 0.50.0
* on_swarm_event and on_connection_handler_event
* replace `Swarm::new` with `Swarm::with_threadpool_executor`
* on_swarm_event and on_connection_handler_event part 2
* on_swarm_event and on_connection_handler_event part 3
* on_swarm_event and on_connection_handler_event part 4
* update libp2p
* libp2p 0.50.0
* rename OutboundQueryCompleted to OutboundQueryProgressed
refs https://github.com/libp2p/rust-libp2p/pull/2712
* remove unused var
* accumulate outbound_query_records until query is finished
* format code
* use p_handler instead of new_handler
https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610
* pass ListenFailure to kademlia
https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664
* use tokio executor in tests
https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776
* use chrono Local::now
instead of deprecated Local::today
* remove unused vars from request_responses tests
* attempt to fix pallet UI tests
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* Add ProtocolName custom type
* Use new ProtocolName in sc_network_common
* Use new ProtocolName in sc_network
* Use new ProtocolName for BEEFY and GRANDPA
* Use new ProtocolName for notifications
* Use new ProtocolName in sc_network (part 2)
* Use new ProtocolName in sc_network_gossip
* Use new ProtocolName in sc_offchain
* Remove unused imports
* Some more fixes
* Add tests
* Fix minor import issues
* Re-export ProtocolName in sc_network
* Revert "Re-export ProtocolName in sc_network"
This reverts commit 8d8ff71927e7750757f29c9bbd88dc0ba181d214.
* Re-export ProtocolName in sc_network
* Remove dependency on sc-network-common from beefy-gadget
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency
* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`
* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls
* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Minimize `sc-authority-discovery` dependency on `sc-network`
* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`
* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`
* Move more methods from `NetworkService` into `NetworkPeers` trait
* Move `NetworkStateInfo` trait into `sc-network-common`
* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`
* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl
* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`
* Remove dependencies on `NetworkService` from most of the methods of `sc-service`
* Address simple review comments
* Remove direct dependency of `sc-network` on `sc-network-light`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Create `sync` module in `sc-network-common`, create `ChainSync` trait there (not used yet), move a bunch of associated data structures from `sc-network-sync`
* Switch from concrete implementation to `ChainSync` trait from `sc-network-common`
* Introduce `OpaqueStateRequest`/`OpaqueStateResponse` to remove generics from `StateSync` trait
* Introduce `OpaqueBlockRequest`/`OpaqueBlockResponse`, make `scheme` module of `sc-network-sync` private
* Surface `sc-network-sync` into `sc-service` and make `sc-network` not depend on it anymore
* Remove now unnecessary dependency from `sc-network`
* Replace crate links with just text since dependencies are gone now
* Remove `warp_sync` re-export from `sc-network-common`
* Update copyright in network-related files
* Address review comments about documentation
* Apply review suggestion
* Rename `extra_requests` module to `metrics`
Co-authored-by: Bastian Köcher <info@kchr.de>
* Move `api.v1.proto` schema into new crate `sc-network-sync`
* Move `sc_network::protocol::sync::state` module into `sc_network_sync::state`
* Move `sc_network::protocol::sync::blocks` module into `sc_network_sync::blocks` and some data structures from `sc_network::protocol::message` module into `sc_network_sync::message`
* Move some data structures from `sc_network::config` and `sc_network::request_responses` into new `sc-network-common` crate
* Move `sc_network::protocol::sync::warm` and `sc_network::warp_request_handler` modules into `sc_network_sync`
* Move `client/network/sync/src/lib.rs` to `client/network/sync/src/lib_old.rs` to preserve history of changes of the file in the next commit
* Move `client/network/src/protocol/sync.rs` on top of `client/network/sync/src/lib.rs` to preserve history of changes
* Move `sc_network::protocol::sync` to `sc_network_sync` with submodules, move message data structures around accordingly
* Move `sc_network::block_request_handler` to `sc_network_sync::block_request_handler`
* Move `sc_network::state_request_handler` to `sc_network_sync::state_request_handler`
* Add re-exports for compatibility reasons
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove unnecessary imports, move one internal re-export into where it is actually used, make one import explicit
* Move a few data structures down into modules
* Use generic parameters in `sc-network` instead of `chain::Client` trait
* Remove unnecessary bound
* Simplify `num_connected_peers`
* Track requested peer counts
* Revert "Track requested peer counts"
This reverts commit 9f1c8704353df6afc17ed7e9f4ab8d8e29466ae4.
* Remove `substrate_sub_libp2p_peerset_num_requested` metric
* Remove two unused functions that I forgot to get rid of in previous commit
* Add a new host function for reporting fatal errors
* Fix one of the wasmtime executor tests
* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only
* Print out errors through `Display` instead of `Debug`
* Switch one more trait to require `Error` for its error instead of only `Debug`
* Align to review comments
* Add stale branches heads to finality notifications
Warning. Previous implementation was sending a notification for
each block between the previous (explicitly) finalized block and
the new finalized one (with an hardcoded limit of 256).
Now finality notification is sent only for the new finalized head and it
contains the hash of the new finalized head, new finalized head header,
a list of all the implicitly finalized blocks and a list of stale
branches heads (i.e. the branches heads that are not part of the
canonical chain anymore).
* Add implicitly finalized blocks list to `ChainEvent::Finalized` message
The list contains all the blocks between the previously finalized block
up to the parent of the currently finalized one, sorted by block number.
`Finalized` messages handler, part of the `MaintainedTransactionPool`
implementation for `BasicPool`, still propagate full set of finalized
blocks to the txpool by iterating over implicitly finalized blocks list.
* Rust fmt
* Greedy evaluation of `stale_heads` during finalization
* Fix outdated assumption in a comment
* Removed a test optimization that is no more relevant
The loop was there to prevent sending to
`peer.network.on_block_finalized` the full list of finalized blocks.
Now only the finalized heads are received.
* Last finalized block lookup not required anymore
* Tests for block finality notifications payloads
* Document a bit tricky condition to avoid duplicate finalization notifications
* More idiomatic way to skip an iterator entry
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Cargo fmt iteration
* Typo fix
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix potential failure when a finalized orphan block is imported
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Dmitry Markin