[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* upgrade libp2p to 0.50.0
* on_swarm_event and on_connection_handler_event
* replace `Swarm::new` with `Swarm::with_threadpool_executor`
* on_swarm_event and on_connection_handler_event part 2
* on_swarm_event and on_connection_handler_event part 3
* on_swarm_event and on_connection_handler_event part 4
* update libp2p
* libp2p 0.50.0
* rename OutboundQueryCompleted to OutboundQueryProgressed
refs https://github.com/libp2p/rust-libp2p/pull/2712
* remove unused var
* accumulate outbound_query_records until query is finished
* format code
* use p_handler instead of new_handler
https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610
* pass ListenFailure to kademlia
https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664
* use tokio executor in tests
https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776
* use chrono Local::now
instead of deprecated Local::today
* remove unused vars from request_responses tests
* attempt to fix pallet UI tests
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* Upgrade to libp2p 0.45.1
* Limit max_negotiating_inbound_streams to 512
* Upgrade prost-build to 0.10
* Set max_negotiating_inbound_streams to 2048
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Fix authority discovery protobuf
* Fix comments in authority-discovery schema
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Add a comment about transport initialization
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Add new hardware and software metrics
* Move sysinfo tests into `mod tests`
* Correct a typo in a comment
* Remove unnecessary `nix` dependency
* Fix the version tests
* Add a `--disable-hardware-benchmarks` CLI argument
* Disable hardware benchmarks in the integration tests
* Remove unused import
* Fix benchmarks compilation
* Move code to a new `sc-sysinfo` crate
* Correct `impl_version` comment
* Move `--disable-hardware-benchmarks` to the chain-specific bin crate
* Move printing out of hardware bench results to `sc-sysinfo`
* Move hardware benchmarks to a separate messages; trigger them manually
* Rename some of the fields in the `HwBench` struct
* Revert changes to the telemetry crate; manually send hwbench messages
* Move sysinfo logs into the sysinfo crate
* Move the `TARGET_OS_*` constants into the sysinfo crate
* Minor cleanups
* Move the `HwBench` struct to the sysinfo crate
* Derive `Clone` for `HwBench`
* Fix broken telemetry connection notification stream
* Prevent the telemetry connection notifiers from leaking if they're disconnected
* Turn the telemetry notification failure log into a debug log
* Rename `--disable-hardware-benchmarks` to `--no-hardware-benchmarks`
* Comment out browser stuff
* Remove browser stuff
* Remove more wasm transport code
* Remove ExtTransport and rework how telemetry initialises.
* Change (most) wasm-timer using code to use std::time
* Rename CI-job
* Aura does not compile for wasm
* Remove testing in the browser on CI
* Update README
* Leave `StreamSink` be
* fmt
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Simplify some Option / Result / ? operator patterns
When those match a combinator exactly.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
* adjust after review
* adjust post-review
* Update to libp2p-0.29.
* Update dependencies.
* Update Cargo.toml
Co-authored-by: Max Inden <mail@max-inden.de>
* Fix tests.
* Fix tests.
* Fix more tests.
* Update to 0.29.1
* Update ed25519-dalek dependency of sp-core.
* Update Cargo.lock.
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Check for url validity when creating TelemetryEndpoints
* Update code that used TelemetryEndpoints::new()
* Update commennts that referred to TelemetryEndpoints::new()
* Add tests for telemetry
* Fix typo and fix code in docs
* Return error on failing to override telemetry
* Use expect instead of suppressing errors on must-be-valid telemetry endpoints
* Update telemetry unit tests to use expect instead of unwrap
* Implement custom deserializer for TelemetryEndpoints
* Fix typo
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Use noise and timeouts on wasm
* Don't use wasm-opt when compiling to wasm
* Forgot lockfile
* Add node about disabling wasm-opt
* Enable timeouts in telemetry on wasm
* Make WASM browser thing compile
* Fix
* updated exit-future (github repo)
* Switch to broadcast crate
* Migrate client/cli
* Switch exit-future to modernize branch
* Small changes
* Switch to cargo version and fix fg tests
* fix basic-authorship
* Fix crash on grafana macro
* Fix grafana macro
* Switch node python version
* Disable record_metrics_slice in grafana macro on wasm
* Update client/grafana-data-source/src/lib.rs
* Revert "Update client/grafana-data-source/src/lib.rs"
This reverts commit 888009a8e0b7051bd4bfbbfdb0448bcf2e2aae93.
* Add wasm support for state machine
* Switch to my own libp2p version
* Revert "Switch to my own libp2p version"
This reverts commit ce613871b59264b3165b45c37943e6560240daa7.
* Revert "Add wasm support for state machine"
This reverts commit de7eaa0694d9534fc3b164621737968e9a6a7c5f.
* Add sc-browser
* Squash
* remove sc-browser
* Fix keystore on wasm
* stubs for removed functions to make env compatible with old runtimes
* Add test (that doesn't work)
* Fix build scripts
* Revert basic-authorship due to no panics
* Revert cli/informant
* Revert consensus
* revert offchain
* Update utils/browser/Cargo.toml
Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>
* export console functions
* Add new chainspec
* Fix ws in chain spec
* revert chainspec
* Fix chainspec
* Use an Option<PathBuf> in keystore instead of cfg flags
* Remove crud
* Only use wasm-timer for instant and systemtime
* Remove telemetry changes
* Assuming this is ok
* Add a KeystoreConfig
* Add stubs back in
* Update libp2p
* Revert "Add stubs back in"
This reverts commit 4690cf1882aa0f99f7f00a58c4080c8aa9b77c36.
* Remove commented js again
* Bump kvdb-web version
* Fix cli
* Switch branch on futures-timer
* Fix tests
* Remove sc-client test build in check-web-wasm because there isn't a good way to build futures-timer with wasm-bindgen support in the build
* Remove more things ^^
* Switch branch on futures-timer back
* Put DB io stats behind a cfg flag
* Fix things
* Don't timeout transports on wasm
* Update branch of futures-timer and fix bad merge
* Spawn informant
* Fix network test
* Fix delay resets
* Changes
* Fix tests
* use wasm_timer for transaction pool
* Fixes
* Switch futures-timer to crates
* Only diagnose futures on native
* Fix sc-network-test tests
* Select log level in js
* Fix syncing ;^)
* Allow disabling colours in the informant
* Use OutputFormat enum for informant
* MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master
* Update futures-diagnose
* Revert "MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master"
This reverts commit baa4ffc94fd968b6660a2c17ba8113e06af15548.
* Pass whole chain spec in start_client
* Get Instant::now to work in transaction pool again
* Informant dep reordering
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
* Entirely update substrate-telemetry to futures 0.3
* Add a Closed error
* Update to libp2p 0.14
* More work
* More work
* More work
* More work
* Fix warnings
* Remove unwrap()
* Work on tests fixing
* Fix network tests
* Fix external network tests
* Update libp2p and restore Yamux in discovery test
* Ignore DNS if initializatio nfails
* Restore variables ordering
* Forgot browser-utils
* Fix downfall after merge
* Fix tests
* Adding script for rename, could be applicable for nodes on top of it, too
* add stderr and gitlab ci features
* apply script
* fix now minor details in expected stderr
* Update the Cargo.lock
* fix name: sc-transaction -> sc-tracing
* fix rename in script, too
* Adding first rough ouline of the repository structure
* Remove old CI stuff
* add title
* formatting fixes
* move node-exits job's script to scripts dir
* Move docs into subdir
* move to bin
* move maintainence scripts, configs and helpers into its own dir
* add .local to ignore
* move core->client
* start up 'test' area
* move test client
* move test runtime
* make test move compile
* Add dependencies rule enforcement.
* Fix indexing.
* Update docs to reflect latest changes
* Moving /srml->/paint
* update docs
* move client/sr-* -> primitives/
* clean old readme
* remove old broken code in rhd
* update lock
* Step 1.
* starting to untangle client
* Fix after merge.
* start splitting out client interfaces
* move children and blockchain interfaces
* Move trie and state-machine to primitives.
* Fix WASM builds.
* fixing broken imports
* more interface moves
* move backend and light to interfaces
* move CallExecutor
* move cli off client
* moving around more interfaces
* re-add consensus crates into the mix
* fix subkey path
* relieve client from executor
* starting to pull out client from grandpa
* move is_decendent_of out of client
* grandpa still depends on client directly
* lemme tests pass
* rename srml->paint
* Make it compile.
* rename interfaces->client-api
* Move keyring to primitives.
* fixup libp2p dep
* fix broken use
* allow dependency enforcement to fail
* move fork-tree
* Moving wasm-builder
* make env
* move build-script-utils
* fixup broken crate depdencies and names
* fix imports for authority discovery
* fix typo
* update cargo.lock
* fixing imports
* Fix paths and add missing crates
* re-add missing crates
Aaro Altonen