Commit Graph

45 Commits

Author SHA1 Message Date
Aaro Altonen 80616f6d03 Integrate litep2p into Polkadot SDK (#2944)
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-08 16:44:13 +00:00
Dmitry Markin 0d93248473 Fix addresses_to_publish_respects_existing_p2p_protocol test in sc-authority-discovery (#3895)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3887.
2024-03-29 13:13:21 +00:00
Aaro Altonen dd1b29c2f8 Revert "chore: update libp2p to 0.52.1 (#14429)" (#14722)
* Revert "chore: update libp2p to 0.52.1 (#14429)"

This reverts commit 59d8b86450.

* Fix dependencies

* Update dependencies

* Update Cargo.lock
2023-08-16 10:06:13 +00:00
Anton 59d8b86450 chore: update libp2p to 0.52.1 (#14429)
* update libp2p to 0.52.0

* proto name now must implement `AsRef<str>`

* update libp2p version everywhere

* ToSwarm, FromBehaviour, ToBehaviour

also LocalProtocolsChange and RemoteProtocolsChange

* new NetworkBehaviour invariants

* replace `Vec<u8>` with `StreamProtocol`

* rename ConnectionHandlerEvent::Custom to NotifyBehaviour

* remove DialError & ListenError invariants

also fix pending_events

* use connection_limits::Behaviour

See https://github.com/libp2p/rust-libp2p/pull/3885

* impl `void::Void` for `BehaviourOut`

also use `Behaviour::with_codec`

* KademliaHandler no longer public

* fix StreamProtocol construction

* update libp2p-identify to 0.2.0

* remove non-existing methods from PollParameters

rename ConnectionHandlerUpgrErr to StreamUpgradeError

* `P2p` now contains `PeerId`, not `Multihash`

* use multihash-codetable crate

* update Cargo.lock

* reformat text

* comment out tests for now

* remove `.into()` from P2p

* confirm observed addr manually

See https://github.com/libp2p/rust-libp2p/blob/master/protocols/identify/CHANGELOG.md#0430

* remove SwarmEvent::Banned

since we're not using `ban_peer_id`, this can be safely removed.
we may want to introduce `libp2p::allow_block_list` module in the future.

* fix imports

* replace `libp2p` with smaller deps in network-gossip

* bring back tests

* finish rewriting tests

* uncomment handler tests

* Revert "uncomment handler tests"

This reverts commit 720a06815887f4e10767c62b58864a7ec3a48e50.

* add a fixme

* update Cargo.lock

* remove extra From

* make void uninhabited

* fix discovery test

* use autonat protocols

confirming external addresses manually is unsafe in open networks

* fix SyncNotificationsClogged invariant

* only set server mode manually in tests

doubt that we need to set it on node since we're adding public addresses

* address @dmitry-markin comments

* remove autonat

* removed unused var

* fix EOL

* update smallvec and sha2

in attempt to compile polkadot

* bump k256

in attempt to build cumulus

---------

Co-authored-by: parity-processbot <>
2023-07-25 11:12:24 +00:00
Anton e4b1aa1811 Upgrade to libp2p 0.51.3 (#13587)
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

https://github.com/paritytech/substrate/pull/13587#discussion_r1141695167

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes #13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981abd69308e9d522ec94905f181439a1b1dba.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820e706e464a0e962a8604861fcb2a7641eb.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4138e2f89517bd7c7281fc78a638ec7040.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
2023-05-12 07:12:51 +00:00
Davide Galassi 480396fe06 Keystore overhaul (iter 2) (#13634)
* Remove bloat about remote keystore

* Update docs and remove unused 'KeystoreRef' trait

* Use wherever possible, MemoryKeystore for testing

* Remove unrequired fully qualified method syntax for Keystore
2023-03-20 18:21:26 +00:00
Davide Galassi f110941b7f Keystore overhaul (#13615)
* Remove 'supported_keys' 'sign_with_any' and 'sign_with_all' from keystore trait

* Remove the aync keystore

* Renaming:
- SyncCryptoStore -> Keystore
- SyncCryptoStorePtr -> KeystorePtr
- KeyStore -> MemoryKeystore

* Fix authority discovery worker and tests

* Rename 'insert_unknown' to 'insert'

* Remove leftover
2023-03-17 11:24:14 +00:00
Aaro Altonen 9ced14e2de Move code from sc-network-common back to sc-network (#13592)
* Move service tests to `client/network/tests`

These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.

* Move some configs from `sc-network-common` to `sc-network`

* Move `NetworkService` traits to `sc-network`

* Move request-responses to `sc-network`

* Remove more stuff

* Remove rest of configs from `sc-network-common` to `sc-network`

* Remove more stuff

* Fix warnings

* Update client/network/src/request_responses.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Fix cargo doc

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-03-14 12:06:40 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Dmitry Markin 8d033b6dfb Use async/await instead of manual polling of NetworkWorker (#13219)
* Convert `NetworkWorker::poll()` into async `next_action()`

* Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`

* Revert "Use `NetworkWorker::next_action` instead of `poll` in `sc-network-test`"

This reverts commit 4b5d851ec864f78f9d083a18a618fbe117c896d2.

* Fix `sc-network-test` to poll `NetworkWorker::next_action`

* Fix `sc_network::service` tests to poll `NetworkWorker::next_action`

* Fix docs

* kick CI

* Factor out `next_worker_message()` & `next_swarm_event()`

* Error handling: replace `futures::pending!()` with `expect()`

* Simplify stream polling in `select!`

* Replace `NetworkWorker::next_action()` with `run()`

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* minor: comment

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Print debug log when network future is shut down

* Evaluate `NetworkWorker::run()` future once before the loop

* Fix client code to match new `NetworkService` interfaces

* Make clippy happy

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Revert "Apply suggestions from code review"

This reverts commit 9fa646d0ed613e5f8623d3d37d1d59ec0a535850.

* Make `NetworkWorker::run()` consume `self`

* Terminate system RPC future if RPC rx stream has terminated.

* Rewrite with let-else

* Fix comments

* Get `best_seen_block` and call `on_block_finalized` via `ChainSync` instead of `NetworkService`

* rustfmt

* make clippy happy

* Tests: schedule wake if `next_action()` returned true

* minor: comment

* minor: fix `NetworkWorker` rustdoc

* minor: amend the rustdoc

* Fix bug that caused `on_demand_beefy_justification_sync` test to hang

* rustfmt

* Apply review suggestions

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-20 12:08:02 +00:00
Michal Kucharczyk c237b82690 BlockId removal: refactor: HeaderBackend::status (#12981)
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

Co-authored-by: parity-processbot <>
2022-12-21 10:25:16 +00:00
Michal Kucharczyk 548955a73f BlockId removal: refactor: HeaderBackend::header (#12874)
* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
2022-12-20 09:43:31 +00:00
Bastian Köcher e876ed310e Authority discovery: Log when we found addresses for an authority (#12960) 2022-12-17 09:25:08 +02:00
Sebastian Kunert 800bc5cd8c Simplify trait bounds in network to prepare for collator-rpc (#12082)
* Hack towards PoC

* Abstract away runtime requirement

* blockchainevents

* Remove bitswap

* Remove unused sync more

* Remove unused features in network

* Re-enable bitswap change

* Remove `Chain` trait bound

* Reimplement blockchain-rpc-events

* Move network to cumulus

* Make AuthorityDiscovery async

* Remove `ProofProvider` requirement from network behaviour

* Extract bitswap

* Adjustments after merge

* Remove HeaderMetadata trait from network

* Introduce NetworkHeaderBackend

* Add comments

* Improve comments

* Move NetworkHeaderBackend to new module

* Improve naming, remove redundand send + sync

* Clean up generics

* Fix CI

* Improve comment and readability

* Remove NetworkHeaderBackend

* Fix Cargo.lock

Co-authored-by: Sebastian Kunert <skunert@Sebastians-MacBook-Pro.fritz.box>
2022-08-31 14:55:46 +02:00
Squirrel a68a80fbae declone and close the door (#12035)
* declone and close the door

* cargo fmt

* remove brackets
2022-08-15 19:38:36 +00:00
Nazar Mokrynskyi a685582bfd Network sync refactoring (part 6) (#11940)
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency

* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`

* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls

* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Minimize `sc-authority-discovery` dependency on `sc-network`

* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`

* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`

* Move more methods from `NetworkService` into `NetworkPeers` trait

* Move `NetworkStateInfo` trait into `sc-network-common`

* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`

* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl

* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`

* Remove dependencies on `NetworkService` from most of the methods of `sc-service`

* Address simple review comments
2022-08-09 18:28:32 +00:00
Roman 1988c4ac1d Upgrade to libp2p 0.45.1 (#11682)
* Upgrade to libp2p 0.45.1

* Limit max_negotiating_inbound_streams to 512

* Upgrade prost-build to 0.10

* Set max_negotiating_inbound_streams to 2048

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Fix authority discovery protobuf

* Fix comments in authority-discovery schema

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Add a comment about transport initialization

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2022-06-17 13:47:14 +00:00
Falco Hirschenberger b581604aa7 Apply some clippy lints (#11154)
* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2022-04-30 21:28:27 +00:00
Bastian Köcher 298fdf3012 authority-discovery: Fix flaky test (#11219)
It can happen that `+ 1` overflows `p.signature[1]` ;D (I have seen this in the CI).
2022-04-13 17:01:36 +02:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
wigy 5fd7fdcfcd Authentication of PeerIds in authority discovery records (#10317)
* Consolidating test and production code

* Signing/verifying authority discovery records with PeerId

Unsigned records cannot be rejected yet, they just produce
a warning in the log.

* Upgrading to libp2p 0.40

* libp2p::identity and sp_core::crypto Ed25519 are compatible

* Rejecting authority records unsigned by peer id can be configured

* Fixes based on review comments

* No command-line argument needed

* info was still too much spam in the logs

* Added tests for both strict and loose validation

* Fixing based on review comments

* Pierre preferred a signing method

* Ooops, I need to slow down

* Update bin/node/cli/src/service.rs

* Reexport libp2p crypto used in sc-network

* Added proto3 compatibility tests. And import noise.

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-12-05 20:17:14 +01:00
Bastian Köcher f037651788 authority-discovery: Support multiple authority ids per peer id (#10259)
* authority-discovery: Support multiple authority ids per peer id

An peer id can be mapped to multiple authority ids, because an authority id is a session key that
could be changed every session. Before this pr the internal authority discovery cache assumed that
each authority id can only be mapped to one peer id. However, this isn't true since we changed the
default implementation of the authority discovery to combine the current and next session authorities.

* Review feedback

* Update client/authority-discovery/src/worker/addr_cache.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Early return on no peer ids

* Update client/authority-discovery/src/worker/addr_cache.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Update types in comment

* FMT

* Add warning

* Update client/authority-discovery/src/worker/addr_cache.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Feedback

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2021-11-17 10:53:31 +01:00
Arkadiy Paronyan e6ff531d0b Warp sync part II (#9284)
* Gap sync

* Gap epoch test

* Simplified network requests

* Update client/db/src/utils.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fixed v1 migration and added some comments

* Next epoch is always regular

* Removed fork tree change

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Added a comment and converted assert to error

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-10-07 09:31:39 +00:00
Liu-Cheng Xu 2562f8c65e Clean up sc-network (#9761)
* Clean up sc-network

- Avoid using clone() for the Copy type `PeerId`.
- Use `find_map` for `filter_map` and `next`.
- Use `Self`.

* More on Copy types

* Cargo +nightly fmt --all

* More ..

* fmt

* Revert vec![default_notif_handshake_message]
2021-09-13 20:11:29 +02:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Arkadiy Paronyan 77a4b980ae Fast sync (#8884)
* State sync

* Importing state fixes

* Bugfixes

* Sync with proof

* Status reporting

* Unsafe sync mode

* Sync test

* Cleanup

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* set_genesis_storage

* Extract keys from range proof

* Detect iter completion

* Download and import bodies with fast sync

* Replaced meta updates tuple with a struct

* Fixed reverting finalized state

* Reverted timeout

* Typo

* Doc

* Doc

* Fixed light client test

* Fixed error handling

* Tweaks

* More UpdateMeta changes

* Rename convert_transaction

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Code review suggestions

* Fixed count handling

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-06-22 09:32:43 +00:00
Pierre Krieger efe2267ec7 Two bugfixes to authority-discovery (#8768) 2021-05-09 21:37:41 +02:00
Pierre Krieger 2fa6f2fbd5 Fix debug_assertion failing in authority discovery (#8599)
* Fix debug_assertion failing in authority discovery

* Improve test

* Change the map_or for invalid addresses

* Remove debug_assertion
2021-04-14 13:19:33 +00:00
Pierre Krieger 1b939bcb53 Check every minute whether authority-discovery key has changed (#8575)
* Check every minute whether authority-discovery key has changed

* Fix test

* Fix comment

* Use HashSet for latest_published_keys

* More fixing

* God I'm tired, sorry
2021-04-08 16:41:23 +00:00
Bastian Köcher 4f4a0c5b38 Make keystore return None when a key doesn't exist (#8163)
* Make keystore return `None` when a key doesn't exist

* Fixes

* More fixes

* Update comment

* Update primitives/keystore/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Address comments

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-02-22 14:24:09 +00:00
Bastian Köcher 33f9becf41 Simplify runtime api error handling (#8114)
* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...
2021-02-15 11:55:40 +00:00
Benjamin Kampmann 8e36d87ca8 Update dependencies ahead of next release (#8015)
Updates dependencies:
parity-db 0.2.2
paste
prometheus 0.11
cfg-if 1.0
strum 0.20
env_logger 0.8
pin-project
prost
nix
platforms
quickcheck 1.0
2021-02-04 19:17:42 +01:00
Pierre Krieger 779c4f8616 Rework priority groups, take 2 (#7700)
* Rework priority groups

* Broken tests fix

* Fix warning causing CI to fail

* [Hack] Try restore backwards-compatibility

* Fix peerset bug

* Doc fixes and clean up

* Error on state mismatch

* Try debug CI

* CI debugging

* [CI debug] Can I please see this line

* Revert "[CI debug] Can I please see this line"

This reverts commit 4b7cf7c1511f579cd818b21d46bd11642dfac5cb.

* Revert "CI debugging"

This reverts commit 9011f1f564b860386dc7dd6ffa9fc34ea7107623.

* Fix error! which isn't actually an error

* Fix Ok() returned when actually Err()

* Tweaks and fixes

* Fix build

* Peerset bugfix

* [Debug] Try outbound GrandPa slots

* Another bugfix

* Revert "[Debug] Try outbound GrandPa slots"

This reverts commit d175b9208c088faad77d9f0ce36ff6f48bd92dd3.

* [Debug] Try outbound GrandPa slots

* Apply suggestions from code review

Co-authored-by: Max Inden <mail@max-inden.de>

* Use consts for hardcoded peersets

* Revert "Try debug CI"

This reverts commit 62c4ad5e79c03d561c714a008022ecac463a597e.

* Renames

* Line widths

* Add doc

Co-authored-by: Max Inden <mail@max-inden.de>
2021-01-07 13:52:39 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Roman Borschel b4ee48ee18 Upgrade to libp2p-0.31. (#7606)
* Upgrade to libp2p-0.31.

* Address line width.

* Add generous incoming connection limit.

* Remove old noise configuration.
2020-11-27 14:29:18 +00:00
Max Inden d692d173f2 client/authority-discovery: Publish and query on exponential interval (#7545)
* client/authority-discovery: Publish and query on exponential interval

When a node starts up publishing and querying might fail due to various
reasons, for example due to being not yet fully bootstrapped on the DHT.
Thus one should retry rather sooner than later. On the other hand, a
long running node is likely well connected and thus timely retries are
not needed. For this reasoning use an exponentially increasing interval
for `publish_interval`, `query_interval` and
`priority_group_set_interval` instead of a constant interval.

* client/authority-discovery/src/interval.rs: Add license header

* .maintain/gitlab: Ensure adder collator tests are run on CI
2020-11-23 16:34:37 +00:00
Bastian Köcher 86d5d9609d Make authority discovery configurable (#7465)
* Make authority discovery configurable

This pr makes the authority discovery configurable. So, instead of
having default values for the query interval, publish interval etc this
pr adds a configuration to make these values changeable. This will be
useful for tests where authority discovery is required.

* Update client/authority-discovery/src/worker.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Update client/authority-discovery/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Update client/authority-discovery/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Update client/authority-discovery/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix compilation

* line width

Co-authored-by: Max Inden <mail@max-inden.de>
2020-10-30 14:59:45 +01:00
Max Inden 653868c01e client/authority-discovery: Remove sentry node logic (#7368)
* client/authority-discovery: Remove sentry node logic

The notion of sentry nodes has been deprecated (see [1] for details).
This commit removes support for sentry nodes in the
`client/authority-discovery` module.

While removing `Role::Sentry` this commit also introduces
`Role::Discover`, allowing a node to discover addresses of authorities
without publishing ones own addresses. This will be needed in Polkadot
for collator nodes.

[1] https://github.com/paritytech/substrate/issues/6845

* client/authority-discovery/service: Improve PeerId comment
2020-10-26 10:06:56 +00:00
Max Inden 0fcf9d6c60 client/network: Make NetworkService::set_priority_group async (#7352)
As done with `NetworkService::{add_to,remove_from}_priority_group`, make
`NetworkService::set_priority_group` async as well. This future-proofs
the API should we ever decide to use a bounded channel between
`NetworkService` and `NetworkWorker`.
2020-10-22 09:15:39 +00:00
Rakan Alhneiti 3aa4bfacfc Async keystore + Authority-Discovery async/await (#7000)
* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-10-08 20:56:35 +00:00
Wei Tang 9fdd4672b0 Use tracing-based subscriber logging (#6825)
* init_logger: switch from log-based to tracing-based and add compatibility layer

* Move tracing profiling subscriber related config realization

* sp-tracing: change profiling to be a layer instead of a subscriber

* Enable profiling layer in cli

* Change all test env_logger init to sp_tracing::try_init_simple

* Remove all local env_logger dependency

* Add missing tracing-subscriber dependency

* frame-sudo: fix tests

* frame-support: fix tests

* Fix frame/pallet and executor tests

* Fix the remaining tests

* Use subscriber's try_init as recommended by @davidbarsky

* Be explict that the tracing-log feature is needed

* Set subscriber writer to stderr

* Shorter line width

* Update cargo lock tracing version

* Fix sc_tracing crate compile

* Fix sc_authority_discovery crate test

* unremove default-features

* Leave enabled to default true

* Warn if global default cannot be set

* Fix unused import

* Remove unused PROXY_TARGET

* Change all reference from rc5 to rc6

* Change all reference of rc2 to rc6

* Fix styling

* Fix typo

* make logger init error'ing

* re-fixing the test issue

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-09-17 11:04:43 +02:00
Max Inden f677b40ed7 client/authority-discovery: Throttle DHT requests (#7018)
* client/authority-discovery: Throttle DHT requests

Instead of passing one DHT query for each authority down to the network
every query interval, only pass MAX_IN_FLIGHT_LOOKUPS at a given point
in time, triggering new ones when previous ones return.

* client/authority-discovery/worker/test: Fix wrong constant
2020-09-07 10:17:28 +00:00
Max Inden 9a17d58960 client/authority-discovery: Limit number of addresses per authority (#6947)
* client/authority-discovery: Test addresses per authority limit

* client/authority-discovery: Limit number of addresses per authority
2020-08-24 14:17:39 +00:00
Max Inden 5c500aa783 client/authority-discovery: Append PeerId to Multiaddr at most once (#6933)
* client/authority-discovery/worker: Extract address getter

* client/authority-discovery: Test for no duplicate p2p components

* client/authority-discovery: Append PeerId to Multiaddr at most once

When collecting the addresses to be published for the local node,
`addresses_to_publish` adds the local nodes `PeerId` to each
`Multiaddr`. Before doing so, ensure the `Multiaddr` does not already
contain one.

* client/authority-discovery: Remove explicit return
2020-08-24 09:37:32 +02:00
Max Inden 2d5ec72331 client/authority-discovery: Introduce AuthorityDiscoveryService (#6760)
* client/authority-discovery: Rename AuthorityDiscovery to XXXWorker

* client/authority-discovery: Introduce AuthorityDiscoveryService

Add a basic `AuthorityDiscoveryService` implementation which enables
callers to get the addresses for a given `AuthorityId` from the local
cache.

* client/authority-discovery: Split into worker and service mod

Move `Service` and `Worker` to their own Rust modules resulting in the
following file structure.

├── build.rs
├── Cargo.toml
└── src
    ├── error.rs
    ├── lib.rs
    ├── service.rs
    ├── tests.rs
    ├── worker
    │   ├── addr_cache.rs
    │   ├── schema
    │   │   └── dht.proto
    │   └── tests.rs
    └── worker.rs

* client/authority-discovery: Cache PeerId -> AuthorityId mapping

* client/authority-discovery: Update priority group on interval

Instead of updating the authority discovery peerset priority group each
time a new DHT value is found, update it regularly on an interval.

This removes the need for deterministic random selection. Instead of
trying to return a random stable set of `Multiaddr`s, the `AddrCache`
now returns a random set on each call.

* client/authority-discovery: Implement Service::get_authority_id

* client/authority-discovery: Use HashMap instead of BTreeMap

* client/authority-discovery: Rework priority group interval

* client/authority-discovery: Fix comment

* bin/node/cli: Update authority discovery constructor

* client/authority-discovery: Fuse from_service receiver

* client/authority-discovery: Remove Rng import

* client/authority-discovery: Ignore Multiaddr without PeerId

* client/authority-discovery/service: Add note on returned None

* client/authority-discovery/addr_cache: Replace double clone with deref
2020-08-12 14:16:40 +00:00