Commit Graph

35 Commits

Author SHA1 Message Date
Pierre Krieger aa95196be3 Extract transactions handling from protocol.rs (#8110)
* Extract transactions handling from protocol.rs

* Oops, boolean

* Do this better

* Update client/network/src/transactions.rs

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* [WIP] Fix handshake

* Finish handshake change

* Bugfix

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2021-02-18 17:04:23 +01:00
Max Inden 3006100977 client/network: Use request response for light client requests (#7895)
* client/network: Re-enable light_client_handler.rs unit tests

* client/network: Add scaffolding for light client using req-resp

* client/network: Make it compile

* client/network: Rename OutEvent SendRequest

* client/network: Restructure light client request client and handler

* client/network: Rename light client request client to sender

* client/network: Remove light client prepare_request

* client/network/src/light: Rework configuration

* client/network: Formatting

* client/network/light: Remove RequestId

* client/network/light: Make request functions methods

* client/network/light: Refactor request wrapping

* client/network/light: Fix warnings

* client/network/light: Serialize request in method

* client/network/light: Make returning response a method

* client/network/light: Depend on request response to timeout requests

* client/network: Fix test compilation

* client/network/light: Re-enable connection test

* client/network/light: Re-enable timeout test

* client/network/light: Re-enable incorrect_response test

* client/network/light: Re-enable wrong_response_type test

* client/network/light: Re-enable retry_count_failures test

* client/network/light: Re-enable issue_request tests

* client/network/light: Re-enable send_receive tests

* client/network/light: Deduplicate test logic

* client/network/light: Remove unused imports

* client/network/light: Handle request failure

* client/network/light: Move generate_protocol_config

* client/network: Fix test compilation

* client/network: Rename light client request client to sender

* client/network: Handle too-many-requests error

* client/network: Update outdated comments

* client/network/light: Choose any peer if none has best block defined

* .maintain: Replace sentry-node with local-docker-test-network

Sentry nodes are deprecated. Thus there is no need for
`.maintain/sentry-node` to spin up a sentry node test environment.
Instead this commit rewrites the setup to contain two full-connected
validators and one light client.

With the steps below one can now spin up a local test network with
two validators, one light-client, Prometheus and Grafana.

- cargo build --release
- sudo docker-compose -f .maintain/local-docker-test-network/docker-compose.yml up

* client/network/light: Handle oneshot cancellation

* client/network/light: Do not reduce retry count on missing peer

* client/network/request-response: Assert in debug request id to be unique

* client/network/light: Choose same limit as block request protocol

* client/network: Report reputation changes via response

Allow request response protocol handlers to issue reputation changes, by
sending them back along with the response payload.

* client/network: Remove resolved TODOs
2021-02-01 15:59:47 +00:00
Pierre Krieger 8e04515912 Add explicit limits to notifications sizes and adjust yamux buffer size (#7925)
* Add explicit limits to notifications sizes and adjust yamux buffer size

* Docfix

* Tests

* Document these 10 bytes
2021-01-19 11:00:37 +00:00
Pierre Krieger 779c4f8616 Rework priority groups, take 2 (#7700)
* Rework priority groups

* Broken tests fix

* Fix warning causing CI to fail

* [Hack] Try restore backwards-compatibility

* Fix peerset bug

* Doc fixes and clean up

* Error on state mismatch

* Try debug CI

* CI debugging

* [CI debug] Can I please see this line

* Revert "[CI debug] Can I please see this line"

This reverts commit 4b7cf7c1511f579cd818b21d46bd11642dfac5cb.

* Revert "CI debugging"

This reverts commit 9011f1f564b860386dc7dd6ffa9fc34ea7107623.

* Fix error! which isn't actually an error

* Fix Ok() returned when actually Err()

* Tweaks and fixes

* Fix build

* Peerset bugfix

* [Debug] Try outbound GrandPa slots

* Another bugfix

* Revert "[Debug] Try outbound GrandPa slots"

This reverts commit d175b9208c088faad77d9f0ce36ff6f48bd92dd3.

* [Debug] Try outbound GrandPa slots

* Apply suggestions from code review

Co-authored-by: Max Inden <mail@max-inden.de>

* Use consts for hardcoded peersets

* Revert "Try debug CI"

This reverts commit 62c4ad5e79c03d561c714a008022ecac463a597e.

* Renames

* Line widths

* Add doc

Co-authored-by: Max Inden <mail@max-inden.de>
2021-01-07 13:52:39 +00:00
Max Inden 3f629f743b client/network: Use request response for block requests (#7478)
* client/network: Add scaffolding for finality req to use req resp
	#sc

* client/network/src/finality_requests: Remove

* client/network/src/behaviour: Pass request id down to sync

* client/network: Use request response for block requests

* client/network: Move handler logic into *_*_handler.rs

* client/network: Track ongoing finality requests in protocol.rs

* client/network: Remove commented out finalization initialization

* client/network: Add docs for request handlers

* client/network/finality_request_handler: Log errors

* client/network/block_request_handler: Log errors

* client/network: Format

* client/network: Handle block request failure

* protocols/network: Fix tests

* client/network/src/behaviour: Handle request sending errors

* client/network: Move response handling into custom method

* client/network/protocol: Handle block response errors

* client/network/protocol: Remove tracking of obsolete requests

* client/network/protocol: Remove block request start time tracking

This will be handled generically via request-responses.

* client/network/protocol: Refactor on_*_request_started

* client/network: Pass protocol config instead of protocol name

* client/network: Pass protocol config in tests

* client/network/config: Document request response configs

* client/network/src/_request_handler: Document protocol config gen

* client/network/src/protocol: Document Peer request values

* client/network: Rework request response to always use oneshot

* client/network: Unified metric reporting for all request protocols

* client/network: Move protobuf parsing into protocol.rs

* client/network/src/protocol: Return pending events after poll

* client/network: Improve error handling and documentation

* client/network/behaviour: Remove outdated error types

* Update client/network/src/block_request_handler.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* Update client/network/src/finality_request_handler.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* client/network/protocol: Reduce reputation on timeout

* client/network/protocol: Refine reputation changes

* client/network/block_request_handler: Set and explain queue length

* client/service: Deny block requests when light client

* client/service: Fix role matching

* client: Enforce line width

* client/network/request_responses: Fix unit tests

* client/network: Expose time to build response via metrics

* client/network/request_responses: Fix early connection closed error

* client/network/protocol: Fix line length

* client/network/protocol: Disconnect on most request failures

* client/network/protocol: Disconnect peer when oneshot is canceled

* client/network/protocol: Disconnect peer even when connection closed

* client/network/protocol: Remove debugging log line

* client/network/request_response: Use Clone::clone for error

* client/network/request_response: Remove outdated comment

With libp2p v0.33.0 libp2p-request-response properly sends inbound
failures on connections being closed.

Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2021-01-05 18:20:54 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
André Silva 1871a95088 grandpa: remove light-client specific block import pipeline (#7546)
* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs
2020-11-23 14:28:55 +00:00
Pierre Krieger 1eae9f5792 Remove necessity to pass ConsensusEngineId when registering notifications protocol (#7549)
* Remove necessity to pass ConsensusEngineId when registering notifications protocol

* Line width

* Fix tests protocol name

* Other renames

* Doc update

* Change issue in TODO
2020-11-18 15:05:35 +00:00
Max Inden 3d94d80309 client/network: Expose number of entries per Kademlia bucket (#7104)
Extend `sub_libp2p_kbuckets_num_nodes` Prometheus metric to expose the
number of nodes per bucket per Kademlia instance instead of only per
Kademlia instance.
2020-09-15 09:34:22 +00:00
Roman Borschel ec47877288 Refactor & detach network metrics. (#6986)
* Refactor sc-network/service metrics.

  1. Aggregate sc-network metrics into a submodule, introducing
  two more sourced metrics to avoid duplicate atomics.

  2. Decouple periodic sc-service network metrics from other
  metrics, so that they can be updated independently.

* Update client/service/src/metrics.rs

* Update client/service/src/metrics.rs
2020-09-06 19:59:05 +02:00
Max Inden 8fd343e39d client/*: Treat protocol name as str and not [u8] (#6967)
* client/*: Treat protocol name as str and not [u8]

Notification protocol names are in practice always valid utf8 strings.
Instead of treating them as such in the type system, thus far they were
casted to a [u8] at creation time.

With this commit protocol names are instead treated as valid utf8
strings throughout the codebase and passed as `Cow<'static, str>`
instead of `Cow<'static, [u8]>`. Among other things this eliminates the
need for string casting when logging.

* client/network: Don't allocate when protocol name is borrowed
2020-08-28 15:34:25 +00:00
Pierre Krieger 1bd6082cf7 Enforce that ProtocolId is a string (#6953)
* Enforce that ProtocolId is a string

* Fix test
2020-08-26 12:27:30 +00:00
Pierre Krieger 74cf119d4a Try un-ignore flaky test (#6923) 2020-08-20 15:10:00 +00:00
Nikolay Volf c347300e3e Move to upstream wasmtime, refactor globals snapshot (#6759)
* refactor globals snapshot

* ignore test

* update pwasm-utils ref

* line width

* add doc comment for internal struct

* add explanation for iteration

* Demote rustdoc to a comment

* use 0.14

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-08-11 15:05:31 +00:00
Pierre Krieger 5060324b91 Ignore flaky test (#6767) 2020-07-30 14:58:30 +02:00
Pierre Krieger 1ab7719314 Add a back-pressure-friendly alternative to NetworkService::write_notifications 🎉 (#6692)
* Add NetworkService::send_notifications

* Doc

* Doc

* API adjustment

* Address concerns

* Make it compile

* Start implementation

* Progress in the implementation

* Change implementation strategy again

* More work before weekend

* Finish changes

* Minor doc fix

* Revert some minor changes

* Apply suggestions from code review

* GroupError -> NotifsHandlerError

* Apply suggestions from code review

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>

* state_transition_waker -> close_waker

* Apply suggestions from code review

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>

* Finish renames in service.rs

* More renames

* More review suggestsions applied

* More review addressing

* Final change

* 512 -> 2048

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
2020-07-29 11:23:19 +00:00
Bastian Köcher 9310f15ac2 Name all the tasks! (#6726)
* Remove any implementation of `Spawn` or `Executor` from our task executors

* Fix compilation

* Rename `SpawnBlockingExecutor`

* Update primitives/core/src/traits.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Fix tests

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-26 12:56:17 +00:00
Bastian Köcher 53be6ec510 Support synching of blocks that are not new_best (#6508)
* Start

* Remove debug println

* Add tests
2020-06-30 14:44:52 +00:00
Cecile Tonglet 4bf044eac6 Ensure the listen addresses are consistent with the transport (#6436)
* Initial commit

Forked at: 82bdf1a891
No parent branch.

* Ensure the listen addresses are consistent with the transport

* Update client/network/src/error.rs

* Update client/network/src/service.rs

* Better implementation

* Fix bad previous impl

* add boot_nodes

* reserved nodes

* test boot nodes

* reserved nodes tests

* add public_addresses and make specific error type

* Update client/network/src/error.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-06-23 17:26:00 +02:00
Roman Borschel 6aa8965f33 Avoid panic on dropping a sc_network::service::out_events::Receiver. (#6458)
* Avoid panic on dropping a `Receiver`.

* CI
2020-06-23 10:51:35 +02:00
Denis Pisarev f9c0c6a719 New CI image (#6223)
* fix (ci): hotfix Docker release

* change (ci): moving to the tested CI image with a proper name

* change (ci): rename substrate-ci-linux

* Reduce the lots_of_incoming_peers_works test load (#6314)

* change (ci): moving to the tested CI image with a proper name

* change (ci): rename substrate-ci-linux

* Reduce the lots_of_incoming_peers_works test load (#6314)

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-06-10 13:12:21 +02:00
Pierre Krieger 07b984b609 Add a test for lots of nodes connecting at the same time (#6247)
* Add a test for lots of nodes connecting at the same time

* Do small change
2020-06-09 16:30:34 +02:00
Pierre Krieger 717fa95bc7 Restore the empty line after the license (#6088) 2020-05-20 13:08:27 +00:00
Nikolay Volf d5411969ac fix whitespace (#6062) 2020-05-18 13:51:28 +02:00
Benjamin Kampmann efc4849f1a Meter block import results via prometheus (#6025) 2020-05-15 17:25:51 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Bastian Köcher 9c5536e01a Expose that BasicQueue expects blocking spawn (#5860)
* Expose that `BasicQueue` expects blocking spawn

Up to now `BasicQueue` expected a closure that to spawn a `Future`.
This was expected to be a closure that spawns a blocking future.
However, this wasn't documented anywhere. This pr introduces a new trait
`SpawnBlocking` that exposes this requirement to the outside.

* Feedback
2020-05-04 19:40:29 +02:00
pscott 0f401e4699 Move spawning tasks from thread pools to Service's TaskManager for block importing (#5647)
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-04-29 18:46:39 +02:00
Pierre Krieger 25c3ab2c1e Fix warning reported in out_events when Registry is None (#5716) 2020-04-22 00:36:12 +02:00
Pierre Krieger ca1c60c2cf Move around definitions in sc_network (#5701) 2020-04-20 14:35:17 +02:00
Pierre Krieger 0fd5643e84 Adjustments to Kademlia-related metrics (#5660)
* Turn kbuckets_num_nodes into a GaugeVec

* random_kademlia_queries -> kademlia_random_queries

* kademalia_random_queries_total now a CounterVec

* Add metrics about records store
2020-04-17 09:11:47 +02:00
Pierre Krieger 239d0998ea Several tweaks to networking Prometheus metrics (#5636) 2020-04-16 15:18:35 +02:00
Pierre Krieger 1e1b0e2767 Give names to channels (#5626)
* Give names to channels

* Fix

* A couple more changes

* More minor tweaks

* Fix test
2020-04-14 14:49:41 +02:00
Pierre Krieger 877a7ab531 Add metrics for the events in the network output channels (#5597)
* Add metrics for the events in the network output channels

* Documentation fixes

* A couple fixes

* Fix panic at destruction

* Rework for direct Prometheus integration

* Don't lock on the Receiver

* Another review address

* Address review

* Update client/network/src/service/out_events.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* Fix bad event name

* Fix descriptions

* Fix names

* client/network/service/out_events: Apply remaining suggestions

Co-authored-by: Max Inden <mail@max-inden.de>
2020-04-09 20:30:34 +02:00
Pierre Krieger 0cc2e4df89 Fix #5516 (#5560)
* Add a failing test

* Make test not freeze

* Fix the bug

* Fix spaces

* Fix tests

* Apply suggestions from code review

Co-Authored-By: Toralf Wittner <tw@dtex.org>

* Make sure test doesn't succeed if nothing happened

* Fix build

* Do the events change

Co-authored-by: Toralf Wittner <tw@dtex.org>
2020-04-08 13:32:22 +02:00