Commit Graph
100 Commits
Author SHA1 Message Date
Max Inden e12c9497d2 Cargo.lock: Update to libp2p-swarm v0.27.2 (#2384)
`libp2p-swarm` `CHANGELOG.md`:

- Have `ToggleProtoHandler` ignore listen upgrade errors when disabled.
  [PR 1945](https://github.com/libp2p/rust-libp2p/pull/1945/files).

Fixes:

https://github.com/paritytech/polkadot/issues/2373
2021-02-04 14:02:33 +01:00
Max Inden 3006100977 client/network: Use request response for light client requests (#7895)
* client/network: Re-enable light_client_handler.rs unit tests

* client/network: Add scaffolding for light client using req-resp

* client/network: Make it compile

* client/network: Rename OutEvent SendRequest

* client/network: Restructure light client request client and handler

* client/network: Rename light client request client to sender

* client/network: Remove light client prepare_request

* client/network/src/light: Rework configuration

* client/network: Formatting

* client/network/light: Remove RequestId

* client/network/light: Make request functions methods

* client/network/light: Refactor request wrapping

* client/network/light: Fix warnings

* client/network/light: Serialize request in method

* client/network/light: Make returning response a method

* client/network/light: Depend on request response to timeout requests

* client/network: Fix test compilation

* client/network/light: Re-enable connection test

* client/network/light: Re-enable timeout test

* client/network/light: Re-enable incorrect_response test

* client/network/light: Re-enable wrong_response_type test

* client/network/light: Re-enable retry_count_failures test

* client/network/light: Re-enable issue_request tests

* client/network/light: Re-enable send_receive tests

* client/network/light: Deduplicate test logic

* client/network/light: Remove unused imports

* client/network/light: Handle request failure

* client/network/light: Move generate_protocol_config

* client/network: Fix test compilation

* client/network: Rename light client request client to sender

* client/network: Handle too-many-requests error

* client/network: Update outdated comments

* client/network/light: Choose any peer if none has best block defined

* .maintain: Replace sentry-node with local-docker-test-network

Sentry nodes are deprecated. Thus there is no need for
`.maintain/sentry-node` to spin up a sentry node test environment.
Instead this commit rewrites the setup to contain two full-connected
validators and one light client.

With the steps below one can now spin up a local test network with
two validators, one light-client, Prometheus and Grafana.

- cargo build --release
- sudo docker-compose -f .maintain/local-docker-test-network/docker-compose.yml up

* client/network/light: Handle oneshot cancellation

* client/network/light: Do not reduce retry count on missing peer

* client/network/request-response: Assert in debug request id to be unique

* client/network/light: Choose same limit as block request protocol

* client/network: Report reputation changes via response

Allow request response protocol handlers to issue reputation changes, by
sending them back along with the response payload.

* client/network: Remove resolved TODOs
2021-02-01 15:59:47 +00:00
Max Inden b692954c6e client/network: Report reputation changes via response (#7958)
* client/network: Report reputation changes via response

When handling a request by a remote peer in a request response handler,
one might want to in- or de-crease the reputation of the peer. E.g. one
might want to decrease the reputation slightly for each request, given
that it forces the local node to do work, or one might want to issue a
larger reputation change due to a malformed request by the remote peer.

Instead of having to pass a peerset handle to each request response
handler, this commit suggests to allow handlers to isssue reputation
changes via the provided `pending_response` `oneshot` channel.

A reputation change issued by a request response handler via the
`pending_response` channel is received by the
`RequestResponsesBehaviour` which passes the reputation change up as an
event to eventually be send to a peerset via a peerset handle.

* client/network/req-resp: Use Vec::new instead of None::<Vec<_>>

* client/network: Rename Response to OutgoingResponse

Given that a request-response request is not called `Request` but
`InomingRequest`, rename a request-response response to
`OutgoingResponse`.

* client/finality-grandpa-warp: Send empty rep change via response
2021-01-26 10:45:41 +00:00
Max Inden 62d551a063 client/network/req-resp: Prevent request id collision (#7957)
* client/network/req-resp: Add unit test for request id collision

* client/network/req-resp: Prevent request id collision

`RequestId` is a monotonically increasing integer, starting at
`1`. A `RequestId` is unique for a single `RequestResponse`
behaviour, but not across multiple `RequestResponse` behaviours. Thus
when handling `RequestId` in the context of multiple
`RequestResponse` behaviours, one needs to couple the protocol name
with the `RequestId` to get a unique request identifier.

This commit ensures that pending requests (`pending_requests`) and
pending responses (`pending_response_arrival_time`) are tracked both by
their protocol name and `RequestId`.

* client/network/req-resp: Remove unused import

* client/network/req-resp: Introduce ProtocolRequestId struct

* client/network/req-resp: Update test doc comment

Treat `RequestId` as an opaque type.

* client/network/req-resp: Improve expect proof
2021-01-25 13:09:45 +00:00
Max Inden 9031c35114 .maintain: Replace sentry-node with local-docker-test-network (#7943)
Sentry nodes are deprecated. Thus there is no need for
`.maintain/sentry-node` to spin up a sentry node test environment.
Instead this commit rewrites the setup to contain two full-connected
validators and one light client.

With the steps below one can now spin up a local test network with
two validators, one light-client, Prometheus and Grafana.

- cargo build --release
- sudo docker-compose -f .maintain/local-docker-test-network/docker-compose.yml up
2021-01-21 09:29:50 +00:00
Max Inden 3927b649cb *: Update to libp2p v0.34.0 (#7888)
* *: Update to libp2p v0.34.0

* client/network: Update bytes, unsigned-varint and asynchronous-codec

* client: Update to prost v0.7
2021-01-15 09:55:42 +00:00
Max Inden ffb5410ba1 client/network: Re-enable light_client_handler.rs unit tests (#7853) 2021-01-08 17:22:55 +01:00
3f629f743b client/network: Use request response for block requests (#7478)
* client/network: Add scaffolding for finality req to use req resp
	#sc

* client/network/src/finality_requests: Remove

* client/network/src/behaviour: Pass request id down to sync

* client/network: Use request response for block requests

* client/network: Move handler logic into *_*_handler.rs

* client/network: Track ongoing finality requests in protocol.rs

* client/network: Remove commented out finalization initialization

* client/network: Add docs for request handlers

* client/network/finality_request_handler: Log errors

* client/network/block_request_handler: Log errors

* client/network: Format

* client/network: Handle block request failure

* protocols/network: Fix tests

* client/network/src/behaviour: Handle request sending errors

* client/network: Move response handling into custom method

* client/network/protocol: Handle block response errors

* client/network/protocol: Remove tracking of obsolete requests

* client/network/protocol: Remove block request start time tracking

This will be handled generically via request-responses.

* client/network/protocol: Refactor on_*_request_started

* client/network: Pass protocol config instead of protocol name

* client/network: Pass protocol config in tests

* client/network/config: Document request response configs

* client/network/src/_request_handler: Document protocol config gen

* client/network/src/protocol: Document Peer request values

* client/network: Rework request response to always use oneshot

* client/network: Unified metric reporting for all request protocols

* client/network: Move protobuf parsing into protocol.rs

* client/network/src/protocol: Return pending events after poll

* client/network: Improve error handling and documentation

* client/network/behaviour: Remove outdated error types

* Update client/network/src/block_request_handler.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* Update client/network/src/finality_request_handler.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* client/network/protocol: Reduce reputation on timeout

* client/network/protocol: Refine reputation changes

* client/network/block_request_handler: Set and explain queue length

* client/service: Deny block requests when light client

* client/service: Fix role matching

* client: Enforce line width

* client/network/request_responses: Fix unit tests

* client/network: Expose time to build response via metrics

* client/network/request_responses: Fix early connection closed error

* client/network/protocol: Fix line length

* client/network/protocol: Disconnect on most request failures

* client/network/protocol: Disconnect peer when oneshot is canceled

* client/network/protocol: Disconnect peer even when connection closed

* client/network/protocol: Remove debugging log line

* client/network/request_response: Use Clone::clone for error

* client/network/request_response: Remove outdated comment

With libp2p v0.33.0 libp2p-request-response properly sends inbound
failures on connections being closed.

Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2021-01-05 18:20:54 +00:00
Max Inden f0b99dd2f2 *: Update to libp2p v0.33.0 (#7759)
* *: Update to libp2p v0.33.0

* client/network: Consistently track request arrival time

With https://github.com/libp2p/rust-libp2p/pull/1886/ one is guaranteed
to receive either a `ResponseSent` or a `InboundFailure` event for each
received inbound request via `RequestResponseEvent::Message`. Given this
guarantee there is no need to track arrival times in a best-effort
manner and thus there is no need to use a LRU cache for arrival times.

* client/offchain: Adjust to PeerId API changes
2021-01-04 11:54:51 +00:00
Max Inden e4e120e9b3 *: Update to libp2p v0.32.2 (#7708)
Version v0.32.2 pins async-tls to the semver compliant v0.11.0 instead
of the yanked async-tls v0.10.2.
2020-12-10 09:22:40 +00:00
Max Indenandparity-processbot 0e885a3a71 Cargo.lock: Update to async-tls 0.10.2 (#2096)
* Cargo.lock: Update to async-tls 0.10.2

* "Update Substrate"

Co-authored-by: parity-processbot <>
2020-12-09 22:23:06 +00:00
Max Inden 0a2636d20c *: Update to libp2p v0.32.0 (#7696)
* *: Update to libp2p v0.32.0

* Cargo.lock: Update async-tls to 0.10.2

* client/network/request_response: Adjust to new request response events

* client/network/request_response.rs: Clean up silently failing responses

* client/network/discovery: Lazily instantiate mdns

* client/network/discovery: Exclude MdnsWrapper for target_os unknown

* client/network/discovery: Fix indentation

* client/network/request-response: Use LruCache to track pending resp time

* client/network/request_responses: Fix early connection closed error

* client/network/request-response: Replace debug_assert with debug

* client/network/request-response: Fix typo

* client/network/request-response: Don't emit event on send_response fail

* client/network/request-response: Revert waker.wake_by_ref()

* client/network/request-resp: Make duration in InboundRequest optional

* client/network/req-resp: Don't emit two events for busy builder

When a response builder is busy incoming requests are dropped.
Previously this was reported both via a `ResponseFailure::Busy` and a
`ReponseFailure::Network(InboundFailure::Omisssion)` event.

With this commit the former is removed, leaving only the latter in
place.
2020-12-09 21:58:22 +00:00
Max Inden b29d6092c2 frame/staking/README: Fix broken link (#7703) 2020-12-09 21:09:38 +01:00
Max Inden b149b1fdc0 *: Update to libp2p v0.31.2 (#7658) 2020-12-02 17:45:49 +00:00
Max Inden e3608da5fa */Cargo.toml: Remove unused dependencies (#7590)
* */Cargo.toml: Remove unused dependencies

Using cargo-udeps to detect unused dependencies.

* client/network/Cargo: Revert dependency removal

* Cargo.lock: Update
2020-11-24 19:56:21 +01:00
Max Indenandparity-processbot 63bf63e442 *: Update authority discovery and remove WorkerConfig (#1953)
* *: Update authority discovery and remove WorkerConfig

With https://github.com/paritytech/substrate/pull/7545 the authority
discovery module queries and publishes addresses on an exponentially
increasing interval. Doing so should make custom configurations
obsolete, as operations are retried in a timely fashion in the first
minutes.

* */Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing

* Revert "*/Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing"

This reverts commit 0785943a1e377454f088814ef20f4432de09da7a.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 377b221e1853b2c383f0c416d686535b545796cb.

* Cargo.lock: Manual Substrate update

* node/test/service/src/lib: Remove unused import

* parachain/test-parachains/adder: Remove unused import

Co-authored-by: parity-processbot <>
2020-11-23 18:23:18 +00:00
Max Inden d692d173f2 client/authority-discovery: Publish and query on exponential interval (#7545)
* client/authority-discovery: Publish and query on exponential interval

When a node starts up publishing and querying might fail due to various
reasons, for example due to being not yet fully bootstrapped on the DHT.
Thus one should retry rather sooner than later. On the other hand, a
long running node is likely well connected and thus timely retries are
not needed. For this reasoning use an exponentially increasing interval
for `publish_interval`, `query_interval` and
`priority_group_set_interval` instead of a constant interval.

* client/authority-discovery/src/interval.rs: Add license header

* .maintain/gitlab: Ensure adder collator tests are run on CI
2020-11-23 16:34:37 +00:00
Max IndenandPierre Krieger 52d261e3c7 .github: Add dependabot config and thus enable dependabot (#7509)
* .github: Add dependabot config and thus enable dependabot

* Update .github/dependabot.yml

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-11-20 13:34:19 +00:00
Max Inden 124b954fad Cargo.lock: Run cargo update (#7553)
* Cargo.lock: Run cargo update

* Cargo.lock: Downgrade cc to v1.0.62

* Cargo.lock: Revert wasm-* updates
2020-11-20 13:59:02 +01:00
Max IndenandAndronik Ordian 9a99607c7e .github/dependabot: Handle Substrate crates manually (#1961)
* .github/dependabot: Handle Substrate crates manually

Handle updates for crates from github.com/paritytech/substrate manually.

* .github/dependabot: Additional prefixes

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>
2020-11-17 11:34:54 +00:00
Max IndenandPierre Krieger 178bc960c5 .github: Add dependabot config and thus enable dependabot (#1937)
* .github: Add dependabot config and thus enable dependabot

* Update .github/dependabot.yml

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-11-17 10:20:46 +00:00
Max Inden 75e365a59b *: Update to libp2p v0.30.0 (#7508)
* *: Update to libp2p v0.30.0

* Cargo.lock: Update

* *: Update to libp2p v0.30.1
2020-11-16 18:49:50 +00:00
Max Inden 80a2be8c4b Cargo.lock: Update dependencies (#1929) 2020-11-08 18:30:13 +01:00
5ff5f64d80 docs/CONTRIBUTING: Update Polkadot pull request process documentation (#7393)
* docs/CONTRIBUTING: Update Polkadot pull request process

* Update docs/CONTRIBUTING.adoc

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
2020-10-30 16:01:01 +01:00
Max Indenandparity-processbot 6d7b42b5ac node/service/src/lib: Do not spawn authority discovery on sentries (#1835)
* node/service/src/lib: Do not spawn authority discovery on sentries

The notion of sentry nodes has been deprecated (see [1] for details).
Support for sentry nodes in the `client/authority-discovery` module has
been removed.

This commit adjusts the instantiation of the authority discovery worker
accordingly, only spawning the module on authority nodes.

[1] https://github.com/paritytech/substrate/issues/6845

* "Update Substrate"

Co-authored-by: parity-processbot <>
2020-10-26 10:31:16 +00:00
Max Inden 653868c01e client/authority-discovery: Remove sentry node logic (#7368)
* client/authority-discovery: Remove sentry node logic

The notion of sentry nodes has been deprecated (see [1] for details).
This commit removes support for sentry nodes in the
`client/authority-discovery` module.

While removing `Role::Sentry` this commit also introduces
`Role::Discover`, allowing a node to discover addresses of authorities
without publishing ones own addresses. This will be needed in Polkadot
for collator nodes.

[1] https://github.com/paritytech/substrate/issues/6845

* client/authority-discovery/service: Improve PeerId comment
2020-10-26 10:06:56 +00:00
Max Inden 0fcf9d6c60 client/network: Make NetworkService::set_priority_group async (#7352)
As done with `NetworkService::{add_to,remove_from}_priority_group`, make
`NetworkService::set_priority_group` async as well. This future-proofs
the API should we ever decide to use a bounded channel between
`NetworkService` and `NetworkWorker`.
2020-10-22 09:15:39 +00:00
Max Inden 96e939f42e client/network: Remove original_request in block request events (#7369)
The `original_request` field of each `block_request::Event` variant is
not used. With that in mind, this commit removes the field.
2020-10-21 20:06:35 +02:00
Max Inden 4e9256aba2 client/network: Allow configuring Kademlia's disjoint query paths (#7356)
The Rust libp2p-kad implementation can require iterative queries to use
disjoint paths for increased resiliency in the presence of potentially
adversarial nodes.

Allow Substrate users to enable this feature via the
`--kademlia-disjoint-query-paths` flag.
2020-10-21 11:08:39 +00:00
Max Inden 918a0c8077 *: Bump async-std to v1.6.5 (#7306)
* *: Bump async-std to v1.6.5

Prevent users from using v1.6.4 which faces issues receiving incoming
TCP connections. See https://github.com/async-rs/async-std/issues/888
for details.

* client/network/src/gossip: Use channel instead of condvar

`async_std::sync::Condvar::wait_timeout` uses
`gloo_timers::callback::Timeout` when compiled for
`wasm32-unknown-unknown`. This timeout implementation does not fulfill
the requirement of being `Send`.

Instead of using a `Condvar` use a `futures::channel::mpsc` to signal
progress from the `QueuedSender` to the background `Future`.

* client/network/Cargo.toml: Remove async-std unstable feature

* client/network/src/gossip: Forward all queued messages

* client/network/gossip: Have QueuedSender methods take &mut self

* client/network/gossip: Move queue_size_limit into QueuedSender

The `queue_size_limit` field is only accessed by `QueuedSender`, thus
there is no need to share it between the background future and the
`QueuedSender`.

* client/network/gossip: Rename background task to future

To be a bit picky the background task is not a task in the sense of an
asynchonous task, but rather a background future in the sense of
`futures::future::Future`.
2020-10-20 09:23:27 +00:00
Max Inden bd2dbc055c client/network: Remove option to disable yamux flow control (#7358)
With the `OnRead` flow control option yamux "send[s] window updates only
when data is read on the receiving end" and not as soon as "a Stream's
receive window drops to 0".

Yamux flow control has proven itself. This commit removes the feature
flag. Yamux flow control is now always enabled.
2020-10-19 21:43:32 +02:00
Max Inden 60e66f6af0 Revert "revert enabling authority discovery by default (#1532)" (#1807)
This reverts commit a7b6c91f80.

The authority discovery module was initially enabled by default on
validator and sentry nodes with commit 59f5eb4. This change was later on
reverted in a7b6c91. With this commit the authority discovery module is
again enabled by default.
2020-10-13 12:09:10 +00:00
Max Inden 0ff724c939 .maintain/monitoring: Add alert when continuous task ends (#7250)
* .maintain/monitoring: Add alert when continuous task ends

Through the `polkadot_tasks_ended_total` Prometheus metric one can tell
when a task ended. Use this metric to alert when specific
known-to-be-continuous tasks end on a node.

* .maintain/monitoring: Don't hard-code task names
2020-10-05 08:40:24 +00:00
Max Inden a772eb27a2 client/cli: Update to fdlimit 0.2.1 (#7249)
fdlimit 0.2.1 contains a small patch allowing compilation on platforms
where libc represents rlimits as u32.
2020-10-02 08:51:21 +00:00
Max Inden 51c0d27aa1 .maintain/monitoring: Normalize alerting rules (#7232)
* .maintain/monitoring: Normalize alerting rules

- Start alert names with their component and end with the describing
adjective.

- Describe alert duration in `message` with `for more than` across all
alerts.

* .maintain/monitoring: Fix alert tests
2020-09-30 08:48:48 +00:00
Max Inden 5c0413aced client/network: Stop sending noise legacy handshake (#7211)
Stop sending legacy handshake payloads, i.e. length-prefixed protobuf
payloads inside a length-prefixed noise frame. Receiving of such legacy
handshakes is still supported, thus backwards compatibility is given.
2020-09-29 09:00:58 +00:00
Max Inden dc81f08621 client/network/src/transport: Use libp2p upgrade builder (#7165)
Instead of building a libp2p `Transport` via a chain of `and_then`,
leverage the `libp2p::core::upgrade::Builder` pattern for authentication
and multiplexing.
2020-09-22 08:48:55 +00:00
Max Inden 3d94d80309 client/network: Expose number of entries per Kademlia bucket (#7104)
Extend `sub_libp2p_kbuckets_num_nodes` Prometheus metric to expose the
number of nodes per bucket per Kademlia instance instead of only per
Kademlia instance.
2020-09-15 09:34:22 +00:00
Max Inden d9e25594e9 client/authority-discovery/src/service: Improve docs (#7059) 2020-09-09 15:08:26 +00:00
Max Inden f677b40ed7 client/authority-discovery: Throttle DHT requests (#7018)
* client/authority-discovery: Throttle DHT requests

Instead of passing one DHT query for each authority down to the network
every query interval, only pass MAX_IN_FLIGHT_LOOKUPS at a given point
in time, triggering new ones when previous ones return.

* client/authority-discovery/worker/test: Fix wrong constant
2020-09-07 10:17:28 +00:00
Max Inden 1d10db3184 frame/authority-discovery: Have authorities() return both current and next (#6788)
* frame/authority-discovery: Have authorities() return both current and next

Authority address lookups on the DHT happen periodically (every 10
mintues) and are rather slow (~10 seconds).

In order to smooth the transition period between two sessions, have the
runtime module return both the current as well as the next authority
set. Thereby the client authority module will:

1. Publish its addresses one session in advance.

2. Prefetch the addresses of authorities of the next session in advance.

* frame/authority-discovery: Deduplicate authority ids

* frame/authority-discovery: Don't dedup on_genesis authorities

* frame/authority-discovery: Remove mut and sort on comparison in tests

* frame/authority-discovery: Use BTreeSet for deduplication
2020-09-02 15:20:51 +00:00
Max Inden aa5a10802c *: Update to Prometheus v0.10.0 (#6964)
* *: Update to Prometheus v0.10.0-rc.1

* *: Update to Prometheus v0.10.0
2020-09-01 11:23:10 +00:00
Max Inden 875daa31bf client/cli/src/config: Warn on low file descriptor limit (#6956)
* client/cli/src/config: Warn on low file descriptor limit

Substrate sets the soft file descriptor limit to the hard limit at
startup. In the case of the latter being low already (< 10_000) a
Substrate node under high demand might run into issues e.g. when opening
up new TCP connections or persisting data to the database.

With this commit a warn message is printed to stderr.

* client/cli/Cargo.toml: Update to fdlimit 0.2.0
2020-08-31 12:37:06 +00:00
Max Indenandparity-processbot 87f6afd862 node/network/bridge: Define protocol names as str (#1655)
* node/network/bridge: Define protocol names as str

* "Update Substrate"

Co-authored-by: parity-processbot <>
2020-08-28 15:51:50 +00:00
Max Inden 8fd343e39d client/*: Treat protocol name as str and not [u8] (#6967)
* client/*: Treat protocol name as str and not [u8]

Notification protocol names are in practice always valid utf8 strings.
Instead of treating them as such in the type system, thus far they were
casted to a [u8] at creation time.

With this commit protocol names are instead treated as valid utf8
strings throughout the codebase and passed as `Cow<'static, str>`
instead of `Cow<'static, [u8]>`. Among other things this eliminates the
need for string casting when logging.

* client/network: Don't allocate when protocol name is borrowed
2020-08-28 15:34:25 +00:00
Max Inden 9a17d58960 client/authority-discovery: Limit number of addresses per authority (#6947)
* client/authority-discovery: Test addresses per authority limit

* client/authority-discovery: Limit number of addresses per authority
2020-08-24 14:17:39 +00:00
Max Inden f04afd596b .maintain/monitoring/alerting-rules: Add fd alert (#6946)
Alert on high file descriptor allocation.
2020-08-24 15:37:07 +02:00
Max Inden 5c500aa783 client/authority-discovery: Append PeerId to Multiaddr at most once (#6933)
* client/authority-discovery/worker: Extract address getter

* client/authority-discovery: Test for no duplicate p2p components

* client/authority-discovery: Append PeerId to Multiaddr at most once

When collecting the addresses to be published for the local node,
`addresses_to_publish` adds the local nodes `PeerId` to each
`Multiaddr`. Before doing so, ensure the `Multiaddr` does not already
contain one.

* client/authority-discovery: Remove explicit return
2020-08-24 09:37:32 +02:00
Max Inden 69e349bd51 *: Update to next libp2p version 0.24.0 (#6891)
* *: Update to next libp2p version (likely v0.24.0)

* Revert "*: Update to next libp2p version (likely v0.24.0)"

This reverts commit ffe1545aba6c2557a2843579de331f3fc1c60743.

* */Cargo.toml: Update to libp2p v0.24.0

* client/network/src/service: Handle ConnectionClosed returning Option

* Cargo.*: Test kad usize conversion

* Revert "Cargo.*: Test kad usize conversion"

This reverts commit ad317879782f982cb4a4c76029a72b5b97e82bec.

* Cargo.lock: Update to libp2p-kad v0.22.1

* client/cli/Cargo.toml: Update to libp2p 0.24.0
2020-08-21 06:25:23 +00:00
Max Inden 6611019e2e client/authority-discovery: Revert query interval change (#6897)
Revert the accidental query interval change from every one minute back
to every 10 minutes.
2020-08-14 16:15:45 +00:00
Max Indenandparity-processbot 7914847950 service/src/lib: Update authority discovery construction (#1563)
* service/src/lib: Update authority discovery construction

https://github.com/paritytech/substrate/pull/6760 introduces the concept
of an authority discovery `Service` allowing one to communicate with an
authority discovery `Worker`, e.g. to learn the `Multiaddr`s for a given
`AuthorityId`.

Along with the new `Service` structure it also alters the authority
discovery constructor to return both a worker and a service. This
commit adjusts the callside of the constructor, ignoring the `Service`
for now.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 04fb79c465f91b55422e22d4ea266f08f4072854.

* Update Substrate

Co-authored-by: parity-processbot <>
2020-08-12 16:06:12 +00:00
Max Inden 2d5ec72331 client/authority-discovery: Introduce AuthorityDiscoveryService (#6760)
* client/authority-discovery: Rename AuthorityDiscovery to XXXWorker

* client/authority-discovery: Introduce AuthorityDiscoveryService

Add a basic `AuthorityDiscoveryService` implementation which enables
callers to get the addresses for a given `AuthorityId` from the local
cache.

* client/authority-discovery: Split into worker and service mod

Move `Service` and `Worker` to their own Rust modules resulting in the
following file structure.

├── build.rs
├── Cargo.toml
└── src
    ├── error.rs
    ├── lib.rs
    ├── service.rs
    ├── tests.rs
    ├── worker
    │   ├── addr_cache.rs
    │   ├── schema
    │   │   └── dht.proto
    │   └── tests.rs
    └── worker.rs

* client/authority-discovery: Cache PeerId -> AuthorityId mapping

* client/authority-discovery: Update priority group on interval

Instead of updating the authority discovery peerset priority group each
time a new DHT value is found, update it regularly on an interval.

This removes the need for deterministic random selection. Instead of
trying to return a random stable set of `Multiaddr`s, the `AddrCache`
now returns a random set on each call.

* client/authority-discovery: Implement Service::get_authority_id

* client/authority-discovery: Use HashMap instead of BTreeMap

* client/authority-discovery: Rework priority group interval

* client/authority-discovery: Fix comment

* bin/node/cli: Update authority discovery constructor

* client/authority-discovery: Fuse from_service receiver

* client/authority-discovery: Remove Rng import

* client/authority-discovery: Ignore Multiaddr without PeerId

* client/authority-discovery/service: Add note on returned None

* client/authority-discovery/addr_cache: Replace double clone with deref
2020-08-12 14:16:40 +00:00
Max Inden 0553dabe32 client/network: Fix wrong metric help text (#6794)
The `sub_libp2p_kademlia_query_duration` metric only has the dimension
`type` not `protocol`.
2020-08-03 09:08:06 +00:00
Max Inden 1365eef4c1 client/network: Expose DHT query duration to Prometheus (#6784)
Expose duration of DHT put and get request as a Prometheus histogram.
2020-08-03 08:30:06 +00:00
Max IndenandPierre Krieger dc33e64643 client/network: Add peers to DHT only if protocols match (#6549)
* client/network/src/discovery: Adjust to Kademlia  API changes

* client/network: Add peers to DHT only if protocols match

With https://github.com/libp2p/rust-libp2p/pull/1628 rust-libp2p allows
manually controlling which peers are inserted into the routing table.
Instead of adding each peer to the routing table automatically, insert
them only if they support the local nodes protocol id (e.g. `dot`)
retrieved via the `identify` behaviour.

For now this works around
https://github.com/libp2p/rust-libp2p/issues/1611. In the future one
might add more requirements. For example one might try to exclude
light-clients.

* Cargo.toml: Remove crates.io patch for libp2p

* client/network/src/behaviour: Adjust to PeerInfo name change

* client/network/src/discovery: Rework Kademlia event matching

* client/network/discovery: Add trace on adding peer to DHT

* client/network/discovery: Retrieve protocol name from kad behaviour

* client/network/discovery: Fix formatting

* client/network: Change DiscoveryBehaviour::add_self_reported signature

* client/network: Document manual insertion strategy

* client/network/discovery: Remove TODO for ignoring DHT address

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-07-29 14:44:15 +02:00
Max Inden e674d64a72 client/network: Adjust wording (#6755)
Rename `NetworkWorker::from_worker` to `NetworkWorker::from_service` as
it is a channel from the `NetworkService` to the `NetworkWorker`.
2020-07-29 11:18:40 +00:00
Max Inden fe9c01fc68 .maintain/monitoring/alerting-rules: Remove HighCPUUsage alert (#6648)
The `HighCPUUsage` alert is based on the `cpu_usage_percentage` metric.
Instead of exposing the overall CPU usage in percent, the metric exposes
the per core usage summed over all cores.

This commit removes the alert for two reasons:

1. Substrate itself does not expose the core count and thus one can not
alert based on the `cpu_usage_percentage` metric.

2. Alerting based on CPU usage is generic and not specific to Substrate
or Blockchains. Thus any CPU usage alert suffice.
2020-07-17 07:43:57 +00:00
Max Inden 6995e5e5e1 client/service/metrics: Rework metric HELP texts (#6647) 2020-07-16 23:07:32 +00:00
Max Inden 59f5eb4c17 *: Enable authority discovery by default (#1395)
* *: Enable authority discovery by default

Instead of having to explicitly enable the authority discovery module on
validator and sentry nodes, this commit enables the module by default.

Today there is no way for non validator or sentry nodes to run the
module. That might change in the future.

* service/src/lib: Fix typo in new_full! for test
2020-07-13 18:19:48 -04:00
Max IndenandPierre Krieger faa72caf91 *: Update to libp2p v0.21.1 (#6559)
* *Cargo.toml: Update versions

* client/network/src/discovery: Adjust to Kademlia  API changes

* client/network: Adjust to one_shot.rs changes

* client/network/discovery: Log address list on trace level

* client/network/discovery: Ignore RoutablePeer and PendingRoutablePeer

* Commit Cargo.lock

* Finish update

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-07-08 14:00:30 +00:00
Max Inden 62f306d972 client/network: Rename DebugInfoBehaviour to PeerInfoBehaviour (#6556)
Information retrieved via `DebugInfoBehaviour` is not only used for
debugging purposes, e.g. disconnecting from nodes not responding to
pings, using external addresses retrieved via indentify, ...

In order for the name to reflect the usage of the module this commit
renames it.
2020-07-08 09:08:47 +00:00
Max Inden 635310c8d4 client/network: Remove unused Result returned by NetworkWorker (#6552) 2020-07-02 15:18:08 +02:00
Max Inden 585ea531a3 .maintain/monitoring/alerting-rules: Adjust transaction queue size alert (#6426)
The transaction queue size alert has been firing with a constant 10
transactions in the queue. While maybe problematic those 10 transactions
don't need to be the same across scrape intervals.

Instead of alerting with a size above 10, alert based on two things:

1. Monotonically increasing queue size

2. Upper limit queue size reached
2020-07-01 10:31:56 +02:00
Max Inden 2cbefce35b .maintain/sentry-node: Remove UI and update Prometheus target (#6473)
Remove burden on user to build polkadot-js apps Docker image locally in
order to get started.

Update Prometheus config fixing target name.
2020-06-25 11:25:28 +02:00
Max Inden bd79b3debc client/network/service: Add primary dimension to connection metrics (#6472)
* client/network/service: Add primary dimension to connection metrics

Two nodes can be interconnected via one or more connections. The first
of those connections is called the primary connection.

This commit adds another dimension to the
`sub_libp2p_connections_{closed,opened}_total` metrics to differentiate
primary and non-primary connections being opened / closed.

By intuition more than one connection between two nodes is rare.
Tracking the fact whether a connection is primary or not will help prove
or disprove this intuition.

* .maintain/monitoring: Ensure to sum over all connections_closed variants

* client/network/service: Rename is_primary to is_first

* client/network/service: Split by metric name with two additional metrics

* Revert ".maintain/monitoring: Ensure to sum over all connections_closed variants"

This reverts commit 2d2f93e414440b9fc9e8f7fae6fe48bd95af6b8f.

* client/network/service: Remove labels from distinct metrics
2020-06-23 17:25:42 +02:00
Max Inden cbfbad6fbb client/authority-discovery: Compare PeerIds and not Multihashes (#6414)
In order to tell whether an address is the local nodes address the
authority discovery module previously compared the Multihash within the
`p2p` Multiaddr protocol.

rust-libp2p recently switched to a new PeerId representation (see [1]).
Multihashes of the same PeerId in the new and the old format don't
equal.

Instead of comparing the Multihashes, this patch ensures the module
compares the PeerIds

[1] https://github.com/libp2p/rust-libp2p/issues/555
2020-06-19 16:01:16 +02:00
Max Inden 737ba2ea20 .gitlab-ci.yml: Use promtool from paritytech/tools:latest image (#6425) 2020-06-19 16:00:33 +02:00
Max Inden fe76ebd548 .maintain/monitoring: Add alerting rule tests (#6343)
* .maintain/monitoring: Add alerting rule tests

* .maintain/monitoring/alerting-rules/alerting-rules.yaml: Break lines

* .gitlab-ci.yml: Add promtool rule testing step
2020-06-19 08:31:42 +02:00
Max Inden b4221cc135 .gitlab-ci.yml: Run promtool on Prometheus alerting rules (#6344)
* .gitlab-ci.yml: Run promtool on Prometheus alerting rules

Add a CI stage to test the Prometheus alerting rules within
`.maintain/monitoring`.

* .gitlab-ci.yml: Switch Prometheus stage to paritytech/tools image

* .gitlab-ci.yml: Follow http redirects in Prometheus stage

* .gitlab-ci.yml: Fix Prometheus stage promtool folder name
2020-06-16 22:49:01 +02:00
Max Inden fb63a7c5bc client/authority-discovery: Don't add own address to priority group (#6370)
* client/authority-discovery: Don't add own address to priority group

In the scenario of a validator publishing the address of its sentry node
to the DHT, said sentry node should not add its own Multiaddr to the
peerset "authority" priority group.

Related to 273f31b.

* client/authority-discovery: Remove unused import PeerId

* client/authority-discovery/tests: Add tcp protocol to multiaddresses
2020-06-16 18:51:21 +02:00
Max Inden aa95c596e6 .maintain/monitoring: Add an initial set of Prometheus alerting rules (#6095)
Create a place to collaborate on Prometheus alerting rules for
Substrate starting with a basic set of rules covering:

- Resource usage
- Block production
- Block finalization
- Transaction queue
- Networking
- ... Others
2020-05-21 16:26:29 +02:00
Max Inden 1ede6d1e36 client/network: Remove default Kademlia DHT in favor of per protocol DHT (#5993)
Substrate was previously running both a chain specific Kademlia DHT as
well as the `/ipfs/kad/1.0.0` Kademlia DHT for backwards compatibility.
(See 4db45a8 for details.)

With the end of the transition period this commit removes the former
only leaving the latter and thus preventing the discovery of
incompatible nodes via Kademlia.
2020-05-12 19:07:36 +02:00
Max Inden 1ce85590ff client/finality-granpda/until_imported: Rework pinning (#5983)
An `UntilImported` stream wraps a `Stream` of incoming messages and
waits for blocks those messages are based on before passing the messages
on.

The above `Stream` of incoming messages implements `Unpin`, thus there
is no need to use `pin_project` on the `UntilImported` struct. Instead
one only has to add the `Unpin` trait bound on the `I` trait parameter.
2020-05-12 11:48:28 +01:00
Max Inden 4084820f0e cli/src/cli.rs: Document --enable-authority-discovery flag (#1085) 2020-05-11 14:14:07 +02:00
Max Inden 117fffb0cb network/src/legacy/gossip: Return bounded channel when testing (#1046)
* network/src/legacy/gossip: Return bounded channel when testing

* Cargo.lock: Update to current Substrate master
2020-05-06 16:00:58 +02:00
Max Inden 327d11025e client/network-gossip/bridge: Use bounded channel (#5748)
* client/network-gossip/bridge: Use bounded channel

Instead of returning an unbounded channel on
`GossipEngine::messages_for` return a bounded channel. For now the
channel length is determined by the amount of past messages cached in
the `ConsensusGossip`.

With a bounded channel, one can't just fire-and-forget style send into
it, but has to first check whether the channel is ready. Thus this
commit restructures `GossipEngine::poll` and introduces a
`ForwardingState` into `GossipEngine`.

* client/network-gossip/bridge: Add quickcheck for different size channels
2020-05-06 14:56:46 +02:00
Max Inden d40bf3cf36 client/authority-discovery: Do not double encode signature (#5901)
Previously, when publishing ones address onto the DHT, the signature
signing those addresses would be SCALE encoded twice.

This commit removes the second encoding and adjusts the tests
to catch future regressions.
2020-05-06 10:52:44 +02:00
Max Inden 324e51fbbf client/authority-discovery: Reduce log level replaced by metrics
Instead of logging value-found-event-handling failures or value-put
failures on error level, log them on debug level only additionally
recording them via Prometheus.

Motivation is that both events can happen in "normal" operations and
thus clutter the logs.
2020-04-24 17:40:48 +02:00
Max Inden e84b06a8c0 client/network-gossip/state_machine: Reduce indentation level (#5714)
* client/network-gossip/src/state_machine: Add unit test for on_incoming

Add two unit tests to ensure `on_incoming` is ingoring discarded
messages and reports and ignores messages from unknown peers.

* client/network-gossip/state_machine: Reduce indentation level

* client/network-gossip/bridge: Remove unnecessary into_iter

* client/network-gossip/state_machine: Report success after register check

* client/network-gossip/state_machine: Error not report on unregistered

`peers` contains all the peers we're connected to. If we receive a
message from a peer not in this list, that means there's an internal
problem within the local client. It's not the fault of the peer in
question.

With the above in mind instead of reducing the reputation of such peer,
log an error.
2020-04-23 09:58:43 +02:00
Max Inden 798de8337b client/authority-discovery: Add metric exposing priority group size (#5707)
Expose the amount of addresses that are passed from the authority
discovery module down to the network peer set manager as a priority
group in order to connect to the corresponding nodes.

```
authority_discovery_priority_group_size 10
```
2020-04-21 10:54:37 +02:00
Max IndenandBenjamin Kampmann 5db336c743 client/network-gossip: Move sink IO outside of state_machine (#5669)
* client/network-gossip: Move sink IO outside of state_machine

`ConsensusGossip` is supposed to be a deterministic state machine.
`GossipEngine` wrapping `ConsensusGossip` should handle IO operations.

This commit moves the `message_sink` IO operations to `GossipEngine`.
More specifically on incoming messages a `GossipEngine` calls
`ConsensusGossip::on_incoming` to validate and register the messages.
`ConsensusGossip` returns the valid messages which are then forwarded by
`GossipEngine` to the upper layer via the `message_sinks`.

* client/network-gossip: Adjust and extend tests

* Update client/network-gossip/src/bridge.rs

Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
2020-04-20 16:40:44 -04:00
Max Inden 70becefb3d service/src/lib: Enable authority discovery on sentry nodes (#984)
* service/src/lib: Enable authority discovery on sentry nodes

When run as a sentry node, the authority discovery module does not
publish any addresses to the dht, but still discovers validators and
sentry nodes of validators.

* bin/node/cli/src/service: Use expressions instead of statements

* Cargo.lock: Run `cargo update`

* service/src/lib: Fix compile error
2020-04-16 20:30:22 +01:00
Max IndenandAndré Silva d742e88e79 client/authority-discovery: Allow to be run by sentry node (#5568)
* client/authority-discovery: Allow to be run by sentry node

When run as a sentry node, the authority discovery module does not
publish any addresses to the dht, but still discovers validators and
sentry nodes of validators.

* client/authority-discovery/src/lib: Wrap lines at 100 characters

* client/authority-discovery: Remove TODO and unused import

* client/authority-discovery: Pass role to new unit tests

* client/authority-discovery: Apply suggestions

Co-Authored-By: André Silva <123550+andresilva@users.noreply.github.com>

* bin/node/cli/src/service: Use expressions instead of statements

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-04-16 18:11:26 +01:00
6bd93825ca client/authority-discovery: Rework error handling (#5631)
* client/authority-discovery: Rework error handling

Instead of `handle_dht_events` returning a `Result<(), Error>`, return
a `Poll<Error>` where `Poll::Pending` signals that there are no more
events to handle and `Poll::Ready(Error)` signals that a fatal error
occured. Non fatal errors are handled within `handle_dht_events`
directly, thus looping in `poll` is not necessary anymore.

* client/authority-discovery: Return () instead of error on termiantion

* Update client/authority-discovery/src/lib.rs

Co-Authored-By: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-04-16 16:09:13 +01:00
Max Inden 1c7525fd4b client/authority-discovery: Increase request counter per request (#5632) 2020-04-16 10:42:08 +02:00
61d64e2ca1 .maintain/sentry-node: Add monitoring to docker-compose stack (#5321)
* Substrate Dashboard example

* Improve README

* Update README_dashboard.md

* Add screenshots

* Minor fix

* Minor fix, image link

* .maintain/sentry-node: Add monitoring to docker-compose stack

With this patch a user can run the following fully configured and
monitored setup with a single command:

`docker-compose -f .maintain/sentry-node/docker-compose.yml up`

- 2 validators in two different network namespaces, connected via one
sentry node.

- Polkadot-js/apps to connect to one of the nodes above.

- Prometheus scraping the 3 Substrate nodes.

- Grafana displaying data from Prometheus with community dashboards

* .maintain/monitoring/grafana: Change default datasource name

* .maintain/monitoring/grafana: Add metric namespace option

* .maintain/monitoring/grafana: Remove `host` metric from most metrics

* .maintain/monitoring/grafana: Remove underscore from metric_namespace

* .maintain/monitoring: Use `instance` label instead of `hostname`

To identify a scrape target, one should use `instance` and not
`hostname` as multiple targets might run on the same node.

See https://prometheus.io/docs/concepts/jobs_instances/ for details.

* .maintain/monitoring: Introduce instance variable

* .maintain/monitoring/grafana: Rename substrate_block_height_number

* .maitain/monitoring/grafana: Use instance instead of host in legend

* .maintain/monitoring: Remove node exporter dependency

* .maintain/sentry-node/prometheus: Simplify configuration

* .maintain/monitoring/grafana: Update README and remove images

* .maintain/sentry-node: Improve docs

* .maintain/monitoring/grafana: Use metric_namespace template variable

* Use --sentry from v0.7.29 instead of a reserved-node

* .maintain/sentry-node: Revert sentry-a using validator-b as bootnode

Co-authored-by: DerFredy - @derfredy:matrix.org <derfredy@gmail.com>
Co-authored-by: david <davidd@custom.home>
2020-04-14 16:08:09 +02:00
Max IndenandAndré Silva 5913969f8d client/authority-discovery: Terminate when network does (#5562)
* client/authority-discovery: Add test ensure termination on network termi

* client/authority-discovery: Terminate when network does

When the dht event stream returns Poll::Ready(None) it is likely due to
the network terminating. When the network terminates due to the node
itself shutting down or due to a fatal error, there is no purpose in
continuing to run the authority discovery module.

* client/authority-discovery/src/lib: Apply suggestions

Co-Authored-By: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-04-09 10:34:51 +02:00
Max IndenandAshley 091335780e client/finality-grandpa: Instrument until-imported queue (#5438)
* client/finality-grandpa: Instrument until-imported queue

The `UntilImported` queue takes as input finality grandpa messages that
depend on blocks that are not yet imported and holds them back until
those blocks are imported.

This commit adds a basic metric, the amount of messages waiting in the
queue, to the module. For now this metric is only available for the
global `UntilImported` queue awaiting blocks for commit and catch-up
messages.

* client/finality-grandpa/src/until_imported: Update metric help text

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-03-31 20:17:14 +02:00
Max Inden c3c8552aaa src/service/src/builder: Fix memory metric exposed in bytes not KiB (#5459)
The library `sysinfo` exposes process memory as kibibytes and not bytes,
thus the value needs to be multiplied by 1024 to comply with the metric
name and the Prometheus base units [1].

[1] https://prometheus.io/docs/practices/naming/#base-units
2020-03-30 15:48:59 +02:00
Max IndenandAndré Silva 2a67e6c437 client/finality-grandpa/src/until_imported: Refactor schedule_wait (#5386)
* client/finality-grandpa/src/until_imported: Refactor schedule_wait

Previously `BlockUntilImported::schedule_wait` took two closures, one to
report ready items and one to report items to await. None of the implementors of
`BlockUntilImported` call both closures. From a symantic perspective it
would as well not make sense to both await something and state that it
is ready.

Instead with this commit `BlockUntilImported::schedule_wait` simply
returns whether the given item needs waiting or is ready to be passed
on.

This reduces complexity by:

- Removing side effects through the two closures.

- Reducing borrowing given that `UntilImported` `ready` and `pending`
don't need to be borrowed from within the two closures.

- Removes the need for trait bounds for the two closures.

* client/finality-grandpa/src/until_imported: Fix comments

Co-Authored-By: André Silva <andre.beat@gmail.com>

Co-authored-by: André Silva <andre.beat@gmail.com>
2020-03-26 13:49:45 +01:00
Max IndenandAndré Silva 77446d3b50 client/finality-grandpa/src/until_imported: Refactor BlockGlobalMessage (#5390)
* client/finality-grandpa/src/until_imported: Refactor BlockGlobalMessage

`BlockGlobalMessage` owns an `inner` which contains (1) a count for the
amount of outstanding blocks to be waited on and (2) the message itself.

Given that both is already wrapped in an `Arc` there is no need to keep
track of the outstanding blocks, given that it simply corresponds to the
amount of strong reference counts on the `Arc` itself.

This commit removes the atomic counter within `inner` and piggy backs on
the `Arc` reference counter instead.

* client/finality-grandpa/src/until_imported: Remove useless match

* client/finality-grandpa/src/until_imported.rs: Remove unused var

Co-Authored-By: André Silva <andre.beat@gmail.com>

* client/finality-grandpa/src/until_imported: Address comment suggestion

Co-authored-by: André Silva <andre.beat@gmail.com>
2020-03-25 19:45:04 +01:00
Max Inden 26a204da21 client/network-gossip/state_machine: Fix log message (#5350)
A gossip validator returns `ValidationResult::Discard` for messages that
should not be processed nor kept due to not being useful for the node.
Examples are out-of-scope messages or messages with bad signatures. The
corresponding log message did not reflect this, thus the update through
this patch.

One caveat is a finality grandpa neighbor packet. Given that the concept
of neighbor packets is handled in the finality grandpa gossip validator
only, valid neighbor packets are always discarded on the
`client/network-gossip` layer.
2020-03-23 11:04:48 +01:00
Max Inden 9201f8abf3 client/network-gossip/src/bridge: Finish when network event stream closes (#5282)
* client/network-gossip/src/bridge: Finish when network even stream closes

Previously within `<GossipEngine as Future>::poll` one would poll the
`network_event_stream` ignoring all messages other than
`Poll::Ready(Some())`. Ignoring `Poll::Ready(None)` leads to a panic on
the next poll of the stream, gien that it is not fused.

By design `network_event_stream` does not close unless an unbounded send
into it fails, or the `NetworkWorker` gets shut down.

> The stream never ends (unless the `NetworkWorker` gets shut down).
> (client/network/src/service.rs)

An `unbounded_send` to fail on an unbounded channel is unlikely. The
`NetworkWorker` shutting down is not unlikely. In such case the
`GossipEngine` should shut down as well.

With this patch a `<GossipEngine as Future>` finishes on
`Poll::Ready(None)` returned from `network_event_stream`.

* client/finality-grandpa/communication: Error on gossip engine finished

Have `<NetworkBridge as Future>::poll` return `Poll::Ready(Err)` instead
of `Poll::Ready(Ok)` to be consistent with the handling of the neighbor
packet worker stream and the gossip validator report stream. Both `Err`
as well as `Ok` shut down the `NetworkBridge` as well as the
`VoterWorker`.

* client/network-gossip/src/bridge: Add regression test

* client/network-gossip: Move substrate test client to dev dependencies

* client/network-gossip: Remove TODO

Addressed in a follow up pull request.

* client/network-gossip/bridge: Put match on newline after loop

* client/finality-grandpa/src/observer: Fix regression test

Make sure the event stream sender side is not dropped till the end.
2020-03-18 20:15:57 +01:00
Max Inden 733d486814 Revert "Substrate Dashboard example (#5284)" (#5293)
This reverts commit 082b66434e.
2020-03-18 11:13:58 +01:00
Max Inden e91d4be998 client/network/src/protocol: Refactor Prometheus metric logic (#5266)
* client/network/src/protocol: Start Prometheus metric help with capital

* client/network/src/protocol: Differentiate metric status as label

Prometheus query language is powerful through its multi-dimensional data
model. Metric names are hirarchical whereas labels enable data to become
multi-dimensional.

Exposing the justification of finality-proof status as a label allows
for more powerful queries.

* client/network/src/protocol: Remove 'Total' from non counter metric help

The word 'total' is reserved for accumulating counters. Counters have to
be monotonically increasing. `obsolete_requests` can decrease, thereby
it is defined as a `Gauge` and not a `Counter`.

For more details on metric naming see
https://prometheus.io/docs/practices/naming/
2020-03-17 11:47:55 +01:00
Max Inden 79fc16e4d7 client/finality-grandpa: Add Prometheus metrics to GossipValidator (#5237)
* client/finality-grandpa: Add Prometheus metrics to GossipValidator

Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.

```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```

* client/finality-grandpa: Add None as Prometheus registry in tests

* client/finality-granpda/src/communication: Refactor metric registration
2020-03-14 12:51:44 +01:00
Max IndenandAshley d2345e8d5c client/authority-discovery: Instrument code with Prometheus (#5195)
* client/authority-discovery: Instrument code with Prometheus

Introduce Prometheus metrics into the authority discovery module
enabling one to observe:

- authority_discovery_times_published_total

- authority_discovery_amount_external_addresses_last_published

- authority_discovery_times_requested_total

- authority_discovery_dht_event_received

* client/authority-discovery/src/lib.rs: Reword metric help texts

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/src/lib.rs: Reword metric help text

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/tests: Fix struct instantiation + basic test

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-03-11 23:59:33 +01:00
Max Inden c9c7d655f1 .maintain/sentry-node/docker-compose: Expose Prometheus endpoin… (#5196)
This patch adds the Prometheus endpoint port of validator A to the
exposed ports to access it from the host network namespace.
2020-03-11 18:32:47 +01:00
Max Inden 0cc3b96076 client/service/src/builder.rs: Add build_info metric (#5192)
* client/service/src/builder.rs: Add build_info metric

Add static Prometheus metric exposing the chain name, the version and
the commit.

* client/service/src/builder.rs: Move node_role to static metrics

The Prometheus metrics `node_role` is static and thus there is no need
to keep a reference of it within `ServiceMetrics`. This follows the
example of the `build_info` metric.

* client/service/src/builder.rs: Adjust indentation
2020-03-10 11:12:32 +01:00
Max Inden e98826977d network/src/legacy: Remove unnecessary super trait on GossipService (#891)
Remove `Send + Sync + 'static` super trait bound on `GossipService`. It
is not required by `GossipService` nor its `impl`s.
2020-03-09 16:35:22 -07:00