* Clean up sc-network
- Avoid using clone() for the Copy type `PeerId`.
- Use `find_map` for `filter_map` and `next`.
- Use `Self`.
* More on Copy types
* Cargo +nightly fmt --all
* More ..
* fmt
* Revert vec![default_notif_handshake_message]
* Comment out browser stuff
* Remove browser stuff
* Remove more wasm transport code
* Remove ExtTransport and rework how telemetry initialises.
* Change (most) wasm-timer using code to use std::time
* Rename CI-job
* Aura does not compile for wasm
* Remove testing in the browser on CI
* Update README
* Leave `StreamSink` be
* fmt
Remove unneeded dependencies and dev-dependencies.
Made self_destruct test not dependent on wasm bin size.
Updated code related to deprecated warning on tracing-subscriber `scope()`
( See https://github.com/tokio-rs/tracing/issues/1429 )
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* network: allow gossiping to light clients
* grandpa: gossip global messages to light clients
* grandpa: don't send neighbor packets to light clients
* grandpa: fix tests
* grandpa: export run_grandpa_observer
* node: run grandpa observer on light client
* node: start network at end
* Use wasm_timer in finality-grandpa
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Allow fallback names for protocols
* Apply suggestions from code review
Co-authored-by: Roman Proskuryakov <humbug@deeptown.org>
* Fix some issues
* Fix compilation after merging master
Co-authored-by: Roman Proskuryakov <humbug@deeptown.org>
* Improve logging in network gossip
This adds some more information to the logging output to get a better
understanding when something fails.
* Update client/network-gossip/src/state_machine.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Sync: Propagate block announcement data
This pr adds a feature to the sync protocol to propagate the data that
we received alongside a block announcement. This is done by adding a
cache that caches the last X block announcement data where X is set to
the number of `in_peers` (giving every peer the chance to send us a
different block). This will be required by parachains to ensure that
even peers who are not connected to a collator receive the data
alongside the block announcement to properly validate it and request the
block.
* Review comment
* Bring back the code and add new variant to ensure we don't insert block
announce data when something wasn't checked
* Also use out_peers
* network-gossip: add metric for number of local messages
* grandpa: fix GossipEngine missing metrics registry parameter
* network-gossip: increase known messages cache size
* network-gossip: fix tests
* grandpa: remove unnecessary clone
Co-authored-by: Max Inden <mail@max-inden.de>
* network-gossip: count registered and expired messages separately
* network-gossip: add comment on known messages cache size
* network-gossip: extend comment with cache size in memory
Co-authored-by: Max Inden <mail@max-inden.de>
* Rework priority groups
* Broken tests fix
* Fix warning causing CI to fail
* [Hack] Try restore backwards-compatibility
* Fix peerset bug
* Doc fixes and clean up
* Error on state mismatch
* Try debug CI
* CI debugging
* [CI debug] Can I please see this line
* Revert "[CI debug] Can I please see this line"
This reverts commit 4b7cf7c1511f579cd818b21d46bd11642dfac5cb.
* Revert "CI debugging"
This reverts commit 9011f1f564b860386dc7dd6ffa9fc34ea7107623.
* Fix error! which isn't actually an error
* Fix Ok() returned when actually Err()
* Tweaks and fixes
* Fix build
* Peerset bugfix
* [Debug] Try outbound GrandPa slots
* Another bugfix
* Revert "[Debug] Try outbound GrandPa slots"
This reverts commit d175b9208c088faad77d9f0ce36ff6f48bd92dd3.
* [Debug] Try outbound GrandPa slots
* Apply suggestions from code review
Co-authored-by: Max Inden <mail@max-inden.de>
* Use consts for hardcoded peersets
* Revert "Try debug CI"
This reverts commit 62c4ad5e79c03d561c714a008022ecac463a597e.
* Renames
* Line widths
* Add doc
Co-authored-by: Max Inden <mail@max-inden.de>
* Remove sc_network::NetworkService::register_notifications_protocol
* Missing calls to .into()
* Wrong crate name
* [WIP] Fix Grandpa tests
* One more passing
* One more. Two to go.
* This one was actually already passing 🎉
* Last one compiles
* Progress
* grandpa: fix voter_persists_its_votes test
* Restore other tests
* Try spawn future later
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Remove necessity to pass ConsensusEngineId when registering notifications protocol
* Line width
* Fix tests protocol name
* Other renames
* Doc update
* Change issue in TODO
* client/*: Treat protocol name as str and not [u8]
Notification protocol names are in practice always valid utf8 strings.
Instead of treating them as such in the type system, thus far they were
casted to a [u8] at creation time.
With this commit protocol names are instead treated as valid utf8
strings throughout the codebase and passed as `Cow<'static, str>`
instead of `Cow<'static, [u8]>`. Among other things this eliminates the
need for string casting when logging.
* client/network: Don't allocate when protocol name is borrowed
* client/network-gossip/bridge: Use bounded channel
Instead of returning an unbounded channel on
`GossipEngine::messages_for` return a bounded channel. For now the
channel length is determined by the amount of past messages cached in
the `ConsensusGossip`.
With a bounded channel, one can't just fire-and-forget style send into
it, but has to first check whether the channel is ready. Thus this
commit restructures `GossipEngine::poll` and introduces a
`ForwardingState` into `GossipEngine`.
* client/network-gossip/bridge: Add quickcheck for different size channels
* client/network-gossip/src/state_machine: Add unit test for on_incoming
Add two unit tests to ensure `on_incoming` is ingoring discarded
messages and reports and ignores messages from unknown peers.
* client/network-gossip/state_machine: Reduce indentation level
* client/network-gossip/bridge: Remove unnecessary into_iter
* client/network-gossip/state_machine: Report success after register check
* client/network-gossip/state_machine: Error not report on unregistered
`peers` contains all the peers we're connected to. If we receive a
message from a peer not in this list, that means there's an internal
problem within the local client. It's not the fault of the peer in
question.
With the above in mind instead of reducing the reputation of such peer,
log an error.
* client/network-gossip: Move sink IO outside of state_machine
`ConsensusGossip` is supposed to be a deterministic state machine.
`GossipEngine` wrapping `ConsensusGossip` should handle IO operations.
This commit moves the `message_sink` IO operations to `GossipEngine`.
More specifically on incoming messages a `GossipEngine` calls
`ConsensusGossip::on_incoming` to validate and register the messages.
`ConsensusGossip` returns the valid messages which are then forwarded by
`GossipEngine` to the upper layer via the `message_sinks`.
* client/network-gossip: Adjust and extend tests
* Update client/network-gossip/src/bridge.rs
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:
- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)
refs #4679
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* Split the Roles bitfield in three
* Forgot to include some changes
* Fix cli test
* More test fixes
* Oh God, merging master broke other tests
* Didn't run the doctests
* Address review
* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine
* Address some review
* Also update the peerset's API to make sense
* Fix peerset tests
* Fix browser node
* client: distinguish between local and network authority
Co-authored-by: André Silva <andre.beat@gmail.com>
A gossip validator returns `ValidationResult::Discard` for messages that
should not be processed nor kept due to not being useful for the node.
Examples are out-of-scope messages or messages with bad signatures. The
corresponding log message did not reflect this, thus the update through
this patch.
One caveat is a finality grandpa neighbor packet. Given that the concept
of neighbor packets is handled in the finality grandpa gossip validator
only, valid neighbor packets are always discarded on the
`client/network-gossip` layer.
* client/network-gossip/src/bridge: Finish when network even stream closes
Previously within `<GossipEngine as Future>::poll` one would poll the
`network_event_stream` ignoring all messages other than
`Poll::Ready(Some())`. Ignoring `Poll::Ready(None)` leads to a panic on
the next poll of the stream, gien that it is not fused.
By design `network_event_stream` does not close unless an unbounded send
into it fails, or the `NetworkWorker` gets shut down.
> The stream never ends (unless the `NetworkWorker` gets shut down).
> (client/network/src/service.rs)
An `unbounded_send` to fail on an unbounded channel is unlikely. The
`NetworkWorker` shutting down is not unlikely. In such case the
`GossipEngine` should shut down as well.
With this patch a `<GossipEngine as Future>` finishes on
`Poll::Ready(None)` returned from `network_event_stream`.
* client/finality-grandpa/communication: Error on gossip engine finished
Have `<NetworkBridge as Future>::poll` return `Poll::Ready(Err)` instead
of `Poll::Ready(Ok)` to be consistent with the handling of the neighbor
packet worker stream and the gossip validator report stream. Both `Err`
as well as `Ok` shut down the `NetworkBridge` as well as the
`VoterWorker`.
* client/network-gossip/src/bridge: Add regression test
* client/network-gossip: Move substrate test client to dev dependencies
* client/network-gossip: Remove TODO
Addressed in a follow up pull request.
* client/network-gossip/bridge: Put match on newline after loop
* client/finality-grandpa/src/observer: Fix regression test
Make sure the event stream sender side is not dropped till the end.
* client/network-gossip: Merge GossipEngine and GossipEngineInner
Given that GossipEngine and GossipEngineInner are not shared between
threads anyone (public interface + background tasks), neither depends on
being Send or Sync. Thus one can merge the two as done in this patch.
One only needs to wrap an `Arc<Mutex<>>` around the whole structure when
the owner (e.g. finality-grandpa) needs to share the gossip engine
between threads.
* client/finality-grandpa: Wrap GossipEngine in Arc Mutex & lock it on use
GossipEngine in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore. Given that
finality-grandpa shares the `NetworkBridge` potentially between threads
its components need to be clonable, thus this patch wraps `GossipEngine`
in an `Arc<Mutex<>>`.
* Open one substream for each notifications protocol
* Fix WASM build
* Apply suggestions from code review
Co-Authored-By: Toralf Wittner <tw@dtex.org>
* Address concerns
* Use unsigned-varint to read the varint
* Use unsigned-varint
* Forgot Cargo.lock
Co-authored-by: Toralf Wittner <tw@dtex.org>
`GossipEngine` spawns two tasks, one for a periodic tick, one to forward
messages from the network to subscribers. These tasks hold an `Arc` to a
`GossipEngineInner`.
To reduce the amount of shared ownership (locking) this patch integrates
the two tasks into a `Future` implementation on the `GossipEngine`
struct. This `Future` implementation can now be called from a single
owner, e.g. the `finality-grandpa` `NetworkBridge`.
As a side effect this removes the requirement on the `network-gossip`
crate to spawn tasks and thereby removes the requirement on the
`finality-grandpa` crate to spawn any tasks.
This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa`, `network` and `network-gossip` as
well as to reduce the amount of unbounded channels. For details see
d9837d7dd, 5f80929dc and 597c0a6c4.
* Make WASM browser thing compile
* Fix
* updated exit-future (github repo)
* Switch to broadcast crate
* Migrate client/cli
* Switch exit-future to modernize branch
* Small changes
* Switch to cargo version and fix fg tests
* fix basic-authorship
* Fix crash on grafana macro
* Fix grafana macro
* Switch node python version
* Disable record_metrics_slice in grafana macro on wasm
* Update client/grafana-data-source/src/lib.rs
* Revert "Update client/grafana-data-source/src/lib.rs"
This reverts commit 888009a8e0b7051bd4bfbbfdb0448bcf2e2aae93.
* Add wasm support for state machine
* Switch to my own libp2p version
* Revert "Switch to my own libp2p version"
This reverts commit ce613871b59264b3165b45c37943e6560240daa7.
* Revert "Add wasm support for state machine"
This reverts commit de7eaa0694d9534fc3b164621737968e9a6a7c5f.
* Add sc-browser
* Squash
* remove sc-browser
* Fix keystore on wasm
* stubs for removed functions to make env compatible with old runtimes
* Add test (that doesn't work)
* Fix build scripts
* Revert basic-authorship due to no panics
* Revert cli/informant
* Revert consensus
* revert offchain
* Update utils/browser/Cargo.toml
Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>
* export console functions
* Add new chainspec
* Fix ws in chain spec
* revert chainspec
* Fix chainspec
* Use an Option<PathBuf> in keystore instead of cfg flags
* Remove crud
* Only use wasm-timer for instant and systemtime
* Remove telemetry changes
* Assuming this is ok
* Add a KeystoreConfig
* Add stubs back in
* Update libp2p
* Revert "Add stubs back in"
This reverts commit 4690cf1882aa0f99f7f00a58c4080c8aa9b77c36.
* Remove commented js again
* Bump kvdb-web version
* Fix cli
* Switch branch on futures-timer
* Fix tests
* Remove sc-client test build in check-web-wasm because there isn't a good way to build futures-timer with wasm-bindgen support in the build
* Remove more things ^^
* Switch branch on futures-timer back
* Put DB io stats behind a cfg flag
* Fix things
* Don't timeout transports on wasm
* Update branch of futures-timer and fix bad merge
* Spawn informant
* Fix network test
* Fix delay resets
* Changes
* Fix tests
* use wasm_timer for transaction pool
* Fixes
* Switch futures-timer to crates
* Only diagnose futures on native
* Fix sc-network-test tests
* Select log level in js
* Fix syncing ;^)
* Allow disabling colours in the informant
* Use OutputFormat enum for informant
* MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master
* Update futures-diagnose
* Revert "MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master"
This reverts commit baa4ffc94fd968b6660a2c17ba8113e06af15548.
* Pass whole chain spec in start_client
* Get Instant::now to work in transaction pool again
* Informant dep reordering
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
Falco Hirschenberger