* Sync: Propagate block announcement data
This pr adds a feature to the sync protocol to propagate the data that
we received alongside a block announcement. This is done by adding a
cache that caches the last X block announcement data where X is set to
the number of `in_peers` (giving every peer the chance to send us a
different block). This will be required by parachains to ensure that
even peers who are not connected to a collator receive the data
alongside the block announcement to properly validate it and request the
block.
* Review comment
* Bring back the code and add new variant to ensure we don't insert block
announce data when something wasn't checked
* Also use out_peers
* Rework priority groups
* Broken tests fix
* Fix warning causing CI to fail
* [Hack] Try restore backwards-compatibility
* Fix peerset bug
* Doc fixes and clean up
* Error on state mismatch
* Try debug CI
* CI debugging
* [CI debug] Can I please see this line
* Revert "[CI debug] Can I please see this line"
This reverts commit 4b7cf7c1511f579cd818b21d46bd11642dfac5cb.
* Revert "CI debugging"
This reverts commit 9011f1f564b860386dc7dd6ffa9fc34ea7107623.
* Fix error! which isn't actually an error
* Fix Ok() returned when actually Err()
* Tweaks and fixes
* Fix build
* Peerset bugfix
* [Debug] Try outbound GrandPa slots
* Another bugfix
* Revert "[Debug] Try outbound GrandPa slots"
This reverts commit d175b9208c088faad77d9f0ce36ff6f48bd92dd3.
* [Debug] Try outbound GrandPa slots
* Apply suggestions from code review
Co-authored-by: Max Inden <mail@max-inden.de>
* Use consts for hardcoded peersets
* Revert "Try debug CI"
This reverts commit 62c4ad5e79c03d561c714a008022ecac463a597e.
* Renames
* Line widths
* Add doc
Co-authored-by: Max Inden <mail@max-inden.de>
* *: Update to libp2p v0.33.0
* client/network: Consistently track request arrival time
With https://github.com/libp2p/rust-libp2p/pull/1886/ one is guaranteed
to receive either a `ResponseSent` or a `InboundFailure` event for each
received inbound request via `RequestResponseEvent::Message`. Given this
guarantee there is no need to track arrival times in a best-effort
manner and thus there is no need to use a LRU cache for arrival times.
* client/offchain: Adjust to PeerId API changes
* Update common block in sync after importing blocks of a peer
This updates the sync code to update the common block of a peer, after
we have imported blocks from this peer. This fixes a bug for when we are
connected to one or more nodes that are doing a full sync as our node.
Nodes in full sync will not announce new blocks, as we don't send import
notifications on full sync. The problem as now that we were connected to
some peer that reported some low number as its best and we tried to sync
these blocks. But, as we did not update the common block of this peer,
we would sync these blocks over and over again. Being captured in some
time warp.
The solution to this problem is that we increase the common number as we
import blocks from this peer.
* Test
* Test name..
* Fix test
* Cleanup some code and write some new regression test
* Implement the ancestor search
* Check that the common number is smaller than the last finalized block
* Update client/network/src/protocol/sync.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update client/network/src/protocol/sync.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update client/network/src/protocol/sync.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Change the way we build the status messages
* Start some new test...
* Finish test
* Rename test
* Update client/network/src/protocol.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* *: Update to libp2p v0.32.0
* Cargo.lock: Update async-tls to 0.10.2
* client/network/request_response: Adjust to new request response events
* client/network/request_response.rs: Clean up silently failing responses
* client/network/discovery: Lazily instantiate mdns
* client/network/discovery: Exclude MdnsWrapper for target_os unknown
* client/network/discovery: Fix indentation
* client/network/request-response: Use LruCache to track pending resp time
* client/network/request_responses: Fix early connection closed error
* client/network/request-response: Replace debug_assert with debug
* client/network/request-response: Fix typo
* client/network/request-response: Don't emit event on send_response fail
* client/network/request-response: Revert waker.wake_by_ref()
* client/network/request-resp: Make duration in InboundRequest optional
* client/network/req-resp: Don't emit two events for busy builder
When a response builder is busy incoming requests are dropped.
Previously this was reported both via a `ResponseFailure::Busy` and a
`ReponseFailure::Network(InboundFailure::Omisssion)` event.
With this commit the former is removed, leaving only the latter in
place.
* Use block requests to check if block responses are correct
Before this pr sync relied on recently announced blocks to check if a
given peer response is correct. However this could lead to situations
where we requested a block from a peer and it gave us the requested, but
we rejected the response because this peer never send us an announcement
for the given block. See the added tests for a reproduction of the
problem.
With this pr, we now take the block request to check if a given response
matches the request. A node should not send us a block response
without a request anyway.
Essentially there is still a bug, because as you see in the test, we are
requesting block 2, while we already have this block imported. It even
happens that we request a block from the network that we have authored.
However a fix for this would require some more refactoring of the sync code.
* Revert change
* Give the test a proper name
* Add moar logging
* Move cheaper checks
* Move checks to common place
* Remove sc_network::NetworkService::register_notifications_protocol
* Missing calls to .into()
* Wrong crate name
* [WIP] Fix Grandpa tests
* One more passing
* One more. Two to go.
* This one was actually already passing 🎉
* Last one compiles
* Progress
* grandpa: fix voter_persists_its_votes test
* Restore other tests
* Try spawn future later
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Log outgoing notifications too
* Update client/network/src/protocol/generic_proto/handler.rs
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Max Inden <mail@max-inden.de>
* chore/error: remove from str conversion and add deprecation notifications
* fixup changes
* fix test looking for gone ::Msg variant
* another test fix
* one is duplicate, the other is not, so duplicates reported are n-1
* darn spaces
Co-authored-by: Andronik Ordian <write@reusable.software>
* remove pointless doc comments of error variants without any value
* low hanging fruits (for a tall person)
* moar error type variants
* avoid the storage modules for now
They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.
* chore remove pointless error generic
* fix test for mocks, add a bunch of non_exhaustive
* max line width
* test fixes due to error changes
* fin
* error outputs... again
* undo stderr adjustments
* Update client/consensus/slots/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove closure clutter
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* more error types
* introduce ApiError
* extract Mock error
* ApiError refactor
* even more error types
* the last for now
* chore unused deps
* another extraction
* reduce should panic, due to extended error messages
* error test happiness
* shift error lines by one
* doc tests
* white space
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Into -> From
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove pointless codec
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* avoid pointless self import
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@parity.io>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Inform sync explicitly about new best block
Instead of "fishing" the new best block out of the processed blocks, we
now tell sync directly that there is a new best block. It also makes
sure that we update the corresponding sync handshake to the new best
block. This is required for parachains as they first import blocks and
declare the new best block after being made aware of it by the relay chain.
* Adds test
* Make sure async stuff had time to run
* client/authority-discovery: Publish and query on exponential interval
When a node starts up publishing and querying might fail due to various
reasons, for example due to being not yet fully bootstrapped on the DHT.
Thus one should retry rather sooner than later. On the other hand, a
long running node is likely well connected and thus timely retries are
not needed. For this reasoning use an exponentially increasing interval
for `publish_interval`, `query_interval` and
`priority_group_set_interval` instead of a constant interval.
* client/authority-discovery/src/interval.rs: Add license header
* .maintain/gitlab: Ensure adder collator tests are run on CI
* Remove necessity to pass ConsensusEngineId when registering notifications protocol
* Line width
* Fix tests protocol name
* Other renames
* Doc update
* Change issue in TODO
* Use inbound peerslot slots when a substream is received, rather than a connection
* Refactor PeerState
* Some bugfixes
* Fix warnings so that CI runs, gmlrlblbl
* Bugfixes
* Update docs
* Apply suggestions from code review
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
* Clean up Banned state
* Refactor connections state
* Fix possibility of Enabled with no Opening or Open connection
* Line width
* Add some debug_asserts! and fix TODO
* Refactor legacy handler
* Rewrite group.rs entirely [part 1]
* Rewrite group.rs entirely [part 2]
* Remove faulty assertion
Because of the asynchronous nature of the behaviour <-> handler communications, it is possible to receive notifications while in the Closing state
* Don't poll the legacy substream is not Open
* Tolerate when not all substreams are accepted
* Remove TODOs
* Dummy commit to make CI log interesting things
* Try race condition fix
* Revert "Try race condition fix"
This reverts commit 0675c659d06195c30f8c5bc13e2d88141d57a3ba.
* Correctly rebuild pending_opening
* Minor tweaks
* Printlns for CI debugging
* Revert "Printlns for CI debugging"
This reverts commit e7852a231f4fc418898767aaa27c9a4358e12e8b.
* Revert "Dummy commit to make CI log interesting things"
This reverts commit 259ddd74088e53e7c6a9b0a62a8d1573a0063ce3.
* mv group.rs ../handler.rs
* Apply suggestions from code review
Co-authored-by: Max Inden <mail@max-inden.de>
* Banned => Backoff
* Mention the actual PeerStates
* OpenDesired -> OpenDesiredByRemote
* OpeningThenClosing
* Add doc links to PeerState
* Simplify increment logic
* One more debug_assert
* debug_assert!
* OpenDesiredByRemote
* Update client/network/src/protocol/generic_proto/behaviour.rs
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
* Update to libp2p-0.29.
* Update dependencies.
* Update Cargo.toml
Co-authored-by: Max Inden <mail@max-inden.de>
* Fix tests.
* Fix tests.
* Fix more tests.
* Update to 0.29.1
* Update ed25519-dalek dependency of sp-core.
* Update Cargo.lock.
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Print an error if an nregistered notifications protocol is used
* Print an error if an nregistered notifications protocol is used
* Update client/network/src/service.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
As done with `NetworkService::{add_to,remove_from}_priority_group`, make
`NetworkService::set_priority_group` async as well. This future-proofs
the API should we ever decide to use a bounded channel between
`NetworkService` and `NetworkWorker`.
The Rust libp2p-kad implementation can require iterative queries to use
disjoint paths for increased resiliency in the presence of potentially
adversarial nodes.
Allow Substrate users to enable this feature via the
`--kademlia-disjoint-query-paths` flag.
* *: Bump async-std to v1.6.5
Prevent users from using v1.6.4 which faces issues receiving incoming
TCP connections. See https://github.com/async-rs/async-std/issues/888
for details.
* client/network/src/gossip: Use channel instead of condvar
`async_std::sync::Condvar::wait_timeout` uses
`gloo_timers::callback::Timeout` when compiled for
`wasm32-unknown-unknown`. This timeout implementation does not fulfill
the requirement of being `Send`.
Instead of using a `Condvar` use a `futures::channel::mpsc` to signal
progress from the `QueuedSender` to the background `Future`.
* client/network/Cargo.toml: Remove async-std unstable feature
* client/network/src/gossip: Forward all queued messages
* client/network/gossip: Have QueuedSender methods take &mut self
* client/network/gossip: Move queue_size_limit into QueuedSender
The `queue_size_limit` field is only accessed by `QueuedSender`, thus
there is no need to share it between the background future and the
`QueuedSender`.
* client/network/gossip: Rename background task to future
To be a bit picky the background task is not a task in the sense of an
asynchonous task, but rather a background future in the sense of
`futures::future::Future`.
With the `OnRead` flow control option yamux "send[s] window updates only
when data is read on the receiving end" and not as soon as "a Stream's
receive window drops to 0".
Yamux flow control has proven itself. This commit removes the feature
flag. Yamux flow control is now always enabled.
* Allow remotes to not open a legacy substream
* No longer actively open legacy substreams
* Misc fixes
* Line width
* Special case first protocol as the one bearing the handshake
* Legacy opening state no longer keeps connection alive
* Remove now-unused code
* Simplify inject_dial_upgrade_error
* [chaos:basic]
* [chaos:basic]
* [chaos:basic]
* sc-network: expose `add_to_priority_group` and `remove_from_priority_group` in `NetworkService`
* sc-network: fix a typo
* Update client/network/src/service.rs
Co-authored-by: Max Inden <mail@max-inden.de>
* s/parse_multiaddr/split_multiaddr_and_peer_id/g
* sc-network: mark new functions as async and add comments
* Apply suggestions from code review
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Max Inden <mail@max-inden.de>
Cecile Tonglet