Commit Graph

61 Commits

Author SHA1 Message Date
Pierre Krieger 3f06fe32f3 Update to libp2p v0.7.0 (#2343)
* Update to libp2p master

* Fix tests

* More tests fixing
2019-04-23 18:46:30 +01:00
Robert Habermeier 1aa6eb9fa8 Polite-grandpa improvements (#2229)
* send neighbor packets in more generic way

* integrate periodic neighbor-packet rebroadcaster

* integrate reporting

* attach callbacks to commit messages for rebroadcasting and reporting

* Tests for commit relay

* crunch up some nice warnings

* exit-scope sub-futures of grandpa

* address small grumbles

* some changes to commit handling
2019-04-16 15:25:46 +08:00
Arkadiy Paronyan e30b362193 Disable gossip for peers running old version (#2245) 2019-04-15 12:18:08 +02:00
Svyatoslav Nikolsky 5725e25448 fix race in sync tests (#2259) 2019-04-12 09:32:00 +02:00
Arkadiy Paronyan 0b2c3ae860 Apply negative rating on explicit ban (#2247)
* Apply negative rating on explicit ban

* Update core/network/src/service.rs

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>
2019-04-10 23:34:41 +02:00
Marek Kotewicz 1142bcde97 simplification of peerset api (#2123)
* Introduction of PeersetHandle

* integrate PeersetHandle with the rest of the codebase

* fix compilation errors

* more tests for peerset, fixed overwriting bug in add_reserved_peer

* Slots data structure and bugfixes for peerset

* bend to pressure

* updated lru-cache to 0.1.2 and updated linked-hash-map to 0.5.2

* peerset discovered list is now a LinkedHashMap

* fix review suggestions

* split back Peerset and PeersetHandle

* test for Peerset::discovered

* applied review suggestions

* fixes to peerset::incoming

* peerset disconnects are all instantaneous

* instantaneous drop in peerset finished

* Peerset::set_reserved_only can also reconnect nodes

* Peerset scores cache uses lru-cache

* remove redundant function call and comment from Peerset::on_set_reserved_only

* add_peer returns SlotState enum

* apply review suggestions

* is_reserved -> is_connected_and_reserved
2019-04-04 19:40:40 +02:00
Robert Habermeier bb95e7d6a2 Initial version of the polite-grandpa networking protocol (#2110)
* Consensus status packet

* Allow for repropagation after status

* More generic gossip

* add a basic view struct and gossip module

* move gossip stuff to the gossip module

* integrate view into gossip

* some reshuffling

* alter rules for keeping one commit at a time in view

* Allow sending addressed messages

* don't cast outgoing votes if we know that we voted before

* Handle one hop messages

* initial run at polite grandpa

* build WASM

* handle neighbor messages

* refactor validator's internals into an Inner struct

* gossip only knows to keep or discard messages. optimize should_send_to

* Periodic rebroadcast

* implement `should_send_to` and message_expired

* track peers' best received commit height

* Pass peer id to topic steam

* kill rebroadcasting network

* Notify about existing peers

* clean up network APIs a bunch

* implement gossip::send_message for direct messages

* refactor network trait

* implement gossip::send_message for direct messages

* get all non set-change tests passing

* treat unknown rebroadcasts as broadcasts

* get all other main tests passing

* remove unimplemented test

* everything compiles

* treat unknown rebroadcasts as broadcasts

* Rebradcast interval

* Apply suggestions from code review

Style

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Style

* some module docs

* address some grumbles + docs

* allow rebroadcast every few minutes

* send_topic && generic context

* some tests for view change

* more grumbles & tests

* use send_peer
2019-04-02 18:09:05 +02:00
DemiMarie-parity a10e86ba5a Refactor DigestItem (#2108)
* Add `start_aura2`.

* .gitignore patch conflict files

and remove one that accidentally got committed

* Fix build

The tests still don’t work.

* Fix compilation errors

* Fix compile errors (again)

* Try (and fail) to fix tests

* Properly deserialize data

Previously, `DigestItem::Consensus` had no separate `DigestItemType`,
so it did not get properly serialized and deserialized.

* Add extra debug logging.  Always allow old seals.

A `RUST_LOG=substrate_aura_consensus cargo test --all -- --nocapture \
tests::authoring_blocks` revealed that old seals were being and
rejected, causing the test to hang.  As a temporary debug measure, allow
old seals unconditionally, so that CI can test if this fixes the
problem.

* Forcibly disable rejection of old seals

* Use old trait, but newer serialization

The old trait for `CompatibleDigestItem` actually worked.  By changing
its implementation, one can ensure that all *new* seals have the modern
form, but *legacy* seals are still decoded correctly.

* Bump impl version

* Squash spurious deprecation warning

`rustc` should not be emitting a deprecation warning in deprecated
code, but it does, so silence it.

* Rip out unused Cargo feature

* Move AURA to aura_primitives

* Respond to code review

* Wrap overly-long line

* Reduce logging verbosity and add target

* Add dependency on `sr-primitives` to `aura_primitives`

* Fix build

It failed with a message about Cargo.lock being out of date.

* core: aura: rename aura engine id const

* core: aura: remove superfluous logging

* core: primitives: add removed semicolons

* core: aura: remove unused import

* core: network: style fix

* runtime: update wasm blobs

* runtime: bump impl_version

* core: primitives: tag all DigestItemType variants explicitly
2019-03-29 12:15:23 +00:00
André Silva d17e34b3c8 core: network: reduce log level of peer drop messages (#2125) 2019-03-27 16:49:19 +01:00
Pierre Krieger 6e394464b8 Replace NodeIndex with PeerId everywhere (#2077)
* Replace NodeIndex with PeerId

* Fix tests

* More test fixing

* Whitespace
2019-03-23 10:34:28 +01:00
Pierre Krieger 90c6f85db5 Add a substrate-peerset crate (#2042)
* Add a substrate-peerset crate

* Some adjustements

* More adjustements

* Use a temporary libp2p branch

* Add back-off mechanism

* Fix RPC tests

* Some adjustements

* Another libp2p bugfix

* Do a round-robin in the peerset

* Use a real dependency instead of a patch for libp2p

* Initialize reserved nodes correctly

* Better diagnostic for no address

* Don't allocate slots if in reserved only

* Ban node on dial failure

* Fix indentation
2019-03-21 14:02:28 +01:00
André Silva e56a5cd00b Minor gossip changes (#2038)
* core: gossip: don't expire messages based on time

* core: gossip: allow forcing resend of gossip messages

* core: grandpa: fix tests
2019-03-19 12:13:05 +01:00
Pierre Krieger 002143d0a2 Remove support for multiple network protocols (#2005)
* Remove support multiple network protocols

* Address concerns

* Add back debug_asserts
2019-03-19 11:56:56 +01:00
Roman Borschel 57387ef585 libp2p-0.5.0 (#1971)
* Update libp2p. Add support for ed25519 node (network) keys.

  * Update networking to the changes from https://github.com/libp2p/rust-libp2p/pull/972.
  * Add support for using ed25519 keys for libp2p networking.
  * Add support for reading libp2p secret keys from (external) files.

* Adapt to changes from https://github.com/libp2p/rust-libp2p/pull/992

* More tests.

* Cosmetics

* Deduplicate tests.

* Remove quickcheck from tests that don't use extra random inputs.

* Remove quickcheck.

* Swap new/default impls for NetworkConfiguration.

* Use libp2p-0.5.0 from crates.io.

* Post-rebase update.

* Remove unnecessary wildcard pattern.

* Combine two overlapping tests.
2019-03-19 11:16:25 +01:00
Pierre Krieger 0a300b061e Remove the error from network-libp2p (#1950) 2019-03-11 11:40:22 +01:00
Bastian Köcher 7927054f86 Update copyrights to 2019 (#1932) 2019-03-06 15:04:28 +01:00
Pierre Krieger bcda2fafa0 Add even more diagnostic messages (#1925)
* Add even more diagnostic messages

* Fix
2019-03-05 22:44:55 +01:00
Robert Habermeier dfb48a2405 Offline fallback for GRANDPA (#1619)
Co-authored-by: André Silva <andre.beat@gmail.com>

* skeleton for finality tracker

* dispatch events when nothing finalized for a long time

* begin integrating finality tracker into grandpa

* add delay field to pending change

* add has_api_with function to sr_version for querying APIs

* partially integrate new force changes into grandpa

* implement forced changes

* get srml-grandpa compiling

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* remove explicit dependence on CoreApi

* increase node runtime version

* integrate grandpa forced changes into node runtime

* add some tests to finality-tracker

* integrate finality tracking into node-runtime

* test forced-change logic

* test forced changes in the authority-set handler

* kill some unneeded bounds in client

* test forced-changes in finality-grandpa and fix logic

* build wasm and finality-tracker is no-std

* restart voter on forced change

* allow returning custom error type from lock_import_and_run

* extract out most DB logic to aux_schema and use atomic client ops

* unify authority set writing

* implement set pausing

* bump runtime version

* note on DB when we pause.

* core: grandpa: integrate forced changes with multiple pending standard changes

* core: grandpa: fix AuthoritySet tests

* runtime: bump impl_version

* core: clear pending justification requests after forced change import

* srml: finality-tracker: use FinalizedInherentData

* core: log requests for clearing justification requests

* core, node: update runtimes

* core: grandpa: fix tests

* core: grandpa: remove todos and add comments

* core: grandpa: use has_api_with from ApiExt

* core: fix tests

* core: grandpa: remove unnecessary mut modifier

* core: replace PostImportActions bitflags with struct

* core: grandpa: restrict genesis on forced authority set change

* core: grandpa: add more docs

* core: grandpa: prevent safety violations in Environment::finalize_block

* core: grandpa: register finality tracker inherent data provider

* core: grandpa: fix tests

* node: update runtime blobs

* core: grandpa: remove outdated todo

* core: aura: fix typo in log message

* core: grandpa: check re-finalization is on canonical chain

* srml: finality-tracker: fix initialization

* node: update runtime wasm

* srml: finality-tracker: don't re-initialize config keys
2019-03-05 15:41:35 +00:00
Gregory Terzian a81f7f48a0 Refactor Sync status updates into a stream of updates (#1858)
* refactor sync provider

* relative use of interval

* typo

* set propagate timeout to 2500ms

* address comments

* fix instant calc

* update intervals
2019-03-02 14:35:16 +01:00
Pierre Krieger c21d7436cc Add an RPC request for the state of the network (#1884)
* Add an RPC request for the state of the network

* Fix concerns

* Fix tests

* Replace comment with TODO

* Rename the RPC
2019-02-28 19:28:38 +01:00
Gregory Terzian 8a72abffdd Remove blocking sync -> import-queue operations (#1818)
* remove blocking sync -> import-queue operations

add specialization to testnet

remove add peer default impl on TestNetFactory

* remove empty brackets from dummy specialization

* nits

* make mut_peers take an fn once

* add SpecializationFactory trait in test

* remove add_peer imple in grandpa, fix typo

* use cmp::max for best importing number comparison

* remove import of non-existent create_peer

* add sender to start message
2019-02-28 12:11:18 +01:00
Gregory Terzian bc15fa31ff Add connected peers to protocol, use in sync provider (#1857)
* add connected peers to protocol, use in sync provider

* use PeerId::random

* address comments`

* docs

* fix import of PeerId

* rewrite rpc tests using PeerId::random

* whitespace

* nits

* remove option around peer id and remove field

* further removal of the option around peer id

* fix rpc tests
2019-02-27 18:50:47 +01:00
Gregory Terzian 5282615416 Use bounded channels for network -> sync -> import (#1874)
* use bounded channels for network -> sync -> import

* bound at 4

* indent

* use return value of handle_network_msg
2019-02-27 09:19:57 +01:00
Pierre Krieger 34fccecc02 Add names to tokio threads (#1866) 2019-02-25 15:40:51 -05:00
Gregory Terzian 077ed00951 Remove blocking operations in SyncOracle implementation (#1852)
* remove blocking operations in SyncOracle implementation

* docs

* docs
2019-02-25 14:00:13 +01:00
Pierre Krieger eaa8f0c97f Add back logging when we ban a peer (#1843)
* Add back logging when we ban a peer

* Make it less verbose
2019-02-22 13:48:59 +01:00
Pierre Krieger f5ba8a2ad9 Report the network PeerId to the telemetry (#1835)
* Report the network PeerId to the telemetry

* Fix accidental paste
2019-02-21 10:03:47 +01:00
Arkadiy Paronyan b3eae17f65 Gossip refactoring (#1811)
* First part of gossip protocol refactoring

* Message validation in GRANDPA

* Reverted to time-based expiration for future round messages

* Removed collect_garbage_for_topic

* Use consensus engine id instead of kind

* Use try_init

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Comment

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Added expiration check on broadcast

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Apply suggestions from code review

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Style

* Style
2019-02-20 12:40:03 +01:00
Gregory Terzian 72bb8ef4c5 Rewrite the BasiQueue using channels (#1327)
* use channels to implement basic import queue

* async justification import

* better conditional for is_done in tests

* reword the test for presence of link

* fix conditional

* trace instead of panic when no link present

* reword expectations when sending to importers

* fix

* debug justification import error

* update expectations

* use NumberFor

* nits

* add general description

* move error handling into closure
2019-02-17 10:13:14 +01:00
Pierre Krieger 9e999cdd81 Make network-libp2p's Service generic over the message (#1708)
* Make network-libp2p's Service generic over the message

* Apply suggestions from code review

Co-Authored-By: tomaka <pierre.krieger1708@gmail.com>

* Fix warning
2019-02-12 15:36:15 +01:00
Stanislav Tkach e60be1ad12 Migrate network, primitives and rpc to the 2018 edition (#1710) 2019-02-06 19:07:48 +01:00
Gregory Terzian a2d2ed69ab Rewrite network protocol/service to use channels (#1340)
* rewrite network protocol/service to use channels

* remove use of unwrap

* re-introduce with_spec

* remove unnecessary mut

* remove unused param

* improve with_spec, add with_gossip

* rename job to task

* style: re-add comma

* remove extra string allocs

* rename use of channel

* turn TODO into FIXME

* remove mut in match

* remove Self in new

* pass headers by value to network service

* remove network sender from service

* remove TODO

* better expect

* rationalize use of network sender in ondemand
2019-02-06 12:54:02 +01:00
Arkadiy Paronyan 641bb7cb46 Stop authoring blocks when offline (#1655)
* Don't author blocks when offline

* Increased canonicalization delay

* Fixed test
2019-02-01 17:17:53 +01:00
Robert Habermeier d027123059 GRANDPA: announce blocks we vote on to peers (#1593)
* announce blocks we vote on to peers (missing network impl)

* Implemented 'announce'

* Fixed test

* improve announce docs and add logging

* Track recently announced blocks
2019-01-28 16:41:36 +00:00
Pierre Krieger a1586df41b Report the average network bandwidth (#1551)
* Report the average network bandwidth

* Fix concerns
2019-01-25 10:34:34 +00:00
Pierre Krieger d30a8e0bc3 Log the clogging networking messages (#1532) 2019-01-23 15:24:25 +01:00
Pierre Krieger cd86643f33 Report when peer is clogged (#1528) 2019-01-23 14:30:20 +01:00
André Silva 399cee310a Sync block justifications (#1410)
* core: sync protocol for justifications

* core: basic test for justification sync

* core: pass block number with justification

* grandpa: request justifications when importing change blocks

* core: pass finality notifications to chain sync

* core: require justifications for pending change blocks on start

* core: avoid requesting justifications from previous failed peers

* core: timeout block justification requests

* core: add some docs

* core: fix unused variables warning

* core: tick pending justifications fetch periodically

* grandpa: add test for syncing justifications

* core: early exit dispatch of pending justifications

* core: style fix

* core: grandpa: change logging level

* core: sync: add missing docs

* core: network: report peer on bad justification

* core: replace mem::replace with Option::take

* core: revert authority set changes on failed block finalization

* core: grandpa: add docs to import_justification

* core: warn on re-finalization of last finalized block

* core: only notify sync with last finality notification

* core: style fix

* core: add docs for PendingJustifications

* core: network: use BlockRequest messages for justification requests

* core: reference issues in todo comments

* core: grandpa: revert authority set changes on db

* core: grandpa: remove inconsistent state warning
2019-01-21 07:04:01 +01:00
Pierre Krieger a0d458aa06 Update to latest libp2p (#1386)
* Update to latest libp2p

* Fix indentations

* Add basic test

* Apply suggestions from code review

Co-Authored-By: tomaka <pierre.krieger1708@gmail.com>

* Remove Mutex from topology

* Remove unused method

* Fix concerns
2019-01-14 13:38:36 +01:00
Xiliang Chen eb3503b0c7 expose peer information via rpc (#1362)
* expose peer information via rpc

* fixes tests

* cleanup

Co-Authored-By: xlc <xlchen1291@gmail.com>

* Update docs

Co-Authored-By: xlc <xlchen1291@gmail.com>

* Add missing docs

* keep original type for PeerInfo best_hash/best_number

* cleanup

* Update mod.rs
2019-01-09 12:18:24 +01:00
Gregory Terzian 3add75910a move import queue to consensus-common (#1282) 2018-12-31 12:33:21 +01:00
André Silva 45d53ad022 Remove grandpa liveness oracle (#1271)
* core: support broadcasting consensus message to all peers

* grandpa: remove liveness oracle

* node: always start grandpa on full nodes

* core: don't check for grandpa justifications on failed block imports

* core: fix network connectivity test
2018-12-15 20:39:48 +01:00
Benjamin Kampmann dbe31230c3 Simplify consensus message sending (#1166)
* Create gossip_consensus_message helper on service, simplify consensus_gossip access
2018-11-27 09:06:58 +01:00
Robert Habermeier 11fe84a742 Minimal switch of substrate-node to GRANDPA /Aura (#1128)
* add beginnings of SRML grandpa library

* get srml-grandpa compiling

* tests for srml-grandpa

* add optional session integration to grandpa SRML

* start integration into node runtime

* Allow extracting pending change from header digest

* Make it compile on wasm

* make tests compile again

* Move Authority Key fetching into service, simplify service factory construction

* Generalize Authority Consensus Setup system

* Add Authority Setup Docs

* Allow CLI params to be extensible

 - move params to structopts
 - split parsing and default command execution
 - add custom config to node
 - extended parsing of custom config
 - extending params via structop's flatten

* Minor fixes on cli extension params:
 - added docs
 - re-add actual app name, rather than node-name
 - make strategy and subcommand optional

* better cli params

* synchronize GRANDPA and normal node authorities

* Implement grandpa::network for gossip consensus

* run_grandpa in Node

* Fix missed merge error

* Integrate grandpa import queue

* more specific type def

* link up linkhalf and import block

* make grandpa future send

* get compiling

* Fix new params convention and license header

* get it running

* rebuild node runtime WASM

* change logging level

* Update node/cli/src/params.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/params.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/runtime/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Clean up and Fixme for mutable config

* Move GrandpaService Integration into grandpa, feature gated but on per default

* Fixing grandpa runtime module test

* Update wasm runtime hashes for tests

* GRANDPA: use post-header hash when logging scheduled changes

* add an extra bit of logging to authorities

* fixing missing constrain

* remove old code

* move `NewAuthorities` to an event in srml-grandpa

* fix node-executor tests to use grandpa log

* Remove GossipConsensus from tests, use newly provided sync-feature, fixes tests

* Update to latest wasm runtimes

* address grumbles

* address grumbles

* only derive deserialize when using std

* Clean up use of Deserialize
2018-11-21 18:42:50 +01:00
Robert Habermeier e37ab01482 Merge branch 'master' into rh-grandpa-dynamic2 2018-11-14 14:32:30 +01:00
Pierre Krieger 74e907f3cb Some tweaks in the network crate (#1108)
* Move Roles to network::config

* Make network::config public

* Move NetworkConfig and NonReservedMode to config

* Move Params to config

* Move node_id() to NetworkManager and fix tests

* Rename Specialization to NetworkSpecialization
2018-11-14 11:25:16 +01:00
Robert Habermeier 21f291a607 Merge branch 'master' into rh-grandpa-dynamic2 2018-11-01 20:31:13 +01:00
Robert Habermeier 0302a1fc5e refactor import queue to use explicit block import references 2018-10-29 15:56:10 +01:00
Robert Habermeier ab6c343aaf Merge branch 'master' into rh-grandpa-dynamic2 2018-10-27 18:12:33 +02:00
guanqun b8b3f5b7d1 fix one typo (#1029) 2018-10-27 18:03:57 +02:00