Commit Graph

51 Commits

Author SHA1 Message Date
Robert Habermeier bb95e7d6a2 Initial version of the polite-grandpa networking protocol (#2110)
* Consensus status packet

* Allow for repropagation after status

* More generic gossip

* add a basic view struct and gossip module

* move gossip stuff to the gossip module

* integrate view into gossip

* some reshuffling

* alter rules for keeping one commit at a time in view

* Allow sending addressed messages

* don't cast outgoing votes if we know that we voted before

* Handle one hop messages

* initial run at polite grandpa

* build WASM

* handle neighbor messages

* refactor validator's internals into an Inner struct

* gossip only knows to keep or discard messages. optimize should_send_to

* Periodic rebroadcast

* implement `should_send_to` and message_expired

* track peers' best received commit height

* Pass peer id to topic steam

* kill rebroadcasting network

* Notify about existing peers

* clean up network APIs a bunch

* implement gossip::send_message for direct messages

* refactor network trait

* implement gossip::send_message for direct messages

* get all non set-change tests passing

* treat unknown rebroadcasts as broadcasts

* get all other main tests passing

* remove unimplemented test

* everything compiles

* treat unknown rebroadcasts as broadcasts

* Rebradcast interval

* Apply suggestions from code review

Style

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Style

* some module docs

* address some grumbles + docs

* allow rebroadcast every few minutes

* send_topic && generic context

* some tests for view change

* more grumbles & tests

* use send_peer
2019-04-02 18:09:05 +02:00
DemiMarie-parity a10e86ba5a Refactor DigestItem (#2108)
* Add `start_aura2`.

* .gitignore patch conflict files

and remove one that accidentally got committed

* Fix build

The tests still don’t work.

* Fix compilation errors

* Fix compile errors (again)

* Try (and fail) to fix tests

* Properly deserialize data

Previously, `DigestItem::Consensus` had no separate `DigestItemType`,
so it did not get properly serialized and deserialized.

* Add extra debug logging.  Always allow old seals.

A `RUST_LOG=substrate_aura_consensus cargo test --all -- --nocapture \
tests::authoring_blocks` revealed that old seals were being and
rejected, causing the test to hang.  As a temporary debug measure, allow
old seals unconditionally, so that CI can test if this fixes the
problem.

* Forcibly disable rejection of old seals

* Use old trait, but newer serialization

The old trait for `CompatibleDigestItem` actually worked.  By changing
its implementation, one can ensure that all *new* seals have the modern
form, but *legacy* seals are still decoded correctly.

* Bump impl version

* Squash spurious deprecation warning

`rustc` should not be emitting a deprecation warning in deprecated
code, but it does, so silence it.

* Rip out unused Cargo feature

* Move AURA to aura_primitives

* Respond to code review

* Wrap overly-long line

* Reduce logging verbosity and add target

* Add dependency on `sr-primitives` to `aura_primitives`

* Fix build

It failed with a message about Cargo.lock being out of date.

* core: aura: rename aura engine id const

* core: aura: remove superfluous logging

* core: primitives: add removed semicolons

* core: aura: remove unused import

* core: network: style fix

* runtime: update wasm blobs

* runtime: bump impl_version

* core: primitives: tag all DigestItemType variants explicitly
2019-03-29 12:15:23 +00:00
Pierre Krieger 6e394464b8 Replace NodeIndex with PeerId everywhere (#2077)
* Replace NodeIndex with PeerId

* Fix tests

* More test fixing

* Whitespace
2019-03-23 10:34:28 +01:00
Arkadiy Paronyan b31bcdfcc3 Keep peer info up to date (#2067)
* Keep peer info up to date

* Docs

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>
2019-03-21 12:55:30 +01:00
André Silva e56a5cd00b Minor gossip changes (#2038)
* core: gossip: don't expire messages based on time

* core: gossip: allow forcing resend of gossip messages

* core: grandpa: fix tests
2019-03-19 12:13:05 +01:00
Arkadiy Paronyan 8d290f6110 Don't mark peers sending expired messages as useless (#1944) 2019-03-07 22:12:57 +01:00
Bastian Köcher 7927054f86 Update copyrights to 2019 (#1932) 2019-03-06 15:04:28 +01:00
Robert Habermeier dfb48a2405 Offline fallback for GRANDPA (#1619)
Co-authored-by: André Silva <andre.beat@gmail.com>

* skeleton for finality tracker

* dispatch events when nothing finalized for a long time

* begin integrating finality tracker into grandpa

* add delay field to pending change

* add has_api_with function to sr_version for querying APIs

* partially integrate new force changes into grandpa

* implement forced changes

* get srml-grandpa compiling

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update core/finality-grandpa/src/authorities.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* remove explicit dependence on CoreApi

* increase node runtime version

* integrate grandpa forced changes into node runtime

* add some tests to finality-tracker

* integrate finality tracking into node-runtime

* test forced-change logic

* test forced changes in the authority-set handler

* kill some unneeded bounds in client

* test forced-changes in finality-grandpa and fix logic

* build wasm and finality-tracker is no-std

* restart voter on forced change

* allow returning custom error type from lock_import_and_run

* extract out most DB logic to aux_schema and use atomic client ops

* unify authority set writing

* implement set pausing

* bump runtime version

* note on DB when we pause.

* core: grandpa: integrate forced changes with multiple pending standard changes

* core: grandpa: fix AuthoritySet tests

* runtime: bump impl_version

* core: clear pending justification requests after forced change import

* srml: finality-tracker: use FinalizedInherentData

* core: log requests for clearing justification requests

* core, node: update runtimes

* core: grandpa: fix tests

* core: grandpa: remove todos and add comments

* core: grandpa: use has_api_with from ApiExt

* core: fix tests

* core: grandpa: remove unnecessary mut modifier

* core: replace PostImportActions bitflags with struct

* core: grandpa: restrict genesis on forced authority set change

* core: grandpa: add more docs

* core: grandpa: prevent safety violations in Environment::finalize_block

* core: grandpa: register finality tracker inherent data provider

* core: grandpa: fix tests

* node: update runtime blobs

* core: grandpa: remove outdated todo

* core: aura: fix typo in log message

* core: grandpa: check re-finalization is on canonical chain

* srml: finality-tracker: fix initialization

* node: update runtime wasm

* srml: finality-tracker: don't re-initialize config keys
2019-03-05 15:41:35 +00:00
Gregory Terzian a81f7f48a0 Refactor Sync status updates into a stream of updates (#1858)
* refactor sync provider

* relative use of interval

* typo

* set propagate timeout to 2500ms

* address comments

* fix instant calc

* update intervals
2019-03-02 14:35:16 +01:00
Gregory Terzian 8a72abffdd Remove blocking sync -> import-queue operations (#1818)
* remove blocking sync -> import-queue operations

add specialization to testnet

remove add peer default impl on TestNetFactory

* remove empty brackets from dummy specialization

* nits

* make mut_peers take an fn once

* add SpecializationFactory trait in test

* remove add_peer imple in grandpa, fix typo

* use cmp::max for best importing number comparison

* remove import of non-existent create_peer

* add sender to start message
2019-02-28 12:11:18 +01:00
Gregory Terzian bc15fa31ff Add connected peers to protocol, use in sync provider (#1857)
* add connected peers to protocol, use in sync provider

* use PeerId::random

* address comments`

* docs

* fix import of PeerId

* rewrite rpc tests using PeerId::random

* whitespace

* nits

* remove option around peer id and remove field

* further removal of the option around peer id

* fix rpc tests
2019-02-27 18:50:47 +01:00
Gregory Terzian 5282615416 Use bounded channels for network -> sync -> import (#1874)
* use bounded channels for network -> sync -> import

* bound at 4

* indent

* use return value of handle_network_msg
2019-02-27 09:19:57 +01:00
Gregory Terzian 077ed00951 Remove blocking operations in SyncOracle implementation (#1852)
* remove blocking operations in SyncOracle implementation

* docs

* docs
2019-02-25 14:00:13 +01:00
Toralf Wittner 4bf4b37185 Limit the set of known extrinsics and blocks. (#1842)
* Limit the set of known extrinsics and blocks.

* Update Cargo.lock

* Always insert and pop oldest entry if needed.

Always inserting first ensures that the element's LRU position is
updated.
2019-02-21 19:24:23 +01:00
Arkadiy Paronyan b3eae17f65 Gossip refactoring (#1811)
* First part of gossip protocol refactoring

* Message validation in GRANDPA

* Reverted to time-based expiration for future round messages

* Removed collect_garbage_for_topic

* Use consensus engine id instead of kind

* Use try_init

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Comment

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Added expiration check on broadcast

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Apply suggestions from code review

Co-Authored-By: arkpar <arkady.paronyan@gmail.com>

* Style

* Style
2019-02-20 12:40:03 +01:00
André Silva 21779b8cf2 Import multiple authority set change blocks (#1808)
* core: implement logic for tracking dag of possible pending changes

* core: move pending justifications dag to its own crate

* core: remove unnecessary clone bounds on dag

* core: request justifications in-order from the dag

* core: dag: rename changes variables to node

* core: dag: allow finalizing blocks not part of dag

* core: dag: track best finalized number

* core: dag: add more tests

* core: sync: clean up pending justifications dag

* core: dag: derive codec decode encode

* core: dag: better error support

* core: dag: add finalization guarded by predicate

* core: grandpa: track multiple authority set changes in dag

* core: dag: add pre-order iterator

* core: grandpa: request justifications on startup

* core: dag: rearrange order of definitions

* core: rename util/dag to util/fork_tree

* core: fork_tree: add docs

* core: fork_tree: add more tests

* core: fork_tree: fix issues found in tests

* core: grandpa: fix authorities tests

* core: grandpa: add docs for is_descendent_of

* core: sync: add docs for PendingJustifications

* core: sync: add test for justification requests across forks

* core: sync: don't resend import or finality notifications in tests

* core: grandpa: add test for importing multiple change blocks

* core: grandpa: fix logic for checking if a block enacts a change

* core: grandpa: fix authorities tests
2019-02-20 00:08:43 +01:00
Toralf Wittner c5d3da32f2 Small refactoring. (#1826) 2019-02-19 17:29:13 -03:00
Gregory Terzian 72bb8ef4c5 Rewrite the BasiQueue using channels (#1327)
* use channels to implement basic import queue

* async justification import

* better conditional for is_done in tests

* reword the test for presence of link

* fix conditional

* trace instead of panic when no link present

* reword expectations when sending to importers

* fix

* debug justification import error

* update expectations

* use NumberFor

* nits

* add general description

* move error handling into closure
2019-02-17 10:13:14 +01:00
Pierre Krieger 9e999cdd81 Make network-libp2p's Service generic over the message (#1708)
* Make network-libp2p's Service generic over the message

* Apply suggestions from code review

Co-Authored-By: tomaka <pierre.krieger1708@gmail.com>

* Fix warning
2019-02-12 15:36:15 +01:00
Arkadiy Paronyan 1dc15be5bd Check for block parent before verification. (#1714)
* Treat verification errors more seriously

* Track obsolete requests

* Check block parent before verification

* Style
2019-02-07 11:58:27 +01:00
Stanislav Tkach e60be1ad12 Migrate network, primitives and rpc to the 2018 edition (#1710) 2019-02-06 19:07:48 +01:00
Gregory Terzian a2d2ed69ab Rewrite network protocol/service to use channels (#1340)
* rewrite network protocol/service to use channels

* remove use of unwrap

* re-introduce with_spec

* remove unnecessary mut

* remove unused param

* improve with_spec, add with_gossip

* rename job to task

* style: re-add comma

* remove extra string allocs

* rename use of channel

* turn TODO into FIXME

* remove mut in match

* remove Self in new

* pass headers by value to network service

* remove network sender from service

* remove TODO

* better expect

* rationalize use of network sender in ondemand
2019-02-06 12:54:02 +01:00
Arkadiy Paronyan 641bb7cb46 Stop authoring blocks when offline (#1655)
* Don't author blocks when offline

* Increased canonicalization delay

* Fixed test
2019-02-01 17:17:53 +01:00
Benjamin Kampmann 15ae7cfef6 Document more TODOs as tickets (#1418)
Went through the TODOs, removed a bunch, which are outdated or nothing more than a regular comment, documented a bunch more as actual tickets and made them FIXMEs and unified their structure (`FIXME #TICKETNO DESC` for local tickets, `FIXME: DESC LINK` for external tickets) for easier in-editor support. Further more remove unnecessary remarks and related old code that I noticed in that instance.
2019-01-30 10:29:48 +01:00
Robert Habermeier ced9e72824 Download unknown but announced forks (#1604)
* test reproducing fork sync issues

* update to new announce_block API

* Download unknown forks

* Reverted download_stale

* Avoid cloning the hash

* Typo
2019-01-28 16:41:33 -03:00
Robert Habermeier d027123059 GRANDPA: announce blocks we vote on to peers (#1593)
* announce blocks we vote on to peers (missing network impl)

* Implemented 'announce'

* Fixed test

* improve announce docs and add logging

* Track recently announced blocks
2019-01-28 16:41:36 +00:00
André Silva 5ddcbe0ca6 core: send justification requests in parallel to block requests (#1563)
* core: send justification requests in parallel to block requests

* core: fix grandpa test
2019-01-25 11:07:26 -03:00
Benjamin Kampmann ab4c70b379 Update dependencies (#1527)
* update to latest env_logger

* roundabout dependency update

* Update to latest vengen

* Update rand to 0.6

* Update to latest rng shuffle API

* Roundabout update of dependencies

* Update rustc_hex

* Update wasm

* Fix logging setup in tests

* revert to fix test
2019-01-25 12:44:41 +01:00
Pierre Krieger d30a8e0bc3 Log the clogging networking messages (#1532) 2019-01-23 15:24:25 +01:00
Pierre Krieger cd86643f33 Report when peer is clogged (#1528) 2019-01-23 14:30:20 +01:00
André Silva 399cee310a Sync block justifications (#1410)
* core: sync protocol for justifications

* core: basic test for justification sync

* core: pass block number with justification

* grandpa: request justifications when importing change blocks

* core: pass finality notifications to chain sync

* core: require justifications for pending change blocks on start

* core: avoid requesting justifications from previous failed peers

* core: timeout block justification requests

* core: add some docs

* core: fix unused variables warning

* core: tick pending justifications fetch periodically

* grandpa: add test for syncing justifications

* core: early exit dispatch of pending justifications

* core: style fix

* core: grandpa: change logging level

* core: sync: add missing docs

* core: network: report peer on bad justification

* core: replace mem::replace with Option::take

* core: revert authority set changes on failed block finalization

* core: grandpa: add docs to import_justification

* core: warn on re-finalization of last finalized block

* core: only notify sync with last finality notification

* core: style fix

* core: add docs for PendingJustifications

* core: network: use BlockRequest messages for justification requests

* core: reference issues in todo comments

* core: grandpa: revert authority set changes on db

* core: grandpa: remove inconsistent state warning
2019-01-21 07:04:01 +01:00
Svyatoslav Nikolsky da1fb3f273 Use changes tries in query_storage RPC (#1082)
* use changes tries in query_storage RPC

* let + match + return + call -> match
2019-01-17 10:08:50 +01:00
Xiliang Chen eb3503b0c7 expose peer information via rpc (#1362)
* expose peer information via rpc

* fixes tests

* cleanup

Co-Authored-By: xlc <xlchen1291@gmail.com>

* Update docs

Co-Authored-By: xlc <xlchen1291@gmail.com>

* Add missing docs

* keep original type for PeerInfo best_hash/best_number

* cleanup

* Update mod.rs
2019-01-09 12:18:24 +01:00
Gregory Terzian 3add75910a move import queue to consensus-common (#1282) 2018-12-31 12:33:21 +01:00
André Silva 45d53ad022 Remove grandpa liveness oracle (#1271)
* core: support broadcasting consensus message to all peers

* grandpa: remove liveness oracle

* node: always start grandpa on full nodes

* core: don't check for grandpa justifications on failed block imports

* core: fix network connectivity test
2018-12-15 20:39:48 +01:00
Benjamin Kampmann dbe31230c3 Simplify consensus message sending (#1166)
* Create gossip_consensus_message helper on service, simplify consensus_gossip access
2018-11-27 09:06:58 +01:00
Robert Habermeier 11fe84a742 Minimal switch of substrate-node to GRANDPA /Aura (#1128)
* add beginnings of SRML grandpa library

* get srml-grandpa compiling

* tests for srml-grandpa

* add optional session integration to grandpa SRML

* start integration into node runtime

* Allow extracting pending change from header digest

* Make it compile on wasm

* make tests compile again

* Move Authority Key fetching into service, simplify service factory construction

* Generalize Authority Consensus Setup system

* Add Authority Setup Docs

* Allow CLI params to be extensible

 - move params to structopts
 - split parsing and default command execution
 - add custom config to node
 - extended parsing of custom config
 - extending params via structop's flatten

* Minor fixes on cli extension params:
 - added docs
 - re-add actual app name, rather than node-name
 - make strategy and subcommand optional

* better cli params

* synchronize GRANDPA and normal node authorities

* Implement grandpa::network for gossip consensus

* run_grandpa in Node

* Fix missed merge error

* Integrate grandpa import queue

* more specific type def

* link up linkhalf and import block

* make grandpa future send

* get compiling

* Fix new params convention and license header

* get it running

* rebuild node runtime WASM

* change logging level

* Update node/cli/src/params.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/params.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/runtime/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update node/cli/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Clean up and Fixme for mutable config

* Move GrandpaService Integration into grandpa, feature gated but on per default

* Fixing grandpa runtime module test

* Update wasm runtime hashes for tests

* GRANDPA: use post-header hash when logging scheduled changes

* add an extra bit of logging to authorities

* fixing missing constrain

* remove old code

* move `NewAuthorities` to an event in srml-grandpa

* fix node-executor tests to use grandpa log

* Remove GossipConsensus from tests, use newly provided sync-feature, fixes tests

* Update to latest wasm runtimes

* address grumbles

* address grumbles

* only derive deserialize when using std

* Clean up use of Deserialize
2018-11-21 18:42:50 +01:00
Svyatoslav Nikolsky fa84cec382 Fetch changes trie roots + CHT-based proof for these roots (#896)
* build CHT for changes tries roots

* collect chnages tries roots proof in key_changes_proof

* flush check_changes_proof

* fixed compilation

* LightDataChecker now has a ref to the blockchain

* continue passing proofs

* new light db tests

* more CHT tests

* more tests for key changes proof when headers are missing

* lost files
2018-11-14 16:06:10 +01:00
Pierre Krieger 74e907f3cb Some tweaks in the network crate (#1108)
* Move Roles to network::config

* Make network::config public

* Move NetworkConfig and NonReservedMode to config

* Move Params to config

* Move node_id() to NetworkManager and fix tests

* Rename Specialization to NetworkSpecialization
2018-11-14 11:25:16 +01:00
Benjamin Kampmann 50adea6220 Consensus Engines Implementation: Aura (#911)
* Generalize BlockImport

 - move ImportBlock, BlockOrigin, ImportResult into shared sr-primitives
 - let Consensus provide  and  traits again
 - update consensus traits to latest development
 - implement traits on client::Client, test_client::TestClient
 - update RHD to use the new import_block API

* Move ImportBlock into consensus-common
* Send import notification in aura tests
* Integrating aura into service
* Make Signatures more generic
* Aura Block Production with the given key
* run aura on the thread pool
* start at exact step start in aura
* Add needed wasm blob, in leiu of better solutions.
* Make API ids consistent with traits and bring upstream for sharing.
* Add decrease_free_balance to Balances module
* Encode `Metadata` once instead of two times
* Bitops include xor
* Upgrade key module.
* Default pages to somewhat bigger.
* Introduce upgrade key into node
* Add `Created` event
2018-10-27 15:59:18 +02:00
Tomasz Drwięga 36625faa9f Pool: parallel ready and runtime changes (#922)
* Revert "Revert runtime changes."

This reverts commit 01a7d1aa83c2918dd63b7dc54eb688d544cfc649.

* Parallel queue reads.

* Avoid recursion in best iterator.
2018-10-18 10:37:29 +02:00
Gav Wood 8bc5242c92 Compact format for block number, nonce, balance (#910)
* Try to make everything compact

* Make `Clone` work

* Fix

* Move contracts and balances over to compact encoding

* Session and timestamp are compact

* Sataking uses compact numbers

* Treasury now compact

* Compact Democracy

* Council is compact

* Fix
2018-10-16 22:47:30 +02:00
Benjamin Kampmann ac4bcf879f Generalize the Consensus Infrastructure (#883)
* Split out Consensus
* Supply ImportQueue through network-service
  - simplify ImportQueue.import_blocks
  - remove Deadlock on import_block
  - Adding Verifier-Trait
  - Implement import_queue provisioning in service; allow cli to import
* Allow to actually customize import queue
* Consensus Gossip: Cache Message hash per Topic
2018-10-16 13:40:33 +02:00
Bastian Köcher db427cb45c Adds construct_simple_protocol macro for simplifying the creation of protocols (#897)
* Make `on_message` take the message as `&mut Option<_>`

* Make `ConsensusGossip` implement `Specialization`

* Move `new_session` into `ConsensusGossip`

* Adds `construct_simple_protocol` macro for simplifying the creation of protocols
2018-10-12 13:10:36 +02:00
Bastian Köcher 12f12bff28 Make all copyrights use the correct year (#894)
* Adds a script for automatically update all copyrights

* Ran the `update-copyright` script
2018-10-10 10:28:41 +02:00
Pierre Krieger 0666759b16 Clean up the networking layer (#881) 2018-10-08 18:20:38 +02:00
Svyatoslav Nikolsky f851dcf41c Filter peers on light nodes (#862)
* do not send OnDemand request to node who, we believe, can't process it

* ignore peers with too old best block on light nodes

* non-weird temporary

* fix condition

* compilation
2018-10-05 13:51:28 -04:00
Pierre Krieger d14700f7b6 The NodeId is always available in the session info (#859) 2018-10-01 18:31:07 +01:00
Arkadiy Paronyan a5a7dd2480 Fixed transaction propagation (#850) 2018-09-30 16:48:41 +01:00
Svyatoslav Nikolsky c54350661d Fetching changes proof from remote nodes (#769)
* changes_trie

* changs_trie: continue

* changes_trie: adding tests

* fixed TODO

* removed obsolete ExtrinsicChanges

* encodable ChangesTrieConfiguration

* removed polkadot fle

* fixed grumbles

* ext_storage_changes_root returns u32

* moved changes trie root to digest

* removed commented code

* read storage values from native code

* fixed grumbles

* fixed grumbles

* missing comma

* key changes proof generation + query

* fix grumbles

* check that changes trie config is not changed by block.finalize()

* fixed changes trie config check
2018-09-29 09:47:29 +01:00