Commit Graph

52 Commits

Author SHA1 Message Date
André Silva 1871a95088 grandpa: remove light-client specific block import pipeline (#7546)
* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs
2020-11-23 14:28:55 +00:00
Pierre Krieger 1eae9f5792 Remove necessity to pass ConsensusEngineId when registering notifications protocol (#7549)
* Remove necessity to pass ConsensusEngineId when registering notifications protocol

* Line width

* Fix tests protocol name

* Other renames

* Doc update

* Change issue in TODO
2020-11-18 15:05:35 +00:00
André Silva 9f61b5b9f4 grandpa: remove duplicate function to fetch local authority id (#7359)
* grandpa: remove duplicate authority_id function

* grandpa: rename is_voter to local_authority_id

* grandpa: cleanup rebuild_voter telemetry event
2020-10-19 17:07:18 +00:00
André Silva f3ea1624f6 frame: remove finality-tracker (#7228)
* frame: remove finality-tracker

* node: remove unused parameter types

* node: bump spec_version
2020-10-15 20:58:51 +00:00
Ashley d1ef8507d7 Add missing fields to the light sync state (#7225)
* Initial draft

* Add an iterator that helps us get most items

* Revert changes to grandpa

* Change fields to just be the grandpa authority set and babe epoch changes

* Only use the fields we need from the shared authority set

* Switch to RPC call

* Revert "Only use the fields we need from the shared authority set"

This reverts commit 6ede87b0c5fe53f251d7cb45951006a7dc8f9b83.

* Add babe_finalized_block_weight from `ashley-improve-sync-state-WIP-loading`

* Fix rpc test

* Move sync state rpc stuff into sc-sync-state-rpc

* Remove as_json_value and remove unwraps from sc-sync-state-rpc

* Add clone_inner to SharedAuthoritySet
2020-10-15 10:10:43 +00:00
Rakan Alhneiti 3aa4bfacfc Async keystore + Authority-Discovery async/await (#7000)
* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-10-08 20:56:35 +00:00
Jon Häggblad 8a0ac4c40c grandpa-rpc: use FinalityProofProvider to check finality for rpc (#6215)
* grandpa-rpc: use FinalityProofProvider to check finality for rpc

* grandpa-rpc: minor tidy

* grandpa-rpc: remove dyn FinalityProofProvider

* grandpa-rpc: remove unused dependencies

* node: move finality_proof_provider setup

* grandpa-rpc: print error reported by finality_proof_provider

* grandpa-rpc: add note about unnecessary encode/decode

* grandpa-rpc: dont encode/decode and use correct hash

* grandpa-rpc: set_id is optional

* grandpa-rpc: create test for prove_finality

* grandpa-rpc: set visibility back to how it was

* grandpa-rpc: remove unused dependency

* grandpa-rpc: minor tidy

* grandpa: doc strings

* grandpa-rpc: rename to prove_finality

* grandpa-rpc: use current set id if none is provided

* grandpa-rpc: remove unnecessary check in test

* node: group finality_proof_provider in rpc_setup

* grandpa: make prove_finality concrete in FinalityProofProvider

* grandpa-rpc: wrap finality output in struct and store as Bytes

* grandpa-rpc: exhaustive error codes and wrap

* grandpa-rpc: let prove_finality take a range instead of a starting point

* grandpa-rpc: fix test for changed API

* grandpa-rpc: fix line length

* grandpa: fix reviewer nits

* node/rpc: fix reviewer comments
2020-09-18 16:39:27 +00:00
André Silva 253f906a4d grandpa: log errors and grandpa unexpected conclusion (#6867) 2020-08-10 21:52:39 +00:00
Hernando Castano 433b7214f5 Add Subscription RPC for Grandpa Finality (#5732)
* Rough skeleton for what I think the RPC should look like

* Create channel for sending justifications

Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.

* WIP: Add subscribers for justifications to Grandpa

Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.

* WIP: Add a struct for managing subscriptions

Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.

* Make naming more clear and lock data in Arc

* Rough idea of what RPC would look like

* Remove code from previous approach

* Missed some things

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Split justification subscription into sender and receiver halves

* Replace RwLock with a Mutex

* Add sample usage from the Service's point of view

* Remove code that referred to "chain_" RPC

* Use the Justification sender/receivers from Grandpa LinkHalf

* Add some PubSub boilerplate

* Add guiding comments

* TMP: comment out to fix compilation

* Return MetaIoHandler from PubSubHandler in create_full

* Uncomment pubsub methods in rpc handler (fails to build)

* node/rpc: make Metadata concrete in create_full to fix compilation

* node: pass in SubscriptionManger to grandpa rpc handler

* grandpa-rpc: use SubscriptionManger to add subscriber

* grandpa-rpc: attempt at setting up the justification stream (fails to build)

* grandpa-rpc: fix compilation of connecting stream to sink

* grandpa-rpc: implement unsubscribe

* grandpa-rpc: update older tests

* grandpa-rpc: add full prefix to avoid confusing rust-analyzer

* grandpa-rpc: add test for pubsub not available

* grandpa-rpc: tidy up leftover code

* grandpa-rpc: add test for sub and unsub of justifications

* grandpa-rpc: minor stylistic changes

* grandpa-rpc: split unit test

* grandpa-rpc: minor stylistic changes in test

* grandpa-rpc: skip returning future when cancelling

* grandpa-rpc: reuse testing executor from sc-rpc

* grandpa-rpc: don't need to use PubSubHandler in tests

* node-rpc: use MetaIoHandler rather than PubSubHandler

* grandpa: log if getting header failed

* grandpa: move justification channel creation into factory function

* grandpa: make the justification sender optional

* grandpa: fix compilation warnings

* grandpa: move justification notification types to new file

* grandpa-rpc: move JustificationNotification to grandpa-rpc

* grandpa-rpc: move JustificationNotification to its own file

* grandpa: rename justification channel pairs

* grandpa: rename notifier types

* grandpa: pass justification as GrandpaJustification to the rpc module

* Move Metadata to sc-rpc-api

* grandpa-rpc: remove unsed error code

* grandpa: fix bug for checking if channel is closed before sendind

* grandpa-rpc: unit test for sending justifications

* grandpa-rpc: update comments for the pubsub test

* grandpa-rpc: update pubsub tests with more steps

* grandpa-rpc: fix pubsub test

* grandpa-rpc: minor indendation

* grandpa-rpc: decode instead of encode in test

* grandpa: fix review comments

* grandpa: remove unused serde dependency

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
2020-08-10 10:31:36 +00:00
Ashley 234e7d0c3d Simplify a few chain components creation APIs related to the service (#6611)
* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-07-09 15:43:04 +02:00
André Silva 4cf1b4fa7b grandpa: minor cleanups in communication module (#6371)
* grandpa: replace Result<(), ()> with Option<()>

* grandpa: replace &Option<T> with Option<&T>

* grandpa: cleanup local id and keystore usages

* grandpa: return bool on check_message_signature

* grandpa: fix erroneous log message on startup

* grandpa: fix test
2020-06-30 18:59:36 +01:00
Rakan Alhneiti 2577dde3d9 Use sign_with for signing grandpa's outgoing message (#6178)
* Use sign_with and stop using `Pair`

* PR feedback

* Remove clone

* Transfer ownership of public to sign_message

* Use Option

* Simplify code

* Fix error message

* Pass keystore as ref

* Pass keystore properly

* Fix tests
2020-06-09 07:39:30 +00:00
Pierre Krieger 717fa95bc7 Restore the empty line after the license (#6088) 2020-05-20 13:08:27 +00:00
Joshy Orndorff 1d8ddd7a98 Make GrandpaBlockImport public (#6068) 2020-05-18 20:13:30 +00:00
Nikolay Volf d5411969ac fix whitespace (#6062) 2020-05-18 13:51:28 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Jon Häggblad b99033368b Fix minor clippy lints in grandpa (#5988)
* grandpa: fix clippy lints about identity conversions

* grandpa: fix clippy lints about unwrap_or_default

* grandpa: fix clippy lints about explicit return

* grandpa: fix clippy lints about unnecessary intermediary

* grandpa: fix clippy lints about to_string

* grandpa: fix clippy lints about unused imports

* grandpa: fix clippy lints about increments

* grandpa: fix clippy lints about unnecessary matches

* grandpa: fix clippy lints about struct arguments

* Fix clippy::redundant_clone

* Fix clippy::clone_on_copy

* Fix clippy::or_fun_call

* Fix clippy::identity_conversion
2020-05-13 17:40:52 +01:00
André Silva fbd2ac8f3b grandpa: report equivocations (#3868)
* session: runtime api for generating session membership proofs

* grandpa: add runtime api for creating equivocation report txs

* grandpa: submit signed equivocation report transactions

* grandpa: use proper equivocation report type

* grandpa: report equivocations

* grandpa: validate equivocation proof

* grandpa: update to finality-grandpa 0.9.1

* grandpa: fix encoding of session membership proof

* grandpa: initialize set id session mapping for genesis session

* grandpa: fix bug in set_id session validation

* fix compilation

* cleanup from merge conflicts

* cleanup crate tomls

* grandpa: refactor equivocation handling to separate trait

* node-template: fix compilation

* fix test compilation

* bump finality-grandpa to v0.10.2

* rpc: fix runtime version test

* CHERRY-PICK #4200: Add documentation to SubmitSignedTransaction and actually make it work

Squashed commit of the following:

commit 4f2cb0b1c588a06f2f3b478bb4b28b5cb29d54b9
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 16:29:33 2019 +0100

    Split the method to avoid confusing type error message.

commit c5bf24eeaaf902add89ed1b046b22c4a4aaeb2cd
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 16:19:55 2019 +0100

    Make accounts optional, fix logic.

commit 97db1ef556e023cf6847e5ffdb036c0e3ea6fb0a
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 10:06:20 2019 +0100

    Remove warning.

commit 535f5c116d1a2e826eaf90c3f7e6798e443d61d8
Merge: 516257217 0f1a5f651
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 07:08:05 2019 +0100

    Merge branch 'master' into td-signed-transactions

commit 516257217bac89fcebd083712f4ea68b7b23b55a
Merge: ac98248c6 2e68c80c2
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Dec 2 13:57:25 2019 +0100

    Merge branch 'master' into td-signed-transactions

commit ac98248c6c56cff381130645a82a13d29933cf83
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:34:52 2019 +0100

    Forgotten import.

commit 67a3c19031506c28e31c6bc4a90fff62d467dd58
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:32:10 2019 +0100

    Fix naming and bounds.

commit 93e768ea9df97a4629fca1f9bc4b108fdb33f876
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:01:05 2019 +0100

    Add documentation to signed transactions and actually make them work.

* grandpa: skip block initialization on report submission method

* primitives: allow transaction pool access by default for offchain calls

* grandpa: unused parameters

* grandpa: remove unused method

* grandpa: enable equivocation reporting

* grandpa: add workaround for parameter encoding

* grandpa: fix localized_payload calls in tests

* fix submit_report_equivocation_extrinsic in runtimes

* node: fix submit transaction test compilation

* node: bump spec_version

* rpc: fix api version test

* grandpa: allow custom equivocation offence type

* grandpa: add test for authorities::next_change_height

* grandpa: cleanup report_equivocation function

* node: move reporting app crypto to node-primitives

* grandpa: move equivocation traits to own module

* grandpa: rename app-crypto crate import

* grandpa: export equivocation types

* node: bump spec_version

* grandpa: rename EquivocationReport to EquivocationProof

* grandpa: add missing docs to primitives

* grandpa: add missing docs to equivocation

* node: fix compilation

* grandpa: add missing docs to pallet

* node: bump spec_version

* fix whitespace

* grandpa: return error on offence reporting

* grandpa: expose session and validator count in proofs through traits

* grandpa: use strong key in module KeyOwnerProofSystem

* grandpa: move key ownership proof to grandpa runtime api

* grandpa: remove unnecessary cloning when checking equivocation proof

* grandpa: make report_equivocation a method in Environment

* support: implement KeyOwnerProofSystem for ()

* grandpa: move KeyOwnerProofSystem to module trait

* test-utils: fix runtime compilation

* grandpa: fix test compilation

* grandpa: fix test compilation after merge

* grandpa: simplify transaction submission types

* grandpa: validate equivocation report in signed extension

* client: fix test

* node: use ValidateEquivocationReport signed extension

* grandpa: expose key ownership proof under opaque type

* grandpa: better docs on key ownership proofs

* grandpa: add note about signed extension

* grandpa: add ValidateEquivocationReport::new

* grandpa: remove skip_initialize_block from runtime api

* grandpa: use new offchain transaction submission API

* grandpa: take set_id in generate_key_ownership_proof

* grandpa: update to finality-grandpa v0.12.2

* grandpa: cleanup usages of AuthoritySet::current

* grandpa: fix test

* grandpa: add mocking utilities for equivocation reporting

* grandpa: add test for equivocation reporting

* grandpa: move SetIdSession initialization

* grandpa: add more tests

* node: enable historical session manager

* node: bump spec_version

* node: use strong key types in KeyOwnerProofSystem definitions

* grandpa: export GrandpaEquivocationOffence type
2020-05-06 17:25:51 +01:00
Jon Häggblad 1f7f8abb33 Expose GRANDPA round state through RPC (#5375)
* grandpa: wire up basic RPC call

* grandpa: make it compile against GRANDPA with expose round state

* grandpa: use shared voter state to expose RPC endpoint

* grandpa: restructure into nested structs

* grandpa: return background rounds too

* grandpa: return error when endpoint not ready

* grandpa: collect grandpa rpc deps

* grandpa: decide to use concrete AuthorityId in finality-grandpa-rpc

* grandpa: remove unncessary type annotation

* grandpa: move error code to const

* grandpa: remove unnecessary WIP comment

* grandpa: remove Id type parameter for SharedVoterState

* grandpa: update tests to add shared_voter_state in parameters

* grandpa: remove old deprecated test

* grandpa: fix getting the correct set_id

* grandpa: make SharedVoterState a struct

* grandpa: wrap shared_voter_state in rpc_setup

* grandpa: replace spaces with tabs

* grandpa: limit RwLock write attempt to 1 sec

* grandpa: add missing doc comments and remove some pub

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-Authored-By: Hernando Castano <HCastano@users.noreply.github.com>

* grandpa: update function name call after change in finality-grandpa

* grandpa: group pub use and only export voter::report

* grandpa: add missing docs

* grandpa: extract out structs used for json serialization

* grandpa: stick to u32 for fields intended for js

* grandpa: move Error type to its own file

* grandpa: group pub use better

* Apply code review suggestion

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* grandpa: use correct version of finality-granpda in rpc crate

* grandpa: add back basic rpc unit test

* grandpa: replace SharedVoterState::new() with empty()

* node: cleanup grandpa::SharedVoterState usage in macro

* grandpa: remove VoterState error variant

* grandpa: enable missing futures compat feature

* grandpa: fix typo in error variant

* grandpa: remove test_utils

* grandpa: allow mocking rpc handler components

* grandpa: rename serialized to report in rpc module

* grandpa: add proper test for RPC

* grandpa: update to finality-grandpa v0.12.1

Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Demi Obenour <demi@parity.io>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-05-04 20:37:22 +01:00
André Silva d2967ba4b6 grandpa: update to finality-grandpa v0.12.0 (#5853)
* grandpa: update to v0.12.0

* grandpa: fix tests

* grandpa: better validation of authority set invariants

* grandpa: avoid duplicating invalid authority list check

* grandpa: add missing doc

* grandpa: better validation of expect proofs

* grandpa: fix test compilation

* grandpa: fix tests

* grandpa: add test for AuthoritySet invariants

* grandpa: bump finality-grandpa to v0.12.0
2020-04-30 20:03:30 +00:00
André Silva 1a9c0fee30 babe, grandpa: restrict info logging during initial sync (#5564)
* babe: restrict info logging during initial sync

* grandpa: restrict info logging during initial sync

* grandpa: fix test compilation

* grandpa: remove afg_log macro mod
2020-04-07 22:11:22 +02:00
Benjamin Kampmann 247822bb33 Additional Metrics collected and exposed via prometheus (#5414)
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-04-04 15:13:35 +02:00
Max Inden 091335780e client/finality-grandpa: Instrument until-imported queue (#5438)
* client/finality-grandpa: Instrument until-imported queue

The `UntilImported` queue takes as input finality grandpa messages that
depend on blocks that are not yet imported and holds them back until
those blocks are imported.

This commit adds a basic metric, the amount of messages waiting in the
queue, to the module. For now this metric is only available for the
global `UntilImported` queue awaiting blocks for commit and catch-up
messages.

* client/finality-grandpa/src/until_imported: Update metric help text

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-03-31 20:17:14 +02:00
André Silva c7a23d836d grandpa: support for hard forking any pending standard changes (#5306)
* grandpa: support for hard forking any pending standard changes

* grandpa: expose authority_set_hard_forks in block import constructor

* grandpa: don't break the public api
2020-03-19 10:37:29 +00:00
Max Inden 79fc16e4d7 client/finality-grandpa: Add Prometheus metrics to GossipValidator (#5237)
* client/finality-grandpa: Add Prometheus metrics to GossipValidator

Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.

```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```

* client/finality-grandpa: Add None as Prometheus registry in tests

* client/finality-granpda/src/communication: Refactor metric registration
2020-03-14 12:51:44 +01:00
Benjamin Kampmann 99ae5342eb removes use of sc_client::Client from sc-rpc (#5063)
* removes use of sc_client::Client from sc-rpc

* remove Client impl from sc-finality-benches

* remove client impl from sc-finality-grandpa

* read_proof accepts iterator

* remove generic Executor param from ExecutorProvider

* fix long ass line

* code style changes

* merge with master

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2020-03-05 16:41:10 +01:00
Gavin Wood ab58a08a53 remove on_exit from grandpa (#5098) 2020-03-05 13:38:16 +01:00
Ashley 883ddfc897 Add more metrics to prometheus (#5034)
* Add a few things

* Add finality_grandpa_round

* fix fg tests

* Nitpicks

* Nitpicks

* Fix name of prometheus crate
2020-03-03 11:36:58 +01:00
Seun Lanlege 5907b0d902 removes use of sc_client::Client from sc_finality_grandpa (#5030)
* removes use of sc_client::Client from sc_finality_grandpa

* code formatting

* code formatting

* removes use of sc_client::Client from sc_finality_grandpa
2020-02-27 12:19:07 +01:00
Pierre Krieger 7a04055814 Open one substream for each notifications protocol (#4909)
* Open one substream for each notifications protocol

* Fix WASM build

* Apply suggestions from code review

Co-Authored-By: Toralf Wittner <tw@dtex.org>

* Address concerns

* Use unsigned-varint to read the varint

* Use unsigned-varint

* Forgot Cargo.lock

Co-authored-by: Toralf Wittner <tw@dtex.org>
2020-02-21 11:06:24 +01:00
André Silva f0043055cc node, node-template: disable GRANDPA observer (#5011)
* node-template: disable grandpa observer

* node: disable grandpa observer

* node: add doc about grandpa-voter infallible task

* grandpa: remove grandpa observer from public API

* grandpa: ignore observer_enabled field in config
2020-02-21 10:50:42 +01:00
Stanislav Tkach ba2362dadd [In Progress] Remove deprecated api (#4973)
* Remove deprecated api

* Revert changes to wasm-build-runner
2020-02-18 23:25:56 +01:00
Max Inden 3f3910ccaf client/network-gossip: Integrate GossipEngine tasks into Future impl (#4767)
`GossipEngine` spawns two tasks, one for a periodic tick, one to forward
messages from the network to subscribers. These tasks hold an `Arc` to a
`GossipEngineInner`.

To reduce the amount of shared ownership (locking) this patch integrates
the two tasks into a `Future` implementation on the `GossipEngine`
struct. This `Future` implementation can now be called from a single
owner, e.g. the `finality-grandpa` `NetworkBridge`.

As a side effect this removes the requirement on the `network-gossip`
crate to spawn tasks and thereby removes the requirement on the
`finality-grandpa` crate to spawn any tasks.

This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa`, `network` and `network-gossip` as
well as to reduce the amount of unbounded channels. For details see
d9837d7dd, 5f80929dc and 597c0a6c4.
2020-02-12 13:15:26 +01:00
Tomasz Drwięga fd03f4e950 Fix warnings. (#4768) 2020-01-29 21:37:26 +01:00
Ashley c2c429877e Switch GrandPa to std futures (replaces #3909) (#4612)
* Switch GrandPa to new futures

* Work on making tests work

* until_imported tests working again

* Work on switching tests to stable futures

* Modifications

* Re-add test as #[ignore]

* Don't ignore

* Add manual unpins

* Remove Header import

* Return concrete Sink type

* Switch to crates.io finality-grandpa version

* Remove use statement that slipped in

* Fix some nitpicks

* Remove unpin from i

* Fixed typo

* Move futures01 to dev-deps

* Fix nitpicks

* Update client/finality-grandpa/src/communication/mod.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* nitpicking

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <andre.beat@gmail.com>
2020-01-24 13:34:42 +01:00
Max Inden 5f80929dce client/finality-grandpa: Reintegrate gossip validator report stream (#4661)
* client/finality-grandpa: Reintegrate gossip validator report stream

The `finality-grandpa` `GossipValidator` is called by the `GossipEngine`
in a synchronous fashion on each gossip message. Its main task is to
decide whether to gossip the given message on, or whether to drop it.

In addition it also updates the reputation of a node's peers based on
the incoming gossip messages. To do so it needs to be able to report the
reputation change which it does through an unbounded channel (in order
to stay synchronous).

Previously the receiving side of this channel would be handled by a new
task, polling the channel and forwarding the changes to a clone of the
`GossipEngine` that it would own.

Instead the receiver of the above mentioned channel is now being polled
by the `NetworkBridge` within its `Future::poll` implementation.
Reputation changes are reported through the already existing
`GossipEngine` instance within `NetworkBridge`.

For details on the overall goal, see d9837d7dd.

* client/finality-grandpa: Remove exit future from test NetworkBridges
2020-01-17 18:28:32 +01:00
Max Inden d9837d7dda client/finality-grandpa: Reintegrate periodic neighbor packet worker (#4631)
The `NeighborPacketWorker` within `client/finality-grandpa` does two
things:

1. It receives neighbor packets from components within
`client/finality-grandpa`, sends them down to the `GossipEngine` in
order for neighboring nodes to receive.

2. It periodically sends out the most recent neighbor packet to the
`GossipEngine`.

In order to send out packets it had a clone to a `GossipEgine` within
an atomic reference counter and a mutex. The `NeighborPacketWorker` was
then spawned onto its own asynchronous task.

Instead of running in its own task, this patch reintegrates the
`NeighborPacketWorker` into the main `client/finality-grandpa` task not
requiring the `NeighborPacketWorker` to own a clone of the
`GossipEngine`.

The greater picture

This is a tiny change within a greater refactoring. The overall goal is
to **simplify** how finality-grandpa interacts with the network and to
**reduce** the amount of **unbounded channels** within the logic.

Why no unbounded channels: Bounding channels is needed for backpressure
and proper scheduling. With unbounded channels there is no way of
telling the producer side to slow down for the consumer side to catch
up.  Rephrased, there is no way for the scheduler to know when to favour
the consumer task over the producer task on a crowded channel and the
other way round for an empty channel.

Reducing the amount of shared ownership simplifies the logic and enables
one to use async-await syntax-suggar, given that one does not need to
hold a lock across poll invocations. Using async-await enables one to
use bounded channels without complex logic.
2020-01-17 13:58:25 +01:00
André Silva d18fbd7940 grandpa: generic voting rule for backing off from best block (#4635)
* grandpa: generic voting rule for backing off from best block

* grandpa: fix tests
2020-01-16 14:53:45 +01:00
Bastian Köcher afc3318f21 Implement runtime version checks in set_code (#4548)
* Implement runtime version checks in `set_code`

Check that the new runtime code given to `set_code` fullfills some
requirements:

- `spec_name` matches
- `spec_version` does not decreases
- `impl_version` does not decreases
- Either `spec_version` and `impl_version` increase

* Make tests almost work

* Some fixes after master merge

* Fix tests

* Add missed file

* Make depedency check happy?

* Remove leftover `sc-executor`

* AHHHHH

* Reset debug stuff

* Remove some 'static

* More 'static

* Some docs

* Update `Cargo.lock`
2020-01-16 13:58:37 +01:00
Ashley 3219be2508 Update the service to std futures (#4447)
* Switch service to futures03

* Fix tests

* Fix service test and cli

* Re-add Executor trait to SpawnTaskHandle

* Fix node-service

* Update babe

* Fix browser node

* Update aura

* Revert back to tokio-executor to fix runtime panic

* Add todo item

* Fix service tests again

* Timeout test futures

* Fix tests

* nits

* Fix service test

* Remove zstd patch

* Re-add futures01 to aura and babe tests as a dev-dep

* Change failing test to tee

* Fix node

* Upgrade tokio

* fix society

* Start switching grandpa to stable futures

* Revert "Start switching grandpa to stable futures"

This reverts commit 9c1976346237637effc07c13f7d0403daf5e71cf.

* Fix utils

* Revert substrate service test

* Revert gitlab

Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
2020-01-14 15:43:45 +01:00
Bastian Köcher fd6b29dd2c Remove requirement on Hash = H256, make Proposer return StorageChanges and Proof (#3860)
* Extend `Proposer` to optionally generate a proof of the proposal

* Something

* Refactor sr-api to not depend on client anymore

* Fix benches

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Apply suggestions from code review

* Introduce new `into_storage_changes` function

* Switch to runtime api for `execute_block` and don't require `H256`
anywhere in the code

* Put the `StorageChanges` into the `Proposal`

* Move the runtime api error to its own trait

* Adds `StorageTransactionCache` to the runtime api

This requires that we add `type NodeBlock = ` to the
`impl_runtime_apis!` macro to work around some bugs in rustc :(

* Remove `type NodeBlock` and switch to a "better" hack

* Start using the transaction cache from the runtime api

* Make it compile

* Move `InMemory` to its own file

* Make all tests work again

* Return block, storage_changes and proof from Blockbuilder::bake()

* Make sure that we use/set `storage_changes` when possible

* Add test

* Fix deadlock

* Remove accidentally added folders

* Introduce `RecordProof` as argument type to be more explicit

* Update client/src/client.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update primitives/state-machine/src/ext.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Integrates review feedback

* Remove `unsafe` usage

* Update client/block-builder/src/lib.rs

Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>

* Update client/src/call_executor.rs

* Bump versions

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
2020-01-10 10:48:32 +01:00
Shawn Tabrizi 7efa62725e Update copyright year (#4532) 2020-01-05 14:05:44 +01:00
Nikolay Volf 6d06a19f41 Refactor to use only chain info (#4516) 2020-01-02 12:46:07 +01:00
Robert Habermeier 8f1b5b840d client: Do not set fork sync request via network-gossip (#4439)
The finality-grandpa module needs two sets of functionalities from the
network:

1. Everything gossip related, e.g. event_stream, write_notification, ...

2. The ability to set a fork sync request for a specific block hash.

Instead of embedding (2) inside of (1) this patch extracts (2) from (1)
having finality-grandpa depend on a `Network` that fulfills the
`network_gossip::Network` trait and that can set block sync requests.

On the one hand this improves the overall structure splitting things
that don't logically belong together. On the other hand it does
reintroduce a lot of trait bounds within finality-grandpa.
2019-12-18 14:17:13 +01:00
Tomasz Drwięga 8778ca7dc8 [big refactor] Remove crate aliasing. (#4395)
* Rename: Phase 1.

* Unify codec.

* Fixing: Phase 2

* Fixing: Phase 3.

* Fixing: Phase 4.

* Fixing: Phase 5.

* Fixing: Phase 6.

* Fixing: Phase 7.

* Fixing: Phase 8. Tests

* Fixing: Phase 9. Tests!!!

* Fixing: Phase 10. Moar tests!

* Finally done!

* More fixes.

* Rename primitives:: to sp_core::

* Apply renames in finality-grandpa.

* Fix benches.

* Fix benches 2.

* Revert node-template.

* Fix frame-system in our modules.
2019-12-16 20:36:49 +08:00
Pierre Krieger c66c191b68 Extract consensus_gossip.rs and put it in its own crate (#4284)
* Extract gossiping system from network

* Finish porting GRANDPA tests

* Try put correct engine ID

* Fix messages encoding

* Fix communication tests

* Use a threads pool to spawn stuff

* Fix compilation everywhere

* Fix bad merge conflict

* Remove dependency on async-std

* Apply suggestions from code review

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* More suggestions

* Remove network startup GP future

* Update to futures_timer

* adjust wait_when_behind test

* Pass correct Roles after handshake

* Revert "adjust wait_when_behind test"

This reverts commit 23cb3a0a6d25ed732c2cd648607bc44ef2ab0919.

* Crate root documentation

* Remove MessageRecipient

* Address concerns

* Fix more concerns

* Forgot Cargo.lock
2019-12-13 19:16:10 +01:00
Ashley a58e8a6e45 Update exit-future and make sc-cli compile on wasm (#4289)
* updated exit-future (github repo)

* Switch to broadcast crate

* Migrate client/cli

* Switch exit-future to modernize branch

* Small changes

* Switch to cargo version and fix fg tests

* Revert "Small changes"

This reverts commit a488106805d220cb4aee9e46a71481424c6d87d5.
2019-12-04 18:35:33 +01:00
Benjamin Kampmann 927e13c13a The crate rename (#4223)
* Adding script for rename, could be applicable for nodes on top of it, too

* add stderr and gitlab ci features

* apply script

* fix now minor details in expected stderr

* Update the Cargo.lock

* fix name: sc-transaction -> sc-tracing

* fix rename in script, too
2019-12-02 11:23:53 +01:00
Benjamin Kampmann bd652793db Remove all (non-dev) client references from frame, activate dependency enforcer (#4184)
* Move transaction pool to primitives

* move backend, errors into primitives

* remove unused client depencies

* Move rpc-api into primitives

* Move peerset back to client

* Move rpc/api back to client, move palette/support/rpc into utils

* move support-rpc into subfolder

* move system-rpc into utils

* move transaction-pool  and -graph back into client

* fix broken imports

* Clean up test primitives

* Make support test utils independent of frame

* remove unnecessary node dependencies from service

* Reactivate dependency script:
 - only enforce the now achieved status quo will remain
 - allow for primitives to depend on /client for now without failing
 - more discriptive error message so people understand, what it wants
 - minor fix to differentiative between ../client and /client (which may be a subfolder)
 - don't allow this to fail anylonger.

* fix doc comment

* 'Should not' rather than 'must not'.

* Revert unwanted dependency changes

* fix faulty import

* fixup derive_more version

* fix wrong import path
2019-11-26 21:38:28 +01:00
Robert Habermeier 5a8146d64e Updating GRANDPA to v0.10.0 (#4174)
* update finality-grandpa to v0.10.0 rev

* add concluded implementation to environment

* also update state when concluding

* switch to the cargo version

* address comment grumbles
2019-11-22 13:18:31 +01:00