Commit Graph

84 Commits

Author SHA1 Message Date
Nazar Mokrynskyi 5896072b86 Network sync refactoring (part 4) (#11412)
* Remove direct dependency of `sc-network` on `sc-network-light`

* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`

* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`

* Create `sync` module in `sc-network-common`, create `ChainSync` trait there (not used yet), move a bunch of associated data structures from `sc-network-sync`

* Switch from concrete implementation to `ChainSync` trait from `sc-network-common`

* Introduce `OpaqueStateRequest`/`OpaqueStateResponse` to remove generics from `StateSync` trait

* Introduce `OpaqueBlockRequest`/`OpaqueBlockResponse`, make `scheme` module of `sc-network-sync` private

* Surface `sc-network-sync` into `sc-service` and make `sc-network` not depend on it anymore

* Remove now unnecessary dependency from `sc-network`

* Replace crate links with just text since dependencies are gone now

* Remove `warp_sync` re-export from `sc-network-common`

* Update copyright in network-related files

* Address review comments about documentation

* Apply review suggestion

* Rename `extra_requests` module to `metrics`

Co-authored-by: Bastian Köcher <info@kchr.de>
2022-07-12 20:34:17 +00:00
André Silva 06cf8ad076 grandpa: fix creation of justification with equivocating precommits in commit (#11302)
* grandpa: fix creation of justification ancestry

we would reject commits that have precommits targeting blocks lower than the
commit target. when there is an equivocation (or if it the commit is not
minimal) it is possible to have such precommits and we should assume that they
are the round base.

* grandpa: bump to 0.16.0

* grandpa: add test for justification with equivocation

* grandpa: fix failing test
2022-06-14 10:41:11 +00:00
Davide Galassi 2bdebbcb00 Grandpa revert procedure (#11162)
* Grandpa revert procedure

* Trigger ci pipeline

* Test rename

* Update client/finality-grandpa/src/authorities.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2022-04-06 16:23:57 +02:00
Adrian Catangiu c1865988df Name changes for GrandPa and Beefy notifications protocols (#10463)
* grandpa: update notif protocol name

* grandpa: add chain id prefix to protocol name

* grandpa: beautify protocol name handling

* grandpa: prepend genesis hash to protocol name

* chain-spec: add optional 'fork_id'

'fork_id' is used to uniquely identify forks of the same chain/network
'ChainSpec' trait provides default 'None' implementation, meaning this
chain hasn't been forked.

* grandpa: protocol_name mod instead of struct

* beefy: add genesis hash prefix to protocol name

* chainspec: add fork_id

* grandpa: simplify protocol name

* grandpa: contain protocol name building logic

* beefy: contain protocol name building logic

* grandpa: fix tests

* fix merge damage

* fix docs reference visibility

Signed-off-by: acatangiu <adrian@parity.io>

* Update client/finality-grandpa/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/finality-grandpa/src/communication/mod.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/beefy/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/beefy/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* avoid using hash default, even for protocol names

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2022-01-05 17:01:44 +00:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
Gavin Wood 1e24e45ea1 Remove Default bound for AccountId (#10403)
* Remove Default for AccountId

* More removals of default

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/authorship/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* More work

* More work

* Remove old code

* More work

* pallet-asset-tx-payment

* tips

* sc-consensus-babe

* sc-finality-grandpa

* sc-consensus-babe-rpc

* sc-cli

* make npos crates accept non-default account (#10420)

* minimal changes to make npos pallets all work

* make this pesky reduce.rs a bit cleaner

* more work

* more work

* Tests build

* Fix imonline tests

* Formatting

* Fixes

* Fixes

* Fix bench

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Fixes

* Formatting

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/finality-grandpa/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/keystore/src/local.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/staking/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update frame/staking/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/runtime/src/traits.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Formatting

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>
2021-12-13 15:03:59 +01:00
Arkadiy Paronyan 4cbbf0cf43 Kill the light client, CHTs and change tries. (#10080)
* Remove light client, change tries and CHTs

* Update tests

* fmt

* Restore changes_root

* Fixed benches

* Cargo fmt

* fmt

* fmt
2021-11-12 14:15:01 +01:00
Bastian Köcher f92e5c5e65 Upgrade tokio to 1.10 (#9575)
* Upgrade tokio to 1.10

* Fix test runner

* Try fix it

* Update Cargo.lock

* Review feedback

* ahhhh

* FML

* FMT

* Fix tests
2021-08-24 14:31:19 +00:00
Arkadiy Paronyan f56232602f Warp sync part I (#9227)
* Started warp sync

* BABE & GRANDPA recovery

* Warp sync protocol

* Sync warp proofs first

* Added basic documentation

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Style changes

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* fmt

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fixed chage trie pruning wrt missing blocks

* Restore parent finalization

* fmt

* fmt

* Revert pwasm-utils bump

* Change error type & check API version

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Build fix

* Fixed target block check

* Formatting

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-08-02 08:27:18 +00:00
Squirrel df59596ec0 Move client consensus parts out of primitives and into client/consensus/api (#9319)
* moved client code out of primitives

* bump ci

* Fixup from merge.

* Removed unused deps thanks to review feedback

* Removing unneeded deps

* updating lock file

* note about rustfmt

* fixed typo to bump ci

* Move lonely CacheKeyId to parent

* cargo fmt

* updating import style

* Update docs/STYLE_GUIDE.md

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-07-30 13:27:17 +00:00
Squirrel aafe64315a Remove extra commas made redundent after rustfmt (#9404)
* Remove extra commas made redundent after rustfmt
2021-07-22 11:06:17 +01:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
André Silva ae5cd339b5 grandpa: cleanup sync bounds (#9127)
* grandpa: cleanup sync bounds

* grandpa: cleanup imports

* remove cargo patch
2021-06-16 12:51:09 +00:00
André Silva 61507e1ba3 grandpa: enable light clients to participate in gossip (#8796)
* network: allow gossiping to light clients

* grandpa: gossip global messages to light clients

* grandpa: don't send neighbor packets to light clients

* grandpa: fix tests

* grandpa: export run_grandpa_observer

* node: run grandpa observer on light client

* node: start network at end

* Use wasm_timer in finality-grandpa

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2021-05-26 11:33:11 +00:00
André Silva 34458e9b9c grandpa: always store justification for the latest finalized block (#8392)
* grandpa: always store justification for best finalized block

* grandpa-warp-sync: add latest justification when finished proving

* grandpa-warp-sync: change logic for sending best justification when finished

* grandpa: test storing best justification

* grandpa: reorder variants in WarpSyncFinished
2021-04-01 09:42:56 +00:00
Bastian Köcher 217c4be226 Make BlockImport and Verifier async (#8472)
* Make grandpa work

* Introduce `SharedData`

* Add test and fix bugs

* Switch to `SharedData`

* Make grandpa tests working

* More Babe work

* Make it async

* Fix fix

* Use `async_trait` in sc-consensus-slots

This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.

* Make grandpa tests compile

* More Babe tests work

* Fix network test

* Start fixing service test

* Finish service-test

* Fix sc-consensus-aura

* Fix fix fix

* More fixes

* Make everything compile *yeah*

* Fix build when we have Rust 1.51

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fix warning

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-03-30 09:19:49 +00:00
André Silva ee8e7f1bcd grandpa: speed up tests (#8439)
* grandpa: tests: add peers with authority role

* grandpa: tests: manually wake-up poll_fn future
2021-03-24 09:56:25 +01:00
Jon Häggblad 0d6884b919 Storing multiple Justifications per block (#7640)
* primitives/runtime: initial changes on supporting multiple Justifications

* primitives/runtime: make Justifications strongly typed

* Encode/decode Justifications

* primitives/runtime: add Justification type

* backend: apply_finality and finalize_block takes a single Justification

* manual-seal: create engine id and let rpc take encoded justification

* backend: skeleton functions for appending justifications

* backend: initial implementation append_justification

Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k

* backend: guard against duplicate consensus engine id

* client/db: add check for block finality

* client/api: add append_justification to in_mem db

* client/light: add no-op append_justification

* network: fix decode call for Justification

* network: only send a single Justification in BlockData

* network: minor comment update

* protocol: update field names to distinguish single justification

* client: further field renames to plural

* client: update function names to plural justifications

* client/db: upgrade existing database for new format

* network: remove dependency on grandpa crate

* db: fix check for finalized block

* grandpa: check for multiple grandpa justifications hwne importing

* backend: update Finalizer trait to take multiple Justifications

* db: remove debugging statements in migration code

* manual-seal: update note about engine id

* db: fix check for finalized block

* client: update variable name to reflect it is now plural

* grandpa: fix incorrect empty Justications in test

* primitives: make Justifications opaque to avoid being empty

* network: fix detecting empty Justification

* runtime: doc strings for Justifications functions

* runtime: add into_justifications

* primitives: check for duplicates in when adding to Justifications

* network/test: use real grandpa engine id in test

* client: fix reviewer comments

* primitives: rename Justifications::push to append

* backend: revert changes to Finalizer trait

* backend: revert mark_finalized

* backend: revert changes to finalize_block

* backend: revert finalized_blocks

* db: add a quick early return for performance

* client: minor reviewer comments

* service/test: use local ConsensusEngineId

* network: add link to issue for sending multiple Justifications

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* network: tweaks to review suggestions

* network: revert change to BlockData for backwards compatibility

* Apply suggestion from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* primitives: update doc comment for Justifications

* client/db/upgrade: avoid grandpa crate dependency

* consensus: revert to single Justification for import_justification

* primitives: improve justifications docs

* style cleanups

* use and_then

* client: rename JUSTIFICATIONS db column

* network: revert to using FRNK in network-test

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-03-17 21:18:16 +00:00
Cecile Tonglet 8031b6eacb Rework telemetry to replace the use of tracing with an object we pass around (#8143)
polkadot companion: paritytech/polkadot#2535
2021-03-11 11:05:45 +01:00
André Silva 24739d1ab0 grandpa: make the VotingRule API async (#8101)
* grandpa: make the VotingRule api async

* grandpa: add docs to VotingRuleResult

* grandpa: formatting

* grandpa: use async blocks

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* grandpa: expose VotingRuleResult

* grandpa: revert some broken changes to async syntax

* grandpa: use finality-grandpa v0.14.0

* grandpa: bump impl_version

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-02-15 18:28:04 +00:00
Bastian Köcher 33f9becf41 Simplify runtime api error handling (#8114)
* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...
2021-02-15 11:55:40 +00:00
Jon Häggblad 20f40fbd12 Cleaner GRANDPA RPC API for proving finality (#7339)
* grandpa: persist block number for last block of authority set

* grandpa: fix authority_set_changes field in tests

* grandpa: fix date on copyright notice

* grandpa-rpc: implement cleaner api for prove finality rpc

* grandpa-rpc: replace the old prove_finality with the new one

* grandpa: undo accidental whitespace change

* grandpa-rpc: start work on redo of the finality_proof RPC API

* grandpa: manual impl of Decode for AuthoritySet

* grandpa: add comment about appending changes for forced changes

* grandpa: flip order in set changes, tidy up some comments

* grandpa: update some of the doc comments

* grandpa: store authority set changes when applying forced changes

* grandpa: simplify finality_proof.rs

* grandpa: move checks and extend tests in finality_proof

* grandpa: address first set of review comments

* grandpa: check that set changes have well-defined start

* grandpa: rework prove_finality and assocated tests

* grandpa: make AuthoritySetChanges tuple struct

* grandpa: add assertions for tracking auth set changes

* grandpa: remove StorageAndProofProvider trait

* grandpa: return more informative results for unexpected input to RPC

* grandpa: tiny tweak to error msg

* grandpa: fix tests

* grandpa: add error specific to finality_proof

* grandpa: fix review comments

* grandpa: proper migration to new AuthoritySet

* grandpa: fix long lines

* grandpa: fix unused warning after merge

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-01-21 22:06:40 +00:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
André Silva e367d57baf Cleanup some warnings (#7816)
* client: cleanup redundant semicolon warnings

* grandpa: remove usage of deprecated compare_and_swap
2021-01-03 23:29:46 +00:00
Pierre Krieger 610585d17f Remove sc_network::NetworkService::register_notifications_protocol and partially refactor Grandpa tests (#7646)
* Remove sc_network::NetworkService::register_notifications_protocol

* Missing calls to .into()

* Wrong crate name

* [WIP] Fix Grandpa tests

* One more passing

* One more. Two to go.

* This one was actually already passing 🎉

* Last one compiles

* Progress

* grandpa: fix voter_persists_its_votes test

* Restore other tests

* Try spawn future later

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2020-12-02 16:35:48 +00:00
André Silva 1871a95088 grandpa: remove light-client specific block import pipeline (#7546)
* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs
2020-11-23 14:28:55 +00:00
Pierre Krieger 1eae9f5792 Remove necessity to pass ConsensusEngineId when registering notifications protocol (#7549)
* Remove necessity to pass ConsensusEngineId when registering notifications protocol

* Line width

* Fix tests protocol name

* Other renames

* Doc update

* Change issue in TODO
2020-11-18 15:05:35 +00:00
André Silva 47815a5936 grandpa: store the authority id that is used for voting per round (#7454)
* grandpa: store the authority id that is used for voting per round

* grandpa: fix tests
2020-11-05 14:36:35 +00:00
André Silva f2925f96a2 grandpa: don't send equivocation reports for local identities (#7372)
* grandpa: don't send equivocation reports for local identities

* grandpa: add test for self-report

* grandpa: fix test compilation

this works on rust nightly but breaks on ci which is using rust stable
2020-10-26 12:47:39 +00:00
André Silva f3ea1624f6 frame: remove finality-tracker (#7228)
* frame: remove finality-tracker

* node: remove unused parameter types

* node: bump spec_version
2020-10-15 20:58:51 +00:00
Rakan Alhneiti 3aa4bfacfc Async keystore + Authority-Discovery async/await (#7000)
* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-10-08 20:56:35 +00:00
Wei Tang 9fdd4672b0 Use tracing-based subscriber logging (#6825)
* init_logger: switch from log-based to tracing-based and add compatibility layer

* Move tracing profiling subscriber related config realization

* sp-tracing: change profiling to be a layer instead of a subscriber

* Enable profiling layer in cli

* Change all test env_logger init to sp_tracing::try_init_simple

* Remove all local env_logger dependency

* Add missing tracing-subscriber dependency

* frame-sudo: fix tests

* frame-support: fix tests

* Fix frame/pallet and executor tests

* Fix the remaining tests

* Use subscriber's try_init as recommended by @davidbarsky

* Be explict that the tracing-log feature is needed

* Set subscriber writer to stderr

* Shorter line width

* Update cargo lock tracing version

* Fix sc_tracing crate compile

* Fix sc_authority_discovery crate test

* unremove default-features

* Leave enabled to default true

* Warn if global default cannot be set

* Fix unused import

* Remove unused PROXY_TARGET

* Change all reference from rc5 to rc6

* Change all reference of rc2 to rc6

* Fix styling

* Fix typo

* make logger init error'ing

* re-fixing the test issue

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-09-17 11:04:43 +02:00
Pierre Krieger 34980ec88a Stop sending messages on legacy substream altogether (#6975)
* Stop sending messages on legacy substream altogether

* Ensure that handshake is sent back even in case of back-pressure

* Update client/network/src/protocol/generic_proto/handler/group.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Also process OpenRequest and Closed

* Also process OpenRequest and Closed

* Fix bad merge

* God I'm so lost with all these merges

* Immediately return Closed

* Add warning for sending on non-registered protocol

* Register GrandPa protocol in tests

* Update client/network/src/protocol/generic_proto/handler/group.rs

Co-authored-by: Max Inden <mail@max-inden.de>

Co-authored-by: Max Inden <mail@max-inden.de>
2020-09-02 15:28:03 +00:00
Hernando Castano 433b7214f5 Add Subscription RPC for Grandpa Finality (#5732)
* Rough skeleton for what I think the RPC should look like

* Create channel for sending justifications

Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.

* WIP: Add subscribers for justifications to Grandpa

Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.

* WIP: Add a struct for managing subscriptions

Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.

* Make naming more clear and lock data in Arc

* Rough idea of what RPC would look like

* Remove code from previous approach

* Missed some things

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Split justification subscription into sender and receiver halves

* Replace RwLock with a Mutex

* Add sample usage from the Service's point of view

* Remove code that referred to "chain_" RPC

* Use the Justification sender/receivers from Grandpa LinkHalf

* Add some PubSub boilerplate

* Add guiding comments

* TMP: comment out to fix compilation

* Return MetaIoHandler from PubSubHandler in create_full

* Uncomment pubsub methods in rpc handler (fails to build)

* node/rpc: make Metadata concrete in create_full to fix compilation

* node: pass in SubscriptionManger to grandpa rpc handler

* grandpa-rpc: use SubscriptionManger to add subscriber

* grandpa-rpc: attempt at setting up the justification stream (fails to build)

* grandpa-rpc: fix compilation of connecting stream to sink

* grandpa-rpc: implement unsubscribe

* grandpa-rpc: update older tests

* grandpa-rpc: add full prefix to avoid confusing rust-analyzer

* grandpa-rpc: add test for pubsub not available

* grandpa-rpc: tidy up leftover code

* grandpa-rpc: add test for sub and unsub of justifications

* grandpa-rpc: minor stylistic changes

* grandpa-rpc: split unit test

* grandpa-rpc: minor stylistic changes in test

* grandpa-rpc: skip returning future when cancelling

* grandpa-rpc: reuse testing executor from sc-rpc

* grandpa-rpc: don't need to use PubSubHandler in tests

* node-rpc: use MetaIoHandler rather than PubSubHandler

* grandpa: log if getting header failed

* grandpa: move justification channel creation into factory function

* grandpa: make the justification sender optional

* grandpa: fix compilation warnings

* grandpa: move justification notification types to new file

* grandpa-rpc: move JustificationNotification to grandpa-rpc

* grandpa-rpc: move JustificationNotification to its own file

* grandpa: rename justification channel pairs

* grandpa: rename notifier types

* grandpa: pass justification as GrandpaJustification to the rpc module

* Move Metadata to sc-rpc-api

* grandpa-rpc: remove unsed error code

* grandpa: fix bug for checking if channel is closed before sendind

* grandpa-rpc: unit test for sending justifications

* grandpa-rpc: update comments for the pubsub test

* grandpa-rpc: update pubsub tests with more steps

* grandpa-rpc: fix pubsub test

* grandpa-rpc: minor indendation

* grandpa-rpc: decode instead of encode in test

* grandpa: fix review comments

* grandpa: remove unused serde dependency

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
2020-08-10 10:31:36 +00:00
André Silva 886d79e0cb grandpa: never overwrite current rounds voter state (#6823)
* grandpa: never overwrite current rounds voter state

* grandpa: add test for voter state overwrite
2020-08-06 11:30:29 +02:00
André Silva ae38a806ed grandpa: report equivocations with unsigned extrinsics (#6656)
* grandpa: use unsigned extrinsics for equivocation reporting

* grandpa: allow signed reports as well

* grandpa: change runtime api for submitting unsigned extrinsics

* grandpa: fix tests

* grandpa: add test for unsigned validation

* grandpa: add benchmark for equivocation proof checking

* offences: fix grandpa benchmark

* grandpa: add proper weight for equivocation reporting extrinsic

* grandpa: fix weight unit
2020-07-17 12:32:20 +02:00
André Silva 4cf1b4fa7b grandpa: minor cleanups in communication module (#6371)
* grandpa: replace Result<(), ()> with Option<()>

* grandpa: replace &Option<T> with Option<&T>

* grandpa: cleanup local id and keystore usages

* grandpa: return bool on check_message_signature

* grandpa: fix erroneous log message on startup

* grandpa: fix test
2020-06-30 18:59:36 +01:00
Bastian Köcher 53be6ec510 Support synching of blocks that are not new_best (#6508)
* Start

* Remove debug println

* Add tests
2020-06-30 14:44:52 +00:00
Rakan Alhneiti 2577dde3d9 Use sign_with for signing grandpa's outgoing message (#6178)
* Use sign_with and stop using `Pair`

* PR feedback

* Remove clone

* Transfer ownership of public to sign_message

* Use Option

* Simplify code

* Fix error message

* Pass keystore as ref

* Pass keystore properly

* Fix tests
2020-06-09 07:39:30 +00:00
Pierre Krieger 717fa95bc7 Restore the empty line after the license (#6088) 2020-05-20 13:08:27 +00:00
Nikolay Volf d5411969ac fix whitespace (#6062) 2020-05-18 13:51:28 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
André Silva fbd2ac8f3b grandpa: report equivocations (#3868)
* session: runtime api for generating session membership proofs

* grandpa: add runtime api for creating equivocation report txs

* grandpa: submit signed equivocation report transactions

* grandpa: use proper equivocation report type

* grandpa: report equivocations

* grandpa: validate equivocation proof

* grandpa: update to finality-grandpa 0.9.1

* grandpa: fix encoding of session membership proof

* grandpa: initialize set id session mapping for genesis session

* grandpa: fix bug in set_id session validation

* fix compilation

* cleanup from merge conflicts

* cleanup crate tomls

* grandpa: refactor equivocation handling to separate trait

* node-template: fix compilation

* fix test compilation

* bump finality-grandpa to v0.10.2

* rpc: fix runtime version test

* CHERRY-PICK #4200: Add documentation to SubmitSignedTransaction and actually make it work

Squashed commit of the following:

commit 4f2cb0b1c588a06f2f3b478bb4b28b5cb29d54b9
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 16:29:33 2019 +0100

    Split the method to avoid confusing type error message.

commit c5bf24eeaaf902add89ed1b046b22c4a4aaeb2cd
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 16:19:55 2019 +0100

    Make accounts optional, fix logic.

commit 97db1ef556e023cf6847e5ffdb036c0e3ea6fb0a
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 10:06:20 2019 +0100

    Remove warning.

commit 535f5c116d1a2e826eaf90c3f7e6798e443d61d8
Merge: 516257217 0f1a5f651
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Tue Dec 3 07:08:05 2019 +0100

    Merge branch 'master' into td-signed-transactions

commit 516257217bac89fcebd083712f4ea68b7b23b55a
Merge: ac98248c6 2e68c80c2
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Dec 2 13:57:25 2019 +0100

    Merge branch 'master' into td-signed-transactions

commit ac98248c6c56cff381130645a82a13d29933cf83
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:34:52 2019 +0100

    Forgotten import.

commit 67a3c19031506c28e31c6bc4a90fff62d467dd58
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:32:10 2019 +0100

    Fix naming and bounds.

commit 93e768ea9df97a4629fca1f9bc4b108fdb33f876
Author: Tomasz Drwięga <tomasz@parity.io>
Date:   Mon Nov 25 17:01:05 2019 +0100

    Add documentation to signed transactions and actually make them work.

* grandpa: skip block initialization on report submission method

* primitives: allow transaction pool access by default for offchain calls

* grandpa: unused parameters

* grandpa: remove unused method

* grandpa: enable equivocation reporting

* grandpa: add workaround for parameter encoding

* grandpa: fix localized_payload calls in tests

* fix submit_report_equivocation_extrinsic in runtimes

* node: fix submit transaction test compilation

* node: bump spec_version

* rpc: fix api version test

* grandpa: allow custom equivocation offence type

* grandpa: add test for authorities::next_change_height

* grandpa: cleanup report_equivocation function

* node: move reporting app crypto to node-primitives

* grandpa: move equivocation traits to own module

* grandpa: rename app-crypto crate import

* grandpa: export equivocation types

* node: bump spec_version

* grandpa: rename EquivocationReport to EquivocationProof

* grandpa: add missing docs to primitives

* grandpa: add missing docs to equivocation

* node: fix compilation

* grandpa: add missing docs to pallet

* node: bump spec_version

* fix whitespace

* grandpa: return error on offence reporting

* grandpa: expose session and validator count in proofs through traits

* grandpa: use strong key in module KeyOwnerProofSystem

* grandpa: move key ownership proof to grandpa runtime api

* grandpa: remove unnecessary cloning when checking equivocation proof

* grandpa: make report_equivocation a method in Environment

* support: implement KeyOwnerProofSystem for ()

* grandpa: move KeyOwnerProofSystem to module trait

* test-utils: fix runtime compilation

* grandpa: fix test compilation

* grandpa: fix test compilation after merge

* grandpa: simplify transaction submission types

* grandpa: validate equivocation report in signed extension

* client: fix test

* node: use ValidateEquivocationReport signed extension

* grandpa: expose key ownership proof under opaque type

* grandpa: better docs on key ownership proofs

* grandpa: add note about signed extension

* grandpa: add ValidateEquivocationReport::new

* grandpa: remove skip_initialize_block from runtime api

* grandpa: use new offchain transaction submission API

* grandpa: take set_id in generate_key_ownership_proof

* grandpa: update to finality-grandpa v0.12.2

* grandpa: cleanup usages of AuthoritySet::current

* grandpa: fix test

* grandpa: add mocking utilities for equivocation reporting

* grandpa: add test for equivocation reporting

* grandpa: move SetIdSession initialization

* grandpa: add more tests

* node: enable historical session manager

* node: bump spec_version

* node: use strong key types in KeyOwnerProofSystem definitions

* grandpa: export GrandpaEquivocationOffence type
2020-05-06 17:25:51 +01:00
Jon Häggblad 1f7f8abb33 Expose GRANDPA round state through RPC (#5375)
* grandpa: wire up basic RPC call

* grandpa: make it compile against GRANDPA with expose round state

* grandpa: use shared voter state to expose RPC endpoint

* grandpa: restructure into nested structs

* grandpa: return background rounds too

* grandpa: return error when endpoint not ready

* grandpa: collect grandpa rpc deps

* grandpa: decide to use concrete AuthorityId in finality-grandpa-rpc

* grandpa: remove unncessary type annotation

* grandpa: move error code to const

* grandpa: remove unnecessary WIP comment

* grandpa: remove Id type parameter for SharedVoterState

* grandpa: update tests to add shared_voter_state in parameters

* grandpa: remove old deprecated test

* grandpa: fix getting the correct set_id

* grandpa: make SharedVoterState a struct

* grandpa: wrap shared_voter_state in rpc_setup

* grandpa: replace spaces with tabs

* grandpa: limit RwLock write attempt to 1 sec

* grandpa: add missing doc comments and remove some pub

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-Authored-By: Hernando Castano <HCastano@users.noreply.github.com>

* grandpa: update function name call after change in finality-grandpa

* grandpa: group pub use and only export voter::report

* grandpa: add missing docs

* grandpa: extract out structs used for json serialization

* grandpa: stick to u32 for fields intended for js

* grandpa: move Error type to its own file

* grandpa: group pub use better

* Apply code review suggestion

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* grandpa: use correct version of finality-granpda in rpc crate

* grandpa: add back basic rpc unit test

* grandpa: replace SharedVoterState::new() with empty()

* node: cleanup grandpa::SharedVoterState usage in macro

* grandpa: remove VoterState error variant

* grandpa: enable missing futures compat feature

* grandpa: fix typo in error variant

* grandpa: remove test_utils

* grandpa: allow mocking rpc handler components

* grandpa: rename serialized to report in rpc module

* grandpa: add proper test for RPC

* grandpa: update to finality-grandpa v0.12.1

Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Demi Obenour <demi@parity.io>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-05-04 20:37:22 +01:00
André Silva d2967ba4b6 grandpa: update to finality-grandpa v0.12.0 (#5853)
* grandpa: update to v0.12.0

* grandpa: fix tests

* grandpa: better validation of authority set invariants

* grandpa: avoid duplicating invalid authority list check

* grandpa: add missing doc

* grandpa: better validation of expect proofs

* grandpa: fix test compilation

* grandpa: fix tests

* grandpa: add test for AuthoritySet invariants

* grandpa: bump finality-grandpa to v0.12.0
2020-04-30 20:03:30 +00:00
Seun Lanlege 4fa5941f44 Move sc-client into sc-service (#5502)
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api

* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa

* drop sc-service from sc-rpc

* drop sc-service from sc-consensus-aura

* drop sc-client from manual-seal and babe

* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli

* drop sc-client from bin/node and bin/node-template

* drop sc-client

* fix tests

* remove check -p sc-client from gitlab.yml

* fix warnings

* fixes ui test

* fix light client tests

* adds associated Client type to AbstractService

* adds UsageProvider to Client

* fixed ui test, again

* tried and failed to get node-cli to compile for wasm

* thanks to tomaka for helping me get node-cli to compile for wasmm

* ui test pls pas 🙏🏾

* all tests passing 🪄

* no_run documentation code

* rm -f documentation code

* ClientProvider

* fix mega trait

* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client

* adds license to sc-consensus

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-04-28 11:59:31 +00:00
Benjamin Kampmann 247822bb33 Additional Metrics collected and exposed via prometheus (#5414)
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-04-04 15:13:35 +02:00
Pierre Krieger 8c03a4fcef Split the Roles in three types (#5520)
* Split the Roles bitfield in three

* Forgot to include some changes

* Fix cli test

* More test fixes

* Oh God, merging master broke other tests

* Didn't run the doctests

* Address review

* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine

* Address some review

* Also update the peerset's API to make sense

* Fix peerset tests

* Fix browser node

* client: distinguish between local and network authority

Co-authored-by: André Silva <andre.beat@gmail.com>
2020-04-03 19:08:14 +02:00
Bastian Köcher 954dca689d Implements mocking of runtime apis (#5448)
* Implements mocking of runtime apis

This pr adds support for easily mock runtime api implementations for
tests by using the `mock_impl_runtime_apis!` macro. The syntax is
similar to `impl_runtime_apis!`. The mocked implementation automatically
implements `ApiExt`, `ApiErrorExt` and `Core` as these are required by
the runtime api traits, but not required in tests or only a subset of them.

* Fix warnings

* Update primitives/api/proc-macro/src/utils.rs

Co-Authored-By: Nikolay Volf <nikvolf@gmail.com>

* Review feedback

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-03-31 13:46:07 +02:00
Max Inden 79fc16e4d7 client/finality-grandpa: Add Prometheus metrics to GossipValidator (#5237)
* client/finality-grandpa: Add Prometheus metrics to GossipValidator

Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.

```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```

* client/finality-grandpa: Add None as Prometheus registry in tests

* client/finality-granpda/src/communication: Refactor metric registration
2020-03-14 12:51:44 +01:00