* Remove sc_network::NetworkService::register_notifications_protocol
* Missing calls to .into()
* Wrong crate name
* [WIP] Fix Grandpa tests
* One more passing
* One more. Two to go.
* This one was actually already passing 🎉
* Last one compiles
* Progress
* grandpa: fix voter_persists_its_votes test
* Restore other tests
* Try spawn future later
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Remove necessity to pass ConsensusEngineId when registering notifications protocol
* Line width
* Fix tests protocol name
* Other renames
* Doc update
* Change issue in TODO
* grandpa: don't send equivocation reports for local identities
* grandpa: add test for self-report
* grandpa: fix test compilation
this works on rust nightly but breaks on ci which is using rust stable
* Asyncify sign_with
* Asyncify generate/get keys
* Complete BareCryptoStore asyncification
* Cleanup
* Rebase
* Add Proxy
* Inject keystore proxy into extensions
* Implement some methods
* Await on send
* Cleanup
* Send result over the oneshot channel sender
* Process one future at a time
* Fix cargo stuff
* Asyncify sr25519_vrf_sign
* Cherry-pick and fix changes
* Introduce SyncCryptoStore
* SQUASH ME WITH THE first commit
* Implement into SyncCryptoStore
* Implement BareCryptoStore for KeystoreProxyAdapter
* authority-discovery
* AURA
* BABE
* finality-grandpa
* offchain-workers
* benchmarking-cli
* sp_io
* test-utils
* application-crypto
* Extensions and RPC
* Client Service
* bin
* Update cargo.lock
* Implement BareCryptoStore on proxy directly
* Simplify proxy setup
* Fix authority-discover
* Pass async keystore to authority-discovery
* Fix tests
* Use async keystore in authority-discovery
* Rename BareCryptoStore to CryptoStore
* WIP
* Remote mutable borrow in CryptoStore trait
* Implement Keystore with backends
* Remove Proxy implementation
* Fix service builder and keystore user-crates
* Fix tests
* Rework authority-discovery after refactoring
* futures::select!
* Fix multiple mut borrows in authority-discovery
* Merge fixes
* Require sync
* Restore Cargo.lock
* PR feedback - round 1
* Remove Keystore and use LocalKeystore directly
Also renamed KeystoreParams to KeystoreContainer
* Join
* Remove sync requirement
* Fix keystore tests
* Fix tests
* client/authority-discovery: Remove event stream dynamic dispatching
With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.
Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.
* Make it compile
* Fix submit_transaction
* Fix block_on issue
* Use await in async context
* Fix manual seal keystore
* Fix authoring_blocks test
* fix aura authoring_blocks
* Try to fix tests for auth-discovery
* client/authority-discovery: Fix lookup_throttling test
* client/authority-discovery: Fix triggers_dht_get_query test
* Fix epoch_authorship_works
* client/authority-discovery: Remove timing assumption in unit test
* client/authority-discovery: Revert changes to termination test
* PR feedback
* Remove deadcode and mark test code
* Fix test_sync
* Use the correct keyring type
* Return when from_service stream is closed
* Convert SyncCryptoStore to a trait
* Fix line width
* Fix line width - take 2
* Remove unused import
* Fix keystore instantiation
* PR feedback
* Remove KeystoreContainer
* Revert "Remove KeystoreContainer"
This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.
* Take a ref of keystore
* Move keystore to dev-dependencies
* Address some PR feedback
* Missed one
* Pass keystore reference - take 2
* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX
Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:
1. No additional mental overhead of a `SyncCryptoStorePtr`.
2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.
Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.
* Remove SyncCryptoStorePtr
* Remove KeystoreContainer & SyncCryptoStorePtr
* PR feedback
* *: Use CryptoStorePtr whereever possible
* *: Define SyncCryptoStore as a pure extension trait of CryptoStore
* Follow up to SyncCryptoStore extension trait
* Adjust docs for SyncCryptoStore as Ben suggested
* Cleanup unnecessary requirements
* sp-keystore
* Use async_std::task::block_on in keystore
* Fix block_on std requirement
* Update primitives/keystore/src/lib.rs
Co-authored-by: Max Inden <mail@max-inden.de>
* Fix wasm build
* Remove unused var
* Fix wasm compilation - take 2
* Revert async-std in keystore
* Fix indent
* Fix version and copyright
* Cleanup feature = "std"
* Auth Discovery: Ignore if from_service is cloed
* Max's suggestion
* Revert async-std usage for block_on
* Address PR feedback
* Fix example offchain worker build
* Address PR feedback
* Update Cargo.lock
* Move unused methods to test helper functions
* Restore accidentally deleted cargo.lock files
* Fix unused imports
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* init_logger: switch from log-based to tracing-based and add compatibility layer
* Move tracing profiling subscriber related config realization
* sp-tracing: change profiling to be a layer instead of a subscriber
* Enable profiling layer in cli
* Change all test env_logger init to sp_tracing::try_init_simple
* Remove all local env_logger dependency
* Add missing tracing-subscriber dependency
* frame-sudo: fix tests
* frame-support: fix tests
* Fix frame/pallet and executor tests
* Fix the remaining tests
* Use subscriber's try_init as recommended by @davidbarsky
* Be explict that the tracing-log feature is needed
* Set subscriber writer to stderr
* Shorter line width
* Update cargo lock tracing version
* Fix sc_tracing crate compile
* Fix sc_authority_discovery crate test
* unremove default-features
* Leave enabled to default true
* Warn if global default cannot be set
* Fix unused import
* Remove unused PROXY_TARGET
* Change all reference from rc5 to rc6
* Change all reference of rc2 to rc6
* Fix styling
* Fix typo
* make logger init error'ing
* re-fixing the test issue
Co-authored-by: Benjamin Kampmann <ben@parity.io>
* Stop sending messages on legacy substream altogether
* Ensure that handshake is sent back even in case of back-pressure
* Update client/network/src/protocol/generic_proto/handler/group.rs
Co-authored-by: Max Inden <mail@max-inden.de>
* Also process OpenRequest and Closed
* Also process OpenRequest and Closed
* Fix bad merge
* God I'm so lost with all these merges
* Immediately return Closed
* Add warning for sending on non-registered protocol
* Register GrandPa protocol in tests
* Update client/network/src/protocol/generic_proto/handler/group.rs
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Max Inden <mail@max-inden.de>
* Rough skeleton for what I think the RPC should look like
* Create channel for sending justifications
Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.
* WIP: Add subscribers for justifications to Grandpa
Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.
* WIP: Add a struct for managing subscriptions
Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.
* Make naming more clear and lock data in Arc
* Rough idea of what RPC would look like
* Remove code from previous approach
* Missed some things
* Update client/rpc-api/src/chain/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update client/rpc-api/src/chain/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Split justification subscription into sender and receiver halves
* Replace RwLock with a Mutex
* Add sample usage from the Service's point of view
* Remove code that referred to "chain_" RPC
* Use the Justification sender/receivers from Grandpa LinkHalf
* Add some PubSub boilerplate
* Add guiding comments
* TMP: comment out to fix compilation
* Return MetaIoHandler from PubSubHandler in create_full
* Uncomment pubsub methods in rpc handler (fails to build)
* node/rpc: make Metadata concrete in create_full to fix compilation
* node: pass in SubscriptionManger to grandpa rpc handler
* grandpa-rpc: use SubscriptionManger to add subscriber
* grandpa-rpc: attempt at setting up the justification stream (fails to build)
* grandpa-rpc: fix compilation of connecting stream to sink
* grandpa-rpc: implement unsubscribe
* grandpa-rpc: update older tests
* grandpa-rpc: add full prefix to avoid confusing rust-analyzer
* grandpa-rpc: add test for pubsub not available
* grandpa-rpc: tidy up leftover code
* grandpa-rpc: add test for sub and unsub of justifications
* grandpa-rpc: minor stylistic changes
* grandpa-rpc: split unit test
* grandpa-rpc: minor stylistic changes in test
* grandpa-rpc: skip returning future when cancelling
* grandpa-rpc: reuse testing executor from sc-rpc
* grandpa-rpc: don't need to use PubSubHandler in tests
* node-rpc: use MetaIoHandler rather than PubSubHandler
* grandpa: log if getting header failed
* grandpa: move justification channel creation into factory function
* grandpa: make the justification sender optional
* grandpa: fix compilation warnings
* grandpa: move justification notification types to new file
* grandpa-rpc: move JustificationNotification to grandpa-rpc
* grandpa-rpc: move JustificationNotification to its own file
* grandpa: rename justification channel pairs
* grandpa: rename notifier types
* grandpa: pass justification as GrandpaJustification to the rpc module
* Move Metadata to sc-rpc-api
* grandpa-rpc: remove unsed error code
* grandpa: fix bug for checking if channel is closed before sendind
* grandpa-rpc: unit test for sending justifications
* grandpa-rpc: update comments for the pubsub test
* grandpa-rpc: update pubsub tests with more steps
* grandpa-rpc: fix pubsub test
* grandpa-rpc: minor indendation
* grandpa-rpc: decode instead of encode in test
* grandpa: fix review comments
* grandpa: remove unused serde dependency
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
* grandpa: replace Result<(), ()> with Option<()>
* grandpa: replace &Option<T> with Option<&T>
* grandpa: cleanup local id and keystore usages
* grandpa: return bool on check_message_signature
* grandpa: fix erroneous log message on startup
* grandpa: fix test
* Use sign_with and stop using `Pair`
* PR feedback
* Remove clone
* Transfer ownership of public to sign_message
* Use Option
* Simplify code
* Fix error message
* Pass keystore as ref
* Pass keystore properly
* Fix tests
* grandpa: wire up basic RPC call
* grandpa: make it compile against GRANDPA with expose round state
* grandpa: use shared voter state to expose RPC endpoint
* grandpa: restructure into nested structs
* grandpa: return background rounds too
* grandpa: return error when endpoint not ready
* grandpa: collect grandpa rpc deps
* grandpa: decide to use concrete AuthorityId in finality-grandpa-rpc
* grandpa: remove unncessary type annotation
* grandpa: move error code to const
* grandpa: remove unnecessary WIP comment
* grandpa: remove Id type parameter for SharedVoterState
* grandpa: update tests to add shared_voter_state in parameters
* grandpa: remove old deprecated test
* grandpa: fix getting the correct set_id
* grandpa: make SharedVoterState a struct
* grandpa: wrap shared_voter_state in rpc_setup
* grandpa: replace spaces with tabs
* grandpa: limit RwLock write attempt to 1 sec
* grandpa: add missing doc comments and remove some pub
* Apply suggestions from code review
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-Authored-By: Hernando Castano <HCastano@users.noreply.github.com>
* grandpa: update function name call after change in finality-grandpa
* grandpa: group pub use and only export voter::report
* grandpa: add missing docs
* grandpa: extract out structs used for json serialization
* grandpa: stick to u32 for fields intended for js
* grandpa: move Error type to its own file
* grandpa: group pub use better
* Apply code review suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* grandpa: use correct version of finality-granpda in rpc crate
* grandpa: add back basic rpc unit test
* grandpa: replace SharedVoterState::new() with empty()
* node: cleanup grandpa::SharedVoterState usage in macro
* grandpa: remove VoterState error variant
* grandpa: enable missing futures compat feature
* grandpa: fix typo in error variant
* grandpa: remove test_utils
* grandpa: allow mocking rpc handler components
* grandpa: rename serialized to report in rpc module
* grandpa: add proper test for RPC
* grandpa: update to finality-grandpa v0.12.1
Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Demi Obenour <demi@parity.io>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api
* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa
* drop sc-service from sc-rpc
* drop sc-service from sc-consensus-aura
* drop sc-client from manual-seal and babe
* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli
* drop sc-client from bin/node and bin/node-template
* drop sc-client
* fix tests
* remove check -p sc-client from gitlab.yml
* fix warnings
* fixes ui test
* fix light client tests
* adds associated Client type to AbstractService
* adds UsageProvider to Client
* fixed ui test, again
* tried and failed to get node-cli to compile for wasm
* thanks to tomaka for helping me get node-cli to compile for wasmm
* ui test pls pas 🙏🏾
* all tests passing 🪄
* no_run documentation code
* rm -f documentation code
* ClientProvider
* fix mega trait
* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client
* adds license to sc-consensus
Co-authored-by: Benjamin Kampmann <ben@parity.io>
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:
- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)
refs #4679
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* Split the Roles bitfield in three
* Forgot to include some changes
* Fix cli test
* More test fixes
* Oh God, merging master broke other tests
* Didn't run the doctests
* Address review
* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine
* Address some review
* Also update the peerset's API to make sense
* Fix peerset tests
* Fix browser node
* client: distinguish between local and network authority
Co-authored-by: André Silva <andre.beat@gmail.com>
* Implements mocking of runtime apis
This pr adds support for easily mock runtime api implementations for
tests by using the `mock_impl_runtime_apis!` macro. The syntax is
similar to `impl_runtime_apis!`. The mocked implementation automatically
implements `ApiExt`, `ApiErrorExt` and `Core` as these are required by
the runtime api traits, but not required in tests or only a subset of them.
* Fix warnings
* Update primitives/api/proc-macro/src/utils.rs
Co-Authored-By: Nikolay Volf <nikvolf@gmail.com>
* Review feedback
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* client/finality-grandpa: Add Prometheus metrics to GossipValidator
Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.
```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```
* client/finality-grandpa: Add None as Prometheus registry in tests
* client/finality-granpda/src/communication: Refactor metric registration
* removes use of sc_client::Client from sc-rpc
* remove Client impl from sc-finality-benches
* remove client impl from sc-finality-grandpa
* read_proof accepts iterator
* remove generic Executor param from ExecutorProvider
* fix long ass line
* code style changes
* merge with master
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
* Start removing last few instances of futures01
* Use to_poll on wasm
* Revert "Use to_poll on wasm"
This reverts commit 1c61728f10d520df5f9b28c415a0db68e478b9c7.
* Fix fg test
* Upgrade network test futures
* Update offchain hyper version
* Update service test
* bump tokio to 0.2.10
* Removed some unneeded tokios
* fixes
* fix run_until_all_full
* Make service test debuggable
* Update client/offchain/src/api/http.rs
Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
* Add service_test to test-int output
* nitpicking
* Finally fix test
* Give up and revert client/serviec/test
* Revert gitlab ci too
Co-authored-by: Demi Obenour <demi@parity.io>
* removes use of sc_client::Client from sc_finality_grandpa
* code formatting
* code formatting
* removes use of sc_client::Client from sc_finality_grandpa
`GossipEngine` spawns two tasks, one for a periodic tick, one to forward
messages from the network to subscribers. These tasks hold an `Arc` to a
`GossipEngineInner`.
To reduce the amount of shared ownership (locking) this patch integrates
the two tasks into a `Future` implementation on the `GossipEngine`
struct. This `Future` implementation can now be called from a single
owner, e.g. the `finality-grandpa` `NetworkBridge`.
As a side effect this removes the requirement on the `network-gossip`
crate to spawn tasks and thereby removes the requirement on the
`finality-grandpa` crate to spawn any tasks.
This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa`, `network` and `network-gossip` as
well as to reduce the amount of unbounded channels. For details see
d9837d7dd, 5f80929dc and 597c0a6c4.
* Switch GrandPa to new futures
* Work on making tests work
* until_imported tests working again
* Work on switching tests to stable futures
* Modifications
* Re-add test as #[ignore]
* Don't ignore
* Add manual unpins
* Remove Header import
* Return concrete Sink type
* Switch to crates.io finality-grandpa version
* Remove use statement that slipped in
* Fix some nitpicks
* Remove unpin from i
* Fixed typo
* Move futures01 to dev-deps
* Fix nitpicks
* Update client/finality-grandpa/src/communication/mod.rs
Co-Authored-By: André Silva <andre.beat@gmail.com>
* nitpicking
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <andre.beat@gmail.com>
* clinet/finality-grandpa: Make round_communication use bounded channel
`round_communication` returns a `Sink` and a `Stream` for outgoing and
incoming messages. The messages send into the `Sink` are forwarded down
to the network as well as send back into the `Stream` to ensure the node
processes its own messages.
So far, to send messages into the `Sink` back into the `Stream`, an
unbounded channel was used. This patch updates `round_communication` and
`OutgoingMessages` to use a bounded channel.
This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa` and `network` as well as to reduce
the amount of unbounded channels. For details see d9837d7dd and
5f80929dc.
* client/finality-grandpa: Import futures03::future::ready at the top
* client/finality-grandpa: Make tests use compat of future 03
* client/finality-grandpa: Do not import ready into scope
Instead of importing futures03::future::ready into the scope, only
import futures::future03 into scope and call ready as furure03::ready.
* consensus, pow: intermediate separation and fail
* Fix compiles
* Update primitives/consensus/common/src/block_import.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Update primitives/consensus/common/src/block_import.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Document what None means for `fork_choice` in block import params
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* client/finality-grandpa: Reintegrate gossip validator report stream
The `finality-grandpa` `GossipValidator` is called by the `GossipEngine`
in a synchronous fashion on each gossip message. Its main task is to
decide whether to gossip the given message on, or whether to drop it.
In addition it also updates the reputation of a node's peers based on
the incoming gossip messages. To do so it needs to be able to report the
reputation change which it does through an unbounded channel (in order
to stay synchronous).
Previously the receiving side of this channel would be handled by a new
task, polling the channel and forwarding the changes to a clone of the
`GossipEngine` that it would own.
Instead the receiver of the above mentioned channel is now being polled
by the `NetworkBridge` within its `Future::poll` implementation.
Reputation changes are reported through the already existing
`GossipEngine` instance within `NetworkBridge`.
For details on the overall goal, see d9837d7dd.
* client/finality-grandpa: Remove exit future from test NetworkBridges
* Extend `Proposer` to optionally generate a proof of the proposal
* Something
* Refactor sr-api to not depend on client anymore
* Fix benches
* Apply suggestions from code review
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Apply suggestions from code review
* Introduce new `into_storage_changes` function
* Switch to runtime api for `execute_block` and don't require `H256`
anywhere in the code
* Put the `StorageChanges` into the `Proposal`
* Move the runtime api error to its own trait
* Adds `StorageTransactionCache` to the runtime api
This requires that we add `type NodeBlock = ` to the
`impl_runtime_apis!` macro to work around some bugs in rustc :(
* Remove `type NodeBlock` and switch to a "better" hack
* Start using the transaction cache from the runtime api
* Make it compile
* Move `InMemory` to its own file
* Make all tests work again
* Return block, storage_changes and proof from Blockbuilder::bake()
* Make sure that we use/set `storage_changes` when possible
* Add test
* Fix deadlock
* Remove accidentally added folders
* Introduce `RecordProof` as argument type to be more explicit
* Update client/src/client.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update primitives/state-machine/src/ext.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Integrates review feedback
* Remove `unsafe` usage
* Update client/block-builder/src/lib.rs
Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>
* Update client/src/call_executor.rs
* Bump versions
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Bastian Köcher