An `UntilImported` stream wraps a `Stream` of incoming messages and
waits for blocks those messages are based on before passing the messages
on.
The above `Stream` of incoming messages implements `Unpin`, thus there
is no need to use `pin_project` on the `UntilImported` struct. Instead
one only has to add the `Unpin` trait bound on the `I` trait parameter.
* Make `Digest` support `StorageAppend`
This adds support for `StorageAppend` to `Digest`. Digest is just a
wrapper around a `Vec` and we abuse the fact that SCALE does not puts
any special marker into the encoding for structs. So, we can just append
to the encoded Digest. A test is added that ensures, if the `Digest`
format ever changes, we remove this optimization.
* Update weight
* Update frame/support/src/storage/mod.rs
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* Update frame/system/src/lib.rs
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* grandpa: wire up basic RPC call
* grandpa: make it compile against GRANDPA with expose round state
* grandpa: use shared voter state to expose RPC endpoint
* grandpa: restructure into nested structs
* grandpa: return background rounds too
* grandpa: return error when endpoint not ready
* grandpa: collect grandpa rpc deps
* grandpa: decide to use concrete AuthorityId in finality-grandpa-rpc
* grandpa: remove unncessary type annotation
* grandpa: move error code to const
* grandpa: remove unnecessary WIP comment
* grandpa: remove Id type parameter for SharedVoterState
* grandpa: update tests to add shared_voter_state in parameters
* grandpa: remove old deprecated test
* grandpa: fix getting the correct set_id
* grandpa: make SharedVoterState a struct
* grandpa: wrap shared_voter_state in rpc_setup
* grandpa: replace spaces with tabs
* grandpa: limit RwLock write attempt to 1 sec
* grandpa: add missing doc comments and remove some pub
* Apply suggestions from code review
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-Authored-By: Hernando Castano <HCastano@users.noreply.github.com>
* grandpa: update function name call after change in finality-grandpa
* grandpa: group pub use and only export voter::report
* grandpa: add missing docs
* grandpa: extract out structs used for json serialization
* grandpa: stick to u32 for fields intended for js
* grandpa: move Error type to its own file
* grandpa: group pub use better
* Apply code review suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* grandpa: use correct version of finality-granpda in rpc crate
* grandpa: add back basic rpc unit test
* grandpa: replace SharedVoterState::new() with empty()
* node: cleanup grandpa::SharedVoterState usage in macro
* grandpa: remove VoterState error variant
* grandpa: enable missing futures compat feature
* grandpa: fix typo in error variant
* grandpa: remove test_utils
* grandpa: allow mocking rpc handler components
* grandpa: rename serialized to report in rpc module
* grandpa: add proper test for RPC
* grandpa: update to finality-grandpa v0.12.1
Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Demi Obenour <demi@parity.io>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api
* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa
* drop sc-service from sc-rpc
* drop sc-service from sc-consensus-aura
* drop sc-client from manual-seal and babe
* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli
* drop sc-client from bin/node and bin/node-template
* drop sc-client
* fix tests
* remove check -p sc-client from gitlab.yml
* fix warnings
* fixes ui test
* fix light client tests
* adds associated Client type to AbstractService
* adds UsageProvider to Client
* fixed ui test, again
* tried and failed to get node-cli to compile for wasm
* thanks to tomaka for helping me get node-cli to compile for wasmm
* ui test pls pas 🙏🏾
* all tests passing 🪄
* no_run documentation code
* rm -f documentation code
* ClientProvider
* fix mega trait
* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client
* adds license to sc-consensus
Co-authored-by: Benjamin Kampmann <ben@parity.io>
* babe: restrict info logging during initial sync
* grandpa: restrict info logging during initial sync
* grandpa: fix test compilation
* grandpa: remove afg_log macro mod
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:
- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)
refs #4679
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* Split the Roles bitfield in three
* Forgot to include some changes
* Fix cli test
* More test fixes
* Oh God, merging master broke other tests
* Didn't run the doctests
* Address review
* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine
* Address some review
* Also update the peerset's API to make sense
* Fix peerset tests
* Fix browser node
* client: distinguish between local and network authority
Co-authored-by: André Silva <andre.beat@gmail.com>
* client/finality-grandpa: Instrument until-imported queue
The `UntilImported` queue takes as input finality grandpa messages that
depend on blocks that are not yet imported and holds them back until
those blocks are imported.
This commit adds a basic metric, the amount of messages waiting in the
queue, to the module. For now this metric is only available for the
global `UntilImported` queue awaiting blocks for commit and catch-up
messages.
* client/finality-grandpa/src/until_imported: Update metric help text
Co-Authored-By: Ashley <ashley.ruglys@gmail.com>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* Implements mocking of runtime apis
This pr adds support for easily mock runtime api implementations for
tests by using the `mock_impl_runtime_apis!` macro. The syntax is
similar to `impl_runtime_apis!`. The mocked implementation automatically
implements `ApiExt`, `ApiErrorExt` and `Core` as these are required by
the runtime api traits, but not required in tests or only a subset of them.
* Fix warnings
* Update primitives/api/proc-macro/src/utils.rs
Co-Authored-By: Nikolay Volf <nikvolf@gmail.com>
* Review feedback
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* client/finality-grandpa/src/until_imported: Refactor schedule_wait
Previously `BlockUntilImported::schedule_wait` took two closures, one to
report ready items and one to report items to await. None of the implementors of
`BlockUntilImported` call both closures. From a symantic perspective it
would as well not make sense to both await something and state that it
is ready.
Instead with this commit `BlockUntilImported::schedule_wait` simply
returns whether the given item needs waiting or is ready to be passed
on.
This reduces complexity by:
- Removing side effects through the two closures.
- Reducing borrowing given that `UntilImported` `ready` and `pending`
don't need to be borrowed from within the two closures.
- Removes the need for trait bounds for the two closures.
* client/finality-grandpa/src/until_imported: Fix comments
Co-Authored-By: André Silva <andre.beat@gmail.com>
Co-authored-by: André Silva <andre.beat@gmail.com>
* client/finality-grandpa/src/until_imported: Refactor BlockGlobalMessage
`BlockGlobalMessage` owns an `inner` which contains (1) a count for the
amount of outstanding blocks to be waited on and (2) the message itself.
Given that both is already wrapped in an `Arc` there is no need to keep
track of the outstanding blocks, given that it simply corresponds to the
amount of strong reference counts on the `Arc` itself.
This commit removes the atomic counter within `inner` and piggy backs on
the `Arc` reference counter instead.
* client/finality-grandpa/src/until_imported: Remove useless match
* client/finality-grandpa/src/until_imported.rs: Remove unused var
Co-Authored-By: André Silva <andre.beat@gmail.com>
* client/finality-grandpa/src/until_imported: Address comment suggestion
Co-authored-by: André Silva <andre.beat@gmail.com>
* grandpa: support for hard forking any pending standard changes
* grandpa: expose authority_set_hard_forks in block import constructor
* grandpa: don't break the public api
* client/network-gossip/src/bridge: Finish when network even stream closes
Previously within `<GossipEngine as Future>::poll` one would poll the
`network_event_stream` ignoring all messages other than
`Poll::Ready(Some())`. Ignoring `Poll::Ready(None)` leads to a panic on
the next poll of the stream, gien that it is not fused.
By design `network_event_stream` does not close unless an unbounded send
into it fails, or the `NetworkWorker` gets shut down.
> The stream never ends (unless the `NetworkWorker` gets shut down).
> (client/network/src/service.rs)
An `unbounded_send` to fail on an unbounded channel is unlikely. The
`NetworkWorker` shutting down is not unlikely. In such case the
`GossipEngine` should shut down as well.
With this patch a `<GossipEngine as Future>` finishes on
`Poll::Ready(None)` returned from `network_event_stream`.
* client/finality-grandpa/communication: Error on gossip engine finished
Have `<NetworkBridge as Future>::poll` return `Poll::Ready(Err)` instead
of `Poll::Ready(Ok)` to be consistent with the handling of the neighbor
packet worker stream and the gossip validator report stream. Both `Err`
as well as `Ok` shut down the `NetworkBridge` as well as the
`VoterWorker`.
* client/network-gossip/src/bridge: Add regression test
* client/network-gossip: Move substrate test client to dev dependencies
* client/network-gossip: Remove TODO
Addressed in a follow up pull request.
* client/network-gossip/bridge: Put match on newline after loop
* client/finality-grandpa/src/observer: Fix regression test
Make sure the event stream sender side is not dropped till the end.
* Version bump
* Split generate_changelog.sh into separate script
Can be run in the format `generate_changelog.sh $previous_version $version`.
* remove early exit from publish_draft_release.sh
* adding changelog
* ci: change last_github_release to also find pre-releases
Co-authored-by: Martin Pugh <pugh@s3kr.it>
* client/finality-grandpa: Add Prometheus metrics to GossipValidator
Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.
```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```
* client/finality-grandpa: Add None as Prometheus registry in tests
* client/finality-granpda/src/communication: Refactor metric registration
* Use CLI to configure max instances cache
* Fix tests
* Move default value into CLI
* Use SmallVec
* Apply review comments
* Get rid of `SmallVec`
Co-authored-by: Bastian Köcher <git@kchr.de>
* grandpa: only gossip commits to peers on the same set
* grandpa: track commits uniquely by round and set
* grandpa: fix communication test
* grandpa: add tests for commit gossip handling
* grandpa: add missing docs
* client/network-gossip: Merge GossipEngine and GossipEngineInner
Given that GossipEngine and GossipEngineInner are not shared between
threads anyone (public interface + background tasks), neither depends on
being Send or Sync. Thus one can merge the two as done in this patch.
One only needs to wrap an `Arc<Mutex<>>` around the whole structure when
the owner (e.g. finality-grandpa) needs to share the gossip engine
between threads.
* client/finality-grandpa: Wrap GossipEngine in Arc Mutex & lock it on use
GossipEngine in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore. Given that
finality-grandpa shares the `NetworkBridge` potentially between threads
its components need to be clonable, thus this patch wraps `GossipEngine`
in an `Arc<Mutex<>>`.
* removes use of sc_client::Client from sc-rpc
* remove Client impl from sc-finality-benches
* remove client impl from sc-finality-grandpa
* read_proof accepts iterator
* remove generic Executor param from ExecutorProvider
* fix long ass line
* code style changes
* merge with master
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
This updates `parity-scale-codec` to `1.2.0`, which includes multiple
performance improvements and a fix that bounds the capacity of a vector
at decoding.
Benjamin Kampmann