* Refactor sc-network/service metrics.
1. Aggregate sc-network metrics into a submodule, introducing
two more sourced metrics to avoid duplicate atomics.
2. Decouple periodic sc-service network metrics from other
metrics, so that they can be updated independently.
* Update client/service/src/metrics.rs
* Update client/service/src/metrics.rs
* Update to libp2p-0.23.
Thereby incorporate bandwidth measurement along the
lines previously done by libp2p itself.
* Tweak dependencies for wasm32 compilation.
For wasm32 we need to enable unstable features to
make `task::Builder::local` available.
* Simplify dependencies.
* Simplify.
Leave the calculation of bytes sent/received per second
to the outer layers of the code, subject to their own
individual update intervals.
* Cleanup
* Re-add lost dev dependency.
* Avoid division by zero.
* Remove redundant metric.
* Enable sending of noise legacy handshakes.
* Add comment about monotonic gauge.
* CI
* Rough skeleton for what I think the RPC should look like
* Create channel for sending justifications
Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.
* WIP: Add subscribers for justifications to Grandpa
Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.
* WIP: Add a struct for managing subscriptions
Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.
* Make naming more clear and lock data in Arc
* Rough idea of what RPC would look like
* Remove code from previous approach
* Missed some things
* Update client/rpc-api/src/chain/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update client/rpc-api/src/chain/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Split justification subscription into sender and receiver halves
* Replace RwLock with a Mutex
* Add sample usage from the Service's point of view
* Remove code that referred to "chain_" RPC
* Use the Justification sender/receivers from Grandpa LinkHalf
* Add some PubSub boilerplate
* Add guiding comments
* TMP: comment out to fix compilation
* Return MetaIoHandler from PubSubHandler in create_full
* Uncomment pubsub methods in rpc handler (fails to build)
* node/rpc: make Metadata concrete in create_full to fix compilation
* node: pass in SubscriptionManger to grandpa rpc handler
* grandpa-rpc: use SubscriptionManger to add subscriber
* grandpa-rpc: attempt at setting up the justification stream (fails to build)
* grandpa-rpc: fix compilation of connecting stream to sink
* grandpa-rpc: implement unsubscribe
* grandpa-rpc: update older tests
* grandpa-rpc: add full prefix to avoid confusing rust-analyzer
* grandpa-rpc: add test for pubsub not available
* grandpa-rpc: tidy up leftover code
* grandpa-rpc: add test for sub and unsub of justifications
* grandpa-rpc: minor stylistic changes
* grandpa-rpc: split unit test
* grandpa-rpc: minor stylistic changes in test
* grandpa-rpc: skip returning future when cancelling
* grandpa-rpc: reuse testing executor from sc-rpc
* grandpa-rpc: don't need to use PubSubHandler in tests
* node-rpc: use MetaIoHandler rather than PubSubHandler
* grandpa: log if getting header failed
* grandpa: move justification channel creation into factory function
* grandpa: make the justification sender optional
* grandpa: fix compilation warnings
* grandpa: move justification notification types to new file
* grandpa-rpc: move JustificationNotification to grandpa-rpc
* grandpa-rpc: move JustificationNotification to its own file
* grandpa: rename justification channel pairs
* grandpa: rename notifier types
* grandpa: pass justification as GrandpaJustification to the rpc module
* Move Metadata to sc-rpc-api
* grandpa-rpc: remove unsed error code
* grandpa: fix bug for checking if channel is closed before sendind
* grandpa-rpc: unit test for sending justifications
* grandpa-rpc: update comments for the pubsub test
* grandpa-rpc: update pubsub tests with more steps
* grandpa-rpc: fix pubsub test
* grandpa-rpc: minor indendation
* grandpa-rpc: decode instead of encode in test
* grandpa: fix review comments
* grandpa: remove unused serde dependency
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
* Remove service components and add build_network, build_offchain_workers etc
* Improve transaction pool api
* Remove commented out line
* Add PartialComponents
* Add BuildNetworkParams, documentation
* Remove unused imports in tests
* Apply suggestions from code review
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* Remove unused imports in node-bench
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* Simplify a few chain components creation APIs related to the service
* Fix basic-authorship doc tests
* Remove DefaultQueue
* Update client/service/src/builder.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Move ExecutionExtensions comment around
* Remove unused BlakeTwo256
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Improve transaction submission
Before this pr the transaction pool validated each transaction, even if
the transaction was already known to the pool. This pr changes the
behavior to first check if we are already aware of a transaction and
thus, to only validate them if we don't know them yet. However, there is
still the possibility that a given transaction is validated multiple
times. This can happen if the transaction is added the first time, but
is not yet validated and added to the validated pool.
Besides that, this pr fixes the wrong metrics of gossiped transactions
in the network. It also moves some metrics to the transaction pool api,
to better track when a transaction actually is scheduled for validation.
* Make sure we don't submit the same transaction twice from the network concurrently
* Remove added listener call
* Feedback
* Ignore banned on resubmit
* Make NumberOrHex a common primitive.
* Update primitives/rpc/src/number.rs
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* Bump jsonrpc pubsub, core, http, and ws
Right now these are the packages which _need_ to be updated
so I can just the latest `jsonrpc-pubsub` code. Once a release
it cut upstream the rest of the dependencies should be updated
as well.
* Use jsonrpc-pubsub's SubscriptionManager
This places sc-rpc-api::Subscriptions
* Bump jsonrpc-core outside of sc-rpc-*
* Update client/rpc tests
Right now one of the `author` tests is failing, I
need to think a bit about how best to fix it.
* Remove Subscriptions manager
There's no need for this implementation since we're
using the one from `jsonrpc-pubsub` now
* Fix author RPC test
This test used to check for a numerial subscription ID,
whereas now it uses a string based ID which is the default
provided by `jsonrpc-pubsub`'s subscription manager.
* Remove unused NumericIdProvider
* Add missing bracket
Removed one too many with that last one, lol
* Bump `jsonrpc` to v14.2
There's an exception though. `jsonrpc-derive` cannot be bumped
past v14.0.5 just yet since it has a dependency on `quote` pinned
to v1.0.1. This means that at the moment it won't build on Substrate
since it's using v1.0.3.
* Track `jsonrpc-derive` master branch
* Bump `quote` version to v1.0.6
* Bump `jsonrpc-derive` to v14.2.1
This includes support for `quote` v1.0.6
* Use exact version for jsonrpc crates
Doing this to make sure any updates in jsonrpc don't
accidently trickle down to Polkadot.
Co-authored-by: André Silva <andre.beat@gmail.com>
* Make tx pool aware of retracted fork blocks
* Make it compile
* Update client/transaction-pool/src/lib.rs
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* Fix doc test
* Simplify the implementation
* Send tree route as arc to prevent heavy clones
* Switch to use `ExtrinsicHash` to make it more clear
* Fix benchmark
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* Bump jsonrpc pubsub, core, http, and ws
Right now these are the packages which _need_ to be updated
so I can just the latest `jsonrpc-pubsub` code. Once a release
it cut upstream the rest of the dependencies should be updated
as well.
* Use jsonrpc-pubsub's SubscriptionManager
This places sc-rpc-api::Subscriptions
* Bump jsonrpc-core outside of sc-rpc-*
* Update client/rpc tests
Right now one of the `author` tests is failing, I
need to think a bit about how best to fix it.
* Remove Subscriptions manager
There's no need for this implementation since we're
using the one from `jsonrpc-pubsub` now
* Fix author RPC test
This test used to check for a numerial subscription ID,
whereas now it uses a string based ID which is the default
provided by `jsonrpc-pubsub`'s subscription manager.
* Remove unused NumericIdProvider
* Add missing bracket
Removed one too many with that last one, lol
* Bump `jsonrpc` to v14.2
There's an exception though. `jsonrpc-derive` cannot be bumped
past v14.0.5 just yet since it has a dependency on `quote` pinned
to v1.0.1. This means that at the moment it won't build on Substrate
since it's using v1.0.3.
* Track `jsonrpc-derive` master branch
* Bump `quote` version to v1.0.6
* Bump `jsonrpc-derive` to v14.2.1
This includes support for `quote` v1.0.6
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api
* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa
* drop sc-service from sc-rpc
* drop sc-service from sc-consensus-aura
* drop sc-client from manual-seal and babe
* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli
* drop sc-client from bin/node and bin/node-template
* drop sc-client
* fix tests
* remove check -p sc-client from gitlab.yml
* fix warnings
* fixes ui test
* fix light client tests
* adds associated Client type to AbstractService
* adds UsageProvider to Client
* fixed ui test, again
* tried and failed to get node-cli to compile for wasm
* thanks to tomaka for helping me get node-cli to compile for wasmm
* ui test pls pas 🙏🏾
* all tests passing 🪄
* no_run documentation code
* rm -f documentation code
* ClientProvider
* fix mega trait
* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client
* adds license to sc-consensus
Co-authored-by: Benjamin Kampmann <ben@parity.io>
* make new contructor
* add metrics to txpool
* fix review
* fix doc comment
* change to counters
* Update client/transaction-pool/src/metrics.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* Update client/transaction-pool/src/metrics.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* Update client/transaction-pool/src/metrics.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* Update client/transaction-pool/src/lib.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* Update client/transaction-pool/src/lib.rs
Co-Authored-By: Max Inden <mail@max-inden.de>
* use dedicated wrapper
Co-authored-by: Max Inden <mail@max-inden.de>
* Add new RPC method to get the chain type
This adds a new RPC method to get the chain type of the running chain.
The chain type needs to be specified in the chain spec. This should make
it easier for tools/UI to display extra information without needing to
rely on parsing the chain name.
* Update client/rpc-api/src/system/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Primitive crate
* Feedback
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:
- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)
refs #4679
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* Split the Roles bitfield in three
* Forgot to include some changes
* Fix cli test
* More test fixes
* Oh God, merging master broke other tests
* Didn't run the doctests
* Address review
* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine
* Address some review
* Also update the peerset's API to make sense
* Fix peerset tests
* Fix browser node
* client: distinguish between local and network authority
Co-authored-by: André Silva <andre.beat@gmail.com>
* Add KEY_KIND_ID to the public trait
This change is being introduced for the purpose of identifying a public
key with it's identifier and algorithm "kind".
* Use `sign_with` as implemented in BareCryptoStore
* Implement `sign_with` in sc_keystore
* Fix inconsistencies, use *_KIND_ID in sp_core testing
* Rename KeyKindId to CryptoTypeId
* Remove pair-returning functions from BareCryptoStore trait
* Define CryptoTypeId in app-crypto macros
* Add functions to get keys supported by keystore
* Fix sign_with signature to include CryptoTypePublicPair
* Add `sign_with_any` and `sign_with_all`
* Use keystore.sign_with in auth_discovery
* Rename get_supported_keys -> supported_keys
* Added headers to function docstrings
* Use chain instead of extending a temp vector
* Fixed some code formatting
* Restrict size of CryptoTypeId
This is to be able to use Encode/Decode derives and the overcome having
the size being unknown at compile-time.
* Implement sign_with in the trait itself
* Remove whitespace
* Use key_type also as a CryptoTypeId in app_crypto macros
* Rename `get_keys` to `keys` in BareCryptoStore
* Remove usage of key_pair funcs in tests
* Adjust docstring for *_CYPTO_ID constants
* Fix failures
* Simplify mapping on keys
* Remove one let
* Fixed typo
* PR feedback
* remove whitespace
* Zip keys and signatures
* Use into_iter & remove cloned
* Pass index to MissingSignature
* Use typed errors instead of strings for BareCryptoStore
* Implement Debug for trait error
* Use hashsets for better performance for supported_keys
* Make sure keys are inserted into the keystore
* Make sign_with_all return type consistent with `sign_with`
* Rename Error to BareCryptoStoreError
* Rename CRYPT_TYPE_ID -> CRYPTO_ID
* Remove unnecessary CRYPTO_ID declaration in Public trait
* Convert pub key to CryptoTypePublicPair
* Fix use
* Fix code style
* Implement From on CryptoTypePublicPair in app_crypto macros
* Change CryptoTypePublicPair to a struct
* Implement Display on CryptoTypePublicPair
* Pass CryptoTypePublicPair to MissingSignature error
* Adjust docs according to function signature
* Unify keys implementation
* Fix RPC author tests
* Fix stackoverflow
* Tabify spaces
* Pass KeyTypeId to error for easier debugging
* Fix asserts
* Use ToHex to format public key
* Use constants from sp_core
* Rename testing KeyTypeId constants
* Please compiler
* Restore KeyTypeId names
apparently, they're not only used in tests
* Use BareCryptoStoreError instead of String
* Document return value
* Fix borrow check
* Convert to hashset internally
* WIP - iter_keys
* Return raw_public_keys
* Address PR feedback
* Address PR Feedback
* Fix hexdisplay import error
* Update primitives/core/src/traits.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* adds state_queryStorageAt
* make at param for query_storage_at optional
* Update client/rpc/src/state/state_full.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* adds query_storage_at to StateBackend
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Move runtime upgrade to `frame-executive`
Instead of storing the runtime upgraded in the space of `frame-system`,
this moves it to `frame-executive`. We also start storing the
`spec_version` and `impl_version` the last runtime upgrade was applied.
This scheme has multiple advantages:
- We don't need to make sure that runtime upgrade was set.
- This will work in the future when we want to test a runtime upgrade,
because the `on_runtime_upgrade` will be executed automatically when the
runtime changes.
* Move `LAST_RUNTIME_UPGRADE` key to `frame-executive`
* Add some more documentation
* Make sure `spec_version` always increases between runtime upgrades
* Fix test
* Upgrade `parity-multiaddr` to fix compilation
* Also check if the `spec_name` changed
* Remove `RuntimeUpgraded` storage entry
* Don't run on genesis
* removes use of sc_client::Client from sc-rpc
* remove Client impl from sc-finality-benches
* remove client impl from sc-finality-grandpa
* read_proof accepts iterator
* remove generic Executor param from ExecutorProvider
* fix long ass line
* code style changes
* merge with master
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
Pierre Krieger