After some discussion with @kogeler after the we added the rate-limit
middleware it may slow down
the rpc call timings metrics significantly because it works as follows:
1. The rate limit guard is checked when the call comes and if a slot is
available -> process the call
2. If no free spot is available then the call will be sleeping
`jitter_delay + min_time_rate_guard` then woken up and checked at most
ten times
3. If no spot is available after 10 iterations -> the call is rejected
(this may take tens of seconds)
Thus, this PR adds a label "is_rate_limited" to filter those out on the
metrics "substrate_rpc_calls_time" and "substrate_rpc_calls_finished".
I had to merge two middleware layers Metrics and RateLimit to avoid
shared state in a hacky way.
---------
Co-authored-by: James Wilson <james@jsdw.me>
The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.
This PR adds two new CLI options:
```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```
Add RPC server rate limiting which can be utilized by the CLI
`--rpc-rate-limit <calls/per minute>`
Resolves first part of
https://github.com/paritytech/polkadot-sdk/issues/3028
//cc @PierreBesson @kogeler you might be interested in this one
---------
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Close#2992
Breaking changes:
- rpc server grafana metric `substrate_rpc_requests_started` is removed
(not possible to implement anymore)
- rpc server grafana metric `substrate_rpc_requests_finished` is removed
(not possible to implement anymore)
- rpc server ws ping/pong not ACK:ed within 30 seconds more than three
times then the connection will be closed
Added
- rpc server grafana metric `substrate_rpc_sessions_time` is added to
get the duration for each websocket session
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
- Less dependencies for the clients in particular
- Return type requires Clone in method call responses
- Moved to tokio channels
- Async subscription API (not used in this PR)
Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate
The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.
Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.
The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.
Hopefully the last release prior to 1.0, sorry in advance for a big PR
Previous attempt: https://github.com/paritytech/substrate/pull/13992
Resolves https://github.com/paritytech/polkadot-sdk/issues/748, resolves
https://github.com/paritytech/polkadot-sdk/issues/627
For tools such that is using the `Miner` it's useful to know whether a
solution was trimmed or not and also how much that was trimmed.
---------
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
* companion for #13384
* adjust parsing RPC address from process output
* update rpc cli
* update lockfile for {"substrate"}
* bump zombienet v1.3.48
* bump zombienet version
* allow zombienet-tests-misc-upgrade-node to fail
* add comment and issue link to allowed_failure
* grumbles: disable failed job
* disabled the correct test
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Javier Viola <javier@parity.io>
* staking miner: less aggresive submissions
We have noticed that the staking-miner performs many concurrent RPC calls (more than 256).
Probably because these batch request are getting bigger because the state is growing.
So let's relax this and mine solutions sequentially i.e, mine solution
one solution at the time and not in concurrently.
* add check if self hasn't submitted after mining
* frame epm: expose feasibity_check in miner
The goal with this commit is to expose the `feasibity_check` such
that anyone that implements the `MinerConfig trait` can utilize it
* cleanup
* fix tests
* jsonrpsee v0.16
add backwards compatibility
run old http server on http only
* cargo fmt
* update jsonrpsee 0.16.1
* less verbose cors log
* fix nit in log: WS -> HTTP
* revert needless changes in Cargo.lock
* remove unused features in tower
* fix nits; add client-core feature
* jsonrpsee v0.16.2
* staking miner: remove needless queue check
If the queue is full and the "mined solution" is better than solutions in the queue according to the "strategy"
then the solution with worse score will be kicked out from the queue.
Thus, the check can be removed completly.
* fix compile warns
* staking miner: CLI request and connection timeout
Make it possible to configure the `connection timeout` and `request timeout` via CLI flags.
* cargo fmt
* bump jsonrpsee to fix#11480
In addition it adds WebSocket server-side pings which is configured to
send out pings periodically every 30 seconds.
* use released crates.io version
* Update Cargo.toml
The westend runtime was missing the fast runtime for `EpochDuration`
which caused the epoch be 1 hour which isn't fast.
This PR changes the `EpochDuration` to 2 minutes when `--fast-runtime`
is enabled, the same as polkadot and kusama.
* staking miner: use config for emergency solution
Fixes#4678
* bump jsonrpsee
* run `monitor_cmd_for` until the connection is closed
* new tokio task for submit_and_watch xt
* re-use header subscription
* update jsonrpsee + simplify code
* revert polkadot runtime changes
* feat: add `ensure_no_better_solution` function
* storage access for submissions and indices
* check ensure_no_previous_solution before remote ext
* fix todos
* grumbles: Perbill::from_percent
* hacky fix
* use modified EPM pallet and various fixes
* diener update --substrate --branch na-epm-pub
* Revert "diener update --substrate --branch na-epm-pub"
This reverts commit b3b9a58c9313372c8f21cf247ba0c8528d9953c0.
* update substrate
* tokio spawn on concurrent stuff
* cleanup
* Update utils/staking-miner/src/monitor.rs
* Update utils/staking-miner/src/monitor.rs
* more cleanup
* fix nits
* address grumbles
* only run batch reqs when signed phase
* better help menu for submission strategy CLI
* add tests for submission strategy
* A clean new attempt
* Checkpoint to move remote.
* A lot of dependency wiring to make it feature gated.
* bad macro, bad macro.
* refactor(remote ext): use jsonrpsee
* refactor(remote ext): use jsonrpsee
* Undo the DB mess.
* fix(remote ext): use max limit `u32::MAX`
* resolve TODOs
* jsonrpsee switch to `hyper` as backend
* Update utils/frame/try-runtime/remote-externalities/src/lib.rs
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
* update jsonrpsee
* remove boiler-plate
* suppress warnings to CI happy
* Unbreak his build
* Use option
* fix nit; make it work again
* fix err message.
* Update utils/frame/remote-externalities/Cargo.toml
* Fix uri stuff
* remove needless clone
Co-authored-by: kianenigma <kian.peymani@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>