* client/beefy: create communication module and move gossip there
* client/beefy: move beefy_protocol_name module to communication
* client/beefy: move notification module under communication
* client/beefy: add incoming request_response protocol handler
* client/beefy: keep track of connected peers and their progress
* client/beefy: add logic for generating Justif requests
* client/beefy: cancel outdated on-demand justification requests
* try Andre's suggestion for JustificationEngine
* justif engine add justifs validation
* client/beefy: impl OnDemandJustificationsEngine async next()
* move beefy proto name test
* client/beefy: initialize OnDemandJustificationsEngine
* client/tests: allow for custom req-resp protocols
* client/beefy: on-demand-justif: implement simple peer selection strategy
* client/beefy: fix voter initialization
Fix corner case where voter gets a single burst of finality
notifications just when it starts.
The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.
Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.
Add regression test for this.
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: make sure justif requests are always out for mandatory blocks
* client/beefy: add test for on-demand justifications sync
* client/beefy: tweak main loop event processing order
* client/beefy: run on-demand-justif-handler under same async task as voter
* client/beefy: add test for known-peers
* client/beefy: reorg request-response module
* client/beefy: add issue references for future work todos
* client/beefy: consolidate on-demand-justifications engine state machine
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: fix for polkadot companion
* client/beefy: implement review suggestions
* cargo fmt and clippy
* fix merge damage
* fix rust-doc
* fix merge damage
* fix merge damage
* client/beefy: add test for justif proto name
Signed-off-by: acatangiu <adrian@parity.io>
* beefy: use VersionedFinalityProof instead of SignedCommitment.
* Change the exposed RPC API to support versioned proofs.
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* client/beefy: don't accept vote for older rounds
* client/beefy: clean up and reorg the worker struct
* client/beefy: first step towards Full BEEFY
The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.
There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.
Implementation details:
- Introduced 'VoterOracle' to separate the voting decisions logic,
and track new/pending sessions.
- New sessions get queued up with the worker operating either:
1. up-to-date - all mandatory blocks leading up to current GRANDPA
finalized: queue has ONE element, the 'current session' where
`mandatory_done == true`,
2. lagging behind GRANDPA: queue has [1, N] elements, where all
`mandatory_done == false`.
In this state, everytime a session gets its mandatory block
BEEFY finalized, the session is popped off the queue,
eventually getting to operating mode `1. up-to-date`.
- Votes get triaged and those that fall withing the `VoterOracle`
allowed window get processed, the others get dropped if stale,
or buffered for later processing (when they reach the window).
- Worker general code was also updated to fall in one of two roles:
1. react to external events and change internal 'state',
2. generate events/votes based on internal 'state'.
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: sketch idea for block import and sync
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: add BEEFY block import
* client/beefy: process justifications from block import
* client/beefy: add TODOs for sync protocol
* client/beefy: add more docs and comments
* client/beefy-rpc: fix RPC error
* client/beefy: verify justification validity on block import
* client/beefy: more tests
* client/beefy: small fixes
- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.
* client/beefy: remove invalid justifications at block import
* todo: beefy block import tests
* RFC: ideas for multiple justifications per block
* Revert "RFC: ideas for multiple justifications per block"
This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.
* client/beefy: append justif to backend on block import
* client/beefy: groundwork for block import test
* client/beefy: groundwork2 for block import test
* client/beefy: groundwork3 for block import test
* client/beefy: add block import test
* client/beefy: add required trait bounds to block import builder
* remove client from beefy block import, backend gets the job done
Signed-off-by: acatangiu <adrian@parity.io>
* bump jsonrpsee to fix#11480
In addition it adds WebSocket server-side pings which is configured to
send out pings periodically every 30 seconds.
* use released crates.io version
* Update Cargo.toml
* Add tokio
* No need to map CallError to CallError
* jsonrpsee proc macros (#9673)
* port error types to `JsonRpseeError`
* migrate chain module to proc macro api
* make it compile with proc macros
* update branch
* update branch
* update to jsonrpsee master
* port system rpc
* port state rpc
* port childstate & offchain
* frame system rpc
* frame transaction payment
* bring back CORS hack to work with polkadot UI
* port babe rpc
* port manual seal rpc
* port frame mmr rpc
* port frame contracts rpc
* port finality grandpa rpc
* port sync state rpc
* resolve a few TODO + no jsonrpc deps
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* fix nit in frame system api
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* update jsonrpsee
* update jsonrpsee
* cleanup rpc-servers crate
* jsonrpsee: add host and origin filtering (#9787)
* add access control in the jsonrpsee servers
* use master
* fix nits
* rpc runtime_version safe
* fix nits
* fix grumbles
* remove unused files
* resolve some todos
* jsonrpsee more cleanup (#9803)
* more cleanup
* resolve TODOs
* fix some unwraps
* remove type hints
* update jsonrpsee
* downgrade zeroize
* pin jsonrpsee rev
* remove unwrap nit
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* RPC tests for state, author and system (#9859)
* Fix test runner
* Impl Default for SubscriptionTaskExecutor
* Keep the minimul amount of code needed to compile tests
* Re-instate `RpcSession` (for now)
* cleanup
* Port over RPC tests
* Add tokio
* No need to map CallError to CallError
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* offchain rpc tests
* Address todos
* fmt
Co-authored-by: James Wilson <james@jsdw.me>
* fix drop in state test
* update jsonrpsee
* fix ignored system test
* fix chain tests
* remove some boiler plate
* Port BEEFY RPC (#9883)
* Merge master
* Port beefy RPC (ty @niklas!)
* trivial changes left over from merge
* Remove unused code
* Update jsonrpsee
* fix build
* make tests compile again
* beefy update jsonrpsee
* fix: respect rpc methods policy
* update cargo.lock
* update jsonrpsee
* update jsonrpsee
* downgrade error logs
* update jsonrpsee
* Fix typo
* remove unused file
* Better name
* Port Babe RPC tests
* Put docs back
* Resolve todo
* Port tests for System RPCs
* Resolve todo
* fix build
* Updated jsonrpsee to current master
* fix: port finality grandpa rpc tests
* Move .into() outside of the match
* more review grumbles
* jsonrpsee: add `rpc handlers` back (#10245)
* add back RpcHandlers
* cargo fmt
* fix docs
* fix grumble: remove needless alloc
* resolve TODO
* fmt
* Fix typo
* grumble: Use constants based on BASE_ERROR
* grumble: DRY whitelisted listening addresses
grumble: s/JSONRPC/JSON-RPC/
* cleanup
* grumbles: Making readers aware of the possibility of gaps
* review grumbles
* grumbles
* remove notes from niklasad1
* Update `jsonrpsee`
* fix: jsonrpsee features
* jsonrpsee: fallback to random port in case the specified port failed (#10304)
* jsonrpsee: fallback to random port
* better comment
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* address grumbles
* cargo fmt
* addrs already slice
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update jsonrpsee to 092081a0a2b8904c6ebd2cd99e16c7bc13ffc3ae
* lockfile
* update jsonrpsee
* fix warning
* Don't fetch jsonrpsee from crates
* make tests compile again
* fix rpc tests
* remove unused deps
* update tokio
* fix rpc tests again
* fix: test runner
`HttpServerBuilder::builder` fails unless it's called within tokio runtime
* cargo fmt
* grumbles: fix subscription aliases
* make clippy happy
* update remaining subscriptions alias
* cleanup
* cleanup
* fix chain subscription: less boiler plate (#10285)
* fix chain subscription: less boiler plate
* fix bad merge
* cargo fmt
* Switch to jsonrpsee 0.5
* fix build
* add missing features
* fix nit: remove needless Box::pin
* Integrate jsonrpsee metrics (#10395)
* draft metrics impl
* Use latest api
* Add missing file
* Http server metrics
* cleanup
* bump jsonrpsee
* Remove `ServerMetrics` and use a single middleware for both connection counting (aka sessions) and call metrics.
* fix build
* remove needless Arc::clone
* Update to jsonrpsee 0.6
* lolz
* fix metrics
* Revert "lolz"
This reverts commit eed6c6a56e78d8e307b4950f4c52a1c3a2322ba1.
* fix: in-memory rpc support subscriptions
* commit Cargo.lock
* Update tests to 0.7
* fix TODOs
* ws server: generate subscriptionIDs as Strings
Some libraries seems to expect the subscription IDs to be Strings, let's not break
this in this PR.
* Increase timeout
* Port over tests
* cleanup
* Using error codes from the spec
* fix clippy
* cargo fmt
* update jsonrpsee
* fix nits
* fix: rpc_query
* enable custom subid gen through spawn_tasks
* remove unsed deps
* unify tokio deps
* Revert "enable custom subid gen through spawn_tasks"
This reverts commit 5c5eb70328fe39d154fdb55c56e637b4548cf470.
* fix bad merge of `test-utils`
* fix more nits
* downgrade wasm-instrument to 0.1.0
* [jsonrpsee]: enable custom RPC subscription ID generatation (#10731)
* enable custom subid gen through spawn_tasks
* fix nits
* Update client/service/src/builder.rs
Co-authored-by: David <dvdplm@gmail.com>
* add Poc; needs jsonrpsee pr
* update jsonrpsee
* add re-exports
* add docs
Co-authored-by: David <dvdplm@gmail.com>
* cargo fmt
* fmt
* port RPC-API dev
* Remove unused file
* fix nit: remove async trait
* fix doc links
* fix merge nit: remove jsonrpc deps
* kill namespace on rpc apis
* companion for jsonrpsee v0.10 (#11158)
* companion for jsonrpsee v0.10
* update versions v0.10.0
* add some fixes
* spelling
* fix spaces
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* send error before subs are closed
* fix unsubscribe method names: chain
* fix tests
* jsonrpc server: print binded local address
* grumbles: kill SubscriptionTaskExecutor
* Update client/sync-state-rpc/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* sync-state-rpc: kill anyhow
* no more anyhow
* remove todo
* jsonrpsee: fix bad params in subscriptions. (#11251)
* update jsonrpsee
* fix error responses
* revert error codes
* dont do weird stuff in drop impl
* rpc servers: remove needless clone
* Remove silly constants
* chore: update jsonrpsee v0.12
* commit Cargo.lock
* deps: downgrade git2
* feat: CLI flag max subscriptions per connection
* metrics: use old logging format
* fix: read WS address from substrate output (#11379)
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Maciej Hirsz <hello@maciej.codes>
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Upgraded dependencies
* Adapting code to scale v3
* Empty commit to trigger CI
* Triggering CI
* Fixing UI test
* Remove superfluous dev-dep added by #9228
* Cryout for CI
* beefy: add dummy latest_finalized() RPC
* beefy: rpc latest_best_beefy() using shared mem
* beefy: rpc populate latest_best_beefy()
* beefy: rpc handle readiness
* beefy: best block over channel - wip
Not working because channel can't be simply opened and receiver passed
to `rpc_extensions_builder` because `rpc_extensions_builder` has to be
`Fn` and not `FnOnce`... and and Receiver side of mpsc can't be cloned
yay!..
* beefy: make notification channels payload-agnostic
* beefy: use notification mechanism instead of custom channel
* beefy: add tracing key to notif channels
* sc-utils: add notification channel - wip
* beefy: use sc-utils generic notification channel
* grandpa: use sc-utils generic notification channel
* fix grumbles
* beefy-rpc: get best block header instead of number
* beefy-rpc: rename to `beefy_getFinalizedHead`
* fix nitpicks
* client-rpc-notifications: move generic Error from struct to fn
* beefy: use header from notification instead of getting from database
* beefy-rpc: get best block hash instead of header
* beefy-rpc: fix and improve latestHead test
* beefy-rpc: bubble up errors from rpc-handler instantiation
* update lockfile
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* fix errors and warnings
* fix nit
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Introduce `SecretUri`
* `inspect-key`: Adds support for `expect-public`
`expect-public` can be used to check that a given secret uri corresponds to the given public key.
This is mainly useful when the secret uri is protected by a password and a new derived account
should be generated. With `--expect-public` the user can pass the public key/account-id of the
"base" secret uri aka the one without any derivation to ensure the correct password was inserted.
* Fixes
* 🤦
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Review feedback
* FMT
* Bump the versions
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Intend to reactivate cargo-unleash check
It appears the bug it was deactivated for has been resolved a while ago. Trying to reactivate the checks.
* adding missing cargo.toml metadata for BEEFY crates
* fix wrong version reference
* matching up versions
* disable faulty cache
* switching more versions to prerelease
* Revert "disable faulty cache"
This reverts commit 411a12ae444a9695a8bfea4458a868438d870b06.
* bump minor of sc-allocator to fix already-published-issue
* fixup another pre-released dependency problem
* temp switch to latest unleash
* fixing dependency version and features
* prometheus endpoint has also been changed
* fixing proposer metrics versioning
* fixing hex feature for beefy
* fix generate-bags feature selection
* fixup Cargo.lock
* upgrade prometheus dependencies
* missed one
* switch to latest release
Adrian Catangiu