* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency
* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`
* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls
* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Minimize `sc-authority-discovery` dependency on `sc-network`
* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`
* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`
* Move more methods from `NetworkService` into `NetworkPeers` trait
* Move `NetworkStateInfo` trait into `sc-network-common`
* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`
* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl
* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`
* Remove dependencies on `NetworkService` from most of the methods of `sc-service`
* Address simple review comments
Before `highestBlock` was an optional that was omitted when it was `None`. We recently changed the
way the `highestBlock` is determined, this resulted in having this value in 99.99% of the time being
`None` when the node is syncing blocks at the tip. Now we always return a block for `highestBlock`.
If sync doesn't return us any best seen block, we return our own local best block as `highestBlock`.
This should mainly reflect the same behavior to before we changed the way the best seen block is determined.
* Add tokio
* No need to map CallError to CallError
* jsonrpsee proc macros (#9673)
* port error types to `JsonRpseeError`
* migrate chain module to proc macro api
* make it compile with proc macros
* update branch
* update branch
* update to jsonrpsee master
* port system rpc
* port state rpc
* port childstate & offchain
* frame system rpc
* frame transaction payment
* bring back CORS hack to work with polkadot UI
* port babe rpc
* port manual seal rpc
* port frame mmr rpc
* port frame contracts rpc
* port finality grandpa rpc
* port sync state rpc
* resolve a few TODO + no jsonrpc deps
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* fix nit in frame system api
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* update jsonrpsee
* update jsonrpsee
* cleanup rpc-servers crate
* jsonrpsee: add host and origin filtering (#9787)
* add access control in the jsonrpsee servers
* use master
* fix nits
* rpc runtime_version safe
* fix nits
* fix grumbles
* remove unused files
* resolve some todos
* jsonrpsee more cleanup (#9803)
* more cleanup
* resolve TODOs
* fix some unwraps
* remove type hints
* update jsonrpsee
* downgrade zeroize
* pin jsonrpsee rev
* remove unwrap nit
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* RPC tests for state, author and system (#9859)
* Fix test runner
* Impl Default for SubscriptionTaskExecutor
* Keep the minimul amount of code needed to compile tests
* Re-instate `RpcSession` (for now)
* cleanup
* Port over RPC tests
* Add tokio
* No need to map CallError to CallError
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* offchain rpc tests
* Address todos
* fmt
Co-authored-by: James Wilson <james@jsdw.me>
* fix drop in state test
* update jsonrpsee
* fix ignored system test
* fix chain tests
* remove some boiler plate
* Port BEEFY RPC (#9883)
* Merge master
* Port beefy RPC (ty @niklas!)
* trivial changes left over from merge
* Remove unused code
* Update jsonrpsee
* fix build
* make tests compile again
* beefy update jsonrpsee
* fix: respect rpc methods policy
* update cargo.lock
* update jsonrpsee
* update jsonrpsee
* downgrade error logs
* update jsonrpsee
* Fix typo
* remove unused file
* Better name
* Port Babe RPC tests
* Put docs back
* Resolve todo
* Port tests for System RPCs
* Resolve todo
* fix build
* Updated jsonrpsee to current master
* fix: port finality grandpa rpc tests
* Move .into() outside of the match
* more review grumbles
* jsonrpsee: add `rpc handlers` back (#10245)
* add back RpcHandlers
* cargo fmt
* fix docs
* fix grumble: remove needless alloc
* resolve TODO
* fmt
* Fix typo
* grumble: Use constants based on BASE_ERROR
* grumble: DRY whitelisted listening addresses
grumble: s/JSONRPC/JSON-RPC/
* cleanup
* grumbles: Making readers aware of the possibility of gaps
* review grumbles
* grumbles
* remove notes from niklasad1
* Update `jsonrpsee`
* fix: jsonrpsee features
* jsonrpsee: fallback to random port in case the specified port failed (#10304)
* jsonrpsee: fallback to random port
* better comment
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* address grumbles
* cargo fmt
* addrs already slice
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update jsonrpsee to 092081a0a2b8904c6ebd2cd99e16c7bc13ffc3ae
* lockfile
* update jsonrpsee
* fix warning
* Don't fetch jsonrpsee from crates
* make tests compile again
* fix rpc tests
* remove unused deps
* update tokio
* fix rpc tests again
* fix: test runner
`HttpServerBuilder::builder` fails unless it's called within tokio runtime
* cargo fmt
* grumbles: fix subscription aliases
* make clippy happy
* update remaining subscriptions alias
* cleanup
* cleanup
* fix chain subscription: less boiler plate (#10285)
* fix chain subscription: less boiler plate
* fix bad merge
* cargo fmt
* Switch to jsonrpsee 0.5
* fix build
* add missing features
* fix nit: remove needless Box::pin
* Integrate jsonrpsee metrics (#10395)
* draft metrics impl
* Use latest api
* Add missing file
* Http server metrics
* cleanup
* bump jsonrpsee
* Remove `ServerMetrics` and use a single middleware for both connection counting (aka sessions) and call metrics.
* fix build
* remove needless Arc::clone
* Update to jsonrpsee 0.6
* lolz
* fix metrics
* Revert "lolz"
This reverts commit eed6c6a56e78d8e307b4950f4c52a1c3a2322ba1.
* fix: in-memory rpc support subscriptions
* commit Cargo.lock
* Update tests to 0.7
* fix TODOs
* ws server: generate subscriptionIDs as Strings
Some libraries seems to expect the subscription IDs to be Strings, let's not break
this in this PR.
* Increase timeout
* Port over tests
* cleanup
* Using error codes from the spec
* fix clippy
* cargo fmt
* update jsonrpsee
* fix nits
* fix: rpc_query
* enable custom subid gen through spawn_tasks
* remove unsed deps
* unify tokio deps
* Revert "enable custom subid gen through spawn_tasks"
This reverts commit 5c5eb70328fe39d154fdb55c56e637b4548cf470.
* fix bad merge of `test-utils`
* fix more nits
* downgrade wasm-instrument to 0.1.0
* [jsonrpsee]: enable custom RPC subscription ID generatation (#10731)
* enable custom subid gen through spawn_tasks
* fix nits
* Update client/service/src/builder.rs
Co-authored-by: David <dvdplm@gmail.com>
* add Poc; needs jsonrpsee pr
* update jsonrpsee
* add re-exports
* add docs
Co-authored-by: David <dvdplm@gmail.com>
* cargo fmt
* fmt
* port RPC-API dev
* Remove unused file
* fix nit: remove async trait
* fix doc links
* fix merge nit: remove jsonrpc deps
* kill namespace on rpc apis
* companion for jsonrpsee v0.10 (#11158)
* companion for jsonrpsee v0.10
* update versions v0.10.0
* add some fixes
* spelling
* fix spaces
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* send error before subs are closed
* fix unsubscribe method names: chain
* fix tests
* jsonrpc server: print binded local address
* grumbles: kill SubscriptionTaskExecutor
* Update client/sync-state-rpc/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* sync-state-rpc: kill anyhow
* no more anyhow
* remove todo
* jsonrpsee: fix bad params in subscriptions. (#11251)
* update jsonrpsee
* fix error responses
* revert error codes
* dont do weird stuff in drop impl
* rpc servers: remove needless clone
* Remove silly constants
* chore: update jsonrpsee v0.12
* commit Cargo.lock
* deps: downgrade git2
* feat: CLI flag max subscriptions per connection
* metrics: use old logging format
* fix: read WS address from substrate output (#11379)
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Maciej Hirsz <hello@maciej.codes>
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove unnecessary imports, move one internal re-export into where it is actually used, make one import explicit
* Move a few data structures down into modules
* Use generic parameters in `sc-network` instead of `chain::Client` trait
* Remove unnecessary bound
* Add a new host function for reporting fatal errors
* Fix one of the wasmtime executor tests
* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only
* Print out errors through `Display` instead of `Debug`
* Switch one more trait to require `Error` for its error instead of only `Debug`
* Align to review comments
* Add stale branches heads to finality notifications
Warning. Previous implementation was sending a notification for
each block between the previous (explicitly) finalized block and
the new finalized one (with an hardcoded limit of 256).
Now finality notification is sent only for the new finalized head and it
contains the hash of the new finalized head, new finalized head header,
a list of all the implicitly finalized blocks and a list of stale
branches heads (i.e. the branches heads that are not part of the
canonical chain anymore).
* Add implicitly finalized blocks list to `ChainEvent::Finalized` message
The list contains all the blocks between the previously finalized block
up to the parent of the currently finalized one, sorted by block number.
`Finalized` messages handler, part of the `MaintainedTransactionPool`
implementation for `BasicPool`, still propagate full set of finalized
blocks to the txpool by iterating over implicitly finalized blocks list.
* Rust fmt
* Greedy evaluation of `stale_heads` during finalization
* Fix outdated assumption in a comment
* Removed a test optimization that is no more relevant
The loop was there to prevent sending to
`peer.network.on_block_finalized` the full list of finalized blocks.
Now only the finalized heads are received.
* Last finalized block lookup not required anymore
* Tests for block finality notifications payloads
* Document a bit tricky condition to avoid duplicate finalization notifications
* More idiomatic way to skip an iterator entry
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Cargo fmt iteration
* Typo fix
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix potential failure when a finalized orphan block is imported
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Removal of light client from substrate
* add missing import
* These tests relate to there being light and non light clients.
* removing lightnodes from test
* cargo fmt
* not needed
* LightDataChecker not needed any longer
* cargo fmt
* Update client/service/test/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/service/test/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* cargo fmt
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Use tokio runtime handle instead of TaskExecutor abstraction
Before this pr we had the `TaskExecutor` abstraction which theoretically
allowed that any futures executor could have been used. However, this
was never tested and is currently not really required. Anyone running a
node currently only used tokio and nothing else (because this was hard
coded in CLI). So, this pr removes the `TaskExecutor` abstraction and
relies directly on the tokio runtime handle.
Besides this changes, this pr also makes sure that the http and ws rpc
server use the same tokio runtime. This fixes a panic that occurred when
you drop the rpc servers inside an async function (tokio doesn't like
that a tokio runtime is dropped in the async context of another tokio
runtime).
As we don't use any custom runtime in the http rpc server anymore, this
pr also removes the `rpc-http-threads` cli argument. If external parties
complain that there aren't enough threads for the rpc server, we could
bring support for increasing the thread count of the tokio runtime.
* FMT
* Fix try runtime
* Fix integration tests and some other optimizations
* Remove warnings
* sp-utils => sc-utils
* cargo fmt
* These files are now in the client so should be licensed as GPL3
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* implement "auto" database backend in client/db, in progress, #9201
* move fn supports_ref_counting from DatabaseSource enum to Database trait to make it work correctly for all types of dbs
* update kvdb_rocksdb to 0.13 and use it's new config feature to properly auto start existing database
* tests for auto database reopening
* introduce OpenDbError to cleanup opening database error handling and handle case when database is not enabled at the compile time
* cargo fmt strings again
* cargo fmt strings again
* rename DataSettingsSrc to fix test compilation
* fix the call to the new kvdb-rocksdb interdace in tests to fix compilation
* simplify OpenDbError and make it compile even when paritydb and rocksdb are disabled
* cargo fmt
* fix compilation without flag with-parity-db
* fix unused var compilation warning
* support different paths for rocksdb and paritydb in DatabaseSouce::Auto
* support "auto" database option in substrate cli
* enable Lz4 compression for some of the parity-db colums as per review suggestion
* applied review suggestions
* Decouples light-sync state from chain spec
This decouples the light-sync state from chain spec. First, the
light-sync state currently only works with BABE+Grandpa, so not
all *Substrate* based chains can use this feature. The next problem was
also that this pulled the `sc-consensus-babe` and `sc-finality-grandpa`
crate into `sc-chain-spec`.
If a chain now wants to support the light-sync state, it needs to add
the `LightSyncStateExtension` to the chain spec as an extension. This is
documented in the crate level docs of `sc-sync-state-rpc`. If this
extension is not available, `SyncStateRpc` fails at initialization.
* Fix compilation for browser
* Fmt
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Make it possible to override maximum payload of RPC
* Finish it.
* remove todo.
* Update client/cli/src/commands/run_cmd.rs
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Incorporate suggestions
* Thread rpc_max_payload from configuration to trace_block
* Try obey line gitlab/check_line_width.sh
* update state rpc tests
* Improve readbility
* Apply suggestions from code review
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
* Lol
* Yeah
* Moare
* adaasda
* Convert AURA to new pallet macro
* AURA: Switch to `CurrentSlot` instead of `LastTimestamp`
This switches AURA to use `CurrentSlot` instead of `LastTimestamp`.
* Add missing file
* Update frame/aura/src/migrations.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Remove the runtime side provide inherent code
* Use correct weight
* Add TODO
* Remove the Inherent from AURA
* 🤦
* Remove unused stuff
* Update primitives authorship
* Fix babe inherent data provider
* Fix consensus-uncles
* Fix BABE
* Do some further changes to authorship primitives... :D
* More work
* Make it compile the happy path
* Make it async!
* Take hash
* More stuff
* Hacks
* Revert "Hacks"
This reverts commit cfffad88668cfdebf632a59c4fbfada001ef8251.
* Fix
* Make `execute_block` return the final block header
* Move Aura digest stuff
* Make it possible to disable equivocation checking
* Fix fix fix
* Some refactorings
* Comment
* Fixes fixes fixes
* More cleanups
* Some love
* Better love
* Make slot duration being exposed as `Duration` to the outside
* Some slot info love
* Add `build_aura_worker` utility function
* Copy copy copy
* Some stuff
* Start fixing pow
* Fix pow
* Remove some bounds
* More work
* Make grandpa work
* Make slots use `async_trait`
* Introduce `SharedData`
* Add test and fix bugs
* Switch to `SharedData`
* Make grandpa tests working
* More Babe work
* Make grandpa work
* Introduce `SharedData`
* Add test and fix bugs
* Switch to `SharedData`
* Make grandpa tests working
* More Babe work
* Make it async
* Fix fix
* Use `async_trait` in sc-consensus-slots
This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.
* Make grandpa tests compile
* More Babe tests work
* Fix network test
* Start fixing service test
* Finish service-test
* Fix sc-consensus-aura
* Fix fix fix
* More fixes
* Make everything compile *yeah*
* Make manual-seal compile
* More fixes
* Start fixing Aura
* Fix Aura tests
* Fix Babe tests
* Make everything compile
* Move code around and switch to async_trait
* Fix Babe
* Docs docs docs
* Move to FRAME
* Fix fix fix
* Make everything compile
* Last cleanups
* Fix integration test
* Change slot usage of the timestamp
* We really need to switch to `impl-trait-for-tuples`
* Update primitives/inherents/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update primitives/inherents/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update primitives/inherents/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Some extra logging
* Remove dbg!
* Update primitives/consensus/common/src/import_queue/basic_queue.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Add boilerplate for JSON-RPC layer for reserved nodes
* Add more boilerplate for JSON-RPC layer for reserved nodes
* Make JSON-RPC layer for reserved nodes async
* Use more realistic data in reserver_peers tests
* Make JSON-RPC layer for reserved nodes blocking
* Apply tomaka's suggestion to reduce .into_iter() for an iter
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Simplify some Option / Result / ? operator patterns
When those match a combinator exactly.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
* adjust after review
* adjust post-review
* Add MMR custom RPC.
* Change RuntimeApi to avoid hardcoding leaf type.
* Properly implement the new RuntimeAPI and wire up RPC.
* Extract Offchain DB as separate execution extension.
* Enable offchain DB access for offchain calls.
* Fix offchain_election tests.
* Skip block initialisation for proof generation.
* Fix integration test setup.
* Fix offchain tests. Not sure how I missed them earlier 🤷.
* Fix long line.
* One more test missing.
* Update mock for multi-phase.
* Address review grumbbles.
* Address review grumbles.
* Fix line width of a comment
* Sync: Propagate block announcement data
This pr adds a feature to the sync protocol to propagate the data that
we received alongside a block announcement. This is done by adding a
cache that caches the last X block announcement data where X is set to
the number of `in_peers` (giving every peer the chance to send us a
different block). This will be required by parachains to ensure that
even peers who are not connected to a collator receive the data
alongside the block announcement to properly validate it and request the
block.
* Review comment
* Bring back the code and add new variant to ensure we don't insert block
announce data when something wasn't checked
* Also use out_peers
* Remove `RpcMetrics` weirdness
The metrics was returning an error when prometheus was not given. This
was a really weird setup, especially when compared to all other metrics
that just do nothing if there is no registry.
* Fix browser build
* Inform sync explicitly about new best block
Instead of "fishing" the new best block out of the processed blocks, we
now tell sync directly that there is a new best block. It also makes
sure that we update the corresponding sync handshake to the new best
block. This is required for parachains as they first import blocks and
declare the new best block after being made aware of it by the relay chain.
* Adds test
* Make sure async stuff had time to run
Nazar Mokrynskyi