* Try to fix flaky `temp-base-path-work` test
The test is most of the time failing when checking if the database path was deleted. The assumption
is that it takes a little bit more time by the OS to actually clean up the temp path under high
load. The pr tries to fix this by checking multiple times if the path was deleted. Besides that it
also ensures that the tests that require the benchmark feature don't fail when compiled without the feature.
* ".git/.scripts/commands/fmt/fmt.sh"
* Capture signals earlier
* Rewrite tests to let them having one big timeout
* Remove unneeded dep
* Update bin/node/cli/tests/common.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Review feedback
* Update bin/node/cli/tests/common.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
---------
Co-authored-by: command-bot <>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Make CLI state pruning optional again
The state pruning setting is stored in the database when it is created. In later runs it is fine to
drop the `--state-pruning` CLI argument as the setting is stored in the database. The state db will
only return an error if the stored state pruning doesn't match the state pruning given via CLI.
Recently we improved the state pruning CLI handling and accidentally made the state pruning value
always present (as we set some default value for the clap). If we could find out if a user has
passed a value or the default value was taken, we could keep the default value in the CLI interface,
but clap isn't supporting this right now. So, we need to go back and make `state_pruning` an
optional with the default written into the docs.
It also adds a test to ensure that we don't break this behavior again.
* More docs
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* Add tokio
* No need to map CallError to CallError
* jsonrpsee proc macros (#9673)
* port error types to `JsonRpseeError`
* migrate chain module to proc macro api
* make it compile with proc macros
* update branch
* update branch
* update to jsonrpsee master
* port system rpc
* port state rpc
* port childstate & offchain
* frame system rpc
* frame transaction payment
* bring back CORS hack to work with polkadot UI
* port babe rpc
* port manual seal rpc
* port frame mmr rpc
* port frame contracts rpc
* port finality grandpa rpc
* port sync state rpc
* resolve a few TODO + no jsonrpc deps
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Update bin/node/rpc-client/src/main.rs
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* fix nit in frame system api
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* update jsonrpsee
* update jsonrpsee
* cleanup rpc-servers crate
* jsonrpsee: add host and origin filtering (#9787)
* add access control in the jsonrpsee servers
* use master
* fix nits
* rpc runtime_version safe
* fix nits
* fix grumbles
* remove unused files
* resolve some todos
* jsonrpsee more cleanup (#9803)
* more cleanup
* resolve TODOs
* fix some unwraps
* remove type hints
* update jsonrpsee
* downgrade zeroize
* pin jsonrpsee rev
* remove unwrap nit
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* RPC tests for state, author and system (#9859)
* Fix test runner
* Impl Default for SubscriptionTaskExecutor
* Keep the minimul amount of code needed to compile tests
* Re-instate `RpcSession` (for now)
* cleanup
* Port over RPC tests
* Add tokio
* No need to map CallError to CallError
* Port over system_ rpc tests
* Make it compile
* Use prost 0.8
* Use prost 0.8
* Make it compile
* Ignore more failing tests
* Comment out WIP tests
* Update lockfile
* No more juggling tokio versions
* No more wait_for_stop ?
* Remove browser-testing
* Arguments must be arrays
* Use same argument names
* Resolve todo: no wait_for_stop for WS server
Add todo: is parse_rpc_result used?
Cleanup imports
* fmt
* log
* One test passes
* Comment out more tests that aren't ported
* Comment out more tests
* Fix tests after merge
* Subscription test
* Invalid nonce test
* Pending exts
* WIP removeExtrinsic test
* Test remove_extrinsic
* Make state test: should_return_storage work
* Uncomment/fix the other non-subscription related state tests
* test: author_insertKey
* test: author_rotateKeys
* Get rest of state tests passing
* asyncify a little more
* Add todo to note #msg change
* Crashing test for has_session_keys
* Fix error conversion to avoid stack overflows
Port author_hasSessionKeys test
fmt
* test author_hasKey
* Add two missing tests
Add a check on the return type
Add todos for James's concerns
* offchain rpc tests
* Address todos
* fmt
Co-authored-by: James Wilson <james@jsdw.me>
* fix drop in state test
* update jsonrpsee
* fix ignored system test
* fix chain tests
* remove some boiler plate
* Port BEEFY RPC (#9883)
* Merge master
* Port beefy RPC (ty @niklas!)
* trivial changes left over from merge
* Remove unused code
* Update jsonrpsee
* fix build
* make tests compile again
* beefy update jsonrpsee
* fix: respect rpc methods policy
* update cargo.lock
* update jsonrpsee
* update jsonrpsee
* downgrade error logs
* update jsonrpsee
* Fix typo
* remove unused file
* Better name
* Port Babe RPC tests
* Put docs back
* Resolve todo
* Port tests for System RPCs
* Resolve todo
* fix build
* Updated jsonrpsee to current master
* fix: port finality grandpa rpc tests
* Move .into() outside of the match
* more review grumbles
* jsonrpsee: add `rpc handlers` back (#10245)
* add back RpcHandlers
* cargo fmt
* fix docs
* fix grumble: remove needless alloc
* resolve TODO
* fmt
* Fix typo
* grumble: Use constants based on BASE_ERROR
* grumble: DRY whitelisted listening addresses
grumble: s/JSONRPC/JSON-RPC/
* cleanup
* grumbles: Making readers aware of the possibility of gaps
* review grumbles
* grumbles
* remove notes from niklasad1
* Update `jsonrpsee`
* fix: jsonrpsee features
* jsonrpsee: fallback to random port in case the specified port failed (#10304)
* jsonrpsee: fallback to random port
* better comment
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update client/rpc-servers/src/lib.rs
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* address grumbles
* cargo fmt
* addrs already slice
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
* Update jsonrpsee to 092081a0a2b8904c6ebd2cd99e16c7bc13ffc3ae
* lockfile
* update jsonrpsee
* fix warning
* Don't fetch jsonrpsee from crates
* make tests compile again
* fix rpc tests
* remove unused deps
* update tokio
* fix rpc tests again
* fix: test runner
`HttpServerBuilder::builder` fails unless it's called within tokio runtime
* cargo fmt
* grumbles: fix subscription aliases
* make clippy happy
* update remaining subscriptions alias
* cleanup
* cleanup
* fix chain subscription: less boiler plate (#10285)
* fix chain subscription: less boiler plate
* fix bad merge
* cargo fmt
* Switch to jsonrpsee 0.5
* fix build
* add missing features
* fix nit: remove needless Box::pin
* Integrate jsonrpsee metrics (#10395)
* draft metrics impl
* Use latest api
* Add missing file
* Http server metrics
* cleanup
* bump jsonrpsee
* Remove `ServerMetrics` and use a single middleware for both connection counting (aka sessions) and call metrics.
* fix build
* remove needless Arc::clone
* Update to jsonrpsee 0.6
* lolz
* fix metrics
* Revert "lolz"
This reverts commit eed6c6a56e78d8e307b4950f4c52a1c3a2322ba1.
* fix: in-memory rpc support subscriptions
* commit Cargo.lock
* Update tests to 0.7
* fix TODOs
* ws server: generate subscriptionIDs as Strings
Some libraries seems to expect the subscription IDs to be Strings, let's not break
this in this PR.
* Increase timeout
* Port over tests
* cleanup
* Using error codes from the spec
* fix clippy
* cargo fmt
* update jsonrpsee
* fix nits
* fix: rpc_query
* enable custom subid gen through spawn_tasks
* remove unsed deps
* unify tokio deps
* Revert "enable custom subid gen through spawn_tasks"
This reverts commit 5c5eb70328fe39d154fdb55c56e637b4548cf470.
* fix bad merge of `test-utils`
* fix more nits
* downgrade wasm-instrument to 0.1.0
* [jsonrpsee]: enable custom RPC subscription ID generatation (#10731)
* enable custom subid gen through spawn_tasks
* fix nits
* Update client/service/src/builder.rs
Co-authored-by: David <dvdplm@gmail.com>
* add Poc; needs jsonrpsee pr
* update jsonrpsee
* add re-exports
* add docs
Co-authored-by: David <dvdplm@gmail.com>
* cargo fmt
* fmt
* port RPC-API dev
* Remove unused file
* fix nit: remove async trait
* fix doc links
* fix merge nit: remove jsonrpc deps
* kill namespace on rpc apis
* companion for jsonrpsee v0.10 (#11158)
* companion for jsonrpsee v0.10
* update versions v0.10.0
* add some fixes
* spelling
* fix spaces
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* send error before subs are closed
* fix unsubscribe method names: chain
* fix tests
* jsonrpc server: print binded local address
* grumbles: kill SubscriptionTaskExecutor
* Update client/sync-state-rpc/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/rpc/src/chain/chain_full.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* sync-state-rpc: kill anyhow
* no more anyhow
* remove todo
* jsonrpsee: fix bad params in subscriptions. (#11251)
* update jsonrpsee
* fix error responses
* revert error codes
* dont do weird stuff in drop impl
* rpc servers: remove needless clone
* Remove silly constants
* chore: update jsonrpsee v0.12
* commit Cargo.lock
* deps: downgrade git2
* feat: CLI flag max subscriptions per connection
* metrics: use old logging format
* fix: read WS address from substrate output (#11379)
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Maciej Hirsz <hello@maciej.codes>
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* sc-client-db: utils::open_database(...) — return OpenDbError so that the caller could tell the `OpenDbError::DoesNotExist` clearly
* sc-client-db: utils::open_database(..) — accept the `create: bool` argument
* sc-client-db: pruning — optional argument in the DatabaseSettings
* sc-state-db: Split `Error<E>` into separate `Error<E>` and `StateDbError`
* StateDb::open: choose the pruning-mode depending on the requested and stored values
* sc-state-db: test for different combinations of stored and requested pruning-modes
* CLI-argument: mark the unsafe-pruning as deprecated
* Fix tests
* tests: do not specify --pruning when running the substrate over the existing storage
* fix types for benches
* cargo fmt
* Check whether the pruning-mode and sync-mode are compatible
* cargo fmt
* parity-db: 0.3.11 -> 0.3.12
* sc-state-db: MetaDb::set_meta — a better doc-test
* cargo fmt
* make MetaDb read-only again!
* Remove the stray newline (and run the CI once again please)
* Last nitpicks
* A more comprehensive error message
* Add new hardware and software metrics
* Move sysinfo tests into `mod tests`
* Correct a typo in a comment
* Remove unnecessary `nix` dependency
* Fix the version tests
* Add a `--disable-hardware-benchmarks` CLI argument
* Disable hardware benchmarks in the integration tests
* Remove unused import
* Fix benchmarks compilation
* Move code to a new `sc-sysinfo` crate
* Correct `impl_version` comment
* Move `--disable-hardware-benchmarks` to the chain-specific bin crate
* Move printing out of hardware bench results to `sc-sysinfo`
* Move hardware benchmarks to a separate messages; trigger them manually
* Rename some of the fields in the `HwBench` struct
* Revert changes to the telemetry crate; manually send hwbench messages
* Move sysinfo logs into the sysinfo crate
* Move the `TARGET_OS_*` constants into the sysinfo crate
* Minor cleanups
* Move the `HwBench` struct to the sysinfo crate
* Derive `Clone` for `HwBench`
* Fix broken telemetry connection notification stream
* Prevent the telemetry connection notifiers from leaking if they're disconnected
* Turn the telemetry notification failure log into a debug log
* Rename `--disable-hardware-benchmarks` to `--no-hardware-benchmarks`
* Removal of light client from substrate
* add missing import
* These tests relate to there being light and non light clients.
* removing lightnodes from test
* cargo fmt
* not needed
* LightDataChecker not needed any longer
* cargo fmt
* Update client/service/test/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/service/test/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* cargo fmt
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Use tokio runtime handle instead of TaskExecutor abstraction
Before this pr we had the `TaskExecutor` abstraction which theoretically
allowed that any futures executor could have been used. However, this
was never tested and is currently not really required. Anyone running a
node currently only used tokio and nothing else (because this was hard
coded in CLI). So, this pr removes the `TaskExecutor` abstraction and
relies directly on the tokio runtime handle.
Besides this changes, this pr also makes sure that the http and ws rpc
server use the same tokio runtime. This fixes a panic that occurred when
you drop the rpc servers inside an async function (tokio doesn't like
that a tokio runtime is dropped in the async context of another tokio
runtime).
As we don't use any custom runtime in the http rpc server anymore, this
pr also removes the `rpc-http-threads` cli argument. If external parties
complain that there aren't enough threads for the rpc server, we could
bring support for increasing the thread count of the tokio runtime.
* FMT
* Fix try runtime
* Fix integration tests and some other optimizations
* Remove warnings
* Store the database in a role specific subdirectory
This is a cleaned up version of #8658 fixing #6880
polkadot companion: paritytech/polkadot#2923
* Disable prometheus in tests
* Also change p2p port
* Fix migration logic
* Use different identification file for rocks and parity db
Add tests for paritydb migration
Remove unneeded dependencies and dev-dependencies.
Made self_destruct test not dependent on wasm bin size.
Updated code related to deprecated warning on tracing-subscriber `scope()`
( See https://github.com/tokio-rs/tracing/issues/1429 )
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Fix tracing tests (#8022)
* Fix tracing tests
The tests were not working properly.
1. Some test was setting a global subscriber, this could lead to racy
conditions with other tests.
2. A logging test called `process::exit` which is completly wrong.
* Update client/tracing/src/lib.rs
Co-authored-by: David <dvdplm@gmail.com>
* Review comments
Co-authored-by: David <dvdplm@gmail.com>
* Fix tracing spans are not being forwarded to spawned task (#8009)
* Fix tracing spans are not being forwarded to spawned task
There is a bug that tracing spans are not forwarded to spawned task. The
problem was that only the telemetry span was forwarded. The solution to
this is to use the tracing provided `in_current_span` to capture the
current active span and pass the telemetry span explictely. We will now
always enter the span when the future is polled. This is essentially the
same strategy as tracing is doing with its `Instrumented`, but now
extended for our use case with having multiple spans active.
* More tests
* Proper test for telemetry and prefix span
* WIP
* Fix test (need to create & enter the span at the same time)
* WIP
* Remove telemtry_span from sc_service config
* CLEANUP
* Update comment
* Incorrect indent
* More meaningful name
* Dedent
* Naming XD
* Attempt to make a more complete test
* lint
* Missing licenses
* Remove user data
* CLEANUP
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* CLEANUP
* Apply suggestion
* Update bin/node/cli/tests/telemetry.rs
Co-authored-by: David <dvdplm@gmail.com>
* Wrapping lines
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
* Add BlockStream Enum and utility fn
* WIP: Modify import closure to work with BlockStream
* Fix trait bounds
* Working prototype
* Revamp block importing
* Add export_import_flow tests
* Add comments and clean code
* Add more comments in the import fn
* Add link code to import function
* Add condition when returning Ready(Ok(()) to make sure we've imported every block
* Add check for imported blocks in JSON case
* Use rest pattern
* Fix compilation error for undeclared variable
* Add polling and waker before pending
* Print read_block_count instead of count
* Simplify binary cli option with structopt
* Update test to reflect changes in CLI api
* Change Stream to take SignedBlock<B> instead of B
* Add comments to BlockStream
* Move out logic to smaller functions for clearer code
* Remove result over import_blocks return type
* Check for error in command output rather than simply checking command exit status
* Revamp export/import/revert testing
* Fix minor typos and formatting errors
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove unnecessary if condition in terminating condition
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Explicit error instead of returning it as a string
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Pass BlockStream to log_importing_status_updates and simplify matching arms for block stream
* Use .contains() instead of regex match
* Line break in match block; return future::ready instead of poll_fn
* Update Cargo.lock
* Add check so that queue doesn't grow too big
* Use Iterator instead of Stream
* Remove allow dead_code
* Remove outdated comments
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Return Errors instead of logging them
* Simplify match arms
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove check before terminating block import
* Apply suggestions from code review
* Check that queue is not full BEFORE calling
* Revert "Remove check before terminating block import"
This reverts commit 377823c0a648a3eb2e61185a257a61023067893d.
* Improve unit tests to make sure we actually import blocks
* Remove Unpin implementation for BlockIter
* Add prototype of enum for ImportStates
* Add working prototype for StateMachine
* Add comments for clearer code
* Add sleep before calling Waker when waiting for import queue
* Add Speedometer
* add dbg!(&log) for test debugging
* Fix lines with more than 100 cols
* Fix regex capture for test
* Update regexes to take to capture the whole number
* Rename Cmd to Command
Co-authored-by: Gavin Wood <gavin@parity.io>
* Actually rename Cmd to Command
* Apply suggestions from code review
Co-authored-by: Gavin Wood <gavin@parity.io>
* Fix compilation errors for tests
* Fix compilation errors from code review suggestion
* Update bin/node/cli/tests/export_import_flow.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
* Initial commit
Forked at: ad90ab7ec9
Parent branch: origin/master
* Increase killing grace period of CLI tests and display more info
* Use --dev everywhere possible
* Put pruning mode to its own params struct
* Add pruning params to export-blocks command
* Added missing file
* Removed not-dev mode in tests
* Add pruning mode to the revert command
* Decrease killing grace period again
* Move back unsafe_pruning to import_params
* Applied proposed changes
Related to #4776
Related to https://github.com/paritytech/polkadot/pull/832
To summarize the changes:
1. I did not manage to validate with types the service's Configuration. But I did reduce the possibility of errors by moving all the "fill" functions to their respective structopts
2. I split params.rs to multiple modules: one module params for just CLI parameters and one module commands for CLI subcommands (and RunCmd). Every command and params are in their own file so things are grouped better together and easier to remove
3. I removed the run and run_subcommand helpers as they are not helping much anymore. Running a command is always a set of 3 commands: 1. init 2. update config 3. run. This still allow the user to change the config before arguments get parsed or right after.
4. I added tests for all subcommands.
5. [deleted]
Overall the aim is to improve the situation with the Configuration and the optional parameters, add tests, make the API more consistent and simpler.
* Fix CLI setup again
We need to set `config_dir` and `database_path` for almost every
command.
This fixes `purge-chain` and also adds a test to make sure we don't
break it again.
* Adds missing test files
* Split methods
These are a few changes I missed during the refactoring.
1. Initialization issue and boilerplate
Most importantly: part of the `Configuration` initialization was done in `sc_cli::init`. This means the user can not benefit from this initialization boilerplate if they have multiple `Configuration` since `sc_cli::init` can only be called once.
2. Boilerplate for `VersionInfo` and `Configuration`
I'm also answering to the critic of @bkchr on the initialization using version: https://github.com/paritytech/substrate/pull/4692/files/bea809d4c14a2ede953227ac885e3b3f9771c548#r372047238 This will allow initializing a `Configuration` and provide the version by default.
3. Loading the `chain_spec` explicitly
In the past it was done automatically but in some cases we want to delay this. I moved the code to `Configuration.load_spec()` so it can be called later on. `chain_spec` can also be written directly to the `Configuration` without using this `load_spec` helper.
4. [deleted]
5. Fixing issue that prevents the user to override the port
In the refactoring I introduced a bug by mistake that could potentially prevent the CLI user to override the ports if defaults where provided for these ports (only on cumulus).
6. Change task_executor from Box to Arc
This is useful for cumulus where we have 2 nodes with 2 separate Configuration that need to spawn tasks to the same runtime.
7. Renamed TasksExecutorRequired to TaskExecutor
For consistency.
This is related to https://github.com/paritytech/cumulus/issues/24
This is the continuation (and hopefully the end of) #4692
Gavin Wood