Still allows custom message hasher, but ties together the crypto
types used for private+public keys and the signature.
Signed-off-by: acatangiu <adrian@parity.io>
Before libp2p 0.50.0 we used a quorum of one to fetch records from the DHT. In the pr that upgraded
to libp2p 0.50.0 we accidentally changed this behavior. This pr brings back the old behavior of
using a qorum of one and thus, a faster discovery. After finding the first value, we directly finish
the query. There was also another behavior change in libp2p, they stopped automatic caching on
remote nodes. This pr also brings back the remote caching on nodes that are nearest to the key from
our point of view of the network.
The pr that changed the behavior in libp2p: https://github.com/libp2p/rust-libp2p/pull/2712
* Adds test
* Ensure we are using the runtime version of the override/substitute wasm
* Update client/service/src/client/call_executor.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* upgrade libp2p to 0.50.0
* on_swarm_event and on_connection_handler_event
* replace `Swarm::new` with `Swarm::with_threadpool_executor`
* on_swarm_event and on_connection_handler_event part 2
* on_swarm_event and on_connection_handler_event part 3
* on_swarm_event and on_connection_handler_event part 4
* update libp2p
* libp2p 0.50.0
* rename OutboundQueryCompleted to OutboundQueryProgressed
refs https://github.com/libp2p/rust-libp2p/pull/2712
* remove unused var
* accumulate outbound_query_records until query is finished
* format code
* use p_handler instead of new_handler
https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610
* pass ListenFailure to kademlia
https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664
* use tokio executor in tests
https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776
* use chrono Local::now
instead of deprecated Local::today
* remove unused vars from request_responses tests
* attempt to fix pallet UI tests
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* restart CI
* Aura: Do not verify on state import
When we import the state, we can not fetch authorities to verify the seal etc. So, we can directly
skip any verification.
* Skip checks as well for gap sync
* Update client/consensus/aura/src/import_queue.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Review comment
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
We don't canonicalize on archive nodes and thus `best_canonical` always returned `None`. So, the
moment such a node tried to force canonicalize, it was trapped in some endless loop.
This pr solves this by renaming `best_canonical` to `last_canonicalized` and also making the return
value more clear by introducing a custom enum `LastCanonicalized`.
* Print "stalled" task on shutdown
When the node is shutting down, we give the Tokio runtime 60 seconds to shutdown. If after these 60
seconds there are still running tasks, we now print these tasks. This should help debugging nodes
that have stalled tasks.
This pr introduces a `TaskRegistry` that keeps track of all running tasks. Each task registers and
unregisters itself in this `TaskRegistry`.
* Fix rustdoc
* Update client/service/src/lib.rs
* Make CLI state pruning optional again
The state pruning setting is stored in the database when it is created. In later runs it is fine to
drop the `--state-pruning` CLI argument as the setting is stored in the database. The state db will
only return an error if the stored state pruning doesn't match the state pruning given via CLI.
Recently we improved the state pruning CLI handling and accidentally made the state pruning value
always present (as we set some default value for the clap). If we could find out if a user has
passed a value or the default value was taken, we could keep the default value in the CLI interface,
but clap isn't supporting this right now. So, we need to go back and make `state_pruning` an
optional with the default written into the docs.
It also adds a test to ensure that we don't break this behavior again.
* More docs
* babe: allow skipping epochs in pallet
* babe: detect and skip epochs on client
* babe: cleaner epoch util functions
* babe: add test for runtime handling of skipped epochs
* babe: simpler implementation of client handling of skipped epochs
* babe: test client-side handling of skipped epochs
* babe: add comments on client-side skipped epochs
* babe: remove emptyline
* babe: make it resilient to forks
* babe: typo
* babe: overflow-safe math
* babe: add test for skipping epochs across different forks
* Fix tests
* FMT
Co-authored-by: Bastian Köcher <info@kchr.de>
* txpool: don't validate block transactions if the pool is empty
Fix shall prevent from wasting the CPU during the major sync. Block
transaction don't need to be re-validated when the txpool is empty.
Fixes: #12903
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
This is a minor clean up.
There was a minor bug in TestApi::add_block: the best block flag for
given number was not cleared when the new best block with the same
number was added.
The usage of is_best_block argument in add_block method in txpool tests
were aligned with the BestBlock event.
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* rpc/chain_head: Add event structure for serialization
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add tests for events
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add API trait for `chainHead`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add RPC errors
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Manage subscription ID tracking for pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add tests for subscription management
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Constructor for the API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Placeholders for API implementation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Accept RPC subscription sink
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Generate the runtime API event
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `follow` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `body` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `header` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `storage` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `call` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement the `unpin` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update `Cargo.lock`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement `getGenesis` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Parse params from hex string
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Constuct API with genesis hash
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add the finalized block to reported tree route
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Export the API and events for better ergonomics
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Add test module with helper functions
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test block events from the `follow` pubsub
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test `genesisHash` getter
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test `header` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test `body` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test calling into the runtime API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test runtime for the `follow` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Add runtime code changes for `follow` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Remove space from rustdoc
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Use the `child_key` for storage queries
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Test `storage` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Test child trie query for `storage` method
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Event serialization typo
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Remove subscription aliases
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add `NetworkConfig` parameter
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Named parameters as camelCase if present
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Implement From<ApiError> for RuntimeEvents
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Handle pruning of the best block in finalization window
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Generate initial block events
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Verify that initial in-memory blocks are reported
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Verify the finalized event with forks and pruned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Separate logic for generating initial events
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Handle stopping a subscription ID
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Submit events until the "Stop" event is triggered
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Separate logic for handling new and finalized blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Extend subscription logic with subId handle
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust to the new subscription mngmt API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Refuse RuntimeAPI calls without the runtime flag
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Verify RuntimeAPI calls without runtime flag
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Add best block per subscription
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Check storage keys for prefixes
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Check storage queries with invalid prefixes
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Allow maximum number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Test the maximum number of pinned blocks
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Adjust to origin/master and apply clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* client/service: Enable the `chainHead` API
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Stop subscription on client disconnect and add debug logs
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Fix sending `Stop` on subscription exit
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Check best block is descendent of latest finalized
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* chain_head/tests: Report events before pruning the best block
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Nonrecursive initial block generation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Generate initial events on subscription executor
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Reduce dev-dependencies for tokio
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* rpc/chain_head: Accept empty parameters
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Use debug of `HexDisplay` for full format
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Enable subscription ID
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Use jsonrpsee 16.2 camelCase feature for paramaters
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Use `NonZeroUsize` for `NetworkConfig` param
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* rpc/chain_head: Rename `runtime_updates` to `has_runtime_updates`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Set genesis block data using the built genesis block
* Make resolve_state_version_from_wasm a separate function and some small refactorings
Useful for the commit following.
* Introduce trait BuildGenesisBlock
Substrate users can use this trait to implement their custom genesis block when constructing the
client.
* Make call_executor test compile
* cargo +nightly fmt --all
* Fix test
* Remove unnecessary clone
* FMT
* Apply review suggestions
* Revert changes to new_full_client() and new_full_parts() signature
* Remove needless `Block` type in `resolve_state_version_from_wasm`
* sc-network-test::Peer: block push methods return hashes vec
This commit reworks the block generation/push methods in
sc-network-test::Peer.
Now methods are providing the vector of hashes that were built.
This allows to get rid of redundant `block_hash_from_id` call, as all
hashes are known just after being built.
Similar approach was taken in BeefyTestNet::generate_blocks_and_sync
method.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* fix
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix missing block number issue on forced canonicalization
There is this issue about missing block numbers on forced canonicalization. I looked over the code
now 10000 times and there are possible ways this can be triggered, but I don't really know how this
is triggered. So, this pr is going to solve the symptom and not the cause. The block number to hash
mapping is set when we import a new best block. Forced canonicalization will now stop at the best
block and it will canonicalize the other blocks later when the best block moved. As the error
reports indicated that this issue mainly happened on major sync, there should not be any forks, so
not doing the canonicalization directly shouldn't be that harmful. All known implementations should
import all blocks as best block on major sync anyway (I mean somewhere there is the bug, but I
didn't yet found it).
I will also do some changes to Cumulus around some potential culprit for this issue.
Closes: https://github.com/paritytech/substrate/issues/12613
* Add some docs
* Fix fix
* Review comments
* Review comments
* jsonrpsee v0.16
add backwards compatibility
run old http server on http only
* cargo fmt
* update jsonrpsee 0.16.1
* less verbose cors log
* fix nit in log: WS -> HTTP
* revert needless changes in Cargo.lock
* remove unused features in tower
* fix nits; add client-core feature
* jsonrpsee v0.16.2
* Ensure that we inform all tasks to stop before starting the 60 seconds shutdown
The change of waiting in maximum 60 seconds for the node to shutdown actually introduced a bug. We
were actually waiting always 60 seconds as we didn't informed our tasks to shutdown. The solution to
this problem is to drop the task manager as this will then inform all tasks to end. It also adds
tests to ensure that the behaviors work as expected. (This should already have been done in the
first pr! :()
* ".git/.scripts/fmt.sh" 1
Co-authored-by: command-bot <>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Fix tests
* Apply review comments
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Don't indefinitely on shutting down Tokio
Now we wait in maximum 60 seconds before we shutdown the node. Tasks are may be leaked and leading
to some data corruption.
* Drink less :thinking_face:
* cli: Improve pruning documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Keep `finalized` notation and remove `canonical` one
* cli: Fix cargo doc
* cli: `PruningModeClap` IR enum
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Convert PruningModeClap into pruning modes
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Use `PruningModeClap`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Rename to `DatabasePruningMode`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Implement `FromStr` instead of `clap::ValueEnum`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Add option documentation back
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* implement crate publishing from CI
* fix indentation
* use resource_group for job exclusivity
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
* correct publish = false
* Remove YAML anchors as GitLab's `extends:` doesn't need it
* Temporarily force cache upload for the new jobs
* Revert `RUSTY_CACHIER_FORCE_UPLOAD`
* pin libp2p-tcp=0.37.0 for sc-telemetry
* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"
This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.
* always collect generated crates
* increase timeout for publish-crates-template
* Force upload the new job cache again
* Revert "Force upload the new job cache again"
This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.
* reformat
* improve timeout explanation
* s/usual/average
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
client/mmr: persisting gadget state across runs
Fixes#12780
* client/mmr: on init do canonicalization catch-up
* client/mmr: add more tests
* client/mmr: persist gadget progress in aux db
* client/mmr: add more tests
* client/mmr: replace async_std with tokio
* remove leftover comment
* address review comments
Signed-off-by: acatangiu <adrian@parity.io>
Adrian Catangiu