Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.
Also needed to nicely test the changes coming to collators in #3168.
### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest
Sometimes you need to debug some issues just by the logs and reconstruct
what happened.
In these scenarios it would be nice to know if a block was imported as
best block, and what it parent was.
So here I propose to change the output of the informant to this:
```
2024-04-05 20:38:22.004 INFO ⋮substrate: [Parachain] ✨ Imported #18 (0xe7b3…4555 -> 0xbd6f…ced7)
2024-04-05 20:38:24.005 INFO ⋮substrate: [Parachain] ✨ Imported #19 (0xbd6f…ced7 -> 0x4dd0…d81f)
2024-04-05 20:38:24.011 INFO ⋮substrate: [jobless-children-5352] 🌟 Imported #42 (0xed2e…27fc -> 0x718f…f30e)
2024-04-05 20:38:26.005 INFO ⋮substrate: [Parachain] ✨ Imported #20 (0x4dd0…d81f -> 0x6e85…e2b8)
2024-04-05 20:38:28.004 INFO ⋮substrate: [Parachain] 🌟 Imported #21 (0x6e85…e2b8 -> 0xad53…2a97)
2024-04-05 20:38:30.004 INFO ⋮substrate: [Parachain] 🌟 Imported #22 (0xad53…2a97 -> 0xa874…890f)
```
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR exports unified hostfunctions needed for parachains. Basicaly
`SubstrateHostFunctions` + `storage_proof_size::HostFunctions`.
Also removes the native executor from the parachain template.
---------
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
This is a refactoring, no changes to the logic are included (if you find
some, report :D).
## Change Overview
In https://github.com/paritytech/polkadot-sdk/issues/2455, dependency on
actual runtimes was removed for the system parachains. This means that
the trait bounds we have on various `start_node_xy` do not check
anything anymore, since they all point to the same runtime. Exception is
asset-hub-polkadot which uses a different key type.
This PR unifies the different nodes as much as possible.
`start_node_impl` is doing the heavy lifting and has been made a bit
more flexible to support the rpc extension instantiation that was there
before.
The generics for `Runtime` and `AuraId` have been removed where
possible. The fake runtime is imported as `FakeRuntime` to make it very
clear to readers that it is not the generic parameter.
Multiple nodes where using the same import queue/start_consensus
closure, they have been unified.
---------
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Egor_P <egor@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Andrei Eres <eresav@me.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Alin Dima <alin@parity.io>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Dónal Murray <donalm@seadanda.dev>
While investigating some pruning issues I found some room for
improvement in the notification pin handling.
**Problem:** It was not possible to define an upper limit on
notification pins. The block pinning cache has a limit, but only handles
bodies and justifications.
After this PR, bookkeeping for notifications is managed in the pinning
worker. A limit can be defined in the worker. If that limit is crossed,
blocks that were pinned for that notification are unpinned, which now
affects the state as well as bodies and justifications. The pinned
blocks cache still has a limit, but should never be hit.
closes#19
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
# Runtime side for PoV Reclaim
## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.
## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime.
2. Provide the HostFunction to the node
3. Enable proof recording during block import
## TODO
- [x] PRDoc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
In CI jobs I see calls to `forklift clean`, which apparently does not
exist.
From CI logs:
```
1m$ forklift clean[0;m
Cargo cache management utility
Usage:
forklift [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
help Help about any command
serve Run forklift coordinator server for current location
start Start detached forklift coordinator server for current location
stop Stop forklift coordinator server for current location
Flags:
-c, --compression string Compression algorithm to use
Available: none, xz (default "zstd")
-h, --help help for forklift
-p, --param stringToString map of additional parameters
ex: -p S3_BUCKET_NAME=my_bucket (default [])
-s, --storage string Storage driver
Available: s3, fs (default "s3")
-v, --verbose string Available: panic, fatal, error, warn, warning, info, debug, trace (default "info")
--version version for forklift
Use "forklift [command] --help" for more information about a command.
```
closes#2567
Followup for https://github.com/paritytech/polkadot-sdk/pull/2331
This PR contains multiple internal cleanups:
1. This gets rid of the functionality in `generate_genesis_block` which
was only used in one benchmark
2. Fixed `transaction_pool` and `transaction_throughput` benchmarks
failing since they require a tokio runtime now.
3. Removed `parachain_id` CLI option from the test parachain
4. Removed `expect` call from `RuntimeResolver`
Smoldot sometimes stops reporting finalized blocks to us. Since we are
recovering from the relay chain with a huge delay on full nodes, we can
not rely on import notifications to set the best block. Because
sometimes we also do not receive finality notifications, the height
check in zombienet fails.
Proper solution is to update smoldot, there have been some changes since
the version we use. But upgrade is blocked by version conflict which
will be resolved with
https://github.com/paritytech/polkadot-sdk/pull/1631.
This PR provides the infrastructure for the pov-reclaim mechanism
discussed in #209. The goal is to provide the current proof size to the
runtime so it can be used to reclaim storage weight.
## New Host Function
- A new host function is provided
[here](https://github.com/skunert/polkadot-sdk/blob/5b317fda3be205f4136f10d4490387ccd4f9765d/cumulus/primitives/pov-reclaim/src/lib.rs#L23).
It returns the size of the current proof size to the runtime. If
recording is not enabled, it returns 0.
## Implementation Overview
- Implement option to enable proof recording during import in the
client. This is currently enabled for `polkadot-parachain`,
`parachain-template` and the cumulus test node.
- Make the proof recorder ready for no-std. It was previously only
enabled for std environments, but we need to record the proof size in
`validate_block` too.
- Provide a recorder implementation that only the records the size of
incoming nodes and does not store the nodes itself.
- Fix benchmarks that were broken by async backing changes
- Provide new externalities extension that is registered by default if
proof recording is enabled.
- I think we should discuss the naming, pov-reclaim was more intuitive
to me, but we could also go with clawback like in the issue.
## Impact of proof recording during import
With proof recording: 6.3058 Kelem/s
Without proof recording: 6.3427 Kelem/s
The measured impact on the importing performance is quite low on my
machine using the block import benchmark. With proof recording I am
seeing a performance hit of 0.585%.
---------
Co-authored-by: command-bot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Remove deprecated `AllPalletsXY` types.
They have been deprecated for nearly 1.5 years now, I think its fine to
remove them.
If anyone feels like we should first put a date on the deprecation as
stated in the deprecation guideline, feel free to speak up. To me it
looks like this has been forgotten and can be directly removed.
The test was a bit flaky on CI.
There was a race condition in the pov-recovery system. If the timing is
bad, it can happen that a block waits for a parent that is already
queued for import. The check if a block has children waiting happens
when we insert into the import queue. So we need to do an additional
check once we receive the import notification for the parent block.
Second issue is that `alice` was missing `--in-peers 0` and `--out-peers
0`, so alice was sometimes still fetching block via sync and the
assertion on the logs in zombienet would fail.
There is another potential issue that I saw once locally. We have a
failing pov-recovery queue that fails from time to time to check that
the retry mechanism does what it should. We now make sure that the same
candidate is never failed twice, so the tests become more predictable.
I noticed that this test broke at some point. The parachain nodes should
only acquire their blocks from the relay chain. But they were connecting
to their peers and started fetching blocks from there.
In this test I now take additional measures so we check that each nodes
really uses pov-recovery to get the blocks.
The `--relay-chain-rpc-urls` CLI flag has been available for a while
now. We have collators with this running and parachain teams are also
using it. It should be fine now to remove the experimental status.
This PR contains some fixes and cleanups for parachain nodes:
1. When using async backing, node no longer complains about being unable
to reach the prospective-parachain subsystem.
2. Parachain warp sync now informs users that the finalized para block
has been retrieved.
```
2023-11-08 13:24:42 [Parachain] 🎉 Received finalized parachain header #5747719 (0xa0aa…674b) from the relay chain.
```
3. When a user supplied an invalid `--relay-chain-rpc-url`, we were
crashing with a very verbose message. Removed the `expect` and improved
the error message.
```
2023-11-08 13:57:56 [Parachain] No valid RPC url found. Stopping RPC worker.
2023-11-08 13:57:56 [Parachain] Essential task `relay-chain-rpc-worker` failed. Shutting down service.
Error: Service(Application(WorkerCommunicationError("RPC worker channel closed. This can hint and connectivity issues with the supplied RPC endpoints. Message: oneshot canceled")))
```
When running with `--relay-chain-rpc-url` we received multiple reports
of high traffic that disappears when `--in-peers-light 0` is set. Indeed
it does not make much sense for light clients to connect to the minimal
node since it is not running the block announce protocol and the
request/response protocol for light clients.
This is intended to alleviate the traffic issues for now.
closes#1896
probably related https://github.com/paritytech/cumulus/issues/2563
Closes#1767
Until now the `purge-chain` command would only remove the `full`
subfolder of the db folder. However there is also the `parachains` db
that currently remains and can cause problems on node restart.
Example wiht old code:
```
polkadot purge-chain --database paritydb --base-path /tmp/some-folder
Are you sure to remove "/tmp/some-folder/chains/polkadot/paritydb/full"? [y/N]: y
"/tmp/some-folder/chains/polkadot/paritydb/full" removed.
```
In this case `/tmp/some-folder/chains/polkadot/paritydb/parachains`
would remain and might cause problem on node restart because of version
conflicts as described in #1767. After this PR the whole
`/tmp/some-folder/chains/polkadot/paritydb` folder will be deleted.
Closes#1671
Adds a `type_id` function to the `Extension` trait, allowing to properly store an retrieve
boxed `Extensions`.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Closes#1372
This fixes the target directories in the polkadot readme. I verified
manually the Ubuntu based install works on a fresh cloud machine. Build
from source also worked as described.
Since #1304 landed `--dev` also works again (not on v1.1.0 though), so I
would consider the install instructions fixed.
* Only calculate tree route when there are multiple leaves
* Update client/service/src/client/client.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* Worker
* Reorganize and unpin onnotification drop
* Pin in state-db, pass block number
* Pin blocks in blockchain db
* Switch to reference counted LRU
* Disable pinning when we keep all blocks
* Fix pinning hint for state-db
* Remove pinning from backend layer
* Improve readability
* Add justifications to test
* Fix justification behaviour
* Remove debug prints
* Convert channels to tracing_unbounded
* Add comments to the test
* Documentation and Cleanup
* Move task start to client
* Simplify cache
* Improve test, remove unwanted log
* Add tracing logs, remove expect for block number
* Cleanup
* Add conversion method for unpin handle to Finalitynotification
* Revert unwanted changes
* Improve naming
* Make clippy happy
* Fix docs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Use `NumberFor` instead of u64 in API
* Hand over weak reference to unpin worker task
* Unwanted
* &Hash -> Hash
* Remove number from interface, rename `_unpin_handle`, LOG_TARGET
* Move RwLock one layer up
* Apply code style suggestions
* Improve comments
* Replace lru crate by schnellru
* Only insert values for pinned items + better docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Improve comments, log target and test
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Allow specification of multiple urls for relay chain rpc nodes
* Add pooled RPC client basics
* Add list of clients to pooled client
* Improve
* Forward requests to dispatcher
* Switch clients on error
* Implement rotation logic
* Improve subscription handling
* Error handling cleanup
* Remove retry from rpc-client
* Improve naming
* Improve documentation
* Improve `ClientManager` abstraction
* Adjust zombienet test
* Add more comments
* fmt
* Apply reviewers comments
* Extract reconnection to extra method
* Add comment to reconnection method
* Clean up some dependencies
* Fix build
* fmt
* Provide alias for cli argument
* Apply review comments
* Rename P* to Relay*
* Improve zombienet test
* fmt
* Fix zombienet sleep
* Simplify zombienet test
* Reduce log clutter and fix starting position
* Do not distribute duplicated imported and finalized blocks
* fmt
* Apply code review suggestions
* Move building of relay chain interface to `cumulus-client-service`
* Refactoring to not push back into channel
* FMT
* Bump to clap4
* Adjust to clap 4 style
* Remove two more deprecated occurences of clap macro
* Remove "deprecated" feature from clap
* Update cargo lock
* Fix group name
* More skipped group names
* Bump clap to 3.2.22
* Replace `from_os_str` with `value_parser`
* Replace `from_str` and `try_from_str` with `value_parser`
* Move possible_values to the new format
* Remove unwanted print
* Add missing match branch
* Update clap to 4.0.9 and make it compile
* Replace deprecated `clap` macro with `command` and `value`
* Move remaining `clap` attributes to `arg`
* Remove no-op value_parsers
* Adjust value_parser for state_version
* Remove "deprecated" feature flag and bump to 4.0.11
* Improve range
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions
* Trigger CI
* Fix unused error warning
* Fix doc errors
* Fix ArgGroup naming conflict
* Change default_value to default_value_t
* Use 1.. instead of 0..
Co-authored-by: Bastian Köcher <git@kchr.de>
* Add minimal overseer gen with dummy subsystems
* Fix dependencies
* no-compile: only client transaction pool missing
* Remove unused imports
* Continue to hack towards PoC
* Continue
* Make mini node compile
* Compiling version with blockchainevents trait
* Continue
* Check in lockfile
* Block with tokio
* update patches
* Update polkadot patches
* Use polkadot-primitives v2
* Fix build problems
* First working version
* Adjust cargo.lock
* Add integration test
* Make integration test work
* Allow startinc collator without relay-chain args
* Make OverseerRuntimeClient async
* Create separate integration test
* Remove unused ChainSelection code
* Remove unused parameters on new-mini
* Connect collator node in test to relay chain nodes
* Make BlockChainRPCClient obsolete
* Clean up
* Clean up
* Reimplement blockchain-rpc-events
* Revert "Allow startinc collator without relay-chain args"
This reverts commit f22c70e16521f375fe125df5616d48ceea926b1a.
* Add `strict_record_validation` to AuthorityDiscovery
* Move network to cumulus
* Remove BlockchainRPCEvents
* Remove `BlockIdTo` and `BlockchainEvents`
* Make AuthorityDiscovery async
* Use hash in OverseerRuntime
* Adjust naming of runtime client trait
* Implement more rpc-client methods
* Improve error handling for `ApiError`
* Extract authority-discovery creationand cleanup
* RPC -> Rpc
* Extract bitswap
* Adjust to changes on master
* Implement `hash` method
* Introduce DummyChainSync, remove ProofProvider and BlockBackend
* Remove `HeaderMetadata` from blockchain-rpc-client
* Make ChainSync work
* Implement NetworkHeaderBackend
* Cleanup
* Adjustments after master merge
* Remove ImportQueue from network parameters
* Remove cargo patches
* Eliminate warnings
* Revert to HeaderBackend
* Add zombienet test
* Implement `status()` method
* Add more comments, improve readability
* Remove patches from Cargo.toml
* Remove integration test in favor of zombienet
* Remove unused dependencies, rename minimal node crate
* Adjust to latest master changes
* fmt
* Execute zombienet test on gitlab ci
* Reuse network metrics
* Chainsync metrics
* fmt
* Feed RPC node as boot node to the relay chain minimal node
* fmt
* Add bootnodes to zombienet collators
* Allow specification of relay chain args
* Apply review suggestions
* Remove unnecessary casts
* Enable PoV recovery for rpc full nodes
* Revert unwanted changes
* Make overseerHandle non-optional
* Add availability-store subsystem
* Add AuxStore and ChainApiSubsystem
* Add availability distribution subsystem
* Improve pov-recovery logging and add RPC nodes to tests
* fmt
* Make availability config const
* lock
* Enable debug logs for pov-recovery in zombienet
* Add log filters to test binary
* Allow wss
* Address review comments
* Apply reviewer comments
* Adjust to master changes
* Apply reviewer suggestions
* Bump polkadot
* Add builder method for minimal node
* Bump substrate and polkadot
* Clean up overseer building
* Add bootnode to two in pov_recovery test
* Fix missing quote in pov recovery zombienet test
* Improve zombienet pov test
* More debug logs for pov-recovery
* Remove reserved nodes like on original test
* Revert zombienet test to master
* Extract json-rpc-client and introduce worker
* Initial rpc worker
* Add error handling
* Use bounded channels for listeners
* Improve naming and clean up
* Use tracing channels
* Improve code readability
* Decrease channel size limit
* Remove unused dependency
* Fix docs
* RPC -> Rpc
* Start worker in initialization method
* Print error in case a distribution channel is full
* Fix docs
* Make `RpcStreamWorker` private
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Use tokio channels and add TODO item
* Remove `Option` from `to_worker_channel`
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update UI test output for rust 1.62.1
* switch ci to staging image to check that everything works
* fix artifacts node-bench-regression-guard
* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning
* Fix more clippy lints
* Make clippy happy by relying on auto-deref were possible
* Add tracking issue to the comments
* pin ci image
Co-authored-by: alvicsam <alvicsam@gmail.com>
* Initial network interface preparations
* Implement get_storage_by_key
* Implement `validators` and `session_index_for_child`
* Implement persisted_validation_data and candidate_pending_availability
* Fix method name for persisted_validation_data and add encoded params
* Implement `retrieve_dmq_contents` and `retrieve_all_inbound_hrmp_channel_contents`
* Implement `prove_read`
* Introduce separate RPC client, expose JsonRpSee errors
* Simplify closure in call_remote_runtime_function
* Implement import stream, upgrade JsonRpSee
* Implement finality stream
* Remove unused method from interface
* Implement `is_major_syncing`
* Implement `wait_on_block`
* Fix tests
* Unify error handling `ApiError`
* Replace WaitError with RelayChainError
* Wrap BlockChainError in RelayChainError
* Unify error handling in relay chain intefaces
* Fix return type of proof method
* Improve error handling of new methods
* Improve error handling and move logging outside of interface
* Clean up
* Remove unwanted changes, clean up
* Remove unused import
* Add format for StatemachineError and remove nused From trait
* Use 'thiserror' crate to simplify error handling
* Expose error for overseer, further simplify error handling
* Reintroduce network interface
* Implement cli option
* Adjust call_state method to use hashes
* Disable PoV recovery when RPC is used
* Add integration test for network full node
* Use Hash instead of BlockId to ensure compatibility with RPC interface
* Fix cargo check warnings
* Implement retries
* Remove `expect` statements from code
* Update jsonrpsee to 0.8.0 and make collator keys optional
* Make cli arguments conflicting
* Remove unused `block_status` method
* Add clippy fixes
* Cargo fmt
* Validate relay chain rpc url
* Clean up dependencies and add one more integration test
* Clean up
* Clean up dependencies of relay-chain-network
* Use hash instead of blockid for rpc methods
* Fix tests
* Update client/cli/src/lib.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Improve error message of cli validation
* Add rpc client constructor
* Do not use debug formatting for errors
* Improve logging for remote runtime methods
* Only retry on transport problems
* Use PHash by value, rename test
* Improve tracing, return error on relay-chain-interface build
* Fix naming, use generics instead of deserializing manually
* Rename RelayChainLocal and RelayChainNetwork
* lock
* Format
* Use impl trait for encodable runtime payload
* Only instantiate full node in tests when we need it
* Upgrade scale-codec to 3.0.0
* Improve expect log
Co-authored-by: Koute <koute@users.noreply.github.com>