[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
This MR contains two major changes and some maintenance cleanup.
## 1. Free Standing Pallet Benchmark Runner
Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).
Introduces a new binary crate: `frame-omni-bencher`.
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.
This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).
It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.
### Example (full example in the Rust docs)
Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```
Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```
Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```
## 2. Building the Benchmark Genesis State in the Runtime
Closes https://github.com/paritytech/polkadot-sdk/issues/2664
This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.
## 3. Simpler args for `PalletCmd::run`
You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.
```patch
runner.sync_run(|config| cmd
- .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+ .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```
## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
This is a refactoring, no changes to the logic are included (if you find
some, report :D).
## Change Overview
In https://github.com/paritytech/polkadot-sdk/issues/2455, dependency on
actual runtimes was removed for the system parachains. This means that
the trait bounds we have on various `start_node_xy` do not check
anything anymore, since they all point to the same runtime. Exception is
asset-hub-polkadot which uses a different key type.
This PR unifies the different nodes as much as possible.
`start_node_impl` is doing the heavy lifting and has been made a bit
more flexible to support the rpc extension instantiation that was there
before.
The generics for `Runtime` and `AuraId` have been removed where
possible. The fake runtime is imported as `FakeRuntime` to make it very
clear to readers that it is not the generic parameter.
Multiple nodes where using the same import queue/start_consensus
closure, they have been unified.
---------
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Egor_P <egor@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Co-authored-by: Andrei Eres <eresav@me.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Alin Dima <alin@parity.io>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Dónal Murray <donalm@seadanda.dev>
# Runtime side for PoV Reclaim
## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.
## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime.
2. Provide the HostFunction to the node
3. Enable proof recording during block import
## TODO
- [x] PRDoc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
This brings functionality to Westend's Coretime Chain runtime, where
previously it was not much more than a shell.
It is assumed that the Coretime pallet will have the same index in the
Westend runtime as it does in Rococo for the runtime calls.
TODO:
- [x] Generate chainspec
- [x] Regenerate weights
- [x] Check hardcoded RuntimeCall weights against relay weights for
transacts
Aura key generation: https://github.com/paritytech/devops/issues/2725
---------
Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Preparation for https://github.com/paritytech/polkadot-sdk/issues/2664
Changes:
- Only require `Hash` instead of `Block` for the benchmarking
- Refactor DB types to do the same
## Integration
This breaking change can easily be integrated into your node via:
```patch
- cmd.run::<Block, ()>(config)
+ cmd.run::<HashingFor<Block>, ()>(config)
```
Status: waiting for CI checks
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
closes#2567
Followup for https://github.com/paritytech/polkadot-sdk/pull/2331
This PR contains multiple internal cleanups:
1. This gets rid of the functionality in `generate_genesis_block` which
was only used in one benchmark
2. Fixed `transaction_pool` and `transaction_throughput` benchmarks
failing since they require a tokio runtime now.
3. Removed `parachain_id` CLI option from the test parachain
4. Removed `expect` call from `RuntimeResolver`
Closes#2326.
This PR both fixes a logic bug and replaces an incorrect name.
## Bug Fix: Respecting custom genesis builder
Prior to this PR the standard logic for creating a genesis block was
repeated inside of cumulus. This PR removes that duplicated logic, and
calls into the proper `BuildGenesisBlock` implementation.
One consequence is that if the genesis block has already been
initialized, it will not be re-created, but rather read from the
database like it is for other node invocations. So you need to watch out
for old unpurged data during the development process. Offchain tools may
need to be updated accordingly. I've already filed
https://github.com/paritytech/zombienet/issues/1519
## Rename: It doesn't export state. It exports head data.
The name export-genesis-state was always wrong, nad it's never too late
to right a wrong. I've changed the name of the struct to
`ExportGenesisHeadCommand`.
There is still the question of what to do with individual nodes' public
CLIs. I have updated the parachain template to a reasonable default that
preserves compatibility with tools that will expect
`export-genesis-state` to still work. And I've chosen not to modify the
public CLIs of any other nodes in the repo. I'll leave it up to their
individual owners/maintains to decide whether that is appropriate.
---------
Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <info@kchr.de>
Since the Polkadot and Kusama runtimes are no longer in the repo, the
relevant systems parachains runtimes also need to be removed. More
context [here](https://github.com/paritytech/polkadot-sdk/issues/603)
and [here](https://github.com/paritytech/polkadot-sdk/pull/1731).
Removes the following:
- `asset-hub-kusama` and `asset-hub-polkadot`
- `bridge-hub-kusama` and `bridge-hub-polkadot`
- `collectives-polkadot`
- `glutton-kusama`
Partially solves #603 and adds to #1731.
The last issue blocking the removal of the Polkadot and Kusama system
parachains from the repo in #1737 is the dependency on the runtime code
through the RuntimeApi in `polkadot-parachain`.
This PR introduces two fake runtimes to satisfy the build requirements
and changes the `new_partial` function to make it not be generic over
the runtimes.
The reason for the second runtime is the different Aura keys used in
Polkadot Asset Hub, as the impl for AuraApi depends on this type.
If this changes the `RuntimeApi` generic could be removed completely
from all functions in `services.rs` and and generic type parameters in
`services.rs` and specified as a concrete type to TFullClient`.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Add collectives and glutton parachain westend runtimes to prepare for
#1737.
The removal of system parachain native runtimes #1737 is blocked until
chainspecs and runtime APIs can be dealt with cleanly (merge of #1256
and follow up PRs).
In the meantime, these additions are ready to be merged to `master`, so
I have separated them out into this PR.
Also marked `bridge-hub-westend` as unimplemented in line with [this
issue](https://github.com/paritytech/parity-bridges-common/issues/2602).
TODO
- [x] add to `command-bot` benchmarks
- [x] add to `command-bot-scripts` benchmarks
- [x] generate weights
---------
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Muharem <ismailov.m.h@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
## Summary
Asset bridging support for AssetHub**Rococo** <-> AssetHub**Wococo** was
added [here](https://github.com/paritytech/polkadot-sdk/pull/1215), so
now we aim to bridge AssetHub**Rococo** and AssetHub**Westend**. (And
perhaps retire AssetHubWococo and the Wococo chains).
## Solution
**bridge-hub-westend-runtime**
- added new runtime as a copy of `bridge-hub-rococo-runtime`
- added support for bridging to `BridgeHubRococo`
- added tests and benchmarks
**bridge-hub-rococo-runtime**
- added support for bridging to `BridgeHubWestend`
- added tests and benchmarks
- internal refactoring by splitting bridge configuration per network,
e.g., `bridge_to_whatevernetwork_config.rs`.
**asset-hub-rococo-runtime**
- added support for asset bridging to `AssetHubWestend` (allows to
receive only WNDs)
- added new xcm router for `Westend`
- added tests and benchmarks
**asset-hub-westend-runtime**
- added support for asset bridging to `AssetHubRococo` (allows to
receive only ROCs)
- added new xcm router for `Rococo`
- added tests and benchmarks
## Deployment
All changes will be deployed as a part of
https://github.com/paritytech/polkadot-sdk/issues/1988.
## TODO
- [x] benchmarks for all pallet instances
- [x] integration tests
- [x] local run scripts
Relates to:
https://github.com/paritytech/parity-bridges-common/issues/2602
Relates to: https://github.com/paritytech/polkadot-sdk/issues/1988
---------
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
This PR addresses several minor issues:
- Fixes the symlink for `asset-hub-rococo.json` chainspec.
- Corrects the `asset-hub-rococo-genesis` invulnerables setup.
- Relocates common bash functions for bridge testing to a separate file
`bridges_common.sh`.
This commit adds Rococo Asset Hub dedicated runtime so we can test new
features here, before merging them in Kusama Asset Hub.
Also adds one such feature: asset transfer over bridge (Rococo AssetHub
<> Wococo AssetHub)
- clone `asset-hub-kusama-runtime` -> `asset-hub-rococo-runtime`
- make it use Rococo primitives, names, assets, constants, etc
- add asset-transfer-over-bridge support to Rococo AssetHub <> Wococo
AssetHub
Fixes#1128
---------
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
When publishing crates, each crate becomes it's own tarball that can't
access files from other crates. So symlink the files to be crate local
and cargo will replace the symlinks with real files at publish time.
We can't just move all of them as it makes the package larger than the
max crates.io package size.
Co-authored-by: Javier Viola <javier@parity.io>
* CI: Fix check-try-runtime
(preparation for the monorepo)
Warnings should be treated as errors here, since this is how it will
be done in the monorepo.
I expect to see two errors in this check now.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Allow deprecated
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove try-runtime command
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* make it compile
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove try-runtime-cli dependency
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* companion 14754: no-beefy flag moved to sc-cli
Signed-off-by: Adrian Catangiu <adrian@parity.io>
* bump substrate ref
Signed-off-by: Adrian Catangiu <adrian@parity.io>
* bump polkadot ref
---------
Signed-off-by: Adrian Catangiu <adrian@parity.io>
* change dir names
* cargo toml updates
* fix crate imports for build
* change chain spec names and PR review rule
* update cli to accept asset-hub
* find/replace benchmark commands
* integration tests
* bridges docs
* more integration tests
* AuraId
* other statemint tidying
* rename statemint mod
* chain spec mod
* rename e2e test dirs
* one more Runtime::Statemine
* benchmark westmint
* rename chain spec name and id
* rename chain spec files
* more tidying in scripts/docs/tests
* rename old dir if exists
* Force people to manually do the move.
(Safer as there could be additional considerations with their setup)
* review touchups
* more renaming
* Update polkadot-parachain/src/command.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* better error message
* do not break on-chain spec_name
* log info message that path has been renamed
* better penpal docs
---------
Co-authored-by: gilescope <gilescope@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
* Runtime code according to the released runtime bridge-hub-kusama_runtime-v9360.compact.compressed.wasm
* Add chains spec for westend with generated by `./scripts/create_bridge_hub_westend_spec.sh ~/Downloads/upload/bridge-hub-kusama_runtime-v9360.compact.compressed.wasm 1002`
* fmt
* Use primitives reexported from `polkadot_primitives` crate root
* restart CI
* Fixes after merge
* update lockfile for {"polkadot", "substrate"}
Co-authored-by: parity-processbot <>
Aaro Altonen