[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
This MR contains two major changes and some maintenance cleanup.
## 1. Free Standing Pallet Benchmark Runner
Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).
Introduces a new binary crate: `frame-omni-bencher`.
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.
This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).
It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.
### Example (full example in the Rust docs)
Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```
Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```
Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```
## 2. Building the Benchmark Genesis State in the Runtime
Closes https://github.com/paritytech/polkadot-sdk/issues/2664
This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.
## 3. Simpler args for `PalletCmd::run`
You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.
```patch
runner.sync_run(|config| cmd
- .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+ .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```
## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
This PR introduces a dedicated module for benchmarks within the
parachain runtime. By segregating benchmarks into their own module, we
achieve a cleaner project structure and improved readability,
facilitating easier maintenance and updates.
### Key Changes:
- **New Benchmarks Module**: A new file `benchmarks.rs` is added,
encapsulating the benchmarking code for various pallets.
- **Refactoring `lib.rs`**: The main runtime library file (`lib.rs`) has
been updated to reflect the extraction of benchmark definitions. By
moving these definitions to `benchmarks.rs`, we reduce clutter in
`lib.rs`, streamlining the runtime's core logic and configuration.
### Benefits of This Refactoring:
- **Focused Benchmarking**: Developers can now easily locate and modify
benchmarks without navigating through the core runtime logic, enabling
targeted performance improvements.
- **Cleaner Codebase**: Segregating benchmarks from the main runtime
logic helps maintain a clean, well-organized codebase, simplifying
navigation and maintenance.
- **Scalability**: As the parachain evolves, adding or updating
benchmarks becomes more straightforward, supporting scalability and
adaptability of the runtime.
### Summary of Changes:
- Created `benchmarks.rs` to house the benchmarking suite.
- Streamlined `lib.rs` by removing the inlined benchmark definitions and
linking to the new benchmarks module.
---------
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Part of https://github.com/paritytech/polkadot-sdk/issues/226
Related https://github.com/paritytech/polkadot-sdk/issues/1833
- Deprecate `CurrencyAdapter` and introduce `FungibleAdapter`
- Deprecate `ToStakingPot` and replace usage with `ResolveTo`
- Required creating a new `StakingPotAccountId` struct that implements
`TypedGet` for the staking pot account ID
- Update parachain common utils `DealWithFees`, `ToAuthor` and
`AssetsToBlockAuthor` implementations to use `fungible`
- Update runtime XCM Weight Traders to use `ResolveTo` instead of
`ToStakingPot`
- Update runtime Transaction Payment pallets to use `FungibleAdapter`
instead of `CurrencyAdapter`
- [x] Blocked by https://github.com/paritytech/polkadot-sdk/pull/1296,
needs the `Unbalanced::decrease_balance` fix
Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`
Related to
https://github.com/paritytech/parity-bridges-common/issues/2538
This PR introduces a refactor of the parachain runtime configuration by
consolidating all pallet configurations into a new module named
`configs`. This change aims to improve the readability and
maintainability of the runtime configuration by centralizing all
configuration parameters.
## Changes
- **Creation of `configs.rs`**: A new file `configs.rs` has been added
under `templates/parachain/runtime/src/`, containing all the runtime
configurations previously scattered across `lib.rs`.
- **Refactoring of `lib.rs`**: The `lib.rs` file has been significantly
slimmed down by removing the inline pallet configurations and importing
them from `configs.rs` instead.
- **Optimization of Import Statements**: Reorganized import statements
to clarify the runtime's dependency structure.
### Benefits
- **Improved Readability**: With configurations being centralized,
developers can now easily locate and review runtime parameters without
navigating through the `lib.rs` file.
This refactor does not introduce any changes to the runtime logic but
improves the project structure for better development experience.
---------
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
This PR exports unified hostfunctions needed for parachains. Basicaly
`SubstrateHostFunctions` + `storage_proof_size::HostFunctions`.
Also removes the native executor from the parachain template.
---------
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
This PR significantly refactors the runtime API implementations to
improve project structure, maintainability, and readability. Key changes
include:
1. **Enhancing Visibility**: Adjusts the visibility of
`RUNTIME_API_VERSIONS` in `impl_runtime_apis.rs` to `pub`, making it
accessible throughout the runtime module.
2. **Centralizing API Implementations**: Introduces a new file,
`apis.rs`, within the parachain template's runtime directory.
3. **Streamlining `lib.rs`**: Updates the main runtime library file to
reflect these structural changes. It removes redundant API
implementations and points `VERSION` to the newly exposed
`RUNTIME_API_VERSIONS` from `apis.rs`, simplifying the overall runtime
configuration.
### Motivations Behind the Refactoring:
- **Improved Project Structure**: Centralizing API implementations in
`apis.rs` offers a clearer, more navigable project structure.
- **Better Readability**: Streamlining `lib.rs` and reducing clutter
enhance readability, making it easier for new contributors to understand
the project layout and logic.
### Summary of Changes:
- Made `RUNTIME_API_VERSIONS` public in `impl_runtime_apis.rs`.
- Added `apis.rs` to centralize runtime API implementations.
- Streamlined `lib.rs` to adjust to the refactored project structure.
Currently the xcm-executor returns an `Unimplemented` error if it
receives any HRMP-related instruction.
What I propose here, which is what we are currently doing in our forked
executor at polimec, is to introduce a trait implemented by the executor
which will handle those instructions.
This way, if parachains want to keep the default behavior, they just use
`()` and it will return unimplemented, but they can also implement their
own logic to establish HRMP channels with other chains in an automated
fashion, without requiring to go through governance.
Our implementation is mentioned in the [polkadot HRMP
docs](https://arc.net/l/quote/hduiivbu), and it was suggested to us to
submit a PR to add these changes to polkadot-sdk.
---------
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: command-bot <>
Removed the `pallet::getter` macro call from storage type definitions
and added the corresponding implementations directly.
fixes#3330
polkadot address: 14JzTPPUd8x8phKi8qLxHgNTnTMg6DUukCLXoWprejkaHXPz
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
The first step towards
https://github.com/paritytech/polkadot-sdk/issues/3155
Brings all templates under the following structure
```
templates
| parachain
| | polkadot-launch
| | runtime --> parachain-template-runtime
| | pallets --> pallet-parachain-template
| | node --> parachain-template-node
| minimal
| | runtime --> minimal-template-runtime
| | pallets --> pallet-minimal-template
| | node --> minimal-template-node
| solochain
| | runtime --> solochain-template-runtime
| | pallets --> pallet-template (the naming is not consistent here)
| | node --> solochain-template-node
```
The only note-worthy changes in this PR are:
- More `Cargo.toml` fields are forwarded to use the one from the
workspace.
- parachain template now has weights and benchmarks
- adds a shell pallet to the minimal template
- remove a few unused deps
A list of possible follow-ups:
- [ ] Unify READMEs, create a parent README for all
- [ ] remove references to `docs.substrate.io` in templates
- [ ] make all templates use `#[derive_impl]`
- [ ] update and unify all licenses
- [ ] Remove polkadot launch, use
https://github.com/paritytech/polkadot-sdk/blob/35349df993ea2e7c4769914ef5d199e787b23d4c/cumulus/zombienet/examples/small_network.toml
instead.