This PR adds a new PolkaVM-based executor to Substrate.
- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.
No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
- Set log to `0.4.20` everywhere
- Lift `log` to the workspace
Starting with a simpler one after seeing
https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw.
This sets the `default-features` to `false` in the root and then
overwrites that in each create to its original value. This is necessary
since otherwise the `default` features are additive and its impossible
to disable them in the crate again once they are enabled in the
workspace.
I am using a tool to do this, so its mostly a test to see that it works
as expected.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Step towards https://github.com/paritytech/polkadot-sdk/issues/1975
As reported
https://github.com/paritytech/polkadot-sdk/issues/1975#issuecomment-1774534225
I'd like to encapsulate crypto related stuff in a dedicated folder.
Currently all cryptographic primitive wrappers are all sparsed in
`substrate/core` which contains "misc core" stuff.
To simplify the process, as the first step with this PR I propose to
move the cryptographic hashing there.
The `substrate/crypto` folder was already created to contains `ec-utils`
crate.
Notes:
- rename `sp-core-hashing` to `sp-crypto-hashing`
- rename `sp-core-hashing-proc-macro` to `sp-crypto-hashing-proc-macro`
- As the crates name is changed I took the freedom to restart fresh from
version 0.1.0 for both crates
---------
Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
Bumps [wasm-instrument](https://github.com/paritytech/wasm-instrument)
from 0.3.0 to 0.4.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/paritytech/wasm-instrument/blob/master/CHANGELOG.md">wasm-instrument's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<p>All notable changes to this project will be documented in this
file.</p>
<p>The format is based on <a
href="https://keepachangelog.com/en/1.0.0/">Keep a Changelog</a>,
and this project adheres to <a
href="https://semver.org/spec/v2.0.0.html">Semantic Versioning</a>.</p>
<p>The semantic versioning guarantees cover the interface to the
substrate runtime which
includes this pallet as a dependency. This module will also add storage
migrations whenever
changes require it. Stability with regard to offchain tooling is
explicitly excluded from
this guarantee: For example adding a new field to an in-storage data
structure will require
changes to frontends to properly display it. However, those changes will
still be regarded
as a minor version bump.</p>
<p>The interface provided to smart contracts will adhere to semver with
one exception: Even
major version bumps will be backwards compatible with regard to already
deployed contracts.
In other words: Upgrading this pallet will not break pre-existing
contracts.</p>
<h2>[Unreleased]</h2>
<h3>New</h3>
<ul>
<li>Add new gas metering method: mutable global + local gas function
<a
href="https://redirect.github.com/paritytech/wasm-instrument/pull/34">#34</a></li>
<li>Account for locals initialization costs
<a
href="https://redirect.github.com/paritytech/wasm-instrument/pull/38">#38</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/paritytech/wasm-instrument/commits/v0.4.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
> **Note**
> Automatic rebases have been disabled on this pull request as it has
been open for over 30 days.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.
There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.
Dependencies:
- [x] PR that upgrades CI to use rust 1.74 is merged.
---------
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.
If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html
@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.
TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Since `sp-state-machine` and `GenesisConfigBuilderRuntimeCaller` always
set `use_native` to be false.
We should remove this param and make `NativeElseWasmExecutor` behave
like its name.
It could make the above components use the correct execution strategy.
Maybe polkadot do not need about `NativeElseWasmExecutor` anymore. But
it is still needed by other chains and it's useful for debugging.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.
The `NotificationService` concept introduces two additional benefits:
* allow protocols to start using custom handshakes
* allow protocols to accept/reject inbound peers
Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.
With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.
This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.
Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556
---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197 but there are no
functional changes compared to that PR
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
This moves the macro related re-exports to `__private` to make it more
obvious for downstream users that they are using an internal api.
---------
Co-authored-by: command-bot <>
Changes the maximum instances count for `wasmtime` to `64`. It also
allows to only pass in maximum `32` for `--max-runtime-instances` as
`256` was way too big. With `64` instances in total and `32` that can be
configured in maximum, there should be enough space to accommodate for
extra instances that are may required to be allocated adhoc.
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
It seems the old strategy have been depracted more than one year.
So maybe it's time to clean up old strategy for wasm executor.
---
polkadot address: 15ouFh2SHpGbHtDPsJ6cXQfes9Cx1gEFnJJsJVqPGzBSTudr
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Koute <koute@users.noreply.github.com>
* Fix std, runtime-benchmarks and try-runtime features
zepter lint propagate-feature --feature try-runtime --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="try-runtime:frame-try-runtime"
zepter lint propagate-feature --feature runtime-benchmarks --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="runtime-benchmarks:frame-benchmarking"
zepter lint propagate-feature --feature std --left-side-feature-missing=ignore --workspace --fix
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add propagate feature CI check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use --locked
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit cf4ff6cc0632269b0a109e547686e5e3314b02de.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* No newline in help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit 5daa06ada8e01f5bebafb9d1c76804dd79bc1006.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit ca15de5729507a564f140a10ec2e87b19516ec4c.
* Fix msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert back to master
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Re-do with Zepter v0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update Zepter to 0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Disable rococo try-runtime check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* More review fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* sc-executor-wasmtime: Do not use absolute path to `Cargo.toml` in test
As the test is executed inside the workspace, `cargo metadata` will automatically detect the correct
`Cargo.toml`. This is required for the mono repo.
* ".git/.scripts/commands/fmt/fmt.sh"
---------
Co-authored-by: command-bot <>
* wasm-builder: Enforce `runtime_version` wasm section
This pr changes the `wasm-builder` to enforce the `runtime_version` wasm section. This wasm section
is being created by the `sp_version::runtime_version` attribute macro. This attribute macro now
exists since quite some time and `runtime_version` also is the only way for parachains to support
reading the `RuntimeVersion` from the runtime.
\# Disabling the check
By default the `WasmBuilder` will now check for this wasm section and if not found, exit with an
error. However, there are situations where you may want to disable this check (like for tests). In
this case there exists the `disable_runtime_version_section_check` function.
```
WasmBuilder::new()
...
...
...
.disable_runtime_version_section_check()
.build()
```
By using this method you get back the old behavior.
* Review comment
* Fix
* Fix issue with `enum-as-inner`
* HoldReason: Improve usage
`HoldReason` was switched recently to use the `composite_enum` attribute that will merge the enums
from all pallets in the runtime to `RuntimeHoldReason`. `pallet-nis` was still requiring that the
variant was passed as constant to call `hold`. The proper implementation is to use the `HoldReason`
from inside the pallet directly when calling `hold`. This is done by adding a `RuntimeHoldReason` as
type to the `Config` trait and requiring that `Currency` is using the same reason. Besides that the
pr changes the name `HoldIdentifier` in `pallet_balances::Config` to `RuntimeHoldReason`.
* Update frame/nis/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Review comment
* Fixes
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WASM executor: add `OutputExceedsBounds` variant to `Error`
Previously this was a `WasmError`, which is intended for runtime construction errors. However this
led to confusion as output-exceeds-bounds occurs due to execution of `validate_block`.
* Fix warning
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
Koute