This PR adds a new PolkaVM-based executor to Substrate.
- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.
No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Bumps [wasm-instrument](https://github.com/paritytech/wasm-instrument)
from 0.3.0 to 0.4.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/paritytech/wasm-instrument/blob/master/CHANGELOG.md">wasm-instrument's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<p>All notable changes to this project will be documented in this
file.</p>
<p>The format is based on <a
href="https://keepachangelog.com/en/1.0.0/">Keep a Changelog</a>,
and this project adheres to <a
href="https://semver.org/spec/v2.0.0.html">Semantic Versioning</a>.</p>
<p>The semantic versioning guarantees cover the interface to the
substrate runtime which
includes this pallet as a dependency. This module will also add storage
migrations whenever
changes require it. Stability with regard to offchain tooling is
explicitly excluded from
this guarantee: For example adding a new field to an in-storage data
structure will require
changes to frontends to properly display it. However, those changes will
still be regarded
as a minor version bump.</p>
<p>The interface provided to smart contracts will adhere to semver with
one exception: Even
major version bumps will be backwards compatible with regard to already
deployed contracts.
In other words: Upgrading this pallet will not break pre-existing
contracts.</p>
<h2>[Unreleased]</h2>
<h3>New</h3>
<ul>
<li>Add new gas metering method: mutable global + local gas function
<a
href="https://redirect.github.com/paritytech/wasm-instrument/pull/34">#34</a></li>
<li>Account for locals initialization costs
<a
href="https://redirect.github.com/paritytech/wasm-instrument/pull/38">#38</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/paritytech/wasm-instrument/commits/v0.4.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
> **Note**
> Automatic rebases have been disabled on this pull request as it has
been open for over 30 days.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.
There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.
Dependencies:
- [x] PR that upgrades CI to use rust 1.74 is merged.
---------
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
* Upgrade wasm crate dependencies
* New wasmi version changed error output a bit
* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts
* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts
Co-authored-by: command-bot <>
* Upgraded dependencies
* Adapting code to scale v3
* Empty commit to trigger CI
* Triggering CI
* Fixing UI test
* Remove superfluous dev-dep added by #9228
* Cryout for CI
* Statically register host WASM functions
* Fix `substrate-test-client` compilation
* Move `ExtendedHostFunctions` to `sp-wasm-interface`
* Fix `sp-runtime-interface` tests' compilation
* Fix `sc-executor-wasmtime` tests' compilation
* Use `runtime_interface` macro in `test-runner`
* Fix `sc-executor` tests' compilation
* Reformatting/`rustfmt`
* Add an extra comment regarding the `H` generic arg in `create_runtime`
* Even more `rustfmt`
* Depend on `wasmtime` without default features in `sp-wasm-interface`
* Bump version of `sp-wasm-interface` to 4.0.1
* Bump `sp-wasm-interface` in `Cargo.lock` too
* Bump all of the `sp-wasm-interface` requirements to 4.0.1
Maybe this will appease cargo-unleash?
* Revert "Bump all of the `sp-wasm-interface` requirements to 4.0.1"
This reverts commit 0f7ccf8e0f371542861121b145ab87af6541ac30.
* Make `cargo-unleash` happy (maybe)
* Use `cargo-unleash` to bump the crates' versions
* Align to review comments
* Introduce `SecretUri`
* `inspect-key`: Adds support for `expect-public`
`expect-public` can be used to check that a given secret uri corresponds to the given public key.
This is mainly useful when the secret uri is protected by a password and a new derived account
should be generated. With `--expect-public` the user can pass the public key/account-id of the
"base" secret uri aka the one without any derivation to ensure the correct password was inserted.
* Fixes
* 🤦
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Review feedback
* FMT
* Bump the versions
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Intend to reactivate cargo-unleash check
It appears the bug it was deactivated for has been resolved a while ago. Trying to reactivate the checks.
* adding missing cargo.toml metadata for BEEFY crates
* fix wrong version reference
* matching up versions
* disable faulty cache
* switching more versions to prerelease
* Revert "disable faulty cache"
This reverts commit 411a12ae444a9695a8bfea4458a868438d870b06.
* bump minor of sc-allocator to fix already-published-issue
* fixup another pre-released dependency problem
* temp switch to latest unleash
* fixing dependency version and features
* prometheus endpoint has also been changed
* fixing proposer metrics versioning
* fixing hex feature for beefy
* fix generate-bags feature selection
* fixup Cargo.lock
* upgrade prometheus dependencies
* missed one
* switch to latest release
* Improve sandbox internal api
This improves the internal sandbox api for the executor implementations.
The main point is to hide the tls in the internal api and not having it
exposed to the outside.
This is especially needed for wasmtime 0.29.0
* Fmt
* Make it nicer
* Add comments and refactor Sandbox module
* Adds some comments
* Add wasmtime instance to the sandbox and delegate calls
* Adds module imports stub
* WIP state holder via *mut
* My take at the problem
* Brings back invoke and instantiate implementation details
* Removes redundant bound
* Code cleanup
* Fixes invoke closure
* Refactors FunctionExecutor to eliminate lifetime
* Wraps `FunctionExecutor::sandbox_store` in `RefCell`
* Renames `FunctionExecutor::heap` to `allocator`
* Wraps `FunctionExecutor::allocator` in `RefCell`
* Refactors FunctionExecutor to `Rc<Inner>` pattern
* Implements scoped TLS for FunctionExecutor
* Fixes wasmi instancing
* Fixes sandbox asserts
* Makes sandbox compile after wasmtime API change
* Uses Vurich/wasmtime for the Lightbeam backend
* Uses wasmtime instead of wasmi for sandbox API results
* Refactors sandbox to use one of the execution backends at a time
* Fixes wasmtime module instantiation
* TEMP vurich branch stuff
* Adds wasmer impl stub
* Adds get global
* Fixes warnings
* Adds wasmer invoke impl
* Implements host function interface for wasmer
* Fixes wasmer instantiation result
* Adds workaround to remove debug_assert
* Fixes import object generation for wasmer
* Attempt to propagate wasmer::Store through sandbox::Store
* Wraps `sandbox::Store::memories` in `RefCell`
* Moves `sandbox::instantiate` to `sandbox::Store`
* Eliminate `RefCell<memories>`
* Implements `HostState::memory_get/set`, removes accidental `borrow_mut`
* Fixes sandbox memory handling for wasmi
* Fix memory allocation
* Resets Cargo.lock to match master
* Fixes compilation
* Refactors sandbox to use TLS for dispatch_thunk propagation to wasmer
* Pass dispatch thunk to the sandbox as a TLS
* Initialize dispatch thunk holder in `SandboxInstance`
* Comment out Wasmtime/Lightbeam sandbox backend
* Revert wasmtime back to mainstream
* Adds SandboxExecutionMethod enum for cli param
* Cleanup sandbox code
* Allow wasmi to access wasmer memory regions
* More cleanup
* Remove debug logging, replace asserts with runtime errors
* Revert "Adds SandboxExecutionMethod enum for cli param"
This reverts commit dcb2b1d3b54145ab51ad2e3fef0d980ba215b596.
* Fixes warnings
* Fixes indentation and line width
* Fix return types condition
* Puts everything related under the `wasmer-sandbox` feature flag
* Fixes warnings
* Address grumbles
* Split instantiate per backend
* More splits
* Refacmemory allocation
* Nitpicks
* Attempt to wrap wasmer memory in protoco enforcing type
* Revert renaming
* WIP wasm buffer proxy API
* Reimplement util::wasmer::MemoryRef to use buffers instead of memory slices
* Adds WasmiMemoryWrapper and MemoryTransfer trait
* Refactor naming
* Perform all memory transfers using MemoryTransfer
* Adds allocating `read`
* Adds comments
* Removes unused imports
* Removes now unused function
* Pulls Cargo.lock from origin/master
* Fix rustdoc
* Removes unused `TransferError`
* Update Cargo.lock
* Removes unused import
* cargo fmt
* Fix feature dependency graph
* Feature should flow from the top level crate
* We should not assume a specific workspace structure
* sc-executor-wasmi does not use the feature
* sc-executor-wasmtime should not know about the feature
* Fix doc typo
* Enable wasmer-sandbox by default (for now)
It will be removed before merge. It is so that the benchbot
uses the wasmer sandbox.
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* Revert "cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs"
This reverts commit d713590ba45387c4204b2ad97c8bd6f6ebabda4e.
* cargo fmt
* Add ci-check to prevent wasmer sandbox build breaking
* Run tests with wasmer-sandbox enabled
* Revert "Run tests with wasmer-sandbox enabled"
This reverts commit cff63156a162f9ffdab23e7cb94a30f44e320f8a.
Co-authored-by: Sergei Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
* mark template and utils as non-publish
* switch to development version for testing
* activate unleash check
* maybe if I disable all rules...
* Fix isolated compilation of `max-encoded-len-derive` with `syn`
error[E0369]: binary operation `==` cannot be applied to type `syn::Path`
--> src/lib.rs:88:29
|
88 | .filter(|attr| attr.path == parse_quote!(max_encoded_len_crate))
| --------- ^^ ----------------------------------- _
| |
| syn::Path
error: aborting due to previous error
For more information about this error, try `rustc --explain E0369`.
Error: could not compile `max-encoded-len-derive`
* WIP: bump changes crates since v3 tag to next breaking
cargo unleash version bump-breaking --changed-since v3.0.0
cargo unleash version set-pre dev --changed-since v3.0.0
FIXME: Don't modify crates that are not yet released, e.g.
`max-encoded-len-derive`
* Update lockfile
* WIP: Bump sp-transaction-pool as well
* WIP: Bump sp-offchain as well
* WIP: Bump frame-system-rpc-runtime-api as well
* WIP: Bump sp-authority-discovery as well
* Manually deactivate dev-deps before `cargo unleash check`
Otherwise we run into `Cycle detected` error.
* Bump sp-consensus-slots
* Add missing Cargo.lock change
* Bump sp-consensus-vrf as well
* Bump sp-keyring as well
* Bump sp-consensus-pow as well
* Try to speed up the `unleash-check` job
Previously, the job took 106 minutes - let's see if explicitly
specifying a `CARGO_TARGET_DIR` will help
* fixup: Ensure the temp target dir exists for unleash check
* Bump pallet-transaction-payment-rpc-runtime-api as well
Needed for Polkadot
* Bump pallet-transaction-payment-rpc as well
Needed for Polkadot
* Try updating crates after patching in the Polkadot CI job
* Use another approach to update patched Substrate crates
* Try to update all sp-core versions in Polkadot CI job
* Simplify sp-core version checking
* Apply another shellcheck lint
* Just do the simplest thing I guess
* Welp don't do --offline then
* Clean up `unleash-check` job triggers
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
* Fix a note in unleash-check cache step
* Add a note about temporary optimization in cargo-unleash
* Pin a newer version of cargo-unleash
Co-authored-by: Igor Matuszewski <xanewok@gmail.com>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
* Move alloc primitive (not used in /pallets)
* Move to alternative location as not shared
* moved crates to different dir
* ren sp_chain_spec to sc_chain_spec_primatives
* merged sc-chain-spec and moved allocation up one.
* no no_std
* nudge
* Bump CI
* emit a custom section from impl_runtime_apis!
This change emits a custom section from the impl_runtime_apis! proc macro.
Each implemented API will result to emitting a link section `runtime_apis`.
During linking all sections with this name will be concatenated and
placed into the final wasm binary under the same name.
* Introduce `runtime_version` proc macro
This macro takes an existing `RuntimeVersion` const declaration, parses
it and emits the version information in form of a linking section.
Ultimately such a linking section will result into a custom wasm
section.
* Parse custom wasm section for runtime version
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Fix sc-executor integration tests
* Nits
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Refactor apis section deserialization
* Fix version decoding
* Reuse uncompressed value for CallInWasm
* Log on decompression error
* Simplify if
* Reexport proc-macro from sp_version
* Merge ReadRuntimeVersionExt
* Export `read_embedded_version`
* Fix test
* Simplify searching for custom section
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Establish the runtime_blob module
Seed it with the existing contents of the `util` module.
* Port wasmtime mutable globals instrumentation into runtime blob APIs
* Opt-out from fast instance reuse
* Minor clean up
* Spaces
* Docs clean up
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Factor out the expects
* Fix the suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* chore/error: remove from str conversion and add deprecation notifications
* fixup changes
* fix test looking for gone ::Msg variant
* another test fix
* one is duplicate, the other is not, so duplicates reported are n-1
* darn spaces
Co-authored-by: Andronik Ordian <write@reusable.software>
* remove pointless doc comments of error variants without any value
* low hanging fruits (for a tall person)
* moar error type variants
* avoid the storage modules for now
They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.
* chore remove pointless error generic
* fix test for mocks, add a bunch of non_exhaustive
* max line width
* test fixes due to error changes
* fin
* error outputs... again
* undo stderr adjustments
* Update client/consensus/slots/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove closure clutter
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* more error types
* introduce ApiError
* extract Mock error
* ApiError refactor
* even more error types
* the last for now
* chore unused deps
* another extraction
* reduce should panic, due to extended error messages
* error test happiness
* shift error lines by one
* doc tests
* white space
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Into -> From
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove pointless codec
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* avoid pointless self import
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@parity.io>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Re-generate READMEs to fix doc links; set readme field in package manifests
* Re-generate READMEs to fix doc links; set readme field in package manifests
* Re-generate READMEs to fix doc links; set readme field in package manifests
* Re-generate READMEs to fix doc links; set readme field in package manifests
* Revert stuff that shouldn't have been committed
* Revert stuff that shouldn't have been committed
* Fix parent relative link generation
* Manually remove this incorrect link for now.
* Bump version
* update test-utils crates to be ready for publishing
* adding changelog
* Adding automaticly generated READMEs
* fixing versions
* another version mishap
Koute