Commit Graph
100 Commits
Author SHA1 Message Date
Koute b0f34e4b29 Add a PolkaVM-based executor (#3458)
This PR adds a new PolkaVM-based executor to Substrate.

- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.

No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
2024-03-12 05:23:06 +00:00
Koute 822082807f Fix wasm-builder not exiting if compilation fails (#3439)
This PR fixes a subtle bug in `wasm-builder` first introduced in
https://github.com/paritytech/polkadot-sdk/pull/1851 (sorry, my bad! I
should have caught this during review) where the status code of the
`cargo` subprocess is not properly checked, which results in builds
silently succeeding when they shouldn't (that is: if we successfully
build a runtime blob, and then modify the code so that it won't compile,
and recompile it again, then the build will succeed and silently use the
*old* blob).

cc @athei This is the bug you were seeing.

[edit]Also fixes a similar PolkaVM-specific bug where I accidentally
used the wrong comparison operator.[/edit]
2024-02-22 10:03:12 +00:00
Koute b179b83264 Make sp-keystore no_std-compatible and fix the build-runtimes-polkavm CI job (#3363)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3352
2024-02-18 15:32:01 +00:00
Koute 402b64caf5 Build more runtimes targeting PolkaVM (#3209)
This PR improves compatibility with RISC-V and PolkaVM, allowing more
runtimes to successfully compile.

In particular, it makes the following changes:

- The `sp-mmr-primitives` and `sp-consensus-beefy` crates
unconditionally required an `std`-only dependency; now they only require
those dependencies when the `std` feature is actually enabled. (Our
RISC-V target is, unlike WASM, a true `no_std` target where you can't
accidentally use stuff from `std` anymore.)
- One of our dependencies (the `bitvec` trace) uses a crate called
`radium` which doesn't compile under RISC-V due to incomplete
autodetection logic in their `build.rs` file. The good news is that this
is already fixed in the newest upstream version of `radium`, and the
newest version of `bitvec` uses it. The bad news is that the newest
version of `bitvec` is not currently released on crates.io, so we can't
use it. I've [created an
issue](https://github.com/ferrilab/ferrilab/issues/5) asking for a new
release, but in the meantime I forked the currently used `radium` 0.7,
[fixed the faulty
logic](https://github.com/paritytech/radium-0.7-fork/commit/ed66c8a294b138c67f93499644051d97d4c7fbda)
and used cargo's patching capabilities to use it for the RISC-V runtime
builds. This might be a little hacky, but it is the least intrusive way
to fix the problem, doesn't affect WASM builds at all, and we can
trivially remove it once a new `bitvec` is released.
- The new runtimes are added to the CI to make sure their compilation
doesn't break.
2024-02-06 14:04:21 +00:00
KouteandBastian Köcher e349fc9ef8 Initial support for building RISC-V runtimes targeting PolkaVM (#3179)
This PR adds initial support for building RISC-V runtimes targeting
PolkaVM.

- Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will
now build a RISC-V runtime instead of a WASM runtime.
- This only adds support for *building* runtimes; running them will need
a PolkaVM-based executor, which I will add in a future PR.
- Only building the minimal runtime is supported (building the Polkadot
runtime doesn't work *yet* due to one of the dependencies).
- The builder now sets a `substrate_runtime` cfg flag when building the
runtimes, with the idea being that instead of doing `#[cfg(not(feature =
"std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're
building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the
whole codebase to use this will be done in a future PR; I deliberately
didn't do this here to keep this PR minimal and reviewable.)
- Further renaming of things (e.g. types, environment variables and proc
macro attributes having "wasm" in their name) to be target-agnostic will
also be done in a future refactoring PR (while keeping backwards
compatibility where it makes sense; I don't intend to break anyone's
workflow or create unnecessary churn).
- This PR also fixes two bugs in the `wasm-builder` crate:
* The `RUSTC` environment variable is now removed when invoking the
compiler. This prevents the toolchain version from being overridden when
called from a `build.rs` script.
* When parsing the `rustup toolchain list` output the `(default)` is now
properly stripped and not treated as part of the version.
- I've also added a minimal CI job that makes sure this doesn't break in
the future. (cc @paritytech/ci)

cc @athei

------

Also, just a fun little tidbit: quickly comparing the size of the built
runtimes it seems that the PolkaVM runtime is slightly smaller than the
WASM one. (`production` build, with the `names` section substracted from
the WASM's size to keep things fair, since for the PolkaVM runtime we're
currently stripping out everything)

- `.wasm`: 625505 bytes
- `.wasm` (after wasm-opt -O3): 563205 bytes
- `.wasm` (after wasm-opt -Os): 562987 bytes
- `.wasm` (after wasm-opt -Oz): 536852 bytes
- `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target
features; I'll add those in another PR once we have an executor working)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-02-03 04:12:12 +00:00
Koute f01781a902 Remove wasm-builder's README (#2525)
Followup of https://github.com/paritytech/polkadot-sdk/pull/2217

This PR deletes the README of the `wasm-builder` crate and moves its
docs back into the rustdoc, [as requested
here](https://github.com/paritytech/polkadot-sdk/pull/2217#discussion_r1406401175).
(:
2023-11-28 14:30:02 +01:00
KouteandBastian Köcher 2610450a18 Build the standard library crates when building the runtimes (#2217)
Our executor currently only supports the WASM MVP feature set, however
nowadays when compiling WASM the Rust compiler has more features enabled
by default.

We do set the `-C target-cpu=mvp` flag to make sure that *our* code gets
compiled in a way that is compatible with our executor, however this
doesn't affect Rust's standard library crates (`std`, `core` and
`alloc`) which are by default precompiled and still can make use of
these extra features.

So in this PR we force the compiler to also compile the standard library
crates for us to make sure that they also only use the MVP features.

I've added the `WASM_BUILD_STD` environment variable which can be used
to disable this behavior if set to `0`.

Unfortunately this *will* slow down the compile times when building
runtimes, but there isn't much that we can do about that.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1755

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-11-27 12:40:27 +02:00
Koute 9d1741eb3c Bump parity-scale-codec to 3.6.1 (#14428) 2023-06-21 11:37:11 +00:00
Koute 8eda0a039f Bump parity-scale-codec to 3.6.1 (#7382) 2023-06-21 19:24:24 +09:00
Kouteandparity-processbot afd1192898 Do not assume AssetIds are Copy (companion for substrate#14158) (#2586)
* Do not assume `AssetId`s are `Copy`

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: parity-processbot <>
2023-05-23 12:09:45 +00:00
Kouteandparity-processbot 194c9edd4a Remove the Copy bound on AssetId (#14158)
* Remove the `Copy` bound on `AssetId`

* Also relax the `Copy` bound in the assets pallet

* Fix warnings on the newest nightly Rust

* Remove some unnecessary `clone()`s

* Try to satisfy clippy

* Remove some more unnecessary `clone()`s

* Add more `.clone()`s for newly merged code

* Also add `clone()`s in the benchmarks

---------

Co-authored-by: parity-processbot <>
2023-05-23 11:34:04 +00:00
Koute 8f6445fb8e Remove the Copy bound on CollectionId in the uniques pallet (#14111)
* Remove the `Copy` bound on `CollectionId` in the uniques pallet

* Also add `clone`s in benchmarks
2023-05-10 09:39:43 +00:00
Koute 59054bd466 Use MaybeUninit when calling getrusage (musl compatibility) (#7086) 2023-04-17 06:44:08 +00:00
Kouteandparity-processbot 795fcf0484 Disable sign-ext WASM feature when building runtimes (#13804)
Co-authored-by: parity-processbot <>
2023-04-03 12:00:31 +00:00
Koute 6c3747ba34 Bump trie-db to 0.27.1 (fix storage iteration bug + change seek behavior with prefix) (#13630) 2023-03-17 16:09:28 +00:00
Koute 3777f2db42 Update substrate to 48e7cb147cb9a27125fd2e82edbcf4d0ed5927c4 (#6851) 2023-03-09 10:51:33 +01:00
Kouteandcommand-bot 03158cd48f Switch to the blake2b_simd crate in sp-core-hashing (#13548)
* Switch to the `blake2b_simd` crate in `sp-core-hashing`

* ".git/.scripts/commands/bench/bench.sh" pallet dev frame_benchmarking

---------

Co-authored-by: command-bot <>
2023-03-07 08:57:16 +00:00
Koute 723d409264 Bump blake2b_simd to 1.0.1 (#2282) 2023-03-07 08:55:57 +00:00
Koute 4778300431 Bump blake2b_simd to 1.0.1 (#6829) 2023-03-07 06:50:08 +00:00
Koute 55263fa2a1 Speed up storage iteration from within the runtime (#13479)
* Speed up storage iteration from within the runtime

* Move the cached iterator into an `Option`

* Use `RefCell` in no_std

* Simplify the code slightly

* Use `Option::replace`

* Update doc comment for `next_storage_key_slow`
2023-03-01 08:58:18 +00:00
Koute 91c595bc78 Further storage iterator refactoring (#13445)
* Remove `Backend::apply_to_key_values_while`

* Add `IterArgs::start_at_exclusive`

* Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while`

* Remove `Backend::apply_to_keys_while`

* Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix`

* Remove unnecessary `to_vec` calls

* Fix unused method warning in no_std

* Remove unnecessary import

* Also check proof sizes in the test

* Iterate over both keys and values in `prove_range_read_with_size` and add a test
2023-02-27 10:45:42 +00:00
Koute 17e055e594 Bump wasmtime to 6.0.0 (#13429)
* Bump `wasmtime` to 6.0.0

* Disable cranelift egraphs
2023-02-23 10:56:11 +00:00
Kouteandparity-processbot 1b30ce5a9d Companion for substrate#13284 (#6653)
* Align iterator use to new substrate

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-02-22 08:36:16 +00:00
Koute f8e3bdad3d Rework storage iterators (#13284)
* Rework storage iterators

* Make sure storage iteration is also accounted for when benchmarking

* Use `trie-db` from crates.io

* Appease clippy

* Bump `trie-bench` to 0.35.0

* Fix tests' compilation

* Update comment to clarify how `IterArgs::start_at` works

* Add extra tests

* Fix iterators on `Client` so that they behave as before

* Add extra `unwrap`s in tests

* More clippy fixes

* Come on clippy, give me a break already

* Rename `allow_missing` to `stop_on_incomplete_database`

* Add `#[inline]` to `with_recorder_and_cache`

* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment

* Simplify code: use `with_trie_db` in `next_storage_key_from_root`

* Remove `expect`s in the benchmarking CLI

* Add extra doc comments

* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)

* Remove a TODO in tests

* Update comment for `StorageIterator::was_complete`

* Update `trie-db` to 0.25.1
2023-02-22 07:49:25 +00:00
Kouteandparity-processbot 2875a0f0b2 Companion for substrate#13160 (#6661)
* Update `wasmtime` to 5.0.0 (+ other deps)

* update lockfile for {"substrate"}

* Update `substrate` to dd4497d8a459d5e3ecac367b991d5dae1f0940c5

* Update `substrate` to 3d60070c1373bc0383aa4f003a5b243c4c4f7118

---------

Co-authored-by: parity-processbot <>
2023-02-07 17:50:11 +01:00
Koute a5fd71ae62 Disable default features for strum (#13326) 2023-02-07 16:29:14 +01:00
Koute 44cbf303c4 Bump wasmtime to 5.0.0 (and a few other deps) (#13160)
* Bump `wasmtime` to 4.0.0 (and a few other deps)

* Use `Error::msg` instead of `anyhow!`

* Bump `wasmtime` to 5.0.0

* Update `Cargo.lock`

* Add `wasmtime` feature to `sp-wasm-interface` dependency
2023-02-06 15:40:50 +00:00
Koute 96ca4bed0e Rework the trie cache (#12982)
* Rework the trie cache

* Align `state-machine` tests

* Bump `schnellru` to 0.1.1

* Fix off-by-one

* Align to review comments

* Bump `ahash` to 0.8.2

* Bump `schnellru` to 0.2.0

* Bump `schnellru` to 0.2.1

* Remove unnecessary bound

* Remove unnecessary loop when calculating maximum memory usage

* Remove unnecessary `mut`s
2023-01-26 05:38:00 +00:00
Koute 9e9b79ec67 Remove dead code (#13213) 2023-01-23 15:08:51 +01:00
Kouteandparity-processbot b39409f2de Companion for substrate#13154 (#6570)
* Remove lifetime from `KeyIterator`

* Remove the lifetime from the method too

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2023-01-18 07:32:32 +00:00
KouteandBastian Köcher 338d75d0a3 Fix the storage_size/state_getStorageSize RPC call (#13154)
* Have `KeyIterator` clone the `prefix` it receives

* Stream keys in `storage_size` RPC and add a runtime limit

* Update client/rpc/Cargo.toml

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/rpc/src/state/utils.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Rename the types to signify that the cancellation is due to a timeout

* Move the test into a `mod tests`

* Add a comment regarding `biased` in `tokio::select`

* Make the `clone` explicit when calling `KeyIterator::{new, new_child}`

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-18 06:28:27 +00:00
Koute 9040eb303c Extend lower bound of manage_lease_period_start from runtime_common::slots (#6318) 2022-11-21 11:40:47 +00:00
Koute 1ab020af9b Remove the wasmtime feature flag (companion for substrate#12684) (#1861)
* Remove the `wasmtime` feature flag

* Update `substrate` and `polkadot` to the newest `master`

* Update `substrate` and `polkadot` to the newest `master`
2022-11-18 15:28:55 +00:00
Koute 9c2561f07a Remove the wasmtime feature flag (companion for substrate#12684) (#6268)
* Remove the `wasmtime` feature flag

* Update `substrate` to the newest `master`

* Update `substrate` to the newest `master`
2022-11-18 14:39:40 +00:00
Koute 4214bc527c Remove the wasmtime feature flag (#12684)
* Remove the `wasmtime` feature flag

* rustfmt
2022-11-18 14:21:44 +01:00
KouteandShawn Tabrizi 5c7ec6909d Remove sp_tasks::spawn API and related code + host functions (#12639)
* Remove `sp_tasks::spawn` API and related code

* Remove `RuntimeTasks::{spawn, join}` host functions

* remove unused

* Remove a few more tests that I forgot to remove

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2022-11-10 12:23:56 +00:00
KouteandOliver Tale-Yazdi 3faa0bd20d Treat near-zero intercept values as zero when calculating weights (#12573)
* Treat near-zero intercept values as zero when calculating weights

* Simplify comparison

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update contract weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2022-10-28 20:39:59 +00:00
92d2977292 Force base weights to be the minimum only when the intercept is negative (#12482)
* Force base weights to be the minimum only when the intercept is negative; emit minimum execution times

* Add an `assert` making sure the intercept is zero when it's supposed to be zero

* Fix template

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_assets

* ".git/.scripts/bench-bot.sh" pallet dev pallet_uniques

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2022-10-20 20:07:52 +00:00
Koute 1b3f211efa Enable the wasmtime-based WASM executor by default (#12486) 2022-10-13 10:31:00 +00:00
Koute eee79a1fa4 Extend the lower bounds of some of the benchmarks to also include 0 (#12386)
* Extend the lower bounds of some of the benchmarks to also include `0`

* Fix verify snippet for `pallet_bounties/spend_funds`
2022-10-07 14:16:41 +00:00
Koute ecc4d5f258 Add @koute to docs/CODEOWNERS and update stale paths (#12408) 2022-10-04 10:01:50 +02:00
Kouteandcommand-bot 8d81065a54 Support running the pallet benchmarks analysis without running the benchmarks (#12361)
* Support running the pallet benchmarks analysis without running the benchmarks

* Rename `override-results` to `json-input` and update the help comment

* ".git/.scripts/fmt.sh" 1

Co-authored-by: command-bot <>
2022-09-28 19:38:12 +00:00
Koute d5c57ffcc1 Bump wasmtime to 1.0.0 (#12317) 2022-09-21 10:21:01 +00:00
Kouteparity-processbotShawn Tabrizi
b1bbdf1192 Improve base weights consistency and make sure they're never zero (#11806)
* Improve base weights consistency and make sure they're never zero

* Switch back to `linregress` crate; add back estimation errors

* Move the test into `mod tests`

* Use all of the samples instead of only the first one

* Use full precision extrinsic base weights and slopes

* Add an extra comment

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2022-09-14 16:01:13 +00:00
Koute b922a25ccd Force rustix to use libc (#12123) 2022-08-29 15:30:37 +00:00
Koute 9c56e79c43 Restore wasmtime's default stack size limit to 1MB (#11993)
* Restore `wasmtime`'s default stack size limit to 1MB

* Add extra comments

* Enforce different maximum call depth in release mode

* Split the call depth limit in two
2022-08-09 11:04:28 +00:00
Kouteandparity-processbot 1a32cdc915 Companion for substrate#11523 (#1390)
* Align to changes in substrate: `sp_io::storage::get` now returns `Bytes`

* Update substrate and polkadot to the newest master

* update lockfile for {"substrate", "polkadot"}

Co-authored-by: parity-processbot <>
2022-07-29 09:34:26 +00:00
Kouteandparity-processbot faeb3d944d Companion for substrate#11523 (#5702)
* Bump `parity-scale-codec` to 3.1.5

* Align tests to changes in substrate

* Apply rustfmt

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2022-07-29 09:07:11 +00:00
KouteandBastian Köcher c4b607d4c9 Prevent double allocation of the payload when calling sp_io::storage::get (#11523)
* Expose allocation stats in `FreeingBumpHeapAllocator`

* Return allocation stats when calling into the runtime

* Bump `parity-scale-codec` to 3.1.3 (fork)

* Prevent double allocation of the payload when calling `sp_io::storage::get`

* Fix tests

* Remove unnecessary `mut`

* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Bump `parity-scale-codec` to 3.1.3

* Fix some of the UI tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-07-29 07:46:15 +00:00
Koute 8f6ba2e75e Improve wasmtime error reporting (#11856)
* Improve `wasmtime` error reporting

* cargo fmt
2022-07-18 11:51:38 +00:00
Koute 4981446a82 Update dependencies (companion for substrate#11722) (#5731)
* Update dependencies

* Update `substrate` to ee3eb8f2448cc1bb978c5d1564febd351c128bb0
2022-06-28 09:13:54 +00:00
Koute 3d89fa3605 Prevent unsoundness in environments with broken madvise(MADV_DONTNEED) (#11722)
* Prevend unsoundness in environments with broken `madvise(MADV_DONTNEED)`

* Add the `std` feature to `rustix` dependency

Apparently not having this breaks compilation on non-nightly toolchains.

* Autodetect the page size when checking whether `madvise` works

* Only make sure that the madvice check doesn't return `Err`
2022-06-28 06:54:56 +00:00
Kouteandparity-processbot bda05cecaf Bump wasmtime to 0.38.0 and zstd to 0.11.2 (companion for substrate#11720) (#5707)
* Bump `wasmtime` to 0.37.0 and `zstd` to 0.11.2

* Bump `wasmtime` to 0.38.0

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2022-06-24 12:29:29 +00:00
Koute 025e1d76f1 Bump wasmtime to 0.38.0 and zstd to 0.11.2 (#11720)
* Bump `wasmtime` to 0.37.0 and `zstd` to 0.11.2

* Bump `wasmtime` to 0.38.0
2022-06-24 11:25:21 +00:00
Koute 9cee3c699d Pump the gossip engine while waiting for the BEEFY runtime pallet (memory leak fix) (#11694)
* Pump the gossip engine while waiting for the BEEFY runtime pallet

This fixes a memory leak when the BEEFY gadget is turned on, but
the runtime doesn't actually use BEEFY.

* Implement `FusedFuture` for `GossipEngine`

* Fuse futures outside of loops
2022-06-20 15:52:43 +00:00
Koute 849a6c35fd Remove without_storage_info for the authorship pallet (#11610)
* Remove `without_storage_info` for the authorship pallet

* Tweak impl bounds style

* Use `defensive_proof` instead of `expect`
2022-06-16 11:48:05 +00:00
Koute a8c1d430ef Fix one-by-off in BoundedSlice::try_from (#11600) 2022-06-06 16:17:41 +00:00
Koute 4f393171d7 Adjust maximum memory pages hard limit for the pooling instantiation strategy (#11482)
* Run `sc-executor-wasmtime` unit tests for all instantiation strategies

* Adjust maximum memory pages hard limit for the pooling instantiation strategy
2022-05-24 05:37:04 +00:00
Kouteandparity-processbot d9eff4ecd4 Switch to pooling copy-on-write instantiation strategy for WASM (companion for Substrate#11232) (#5337)
* Switch to pooling copy-on-write instantiation strategy for WASM

* Fix compilation of `polkadot-test-service`

* Update comments

* Move `max_memory_size` to `Semantics`

* Rename `WasmInstantiationStrategy` to `WasmtimeInstantiationStrategy`

* Update a safety comment

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2022-05-19 13:06:34 +02:00
Koute dd854c16e2 Switch to pooling copy-on-write instantiation strategy for WASM (#11232)
* Switch to pooling copy-on-write instantiation strategy for WASM

* Fix benchmark compilation

* Fix `cargo fmt`

* Fix compilation of another benchmark I've missed

* Cleanups according to review comments

* Move `max_memory_size` to `Semantics`

* Set `memory_guaranteed_dense_image_size` to `max_memory_size`

* Rename `wasm_instantiation_strategy` to `wasmtime_instantiation_strategy`

* Update the doc-comments regarding the instantiation strategy

* Extend the integration tests to test every instantiation strategy

* Don't drop the temporary directory until the runtime is dropped in benchmarks

* Don't drop the temporary directory until the runtime is dropped in tests
2022-05-19 07:32:53 +00:00
Koute 359849b169 Make sure the commit hash is always of the same length in impl_version (#11404)
* Make sure the commit hash is always of the same length in `impl_version`

* Add a comment regarding the length of the commit hash
2022-05-16 12:25:15 +00:00
Koute cd0bfea42d Bump tempfile to 3.3.0 (#1179) 2022-04-28 14:23:04 +00:00
KouteandBastian Köcher 3890fd4cb2 Make wildcard storage change subscriptions RPC-unsafe (#11259)
* When an RPC is rejected because it's RPC-unsafe say why in the error message

* Make wildcard storage change subscriptions RPC-unsafe

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix typo

* Fix tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-04-27 05:35:52 +00:00
Kouteandparity-processbot 9550172cb9 Companion for Substrate#11062 (#1113)
* Align to changes in Substrate

* Align to the newest changes in substrate

* Update `Cargo.lock`

* Add hwbenches to `parachain-template` too

* update lockfile for {"polkadot"}

Co-authored-by: parity-processbot <>
2022-04-26 09:51:18 +00:00
Koute b049d4f1c9 Add hardware benchmark telemetry (Companion for Substrate#11062) (#5206)
* Align to changes in Substrate

* Align to the newest changes in substrate

* Rename `--disable-hardware-benchmarks` to `--no-hardware-benchmarks`

* Fix `polkadot-test-service` compilation

* Fix compilation of test parachains
2022-04-26 09:16:31 +00:00
Kouteandparity-processbot 7a92f1678d Update tracing-log to 0.1.3 (Companion for Substrate#11264) (#5393)
* Update `tracing-log` to 0.1.3

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2022-04-26 09:05:46 +00:00
Koute 94dac682b4 Turn on logger's interest cache (#11264) 2022-04-26 08:05:51 +00:00
Koute 8351ada6a3 Add new hardware and software metrics (#11062)
* Add new hardware and software metrics

* Move sysinfo tests into `mod tests`

* Correct a typo in a comment

* Remove unnecessary `nix` dependency

* Fix the version tests

* Add a `--disable-hardware-benchmarks` CLI argument

* Disable hardware benchmarks in the integration tests

* Remove unused import

* Fix benchmarks compilation

* Move code to a new `sc-sysinfo` crate

* Correct `impl_version` comment

* Move `--disable-hardware-benchmarks` to the chain-specific bin crate

* Move printing out of hardware bench results to `sc-sysinfo`

* Move hardware benchmarks to a separate messages; trigger them manually

* Rename some of the fields in the `HwBench` struct

* Revert changes to the telemetry crate; manually send hwbench messages

* Move sysinfo logs into the sysinfo crate

* Move the `TARGET_OS_*` constants into the sysinfo crate

* Minor cleanups

* Move the `HwBench` struct to the sysinfo crate

* Derive `Clone` for `HwBench`

* Fix broken telemetry connection notification stream

* Prevent the telemetry connection notifiers from leaking if they're disconnected

* Turn the telemetry notification failure log into a debug log

* Rename `--disable-hardware-benchmarks` to `--no-hardware-benchmarks`
2022-04-11 18:46:53 +09:00
Koute 1411e028aa Catch panics on the FFI boundary between the runtime and the host for wasmtime (#11189)
* Catch panics on the FFI boundary between the runtime and the host for `wasmtime`

* Use an already existing test runtime function

* Merge the tests together
2022-04-08 11:26:47 +00:00
Koute ef4e64c641 Improve error handling in approval voting block import (#5283)
* Print out original the runtime API error

* Improved error handling in approval voting block import

* Fix test

* Update comment
2022-04-08 10:26:38 +02:00
Koute 6b4deb5a25 Bump lru to 0.7.4 (#1137) 2022-04-04 13:55:00 +02:00
Kouteandparity-processbot cace3561e3 Update dependencies (companion for Substrate#11135) (#5217)
* Update `regex` to 1.5.5

* Update `libc` to 0.2.121

* Update `handlebars` to 4.2.2

* Update `lru` to 0.7.3

* Update `zeroize` to 1.5.4 and `zeroize_derive` to 1.3.2

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
2022-03-29 17:43:56 +02:00
Koute 83d4be6151 Update lru, regex and thread_local (#11135)
* Update `lru` to 0.7.3

* Update `regex` to 1.5.5

* Update `thread_local` to 1.1.4
2022-03-29 14:19:19 +00:00
KouteandKeith Yeung d932aa0cd4 Bump tokio to 1.17.0 (#10894)
* Bump `tokio` to 0.17.0

* Revert version changes to scale-info

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-03-28 23:42:01 +02:00
b424e5e741 Companion for Substrate#11107 (#5197)
* Rename to BagError

* Additional parameter for 'revert' command

* Set aux revert param to None

* Align to changes in how the WASM executor is configured in `substrate`

* update lockfile for {"substrate"}

* update lockfile for {"substrate"}

* Update substrate

* Update substrate

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: parity-processbot <>
2022-03-24 15:56:57 -04:00
Koute a4041add8a Add extra WASM heap pages when precompiling the runtime blob (#11107)
* Add extra WASM heap pages when precompiling the runtime blob

* Fix compilation

* Fix rustdoc

* Fix rustdoc for real this time

* Fix benches compilation

* Improve the builder in `sc-executor-wasmtime`'s tests
2022-03-24 14:19:17 +01:00
Koute 17b1a58eef Refactor WASM module instantiation (#10480)
* Refactor WASM module instantiation; enable WASM instance pooling

* Disable the `uffd` feature on `wasmtime`

* Restore the original behavior regarding the initial WASM memory size

* Adjust error message

* Remove unnecessary import in the benchmarks

* Preinstantiate the WASM runtime for a slight speedup

* Delete the asserts in `convert_memory_import_into_export`

* `return` -> `break`

* Revert WASM instance pooling for now

* Have `convert_memory_import_into_export` return an error instead of panic

* Update the warning when an import is missing

* Rustfmt and clippy fix

* Fix executor benchmarks' compilation without `wasmtime` being enabled

* rustfmt again

* Align to review comments

* Extend tests so that both imported and exported memories are tested

* Increase the number of heap pages for exported memories too

* Fix `decommit_works` test
2022-03-19 19:11:02 +00:00
Koute ba81ba8048 Get rid of unnecessary use of async-std in non-test code (#10891) 2022-02-24 11:30:25 +00:00
KouteandBastian Köcher 7adde98fb4 Bump tokio to 1.17.0 (#4965)
* Bump `tokio` to 1.17.0

* Bump the `tokio` version in the test service too

* Revert changes to `bridges/relays/client-substrate/Cargo.toml`

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-02-22 12:45:16 +01:00
Koute 5f74b4cf67 Bump tokio to 1.17.0 (#1014) 2022-02-22 11:01:17 +00:00
Koute b9a20ce86b Further reduce the CPU overhead of networking metrics (#10875)
* Simplify `num_connected_peers`

* Track requested peer counts

* Revert "Track requested peer counts"

This reverts commit 9f1c8704353df6afc17ed7e9f4ab8d8e29466ae4.

* Remove `substrate_sub_libp2p_peerset_num_requested` metric

* Remove two unused functions that I forgot to get rid of in previous commit
2022-02-18 12:35:43 +00:00
Koute ee6223327c Reduce CPU overhead of gossip (#10859) 2022-02-16 04:14:00 +00:00
Koute 43c3e709f9 Enable the parking_lot feature for tokio (#10855) 2022-02-15 11:53:45 +00:00
Koute e026077505 Reduce overhead of generating network event metrics (#10839) 2022-02-11 11:35:35 +00:00
Koute 9a31b2c341 Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)
* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments
2022-02-09 09:12:55 +00:00
Koute c328d1e63b Companion for Substrate#10394 (#4471)
* Align PVF executor to changes in Substrate

* Update to the newest `substrate`
2021-12-14 09:47:10 +00:00
Koute 7711f5266e Statically register host WASM functions (#10394)
* Statically register host WASM functions

* Fix `substrate-test-client` compilation

* Move `ExtendedHostFunctions` to `sp-wasm-interface`

* Fix `sp-runtime-interface` tests' compilation

* Fix `sc-executor-wasmtime` tests' compilation

* Use `runtime_interface` macro in `test-runner`

* Fix `sc-executor` tests' compilation

* Reformatting/`rustfmt`

* Add an extra comment regarding the `H` generic arg in `create_runtime`

* Even more `rustfmt`

* Depend on `wasmtime` without default features in `sp-wasm-interface`

* Bump version of `sp-wasm-interface` to 4.0.1

* Bump `sp-wasm-interface` in `Cargo.lock` too

* Bump all of the `sp-wasm-interface` requirements to 4.0.1

Maybe this will appease cargo-unleash?

* Revert "Bump all of the `sp-wasm-interface` requirements to 4.0.1"

This reverts commit 0f7ccf8e0f371542861121b145ab87af6541ac30.

* Make `cargo-unleash` happy (maybe)

* Use `cargo-unleash` to bump the crates' versions

* Align to review comments
2021-12-14 08:26:40 +00:00
eee31418ba Automatically unsubscribe storage listeners when they're dropped (RCP node memory leak fix) (#10454)
* Automatically unsubscribe storage listeners when they're dropped

* Fix tests' compilation in `sc-client-api`

* Add an extra test

* Align to review comments; cleanups

* Update client/api/src/notifications.rs

* FMT

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2021-12-10 11:31:04 +00:00
Koute e5108606eb Fix WASM executor without instance reuse; cleanups and refactoring (#10313)
* Fix WASM executor without instance reuse; cleanups and refactoring

* Align to review comments

* Move the functions for reading/writing memory to `util.rs`

* Only `#[ignore]` the test in debug builds

* More review comments and minor extra comments
2021-11-23 06:35:19 +00:00
Koute 15a0bfb0f6 Clear WASM linear memory on other OSes besides Linux too (#10291) 2021-11-18 11:16:38 +00:00
Koute 59078fcf94 Add a block production benchmark (#10104)
* Add a block production benchmark

* Simplify the block production benchmark

* Cleanups; switch execution strategy to WASM

* Switch WASM execution to `Compiled`

* Reduce the setup cost of the benchmark

Creating all of those extrinsics takes up *a lot* of time, up to the point
where the majority of the time is actually spent *outside* of the code
which we want to benchmark here. So let's only do it once.

* Add a variant of the block production benchmark with proof recording
2021-11-09 13:38:31 +00:00
Koute 3080f793e3 Update wasmtime and related dependencies (companion for Substrate#10149) (#4210)
* Update `wasmtime` and related dependencies

* Update `substrate` to 7adee13ddc48a9d715eb2f9caa37d99713fa0102
2021-11-09 13:12:57 +00:00
Koute ef3eca4198 Bump wasmtime to 0.31.0 (#10149)
* Bump `wasmtime` to 0.31.0

* Bump `itoa` to 0.4.8

* sc-executor-wasmtime: fix `SandboxContext::invoke` which I've broke

* sc-executor-wasmtime: cargo fmt
2021-11-09 12:08:02 +00:00
Koute a7e3d819f8 Refactor sp-sandbox; make sure both sandbox executors are always tested (#10173)
* sp-sandbox: convert executors into normal `mod`s instead of using `include!`

* sp-sandbox: run `cargo fmt` on `host_executor.rs`

* sp-sandbox: abstract away the executors behind traits

* sp_sandbox: always compile both executors when possible

* sc-executor: make sure all sandbox tests run on both sandbox executors

* sc-executor: fix brainfart: actually call into the sandbox through the trait

* sc-runtime-test: fix cargo fmt

* sc-runtime-test: deduplicate executor-specific sandbox test entrypoints

* sc-executor: test each sandbox executor in a separate test

* cargo fmt (Github's conflict resolving thingy broke indentation)
2021-11-08 12:52:11 +00:00
Koute 8c2216c8cd Remove unnecessary allocations when crossing WASM FFI boundary (#10191) 2021-11-05 12:52:34 +00:00
Koute 8c2ea19085 Strip out control codes from the logged messages (#10081)
* Strip out control codes from the logged messages

* Also strip away C1 control codes

* Add extra comments

* Clear the buffer after flushing; rename `write` to `flush`

* Move control code stripping into its own function

* Also strip out control codes from panic messages

* Also strip out Unicode left-to-right/right-to-left control codes
2021-11-01 08:46:32 +00:00
Koute 3d5f093ddb Speed up logging once again (#9981)
* Update `tracing`-related dependencies

* Enable `parking_lot` feature in `tracing-subscriber`

* Add an asynchronous stderr logger

* Make clippy happy

* Add an integration test for the logger

* Refactor `test_logger_filters`'s subprocess machinery into a separate function

* Use a child process instead of hooking into stderr for the test

* Add a doc comment for `MakeStderrWriter`

* Move the initialization into the `MakeStderrWriter`'s constructor

* Add an extra test case to trigger the logger's emergency flush mechanism

* Use the buffer's mutex for asynchronous flushes

* Remove vestigial `nix` dependency from one of the previous commits
2021-10-21 16:54:15 +00:00
Koute dbc97029b6 Update multiple dependencies (#9936) 2021-10-05 14:22:28 +00:00
Koute 400f3e6579 Speed up timestamp generation when logging (#9933)
* Speed up timestamp generation when logging

* Align to review comments.

* Correct a typo
2021-10-05 12:15:18 +00:00
KouteandBastian Köcher 886c42c5c8 Do not propagate host RUSTFLAGS when checking for WASM toolchain (#9926)
* Do not propagate host RUSTFLAGS when checking for WASM toolchain

* Update utils/wasm-builder/src/prerequisites.rs

* Update utils/wasm-builder/src/prerequisites.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-10-04 13:28:51 +00:00