Commit Graph
44 Commits
Author SHA1 Message Date
Koute eee79a1fa4 Extend the lower bounds of some of the benchmarks to also include 0 (#12386)
* Extend the lower bounds of some of the benchmarks to also include `0`

* Fix verify snippet for `pallet_bounties/spend_funds`
2022-10-07 14:16:41 +00:00
Koute ecc4d5f258 Add @koute to docs/CODEOWNERS and update stale paths (#12408) 2022-10-04 10:01:50 +02:00
Kouteandcommand-bot 8d81065a54 Support running the pallet benchmarks analysis without running the benchmarks (#12361)
* Support running the pallet benchmarks analysis without running the benchmarks

* Rename `override-results` to `json-input` and update the help comment

* ".git/.scripts/fmt.sh" 1

Co-authored-by: command-bot <>
2022-09-28 19:38:12 +00:00
Koute d5c57ffcc1 Bump wasmtime to 1.0.0 (#12317) 2022-09-21 10:21:01 +00:00
Kouteparity-processbotShawn Tabrizi
b1bbdf1192 Improve base weights consistency and make sure they're never zero (#11806)
* Improve base weights consistency and make sure they're never zero

* Switch back to `linregress` crate; add back estimation errors

* Move the test into `mod tests`

* Use all of the samples instead of only the first one

* Use full precision extrinsic base weights and slopes

* Add an extra comment

* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts

Co-authored-by: parity-processbot <>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2022-09-14 16:01:13 +00:00
Koute b922a25ccd Force rustix to use libc (#12123) 2022-08-29 15:30:37 +00:00
Koute 9c56e79c43 Restore wasmtime's default stack size limit to 1MB (#11993)
* Restore `wasmtime`'s default stack size limit to 1MB

* Add extra comments

* Enforce different maximum call depth in release mode

* Split the call depth limit in two
2022-08-09 11:04:28 +00:00
KouteandBastian Köcher c4b607d4c9 Prevent double allocation of the payload when calling sp_io::storage::get (#11523)
* Expose allocation stats in `FreeingBumpHeapAllocator`

* Return allocation stats when calling into the runtime

* Bump `parity-scale-codec` to 3.1.3 (fork)

* Prevent double allocation of the payload when calling `sp_io::storage::get`

* Fix tests

* Remove unnecessary `mut`

* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Bump `parity-scale-codec` to 3.1.3

* Fix some of the UI tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-07-29 07:46:15 +00:00
Koute 8f6ba2e75e Improve wasmtime error reporting (#11856)
* Improve `wasmtime` error reporting

* cargo fmt
2022-07-18 11:51:38 +00:00
Koute 3d89fa3605 Prevent unsoundness in environments with broken madvise(MADV_DONTNEED) (#11722)
* Prevend unsoundness in environments with broken `madvise(MADV_DONTNEED)`

* Add the `std` feature to `rustix` dependency

Apparently not having this breaks compilation on non-nightly toolchains.

* Autodetect the page size when checking whether `madvise` works

* Only make sure that the madvice check doesn't return `Err`
2022-06-28 06:54:56 +00:00
Koute 025e1d76f1 Bump wasmtime to 0.38.0 and zstd to 0.11.2 (#11720)
* Bump `wasmtime` to 0.37.0 and `zstd` to 0.11.2

* Bump `wasmtime` to 0.38.0
2022-06-24 11:25:21 +00:00
Koute 9cee3c699d Pump the gossip engine while waiting for the BEEFY runtime pallet (memory leak fix) (#11694)
* Pump the gossip engine while waiting for the BEEFY runtime pallet

This fixes a memory leak when the BEEFY gadget is turned on, but
the runtime doesn't actually use BEEFY.

* Implement `FusedFuture` for `GossipEngine`

* Fuse futures outside of loops
2022-06-20 15:52:43 +00:00
Koute 849a6c35fd Remove without_storage_info for the authorship pallet (#11610)
* Remove `without_storage_info` for the authorship pallet

* Tweak impl bounds style

* Use `defensive_proof` instead of `expect`
2022-06-16 11:48:05 +00:00
Koute a8c1d430ef Fix one-by-off in BoundedSlice::try_from (#11600) 2022-06-06 16:17:41 +00:00
Koute 4f393171d7 Adjust maximum memory pages hard limit for the pooling instantiation strategy (#11482)
* Run `sc-executor-wasmtime` unit tests for all instantiation strategies

* Adjust maximum memory pages hard limit for the pooling instantiation strategy
2022-05-24 05:37:04 +00:00
Koute dd854c16e2 Switch to pooling copy-on-write instantiation strategy for WASM (#11232)
* Switch to pooling copy-on-write instantiation strategy for WASM

* Fix benchmark compilation

* Fix `cargo fmt`

* Fix compilation of another benchmark I've missed

* Cleanups according to review comments

* Move `max_memory_size` to `Semantics`

* Set `memory_guaranteed_dense_image_size` to `max_memory_size`

* Rename `wasm_instantiation_strategy` to `wasmtime_instantiation_strategy`

* Update the doc-comments regarding the instantiation strategy

* Extend the integration tests to test every instantiation strategy

* Don't drop the temporary directory until the runtime is dropped in benchmarks

* Don't drop the temporary directory until the runtime is dropped in tests
2022-05-19 07:32:53 +00:00
Koute 359849b169 Make sure the commit hash is always of the same length in impl_version (#11404)
* Make sure the commit hash is always of the same length in `impl_version`

* Add a comment regarding the length of the commit hash
2022-05-16 12:25:15 +00:00
KouteandBastian Köcher 3890fd4cb2 Make wildcard storage change subscriptions RPC-unsafe (#11259)
* When an RPC is rejected because it's RPC-unsafe say why in the error message

* Make wildcard storage change subscriptions RPC-unsafe

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix typo

* Fix tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-04-27 05:35:52 +00:00
Koute 94dac682b4 Turn on logger's interest cache (#11264) 2022-04-26 08:05:51 +00:00
Koute 8351ada6a3 Add new hardware and software metrics (#11062)
* Add new hardware and software metrics

* Move sysinfo tests into `mod tests`

* Correct a typo in a comment

* Remove unnecessary `nix` dependency

* Fix the version tests

* Add a `--disable-hardware-benchmarks` CLI argument

* Disable hardware benchmarks in the integration tests

* Remove unused import

* Fix benchmarks compilation

* Move code to a new `sc-sysinfo` crate

* Correct `impl_version` comment

* Move `--disable-hardware-benchmarks` to the chain-specific bin crate

* Move printing out of hardware bench results to `sc-sysinfo`

* Move hardware benchmarks to a separate messages; trigger them manually

* Rename some of the fields in the `HwBench` struct

* Revert changes to the telemetry crate; manually send hwbench messages

* Move sysinfo logs into the sysinfo crate

* Move the `TARGET_OS_*` constants into the sysinfo crate

* Minor cleanups

* Move the `HwBench` struct to the sysinfo crate

* Derive `Clone` for `HwBench`

* Fix broken telemetry connection notification stream

* Prevent the telemetry connection notifiers from leaking if they're disconnected

* Turn the telemetry notification failure log into a debug log

* Rename `--disable-hardware-benchmarks` to `--no-hardware-benchmarks`
2022-04-11 18:46:53 +09:00
Koute 1411e028aa Catch panics on the FFI boundary between the runtime and the host for wasmtime (#11189)
* Catch panics on the FFI boundary between the runtime and the host for `wasmtime`

* Use an already existing test runtime function

* Merge the tests together
2022-04-08 11:26:47 +00:00
Koute 83d4be6151 Update lru, regex and thread_local (#11135)
* Update `lru` to 0.7.3

* Update `regex` to 1.5.5

* Update `thread_local` to 1.1.4
2022-03-29 14:19:19 +00:00
KouteandKeith Yeung d932aa0cd4 Bump tokio to 1.17.0 (#10894)
* Bump `tokio` to 0.17.0

* Revert version changes to scale-info

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
2022-03-28 23:42:01 +02:00
Koute a4041add8a Add extra WASM heap pages when precompiling the runtime blob (#11107)
* Add extra WASM heap pages when precompiling the runtime blob

* Fix compilation

* Fix rustdoc

* Fix rustdoc for real this time

* Fix benches compilation

* Improve the builder in `sc-executor-wasmtime`'s tests
2022-03-24 14:19:17 +01:00
Koute 17b1a58eef Refactor WASM module instantiation (#10480)
* Refactor WASM module instantiation; enable WASM instance pooling

* Disable the `uffd` feature on `wasmtime`

* Restore the original behavior regarding the initial WASM memory size

* Adjust error message

* Remove unnecessary import in the benchmarks

* Preinstantiate the WASM runtime for a slight speedup

* Delete the asserts in `convert_memory_import_into_export`

* `return` -> `break`

* Revert WASM instance pooling for now

* Have `convert_memory_import_into_export` return an error instead of panic

* Update the warning when an import is missing

* Rustfmt and clippy fix

* Fix executor benchmarks' compilation without `wasmtime` being enabled

* rustfmt again

* Align to review comments

* Extend tests so that both imported and exported memories are tested

* Increase the number of heap pages for exported memories too

* Fix `decommit_works` test
2022-03-19 19:11:02 +00:00
Koute ba81ba8048 Get rid of unnecessary use of async-std in non-test code (#10891) 2022-02-24 11:30:25 +00:00
Koute b9a20ce86b Further reduce the CPU overhead of networking metrics (#10875)
* Simplify `num_connected_peers`

* Track requested peer counts

* Revert "Track requested peer counts"

This reverts commit 9f1c8704353df6afc17ed7e9f4ab8d8e29466ae4.

* Remove `substrate_sub_libp2p_peerset_num_requested` metric

* Remove two unused functions that I forgot to get rid of in previous commit
2022-02-18 12:35:43 +00:00
Koute ee6223327c Reduce CPU overhead of gossip (#10859) 2022-02-16 04:14:00 +00:00
Koute 43c3e709f9 Enable the parking_lot feature for tokio (#10855) 2022-02-15 11:53:45 +00:00
Koute e026077505 Reduce overhead of generating network event metrics (#10839) 2022-02-11 11:35:35 +00:00
Koute 9a31b2c341 Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)
* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments
2022-02-09 09:12:55 +00:00
Koute 7711f5266e Statically register host WASM functions (#10394)
* Statically register host WASM functions

* Fix `substrate-test-client` compilation

* Move `ExtendedHostFunctions` to `sp-wasm-interface`

* Fix `sp-runtime-interface` tests' compilation

* Fix `sc-executor-wasmtime` tests' compilation

* Use `runtime_interface` macro in `test-runner`

* Fix `sc-executor` tests' compilation

* Reformatting/`rustfmt`

* Add an extra comment regarding the `H` generic arg in `create_runtime`

* Even more `rustfmt`

* Depend on `wasmtime` without default features in `sp-wasm-interface`

* Bump version of `sp-wasm-interface` to 4.0.1

* Bump `sp-wasm-interface` in `Cargo.lock` too

* Bump all of the `sp-wasm-interface` requirements to 4.0.1

Maybe this will appease cargo-unleash?

* Revert "Bump all of the `sp-wasm-interface` requirements to 4.0.1"

This reverts commit 0f7ccf8e0f371542861121b145ab87af6541ac30.

* Make `cargo-unleash` happy (maybe)

* Use `cargo-unleash` to bump the crates' versions

* Align to review comments
2021-12-14 08:26:40 +00:00
eee31418ba Automatically unsubscribe storage listeners when they're dropped (RCP node memory leak fix) (#10454)
* Automatically unsubscribe storage listeners when they're dropped

* Fix tests' compilation in `sc-client-api`

* Add an extra test

* Align to review comments; cleanups

* Update client/api/src/notifications.rs

* FMT

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2021-12-10 11:31:04 +00:00
Koute e5108606eb Fix WASM executor without instance reuse; cleanups and refactoring (#10313)
* Fix WASM executor without instance reuse; cleanups and refactoring

* Align to review comments

* Move the functions for reading/writing memory to `util.rs`

* Only `#[ignore]` the test in debug builds

* More review comments and minor extra comments
2021-11-23 06:35:19 +00:00
Koute 15a0bfb0f6 Clear WASM linear memory on other OSes besides Linux too (#10291) 2021-11-18 11:16:38 +00:00
Koute 59078fcf94 Add a block production benchmark (#10104)
* Add a block production benchmark

* Simplify the block production benchmark

* Cleanups; switch execution strategy to WASM

* Switch WASM execution to `Compiled`

* Reduce the setup cost of the benchmark

Creating all of those extrinsics takes up *a lot* of time, up to the point
where the majority of the time is actually spent *outside* of the code
which we want to benchmark here. So let's only do it once.

* Add a variant of the block production benchmark with proof recording
2021-11-09 13:38:31 +00:00
Koute ef3eca4198 Bump wasmtime to 0.31.0 (#10149)
* Bump `wasmtime` to 0.31.0

* Bump `itoa` to 0.4.8

* sc-executor-wasmtime: fix `SandboxContext::invoke` which I've broke

* sc-executor-wasmtime: cargo fmt
2021-11-09 12:08:02 +00:00
Koute a7e3d819f8 Refactor sp-sandbox; make sure both sandbox executors are always tested (#10173)
* sp-sandbox: convert executors into normal `mod`s instead of using `include!`

* sp-sandbox: run `cargo fmt` on `host_executor.rs`

* sp-sandbox: abstract away the executors behind traits

* sp_sandbox: always compile both executors when possible

* sc-executor: make sure all sandbox tests run on both sandbox executors

* sc-executor: fix brainfart: actually call into the sandbox through the trait

* sc-runtime-test: fix cargo fmt

* sc-runtime-test: deduplicate executor-specific sandbox test entrypoints

* sc-executor: test each sandbox executor in a separate test

* cargo fmt (Github's conflict resolving thingy broke indentation)
2021-11-08 12:52:11 +00:00
Koute 8c2216c8cd Remove unnecessary allocations when crossing WASM FFI boundary (#10191) 2021-11-05 12:52:34 +00:00
Koute 8c2ea19085 Strip out control codes from the logged messages (#10081)
* Strip out control codes from the logged messages

* Also strip away C1 control codes

* Add extra comments

* Clear the buffer after flushing; rename `write` to `flush`

* Move control code stripping into its own function

* Also strip out control codes from panic messages

* Also strip out Unicode left-to-right/right-to-left control codes
2021-11-01 08:46:32 +00:00
Koute 3d5f093ddb Speed up logging once again (#9981)
* Update `tracing`-related dependencies

* Enable `parking_lot` feature in `tracing-subscriber`

* Add an asynchronous stderr logger

* Make clippy happy

* Add an integration test for the logger

* Refactor `test_logger_filters`'s subprocess machinery into a separate function

* Use a child process instead of hooking into stderr for the test

* Add a doc comment for `MakeStderrWriter`

* Move the initialization into the `MakeStderrWriter`'s constructor

* Add an extra test case to trigger the logger's emergency flush mechanism

* Use the buffer's mutex for asynchronous flushes

* Remove vestigial `nix` dependency from one of the previous commits
2021-10-21 16:54:15 +00:00
Koute dbc97029b6 Update multiple dependencies (#9936) 2021-10-05 14:22:28 +00:00
Koute 400f3e6579 Speed up timestamp generation when logging (#9933)
* Speed up timestamp generation when logging

* Align to review comments.

* Correct a typo
2021-10-05 12:15:18 +00:00
KouteandBastian Köcher 886c42c5c8 Do not propagate host RUSTFLAGS when checking for WASM toolchain (#9926)
* Do not propagate host RUSTFLAGS when checking for WASM toolchain

* Update utils/wasm-builder/src/prerequisites.rs

* Update utils/wasm-builder/src/prerequisites.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-10-04 13:28:51 +00:00