This PR provides the infrastructure for the pov-reclaim mechanism
discussed in #209. The goal is to provide the current proof size to the
runtime so it can be used to reclaim storage weight.
## New Host Function
- A new host function is provided
[here](https://github.com/skunert/polkadot-sdk/blob/5b317fda3be205f4136f10d4490387ccd4f9765d/cumulus/primitives/pov-reclaim/src/lib.rs#L23).
It returns the size of the current proof size to the runtime. If
recording is not enabled, it returns 0.
## Implementation Overview
- Implement option to enable proof recording during import in the
client. This is currently enabled for `polkadot-parachain`,
`parachain-template` and the cumulus test node.
- Make the proof recorder ready for no-std. It was previously only
enabled for std environments, but we need to record the proof size in
`validate_block` too.
- Provide a recorder implementation that only the records the size of
incoming nodes and does not store the nodes itself.
- Fix benchmarks that were broken by async backing changes
- Provide new externalities extension that is registered by default if
proof recording is enabled.
- I think we should discuss the naming, pov-reclaim was more intuitive
to me, but we could also go with clawback like in the issue.
## Impact of proof recording during import
With proof recording: 6.3058 Kelem/s
Without proof recording: 6.3427 Kelem/s
The measured impact on the importing performance is quite low on my
machine using the block import benchmark. With proof recording I am
seeing a performance hit of 0.585%.
---------
Co-authored-by: command-bot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Bumps the known_good_semver group with 1 update:
[syn](https://github.com/dtolnay/syn).
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/syn/releases">syn's
releases</a>.</em></p>
<blockquote>
<h2>2.0.38</h2>
<ul>
<li>Fix <em>"method 'peek' has an incompatible type for
trait"</em> error when defining <code>bool</code> as a custom
keyword (<a
href="https://redirect.github.com/dtolnay/syn/issues/1518">#1518</a>,
thanks <a
href="https://github.com/Vanille-N"><code>@Vanille-N</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/syn/commit/43632bfb6c78ee1f952645a268ab1ac4af162977"><code>43632bf</code></a>
Release 2.0.38</li>
<li><a
href="https://github.com/dtolnay/syn/commit/abd2c214b44da64a5e420d72919308300eebc23d"><code>abd2c21</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/syn/issues/1518">#1518</a>
from Vanille-N/master</li>
<li><a
href="https://github.com/dtolnay/syn/commit/6701e6077e15013ef34b15e3ffdae2657e499d83"><code>6701e60</code></a>
Absolute path to <code>bool</code> in
<code>custom_punctuation.rs</code></li>
<li><a
href="https://github.com/dtolnay/syn/commit/7313d242398111423f046386aa0a75548f63d236"><code>7313d24</code></a>
Resolve single_match_else pedantic clippy lint in code generator</li>
<li><a
href="https://github.com/dtolnay/syn/commit/67ab64f3c09e17b23493c7cda498e7edb8830f21"><code>67ab64f</code></a>
Include unexpected token in the test failure message</li>
<li><a
href="https://github.com/dtolnay/syn/commit/137ae33486de3f2652487f8f64436ad1429df496"><code>137ae33</code></a>
Check no remaining token after the first literal</li>
<li><a
href="https://github.com/dtolnay/syn/commit/258e9e8a11d188c1ee1ffb2b069819239999f9ac"><code>258e9e8</code></a>
Ignore single_match_else pedantic clippy lint in test</li>
<li><a
href="https://github.com/dtolnay/syn/commit/92fd50ee8cb52968d9c66fbe6d67638c1f838e26"><code>92fd50e</code></a>
Test docs.rs documentation build in CI</li>
<li>See full diff in <a
href="https://github.com/dtolnay/syn/compare/2.0.37...2.0.38">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[//]: # (dependabot-start)
⚠️ **Dependabot is rebasing this PR** ⚠️
Rebasing might not happen immediately, so don't worry if this takes some
time.
Note: if you make any changes to this PR yourself, they will take
precedence over the rebase.
---
[//]: # (dependabot-end)
Bumps the known_good_semver group with 2 updates:
[clap](https://github.com/clap-rs/clap) and
[syn](https://github.com/dtolnay/syn).
Updates `clap` from 4.4.3 to 4.4.4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/releases">clap's
releases</a>.</em></p>
<blockquote>
<h2>v4.4.4</h2>
<h2>[4.4.4] - 2023-09-18</h2>
<h3>Internal</h3>
<ul>
<li>Update <code>terminal_size</code> to 0.3</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's
changelog</a>.</em></p>
<blockquote>
<h2>[4.4.4] - 2023-09-18</h2>
<h3>Internal</h3>
<ul>
<li>Update <code>terminal_size</code> to 0.3</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/clap-rs/clap/commit/e6e539660f36487e3521ab6835ef1381a21785a4"><code>e6e5396</code></a>
chore: Release</li>
<li><a
href="https://github.com/clap-rs/clap/commit/acbb60c11389d362e3b2d23a7b2a0a0523bd2fa8"><code>acbb60c</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/clap-rs/clap/commit/f09d521450c50f56153f7246489666c71e3b4739"><code>f09d521</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5129">#5129</a>
from cgwalters/widen-terminal-size</li>
<li><a
href="https://github.com/clap-rs/clap/commit/84f99ff979d58e3fe94716144b1800fd81723ed7"><code>84f99ff</code></a>
chore(builder): Bump terminal_size to 0.3</li>
<li><a
href="https://github.com/clap-rs/clap/commit/7f8df272d90afde89e40de086492e1c9f5749897"><code>7f8df27</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5124">#5124</a>
from devinherron/master</li>
<li><a
href="https://github.com/clap-rs/clap/commit/4dff87386a950d8dfa8e89906e9bf54d180d13de"><code>4dff873</code></a>
fix(doc): Fix typo in 03_04_subcommands.md</li>
<li>See full diff in <a
href="https://github.com/clap-rs/clap/compare/v4.4.3...v4.4.4">compare
view</a></li>
</ul>
</details>
<br />
Updates `syn` from 2.0.36 to 2.0.37
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/syn/releases">syn's
releases</a>.</em></p>
<blockquote>
<h2>2.0.37</h2>
<ul>
<li>Work around incorrect future compatibility warning in rustc
1.74.0-nightly</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/syn/commit/96810880f3acbb63415cf4684ab42c82edc31e2a"><code>9681088</code></a>
Release 2.0.37</li>
<li><a
href="https://github.com/dtolnay/syn/commit/fbe3bc2ddcee4192f95697a953330d031030f5a1"><code>fbe3bc2</code></a>
Work around unknown_lints warning on rustc older than 1.64</li>
<li><a
href="https://github.com/dtolnay/syn/commit/75cf912e061ef5a1d97c003f4988f43d7639f5a8"><code>75cf912</code></a>
Ignore more repr_transparent_external_private_fields</li>
<li><a
href="https://github.com/dtolnay/syn/commit/299c782439823b868de4aea3682b41d03351d978"><code>299c782</code></a>
Ignore false repr_transparent_external_private_fields FCW in generated
code</li>
<li>See full diff in <a
href="https://github.com/dtolnay/syn/compare/2.0.36...2.0.37">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Support cfg attribute in host functions definitions
* Added test to check feature gated methods are not included
* Versioned conditional compiled host function are forbidden
* Improve runtime-interface macro docs
* Fix doc
* Apply review suggestion
Co-authored-by: Koute <koute@users.noreply.github.com>
* Better error message
* Rust fmt
* More refinements to the docs
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
* globally upgrade quote to latest 1.0.x (1.0.26)
* globally upgrade syn to final 1.0.x version (1.0.109)
* globally upgrade proc-macro2 to 1.0.56
* upgrade to syn v2.0.13 and fix everything except NestedMeta
* fix parse nested metadata code in decl_runtime_apis.rs
* Port more stuff to syn 2.0
* Make the rest compile
* Ignore error
* update to syn 2.0.14
---------
Co-authored-by: Bastian Köcher <info@kchr.de>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Bump `wasmtime` to 4.0.0 (and a few other deps)
* Use `Error::msg` instead of `anyhow!`
* Bump `wasmtime` to 5.0.0
* Update `Cargo.lock`
* Add `wasmtime` feature to `sp-wasm-interface` dependency
* Catch panics on the FFI boundary between the runtime and the host for `wasmtime`
* Use an already existing test runtime function
* Merge the tests together
* Add a new host function for reporting fatal errors
* Fix one of the wasmtime executor tests
* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only
* Print out errors through `Display` instead of `Debug`
* Switch one more trait to require `Error` for its error instead of only `Debug`
* Align to review comments
* runtime-interface: Implement `register_only` functions
The runtime interface supports versioning of functions. Currently, if you add a new function it will
be used by the runtime automatically. This results in requiring all nodes of a network to upgrade
before the runtime is upgraded, otherwise they will fail to instantiate the new runtime because of
missing host functions. This pr introduces `register_only` functions. This can be used when a new
runtime interface function should be introduced, but the actual usage can be deferred. This means
that nodes will have the host function for this, but the runtime will still use the old version of
the function when being compiled for wasm. However, when a runtime is enacted that uses the new host
function, the "old nodes" will already have the host function and will continue to work.
* Update primitives/runtime-interface/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update primitives/runtime-interface/proc-macro/src/utils.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Statically register host WASM functions
* Fix `substrate-test-client` compilation
* Move `ExtendedHostFunctions` to `sp-wasm-interface`
* Fix `sp-runtime-interface` tests' compilation
* Fix `sc-executor-wasmtime` tests' compilation
* Use `runtime_interface` macro in `test-runner`
* Fix `sc-executor` tests' compilation
* Reformatting/`rustfmt`
* Add an extra comment regarding the `H` generic arg in `create_runtime`
* Even more `rustfmt`
* Depend on `wasmtime` without default features in `sp-wasm-interface`
* Bump version of `sp-wasm-interface` to 4.0.1
* Bump `sp-wasm-interface` in `Cargo.lock` too
* Bump all of the `sp-wasm-interface` requirements to 4.0.1
Maybe this will appease cargo-unleash?
* Revert "Bump all of the `sp-wasm-interface` requirements to 4.0.1"
This reverts commit 0f7ccf8e0f371542861121b145ab87af6541ac30.
* Make `cargo-unleash` happy (maybe)
* Use `cargo-unleash` to bump the crates' versions
* Align to review comments
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* mark template and utils as non-publish
* switch to development version for testing
* activate unleash check
* maybe if I disable all rules...
* Fix isolated compilation of `max-encoded-len-derive` with `syn`
error[E0369]: binary operation `==` cannot be applied to type `syn::Path`
--> src/lib.rs:88:29
|
88 | .filter(|attr| attr.path == parse_quote!(max_encoded_len_crate))
| --------- ^^ ----------------------------------- _
| |
| syn::Path
error: aborting due to previous error
For more information about this error, try `rustc --explain E0369`.
Error: could not compile `max-encoded-len-derive`
* WIP: bump changes crates since v3 tag to next breaking
cargo unleash version bump-breaking --changed-since v3.0.0
cargo unleash version set-pre dev --changed-since v3.0.0
FIXME: Don't modify crates that are not yet released, e.g.
`max-encoded-len-derive`
* Update lockfile
* WIP: Bump sp-transaction-pool as well
* WIP: Bump sp-offchain as well
* WIP: Bump frame-system-rpc-runtime-api as well
* WIP: Bump sp-authority-discovery as well
* Manually deactivate dev-deps before `cargo unleash check`
Otherwise we run into `Cycle detected` error.
* Bump sp-consensus-slots
* Add missing Cargo.lock change
* Bump sp-consensus-vrf as well
* Bump sp-keyring as well
* Bump sp-consensus-pow as well
* Try to speed up the `unleash-check` job
Previously, the job took 106 minutes - let's see if explicitly
specifying a `CARGO_TARGET_DIR` will help
* fixup: Ensure the temp target dir exists for unleash check
* Bump pallet-transaction-payment-rpc-runtime-api as well
Needed for Polkadot
* Bump pallet-transaction-payment-rpc as well
Needed for Polkadot
* Try updating crates after patching in the Polkadot CI job
* Use another approach to update patched Substrate crates
* Try to update all sp-core versions in Polkadot CI job
* Simplify sp-core version checking
* Apply another shellcheck lint
* Just do the simplest thing I guess
* Welp don't do --offline then
* Clean up `unleash-check` job triggers
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
* Fix a note in unleash-check cache step
* Add a note about temporary optimization in cargo-unleash
* Pin a newer version of cargo-unleash
Co-authored-by: Igor Matuszewski <xanewok@gmail.com>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
* Fix incorrect use of syn::exports
Instead of using `syn::exports` we should import the trait from the
quote crate directly.
* Use own macro for test cases to fix compilation with latest syn
* Fix test
Sebastian Kunert