**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
This PR adds a new PolkaVM-based executor to Substrate.
- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.
No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
It seems the old strategy have been depracted more than one year.
So maybe it's time to clean up old strategy for wasm executor.
---
polkadot address: 15ouFh2SHpGbHtDPsJ6cXQfes9Cx1gEFnJJsJVqPGzBSTudr
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Koute <koute@users.noreply.github.com>
* WASM executor: add `OutputExceedsBounds` variant to `Error`
Previously this was a `WasmError`, which is intended for runtime construction errors. However this
led to confusion as output-exceeds-bounds occurs due to execution of `validate_block`.
* Fix warning
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* Upgrade wasm crate dependencies
* New wasmi version changed error output a bit
* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts
* ".git/.scripts/bench-bot.sh" pallet dev pallet_contracts
Co-authored-by: command-bot <>
* Expose allocation stats in `FreeingBumpHeapAllocator`
* Return allocation stats when calling into the runtime
* Bump `parity-scale-codec` to 3.1.3 (fork)
* Prevent double allocation of the payload when calling `sp_io::storage::get`
* Fix tests
* Remove unnecessary `mut`
* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`
* Update client/allocator/src/freeing_bump.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Bump `parity-scale-codec` to 3.1.3
* Fix some of the UI tests
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Refactor WASM module instantiation; enable WASM instance pooling
* Disable the `uffd` feature on `wasmtime`
* Restore the original behavior regarding the initial WASM memory size
* Adjust error message
* Remove unnecessary import in the benchmarks
* Preinstantiate the WASM runtime for a slight speedup
* Delete the asserts in `convert_memory_import_into_export`
* `return` -> `break`
* Revert WASM instance pooling for now
* Have `convert_memory_import_into_export` return an error instead of panic
* Update the warning when an import is missing
* Rustfmt and clippy fix
* Fix executor benchmarks' compilation without `wasmtime` being enabled
* rustfmt again
* Align to review comments
* Extend tests so that both imported and exported memories are tested
* Increase the number of heap pages for exported memories too
* Fix `decommit_works` test
* Add a new host function for reporting fatal errors
* Fix one of the wasmtime executor tests
* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only
* Print out errors through `Display` instead of `Debug`
* Switch one more trait to require `Error` for its error instead of only `Debug`
* Align to review comments
* Start
* Move to ctx
* Make it compile for now
* More work
* Get rid off state-holder
* Use less Refcells
* 🤦
* Don't use RefCell
* Use names for parameters
* Fixes after merge
* Fixes after merge
* Review feedback
* FMT
* Improve sandbox internal api
This improves the internal sandbox api for the executor implementations.
The main point is to hide the tls in the internal api and not having it
exposed to the outside.
This is especially needed for wasmtime 0.29.0
* Fmt
* Make it nicer
* Add comments and refactor Sandbox module
* Adds some comments
* Add wasmtime instance to the sandbox and delegate calls
* Adds module imports stub
* WIP state holder via *mut
* My take at the problem
* Brings back invoke and instantiate implementation details
* Removes redundant bound
* Code cleanup
* Fixes invoke closure
* Refactors FunctionExecutor to eliminate lifetime
* Wraps `FunctionExecutor::sandbox_store` in `RefCell`
* Renames `FunctionExecutor::heap` to `allocator`
* Wraps `FunctionExecutor::allocator` in `RefCell`
* Refactors FunctionExecutor to `Rc<Inner>` pattern
* Implements scoped TLS for FunctionExecutor
* Fixes wasmi instancing
* Fixes sandbox asserts
* Makes sandbox compile after wasmtime API change
* Uses Vurich/wasmtime for the Lightbeam backend
* Uses wasmtime instead of wasmi for sandbox API results
* Refactors sandbox to use one of the execution backends at a time
* Fixes wasmtime module instantiation
* TEMP vurich branch stuff
* Adds wasmer impl stub
* Adds get global
* Fixes warnings
* Adds wasmer invoke impl
* Implements host function interface for wasmer
* Fixes wasmer instantiation result
* Adds workaround to remove debug_assert
* Fixes import object generation for wasmer
* Attempt to propagate wasmer::Store through sandbox::Store
* Wraps `sandbox::Store::memories` in `RefCell`
* Moves `sandbox::instantiate` to `sandbox::Store`
* Eliminate `RefCell<memories>`
* Implements `HostState::memory_get/set`, removes accidental `borrow_mut`
* Fixes sandbox memory handling for wasmi
* Fix memory allocation
* Resets Cargo.lock to match master
* Fixes compilation
* Refactors sandbox to use TLS for dispatch_thunk propagation to wasmer
* Pass dispatch thunk to the sandbox as a TLS
* Initialize dispatch thunk holder in `SandboxInstance`
* Comment out Wasmtime/Lightbeam sandbox backend
* Revert wasmtime back to mainstream
* Adds SandboxExecutionMethod enum for cli param
* Cleanup sandbox code
* Allow wasmi to access wasmer memory regions
* More cleanup
* Remove debug logging, replace asserts with runtime errors
* Revert "Adds SandboxExecutionMethod enum for cli param"
This reverts commit dcb2b1d3b54145ab51ad2e3fef0d980ba215b596.
* Fixes warnings
* Fixes indentation and line width
* Fix return types condition
* Puts everything related under the `wasmer-sandbox` feature flag
* Fixes warnings
* Address grumbles
* Split instantiate per backend
* More splits
* Refacmemory allocation
* Nitpicks
* Attempt to wrap wasmer memory in protoco enforcing type
* Revert renaming
* WIP wasm buffer proxy API
* Reimplement util::wasmer::MemoryRef to use buffers instead of memory slices
* Adds WasmiMemoryWrapper and MemoryTransfer trait
* Refactor naming
* Perform all memory transfers using MemoryTransfer
* Adds allocating `read`
* Adds comments
* Removes unused imports
* Removes now unused function
* Pulls Cargo.lock from origin/master
* Fix rustdoc
* Removes unused `TransferError`
* Update Cargo.lock
* Removes unused import
* cargo fmt
* Fix feature dependency graph
* Feature should flow from the top level crate
* We should not assume a specific workspace structure
* sc-executor-wasmi does not use the feature
* sc-executor-wasmtime should not know about the feature
* Fix doc typo
* Enable wasmer-sandbox by default (for now)
It will be removed before merge. It is so that the benchbot
uses the wasmer sandbox.
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* Revert "cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs"
This reverts commit d713590ba45387c4204b2ad97c8bd6f6ebabda4e.
* cargo fmt
* Add ci-check to prevent wasmer sandbox build breaking
* Run tests with wasmer-sandbox enabled
* Revert "Run tests with wasmer-sandbox enabled"
This reverts commit cff63156a162f9ffdab23e7cb94a30f44e320f8a.
Co-authored-by: Sergei Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* NaN canonicalization
* Introduce a simple stack depth metering
* Be explicit about the wasm features we enable
* Pull the latest latast fix for the pwasm-utils crate
* Disable `wasm_threads` as well.
* Factor out deterministic stack params
* Add more docs
* Remove redundant dep
* Refine comments
* Typo
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Andronik Ordian <write@reusable.software>
* Move alloc primitive (not used in /pallets)
* Move to alternative location as not shared
* moved crates to different dir
* ren sp_chain_spec to sc_chain_spec_primatives
* merged sc-chain-spec and moved allocation up one.
* no no_std
* nudge
* Bump CI
* Decommit instance memory after a runtime call on Linux
* Update documentation for the test
* Remove unfinished comment
* Use saturating_sub.
Also update the doc comment.
* Precise RSS tracking in the test
Instead of tracking RSS for the whole process we just look at the particular mapping that is associated with the linear memory of the runtime instance
* Remove unused import
* Fix unused imports
* Fix the unused imports error for good
* Rollback an accidental change to benches
* Fix the test
* Remove now unneeded code
* emit a custom section from impl_runtime_apis!
This change emits a custom section from the impl_runtime_apis! proc macro.
Each implemented API will result to emitting a link section `runtime_apis`.
During linking all sections with this name will be concatenated and
placed into the final wasm binary under the same name.
* Introduce `runtime_version` proc macro
This macro takes an existing `RuntimeVersion` const declaration, parses
it and emits the version information in form of a linking section.
Ultimately such a linking section will result into a custom wasm
section.
* Parse custom wasm section for runtime version
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Fix sc-executor integration tests
* Nits
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Refactor apis section deserialization
* Fix version decoding
* Reuse uncompressed value for CallInWasm
* Log on decompression error
* Simplify if
* Reexport proc-macro from sp_version
* Merge ReadRuntimeVersionExt
* Export `read_embedded_version`
* Fix test
* Simplify searching for custom section
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Establish the runtime_blob module
Seed it with the existing contents of the `util` module.
* Port wasmtime mutable globals instrumentation into runtime blob APIs
* Opt-out from fast instance reuse
* Minor clean up
* Spaces
* Docs clean up
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Factor out the expects
* Fix the suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* handle executor should_panic test better
* Revert "reduce should panic, due to extended error messages"
This reverts commit c0805940184a62cd9302603ad911c3591e70a60c.
* remove excessive constraints
* remove duplicate documentation messages for error variants
* reduce T: constraints to the abs minimum
* whoops
* fewer bounds again
Co-authored-by: Bernhard Schuster <bernhard@parity.io>
* chore/error: remove from str conversion and add deprecation notifications
* fixup changes
* fix test looking for gone ::Msg variant
* another test fix
* one is duplicate, the other is not, so duplicates reported are n-1
* darn spaces
Co-authored-by: Andronik Ordian <write@reusable.software>
* remove pointless doc comments of error variants without any value
* low hanging fruits (for a tall person)
* moar error type variants
* avoid the storage modules for now
They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.
* chore remove pointless error generic
* fix test for mocks, add a bunch of non_exhaustive
* max line width
* test fixes due to error changes
* fin
* error outputs... again
* undo stderr adjustments
* Update client/consensus/slots/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove closure clutter
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* more error types
* introduce ApiError
* extract Mock error
* ApiError refactor
* even more error types
* the last for now
* chore unused deps
* another extraction
* reduce should panic, due to extended error messages
* error test happiness
* shift error lines by one
* doc tests
* white space
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Into -> From
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove pointless codec
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* avoid pointless self import
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@parity.io>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Dcompoze