Integrate Wasmer into Substrate sandbox environment (#5920)

* Add comments and refactor Sandbox module

* Adds some comments

* Add wasmtime instance to the sandbox and delegate calls

* Adds module imports stub

* WIP state holder via *mut

* My take at the problem

* Brings back invoke and instantiate implementation details

* Removes redundant bound

* Code cleanup

* Fixes invoke closure

* Refactors FunctionExecutor to eliminate lifetime

* Wraps `FunctionExecutor::sandbox_store` in `RefCell`

* Renames `FunctionExecutor::heap` to `allocator`

* Wraps `FunctionExecutor::allocator` in `RefCell`

* Refactors FunctionExecutor to `Rc<Inner>` pattern

* Implements scoped TLS for FunctionExecutor

* Fixes wasmi instancing

* Fixes sandbox asserts

* Makes sandbox compile after wasmtime API change

* Uses Vurich/wasmtime for the Lightbeam backend

* Uses wasmtime instead of wasmi for sandbox API results

* Refactors sandbox to use one  of the execution backends at a time

* Fixes wasmtime module instantiation

* TEMP vurich branch stuff

* Adds wasmer impl stub

* Adds get global

* Fixes warnings

* Adds wasmer invoke impl

* Implements host function interface for wasmer

* Fixes wasmer instantiation result

* Adds workaround to remove debug_assert

* Fixes import object generation for wasmer

* Attempt to propagate wasmer::Store through sandbox::Store

* Wraps `sandbox::Store::memories` in `RefCell`

* Moves `sandbox::instantiate` to `sandbox::Store`

* Eliminate `RefCell<memories>`

* Implements `HostState::memory_get/set`, removes accidental `borrow_mut`

* Fixes sandbox memory handling for wasmi

* Fix memory allocation

* Resets Cargo.lock to match master

* Fixes compilation

* Refactors sandbox to use TLS for dispatch_thunk propagation to wasmer

* Pass dispatch thunk to the sandbox as a TLS

* Initialize dispatch thunk holder in `SandboxInstance`

* Comment out Wasmtime/Lightbeam sandbox backend

* Revert wasmtime back to mainstream

* Adds SandboxExecutionMethod enum for cli param

* Cleanup sandbox code

* Allow wasmi to access wasmer memory regions

* More cleanup

* Remove debug logging, replace asserts with runtime errors

* Revert "Adds SandboxExecutionMethod enum for cli param"

This reverts commit dcb2b1d3b54145ab51ad2e3fef0d980ba215b596.

* Fixes warnings

* Fixes indentation and line width

* Fix return types condition

* Puts everything related under the `wasmer-sandbox` feature flag

* Fixes warnings

* Address grumbles

* Split instantiate per backend

* More splits

* Refacmemory allocation

* Nitpicks

* Attempt to wrap wasmer memory in protoco enforcing type

* Revert renaming

* WIP wasm buffer proxy API

* Reimplement util::wasmer::MemoryRef to use buffers instead of memory slices

* Adds WasmiMemoryWrapper and MemoryTransfer trait

* Refactor naming

* Perform all memory transfers using MemoryTransfer

* Adds allocating `read`

* Adds comments

* Removes unused imports

* Removes now unused function

* Pulls Cargo.lock from origin/master

* Fix rustdoc

* Removes unused `TransferError`

* Update Cargo.lock

* Removes unused import

* cargo fmt

* Fix feature dependency graph

* Feature should flow from the top level crate
	* We should not assume a specific workspace structure
* sc-executor-wasmi does not use the feature
* sc-executor-wasmtime should not know about the feature

* Fix doc typo

* Enable wasmer-sandbox by default (for now)

It will be removed before merge. It is so that the benchbot
uses the wasmer sandbox.

* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* Revert "cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs"

This reverts commit d713590ba45387c4204b2ad97c8bd6f6ebabda4e.

* cargo fmt

* Add ci-check to prevent wasmer sandbox build breaking

* Run tests with wasmer-sandbox enabled

* Revert "Run tests with wasmer-sandbox enabled"

This reverts commit cff63156a162f9ffdab23e7cb94a30f44e320f8a.

Co-authored-by: Sergei Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Parity Benchmarking Bot <admin@parity.io>

substrate/client/executor/wasmtime/src/instance_wrapper.rs

@@ -19,11 +19,15 @@
 //! Defines data and logic needed for interaction with an WebAssembly instance of a substrate
 //! runtime module.
 
-use crate::{imports::Imports, util};
+use crate::{
+	imports::Imports,
+	util::{from_wasmtime_val, into_wasmtime_val},
+};
 
 use sc_executor_common::{
 	error::{Error, Result},
 	runtime_blob,
+	util::checked_range,
 	wasm_runtime::InvokeMethod,
 };
 use sp_wasm_interface::{Pointer, Value, WordSize};
@@ -96,12 +100,16 @@ impl EntryPoint {
 /// routines.
 pub struct InstanceWrapper {
 	instance: Instance,
+
 	// The memory instance of the `instance`.
 	//
 	// It is important to make sure that we don't make any copies of this to make it easier to
 	// proof See `memory_as_slice` and `memory_as_slice_mut`.
 	memory: Memory,
+
+	/// Indirect functions table of the module
 	table: Option<Table>,
+
 	// Make this struct explicitly !Send & !Sync.
 	_not_send_nor_sync: marker::PhantomData<*const ()>,
 }
@@ -147,7 +155,7 @@ impl InstanceWrapper {
 			None => {
 				let memory = get_linear_memory(&instance)?;
 				if !memory.grow(heap_pages).is_ok() {
-					return Err("failed top increase the linear memory size".into())
+					return Err("failed to increase the linear memory size".into())
 				}
 				memory
 			},
@@ -223,11 +231,6 @@ impl InstanceWrapper {
 		self.table.as_ref()
 	}
 
-	/// Returns the byte size of the linear memory instance attached to this instance.
-	pub fn memory_size(&self) -> u32 {
-		self.memory.data_size() as u32
-	}
-
 	/// Reads `__heap_base: i32` global variable and returns it.
 	///
 	/// If it doesn't exist, not a global or of not i32 type returns an error.
@@ -291,32 +294,45 @@ fn get_table(instance: &Instance) -> Option<Table> {
 
 /// Functions related to memory.
 impl InstanceWrapper {
-	/// Read data from a slice of memory into a destination buffer.
+	/// Read data from a slice of memory into a newly allocated buffer.
 	///
 	/// Returns an error if the read would go out of the memory bounds.
-	pub fn read_memory_into(&self, address: Pointer<u8>, dest: &mut [u8]) -> Result<()> {
+	pub fn read_memory(&self, source_addr: Pointer<u8>, size: usize) -> Result<Vec<u8>> {
+		let range = checked_range(source_addr.into(), size, self.memory.data_size())
+			.ok_or_else(|| Error::Other("memory read is out of bounds".into()))?;
+
+		let mut buffer = vec![0; range.len()];
+		self.read_memory_into(source_addr, &mut buffer)?;
+
+		Ok(buffer)
+	}
+
+	/// Read data from the instance memory into a slice.
+	///
+	/// Returns an error if the read would go out of the memory bounds.
+	pub fn read_memory_into(&self, source_addr: Pointer<u8>, dest: &mut [u8]) -> Result<()> {
 		unsafe {
 			// This should be safe since we don't grow up memory while caching this reference and
 			// we give up the reference before returning from this function.
 			let memory = self.memory_as_slice();
 
-			let range = util::checked_range(address.into(), dest.len(), memory.len())
+			let range = checked_range(source_addr.into(), dest.len(), memory.len())
 				.ok_or_else(|| Error::Other("memory read is out of bounds".into()))?;
 			dest.copy_from_slice(&memory[range]);
 			Ok(())
 		}
 	}
 
-	/// Write data to a slice of memory.
+	/// Write data to the instance memory from a slice.
 	///
 	/// Returns an error if the write would go out of the memory bounds.
-	pub fn write_memory_from(&self, address: Pointer<u8>, data: &[u8]) -> Result<()> {
+	pub fn write_memory_from(&self, dest_addr: Pointer<u8>, data: &[u8]) -> Result<()> {
 		unsafe {
 			// This should be safe since we don't grow up memory while caching this reference and
 			// we give up the reference before returning from this function.
 			let memory = self.memory_as_slice_mut();
 
-			let range = util::checked_range(address.into(), data.len(), memory.len())
+			let range = checked_range(dest_addr.into(), data.len(), memory.len())
 				.ok_or_else(|| Error::Other("memory write is out of bounds".into()))?;
 			memory[range].copy_from_slice(data);
 			Ok(())
@@ -442,11 +458,11 @@ impl runtime_blob::InstanceGlobals for InstanceWrapper {
 	}
 
 	fn get_global_value(&self, global: &Self::Global) -> Value {
-		util::from_wasmtime_val(global.get())
+		from_wasmtime_val(global.get())
 	}
 
 	fn set_global_value(&self, global: &Self::Global, value: Value) {
-		global.set(util::into_wasmtime_val(value)).expect(
+		global.set(into_wasmtime_val(value)).expect(
 			"the value is guaranteed to be of the same value; the global is guaranteed to be mutable; qed",
 		);
 	}