feat: Vendor pezkuwi-subxt and pezkuwi-zombienet-sdk into monorepo

- Add pezkuwi-subxt crates to vendor/pezkuwi-subxt - Add pezkuwi-zombienet-sdk crates to vendor/pezkuwi-zombienet-sdk - Convert git dependencies to path dependencies - Add vendor crates to workspace members - Remove test/example crates from vendor (not needed for SDK) - Fix feature propagation issues detected by zepter - Fix workspace inheritance for internal dependencies - All 606 crates now in workspace - All 6919 internal dependency links verified correct - No git dependencies remaining
2025-12-22 23:31:24 +03:00
parent 4c8f281051
commit 70ddb6516f
386 changed files with 76759 additions and 36 deletions
@@ -0,0 +1,112 @@
+# Mechanism to call Rust code from Javascript/Typescript
+
+### Status: proposed | rejected | **accepted** | deprecated
+
+### Deciders: [@pepoviola](https://github.com/pepoviola) [@wirednkod](https://github.com/wirednkod) [@l0r1s](https://github.com/l0r1s)
+
+### Creation date: 18/05/2023
+
+### Update date: -
+
+---
+
+## Context and Problem Statement
+
+The `zombienet-sdk` will be developed in Rust. Our objective is make it easily integrable into existing Typescript/Javascript project. To achieve this goal, we need to find a way to call the Rust code from a Javascript/Typescript program.
+
+Many mechanisms exists for this purpose like Wasm or N(ode)-API but some may or may not fit our use case, for example, executing async code.
+
+---
+
+## Decision drivers
+
+- We can use the standard library (for filesystem or networking in providers).
+
+- We can execute asynchronous code: our goal is not to make the program fully sequential as many operations (e.g: bootstrapping the relaychain nodes) can be done concurrently.
+
+- Easy to package and deploy
+
+---
+
+## Considered Options
+
+- #### WASM
+
+  - [wasm-pack](https://github.com/neon-bindings/neon)
+
+- #### Native node modules (Node-API / V8 / libuv)
+  - [napi-rs](https://github.com/napi-rs/napi-rs)
+
+---
+
+## Prototyping
+
+To demonstrate and learn which options fit the best for our use case, we will create a small test program which will have the following functionalities:
+
+- Has a function taking an arbitratry object and a callback as parameters in the Typescript code, calling the callback with the function result on Rust side.
+- Has a function taking an arbitrary object as parameter and a returning a promise in Typescript, signaling an asynchronous operation on Rust side.
+- Make an HTTP request asynchronously in the Rust code, using a dependency using the standard library.
+
+The prototype assume versions of `rustc` and `cargo` to be `1.69.0`, use of `stable` channel and `Linux` on `amd64` architecture.
+
+
+- ### [Boilerplate app to execute prototype](boilerplate-app-prototype.md)
+
+- ### [Wasm-pack prototype](wasm-prototype.md)
+
+- ### [Napi-rs prototype](napi-prototype.md)
+
+---
+
+## Pros and cons of each options
+
+- ### Napi-rs
+  - Pros 👍
+    - Support many types correctly including typed callback, typed array, class and all JS primitives types (Null, Undefined, Numbers, String, BigInt, ...)
+
+    - Support top level async function because it detects if it needs to be run inside an async runtime (tokio by default)
+
+    - Standard library can be used without limitations, including threading, networking, etc...
+
+    - Extremely well documented with examples
+
+    - Provide full Github action pipeline template to compile on all architecture easily
+
+    - Support complex use cases
+
+    - Used by many big names (Prisma, Parcel, Tailwind, Next.js, Bitwarden)
+
+  - Cons 👎
+    - Node-API is not simple for complex use case
+
+    - Bound to NodeJS, if we want to expose the same logic to others languages (Go, C++, Python, ...) we need to wrap the Rust code inside a dynamic library and adapt to others languages primitives by creating a small adapter over the library
+
+    - Not universally compiled
+
+
+- ### Wasm-pack
+  - Pros 👍
+    - Rich ecosystem and developing fast
+
+    - Used in many places across web, backend (Docker supports WASM)
+
+    - Easy to use and distribute
+
+    - Universally compiled and used across languages (if they support WASM execution)
+
+    - Good for simple use case where you do pure function (taking input, returning output, without side effects like writing to filesystem or making networking calls)
+
+  - Cons 👎
+    - Limited in the use of the standard library, can't access networking/filesystem primitives without having to use WASI which is inconsistent across languages/runtimes
+
+    - Only support 32 bits
+
+    - No support for concurrent programming (async/threads), even if we can returns Promise from WASM exposed functions but could see the light in few months (maybe?)
+
+    - wasm-bindgen types are too generic, for example, we return a JsValue but we would like to be more specific for the type
+
+## Decision outcome
+
+- ### **Napi-rs** for crates dependant on async, filesystem or networking: *support*, *orchestrator*, *test-runner*, *providers* from [schema](https://github.com/paritytech/zombienet-sdk/issues/22)
+
+- ### **Wasm-pack** for the rest of the crates: *configuration* from [schema](https://github.com/paritytech/zombienet-sdk/issues/22)
@@ -0,0 +1,32 @@
+## [Back](001-node-to-rust-foreign-function-interface.md)
+
+## Boilerplate app to execute prototypes
+
+1. Create the new node app :
+
+```bash
+$ mkdir -p ffi-prototype/app && cd ffi-prototype/app && npm init -y
+```
+
+2. Install required packages :
+
+```bash
+[ffi-prototype/app]$ npm i -D @tsconfig/recommended ts-node typescript
+```
+
+3. Add a new script :
+
+```json
+{
+  "scripts": {
+    "build+exec": "tsc && node ./index.js"
+  }
+}
+```
+
+4. Add tsconfig.json
+```json
+{
+  "extends": "@tsconfig/recommended/tsconfig.json"
+}
+```
@@ -0,0 +1,142 @@
+## [Back](001-node-to-rust-foreign-function-interface.md)
+
+## Napi-rs prototype
+___
+
+1. Install the napi CLI
+
+```bash
+[ffi-prototype]$ npm install -g @napi-rs/cli
+```
+
+2. Create a new napi project
+
+```bash
+[ffi-prototype]$ napi new napi-prototype
+```
+
+3. Install cargo dependencies
+
+```bash
+[ffi-prototype/napi-prototype]$ cargo add tokio --features full
+[ffi-prototype/napi-prototype]$ cargo add reqwest --features blocking
+[ffi-prototype/napi-prototype]$ cargo add napi --no-default-features --features napi4,async
+```
+
+4. Copy the following code to `napi-prototype/src/lib.rs`
+
+```rust
+#![deny(clippy::all)]
+
+use std::thread;
+
+use napi::{
+  bindgen_prelude::*,
+  threadsafe_function::{
+    ErrorStrategy, ThreadSafeCallContext, ThreadsafeFunction, ThreadsafeFunctionCallMode,
+  },
+};
+use reqwest;
+
+#[macro_use]
+extern crate napi_derive;
+
+// native async with tokio is supported without annotating a main function
+#[napi]
+pub async fn fetch_promise() -> Result<String> {
+  let body = reqwest::get("https://paritytech.github.io/zombienet/")
+    .await
+    .map_err(|_| napi::Error::from_reason("Error while fetching page"))?
+    .text()
+    .await
+    .map_err(|_| napi::Error::from_reason("Error while extracting body"))?;
+
+  Ok(body)
+}
+
+#[napi]
+pub fn fetch_callback(callback: JsFunction) -> Result<()> {
+  // createa thread safe callback from the JsFunction
+  let thread_safe_callback: ThreadsafeFunction<String, ErrorStrategy::CalleeHandled> = callback
+    .create_threadsafe_function(0, |ctx: ThreadSafeCallContext<String>| {
+      ctx.env.create_string(&ctx.value).map(|s| vec![s])
+    })?;
+
+  // spawn a thread to execute our logic
+  thread::spawn(move || {
+    let response = reqwest::blocking::get("https://paritytech.github.io/zombienet/");
+
+    if response.is_err() {
+      let response = response
+        .map(|_| "".into())
+        .map_err(|_| napi::Error::from_reason("Error while fetching page"));
+
+      // error are returned by calling the callback with an empty response and the error mapped
+      return thread_safe_callback.call(response, ThreadsafeFunctionCallMode::Blocking);
+    }
+
+    let body = response.unwrap().text();
+
+    if body.is_err() {
+      let body = body
+        .map(|_| "".into())
+        .map_err(|_| napi::Error::from_reason("Error while extracting body"));
+
+      return thread_safe_callback.call(body, ThreadsafeFunctionCallMode::Blocking);
+    }
+
+    // result is returned as a string
+    thread_safe_callback.call(Ok(body.unwrap()), ThreadsafeFunctionCallMode::Blocking)
+  });
+
+  Ok(())
+}
+```
+
+5. Build the project :
+```bash
+[ffi-prototype/napi-prototype]$ npm run build
+```
+
+6. Copy artifacts :
+```bash
+[ffi-prorotype/napi-prototype]$ mv napi-prototype.linux-x64-gnu.node index.d.ts index.js npm/linux-x64-gnu
+```
+
+7. Install package in ```ffi-prototype/app``` :
+```bash
+[ffi-prototype/app]$ npm i ../napi-prototype/npm/linux-x64-gnu/
+```
+
+8. Copy the following code to the ```ffi-prototype/app/index.ts``` file :
+
+```ts
+import { fetchCallback, fetchPromise } from "napi-prototype-linux-x64-gnu";
+
+(async () => {
+  fetchCallback((_err: any, result: string) => {
+    console.log(`HTTP request through FFI with callback: ${result.length}`);
+  });
+
+  console.log(
+    `HTTP request through FFI with promise ${(await fetchPromise()).length}`
+  );
+})();
+```
+
+9. Build and execute the app :
+
+```bash
+[ffi-prototype/app]$ npm run build+exec
+```
+
+Expected output:
+```tty
+> app@1.0.0 build+exec
+> tsc && node ./index.js
+
+HTTP request through FFI with promise 12057
+HTTP request through FFI with callback: 12057
+```
+
+That's it !
@@ -0,0 +1,153 @@
+## [Back](001-node-to-rust-foreign-function-interface.md)
+
+## Wasm-pack prototype
+___
+
+1. Install the wasm-pack CLI
+
+```bash
+curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
+```
+
+2. Create a new wasm-pack project
+
+```bash
+[ffi-prototype]$ wasm-pack new wasm-prototype
+```
+
+3. Install cargo dependencies
+```bash
+[ffi-prototype/wasm-prototype]$ cargo add tokio --features full
+[ffi-prototype/wasm-prototype]$ cargo add reqwest --features blocking
+[ffi-prototype/wasm-prototype]$ cargo add wasm-bindgen-futures
+cargo add js-sys
+```
+
+4. Copy the following code to `wasm-prototype/src/lib.rs`
+```rust
+mod utils;
+
+use wasm_bindgen::prelude::*;
+
+// When the `wee_alloc` feature is enabled, use `wee_alloc` as the global
+// allocator.
+#[cfg(feature = "wee_alloc")]
+#[global_allocator]
+static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
+
+#[wasm_bindgen]
+pub async fn fetch_promise() -> Result<String, JsError> {
+    let body = reqwest::get("https://paritytech.github.io/zombienet/")
+        .await
+        .map_err(|_| JsError::new("Error while fetching page"))?
+        .text()
+        .await
+        .map_err(|_| JsError::new("Error while extracting body"))?;
+
+    Ok(body)
+}
+
+#[wasm_bindgen]
+pub fn fetch_callback(callback: &js_sys::Function) -> Result<JsValue, JsValue> {
+    let this = JsValue::null();
+
+    let response = reqwest::blocking::get("https://paritytech.github.io/zombienet/");
+
+    if response.is_err() {
+        return callback.call2(
+            &this,
+            &JsError::new("Error while fetching page").into(),
+            &JsValue::null(),
+        );
+    }
+
+    let body = response.unwrap().text();
+
+    if body.is_err() {
+        return callback.call2(
+            &this,
+            &JsError::new("Error while extracting body").into(),
+            &JsValue::null(),
+        );
+    }
+
+    Ok(body.unwrap().into())
+}
+```
+
+5. Build the project :
+```bash
+[ffi-prototype/wasm-prototype]$ wasm-pack build -t nodejs
+```
+
+Error are shown, this is expected because WASM doesn't support networking primitives, 
+as you can see, we removed the thread call from the fetch_callback function because ```JsValue```
+is using *const u8 under the hood and it's not ```Send``` so can't be passed safely across thread:
+
+```bash
+[INFO]: 🎯  Checking for the Wasm target...
+[INFO]: 🌀  Compiling to Wasm...
+   Compiling mio v0.8.6
+   Compiling parking_lot v0.12.1
+   Compiling serde_json v1.0.96
+   Compiling url v2.3.1
+error[E0432]: unresolved import `crate::sys::IoSourceState`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/io_source.rs:12:5
+   |
+12 | use crate::sys::IoSourceState;
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^ no `IoSourceState` in `sys`
+
+error[E0432]: unresolved import `crate::sys::tcp`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/net/tcp/listener.rs:15:17
+   |
+15 | use crate::sys::tcp::{bind, listen, new_for_addr};
+   |                 ^^^ could not find `tcp` in `sys`
+
+error[E0432]: unresolved import `crate::sys::tcp`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/net/tcp/stream.rs:13:17
+   |
+13 | use crate::sys::tcp::{connect, new_for_addr};
+   |                 ^^^ could not find `tcp` in `sys`
+
+error[E0433]: failed to resolve: could not find `Selector` in `sys`
+   --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/poll.rs:301:18
+    |
+301 |             sys::Selector::new().map(|selector| Poll {
+    |                  ^^^^^^^^ could not find `Selector` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:24:14
+   |
+24 |         sys::event::token(&self.inner)
+   |              ^^^^^ could not find `event` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:38:14
+   |
+38 |         sys::event::is_readable(&self.inner)
+   |              ^^^^^ could not find `event` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:43:14
+   |
+43 |         sys::event::is_writable(&self.inner)
+   |              ^^^^^ could not find `event` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:68:14
+   |
+68 |         sys::event::is_error(&self.inner)
+   |              ^^^^^ could not find `event` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+  --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:99:14
+   |
+99 |         sys::event::is_read_closed(&self.inner)
+   |              ^^^^^ could not find `event` in `sys`
+
+error[E0433]: failed to resolve: could not find `event` in `sys`
+   --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:129:14
+    |
+129 |         sys::event::is_write_closed(&self.inner)
+    |              ^^^^^ could not find `event` in `sys`
+```