pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-04-27 01:07:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Wasm-builder 3.0 (#7532)

Browse Source 
* Build every wasm crate in its own project with wasm-builder

Building all wasm crates in one workspace was a nice idea, however it
just introduced problems:

1. We needed to prune old members, but this didn't worked for old git
deps.
2. We locked the whole wasm workspace while building one crate. This
could lead to infinitely locking the workspace on a crash.

Now we just build every crate in its own project, this means we will
build the dependencies multiple times. While building the dependencies
multiple times, we still decrease the build time by around 30 seconds
for Polkadot and Substrate because of the new parallelism ;)

* Remove the requirement on wasm-builder-runner

This removes the requirement on wasm-builder-runner by using the new
`build_dep` feature of cargo. We use nightly anyway and that enables us
to use this feature. This solves the problem of not mixing
build/proc-macro deps with normal deps. By doing this we get rid off
this complicated project structure and can depend directly on
`wasm-builder`. This also removes all the code from wasm-builder-runner
and mentions that it is deprecated.

* Copy the `Cargo.lock` to the correct folder

* Remove wasm-builder-runner

* Update docs

* Fix deterministic check

Modified-by: Bastian Köcher <git@kchr.de>

* Try to make the ui test happy

* Switch to `SKIP_WASM_BUILD`

* Rename `SKIP_WASM_BINARY` to the correct name...

* Update utils/wasm-builder/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update utils/wasm-builder/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
This commit is contained in:
Bastian Köcher
2020-11-24 10:18:36 +01:00
committed by GitHub
parent e6a6ed9a16
commit 923cb8eea1
36 changed files with 442 additions and 885 deletions
Download Patch File Download Diff File Expand all files Collapse all files
substrate/utils/wasm-builder/src/lib.rs
+31 -101
View File
@@ -25,20 +25,23 @@
//! A project that should be compiled as a Wasm binary needs to:
//!
//! 1. Add a `build.rs` file.
//! 2. Add `substrate-wasm-builder` as dependency into `build-dependencies`.
//! 2. Add `wasm-builder` as dependency into `build-dependencies`.
//!
//! The `build.rs` file needs to contain the following code:
//!
//! ```ignore
//! use wasm_builder_runner::{build_current_project, WasmBuilderSource};
//! ```no_run
//! use substrate_wasm_builder::WasmBuilder;
//!
//! fn main() {
//! build_current_project(
//! // The name of the file being generated in out-dir.
//! "wasm_binary.rs",
//! // How to include wasm-builder, in this case from crates.io.
//! WasmBuilderSource::Crates("1.0.0"),
//! );
//! WasmBuilder::new()
//! // Tell the builder to build the project (crate) this `build.rs` is part of.
//! .with_current_project()
//! // Make sure to export the `heap_base` global, this is required by Substrate
//! .export_heap_base()
//! // Build the Wasm file so that it imports the memory (need to be provided by at instantiation)
//! .import_memory()
//! // Build it.
//! .build()
//! }
//! ```
//!
@@ -49,7 +52,8 @@
//! ```
//!
//! This will include the generated Wasm binary as two constants `WASM_BINARY` and `WASM_BINARY_BLOATY`.
//! The former is a compact Wasm binary and the latter is not compacted.
//! The former is a compact Wasm binary and the latter is the Wasm binary as being generated by the compiler.
//! Both variables have `Option<&'static [u8]>` as type.
//!
//! ### Feature
//!
@@ -63,19 +67,19 @@
//!
//! By using environment variables, you can configure which Wasm binaries are built and how:
//!
//! - `SKIP_WASM_BUILD` - Skips building any wasm binary. This is useful when only native should be recompiled.
//! - `BUILD_DUMMY_WASM_BINARY` - Builds dummy wasm binaries. These dummy binaries are empty and useful
//! for `cargo check` runs.
//! - `WASM_BUILD_TYPE` - Sets the build type for building wasm binaries. Supported values are `release` or `debug`.
//! - `SKIP_WASM_BUILD` - Skips building any Wasm binary. This is useful when only native should be recompiled.
//! If this is the first run and there doesn't exist a Wasm binary, this will set both
//! variables to `None`.
//! - `WASM_BUILD_TYPE` - Sets the build type for building Wasm binaries. Supported values are `release` or `debug`.
//! By default the build type is equal to the build type used by the main build.
//! - `FORCE_WASM_BUILD` - Can be set to force a wasm build. On subsequent calls the value of the variable
//! needs to change. As wasm builder instructs `cargo` to watch for file changes
//! - `FORCE_WASM_BUILD` - Can be set to force a Wasm build. On subsequent calls the value of the variable
//! needs to change. As wasm-builder instructs `cargo` to watch for file changes
//! this environment variable should only be required in certain circumstances.
//! - `WASM_BUILD_RUSTFLAGS` - Extend `RUSTFLAGS` given to `cargo build` while building the wasm binary.
//! - `WASM_BUILD_NO_COLOR` - Disable color output of the wasm build.
//! - `WASM_TARGET_DIRECTORY` - Will copy any build wasm binary to the given directory. The path needs
//! - `WASM_TARGET_DIRECTORY` - Will copy any build Wasm binary to the given directory. The path needs
//! to be absolute.
//! - `WASM_BUILD_TOOLCHAIN` - The toolchain that should be used to build the wasm binaries. The
//! - `WASM_BUILD_TOOLCHAIN` - The toolchain that should be used to build the Wasm binaries. The
//! format needs to be the same as used by cargo, e.g. `nightly-2020-02-20`.
//!
//! Each project can be skipped individually by using the environment variable `SKIP_PROJECT_NAME_WASM_BUILD`.
@@ -92,11 +96,14 @@
//! as well. For example if installing the rust nightly from 20.02.2020 using `rustup install nightly-2020-02-20`,
//! the wasm target needs to be installed as well `rustup target add wasm32-unknown-unknown --toolchain nightly-2020-02-20`.
use std::{env, fs, path::{PathBuf, Path}, process::{Command, self}, io::BufRead};
use std::{env, fs, path::{PathBuf, Path}, process::Command, io::BufRead};
mod builder;
mod prerequisites;
mod wasm_project;
pub use builder::{WasmBuilder, WasmBuilderSelectProject};
/// Environment variable that tells us to skip building the wasm binary.
const SKIP_BUILD_ENV: &str = "SKIP_WASM_BUILD";
@@ -120,87 +127,8 @@ const WASM_BUILD_NO_COLOR: &str = "WASM_BUILD_NO_COLOR";
/// Environment variable to set the toolchain used to compile the wasm binary.
const WASM_BUILD_TOOLCHAIN: &str = "WASM_BUILD_TOOLCHAIN";
/// Build the currently built project as wasm binary.
///
/// The current project is determined by using the `CARGO_MANIFEST_DIR` environment variable.
///
/// `file_name` - The name + path of the file being generated. The file contains the
/// constant `WASM_BINARY`, which contains the built WASM binary.
/// `cargo_manifest` - The path to the `Cargo.toml` of the project that should be built.
pub fn build_project(file_name: &str, cargo_manifest: &str) {
build_project_with_default_rustflags(file_name, cargo_manifest, "");
}
/// Build the currently built project as wasm binary.
///
/// The current project is determined by using the `CARGO_MANIFEST_DIR` environment variable.
///
/// `file_name` - The name + path of the file being generated. The file contains the
/// constant `WASM_BINARY`, which contains the built WASM binary.
/// `cargo_manifest` - The path to the `Cargo.toml` of the project that should be built.
/// `default_rustflags` - Default `RUSTFLAGS` that will always be set for the build.
pub fn build_project_with_default_rustflags(
file_name: &str,
cargo_manifest: &str,
default_rustflags: &str,
) {
if check_skip_build() {
return;
}
let cargo_manifest = PathBuf::from(cargo_manifest);
if !cargo_manifest.exists() {
panic!("'{}' does not exist!", cargo_manifest.display());
}
if !cargo_manifest.ends_with("Cargo.toml") {
panic!("'{}' no valid path to a `Cargo.toml`!", cargo_manifest.display());
}
let cargo_cmd = match prerequisites::check() {
Ok(cmd) => cmd,
Err(err_msg) => {
eprintln!("{}", err_msg);
process::exit(1);
},
};
let (wasm_binary, bloaty) = wasm_project::create_and_compile(
&cargo_manifest,
default_rustflags,
cargo_cmd,
);
let (wasm_binary, wasm_binary_bloaty) = if let Some(wasm_binary) = wasm_binary {
(
wasm_binary.wasm_binary_path_escaped(),
bloaty.wasm_binary_bloaty_path_escaped(),
)
} else {
(
bloaty.wasm_binary_bloaty_path_escaped(),
bloaty.wasm_binary_bloaty_path_escaped(),
)
};
write_file_if_changed(
file_name,
format!(
r#"
pub const WASM_BINARY: Option<&[u8]> = Some(include_bytes!("{wasm_binary}"));
pub const WASM_BINARY_BLOATY: Option<&[u8]> = Some(include_bytes!("{wasm_binary_bloaty}"));
"#,
wasm_binary = wasm_binary,
wasm_binary_bloaty = wasm_binary_bloaty,
),
);
}
/// Checks if the build of the WASM binary should be skipped.
fn check_skip_build() -> bool {
env::var(SKIP_BUILD_ENV).is_ok()
}
/// Environment variable that makes sure the WASM build is triggered.
const FORCE_WASM_BUILD_ENV: &str = "FORCE_WASM_BUILD";
/// Write to the given `file` if the `content` is different.
fn write_file_if_changed(file: impl AsRef<Path>, content: impl AsRef<str>) {
@@ -217,7 +145,9 @@ fn copy_file_if_changed(src: PathBuf, dst: PathBuf) {
if src_file != dst_file {
fs::copy(&src, &dst)
.unwrap_or_else(|_| panic!("Copying `{}` to `{}` can not fail; qed", src.display(), dst.display()));
.unwrap_or_else(
|_| panic!("Copying `{}` to `{}` can not fail; qed", src.display(), dst.display())
);
}
}