Build WASM binaries as part of cargo build (#2868)

* Introduce `wasm-builder` and `wasm-builder-runner` to retire `build.sh`

Make use of `wasm-builder` in `test-runtime`.

* Add build script and remove the wasm project

* Port `node-runtime` to new wasm-builder

* Make `substrate-executor` tests work with `wasm-builder`

* Move `node-template` to `wasm-builder`

* Remove `build.sh` :)

* Remove the last include_bytes

* Adds the missing build.rs files

* Remove `build.sh` from CI

* Debug CI

* Make it work in CI

* CI attempt 3

* Make `substrate-runtime-test` compile on stable

* Ahhh, some missed `include_bytes!`

* AHH

* Add suggestions

* Improve search for `Cargo.lock` and don't panic if it is not found

* Searching from manifest path was no good idea

* Make the `wasm-builder` source better configurable

* Expose the bloaty wasm binary as well

* Make sure to rerun WASM recompilation on changes in dependencies

* Introduce new `WASM_BUILD_TYPE` env and make sure to call `build.rs` on
changes to env variables

* Remove `build.sh` from READMEs

* Rename the projects

* Fixes CI

* Update lock file

* Fixes merge-conflict

* Apply suggestions from code review

Co-Authored-By: TriplEight <denis.pisarev@parity.io>

* Try to make windows happy

* Replace all back slashes in paths with slashes

* Apply suggestions from code review

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Use cargo from `CARGO` env variable

* Fix compilation

* Use `rustup` for running the nightly build

* Make individual projects skipable

* Fix compilation

* Fixes compilation

* Build all WASM projects in one workspace

* Replace more back slashes!

* Remove `inlcude_bytes!`

* Adds some documentation

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* More review comments

* Update `Cargo.lock`

* Set license

* Apply suggestions from code review

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* More review comments + adds `TRIGGER_WASM_BUILD` env

* Fix doc tests

* Increase version + update README

* Switch crates.io version of `wasm-builder`

* Update README

* Switch to released version of `wasm-builder-runner`
This commit is contained in:
Bastian Köcher
2019-07-04 11:34:06 +02:00
committed by GitHub
parent eca9c36b75
commit fe08221479
57 changed files with 1187 additions and 11869 deletions
@@ -0,0 +1,168 @@
// Copyright 2019 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
//! # WASM builder is a utility for building a project as a WASM binary
//!
//! The WASM builder is a tool that integrates the process of building the WASM binary of your project into the main
//! `cargo` build process.
//!
//! ## Project setup
//!
//! A project that should be compiled as a WASM binary needs to:
//!
//! 1. Add a `build.rs` file.
//! 2. Add `substrate-wasm-builder-runner` as dependency into `build-dependencies`.
//! 3. Add a feature called `no-std`.
//!
//! The `build.rs` file needs to contain the following code:
//!
//! ```ignore
//! use wasm_builder_runner::{build_current_project, WasmBuilderSource};
//!
//! fn main() {
//! build_current_project(
//! // The name of the file being generated in out-dir.
//! "wasm_binary.rs",
//! // How to include wasm-builder, in this case from crates.io.
//! WasmBuilderSource::Crates("1.0.0"),
//! );
//! }
//! ```
//!
//! The `no-std` feature will be enabled by WASM builder while compiling your project to WASM.
//!
//! As the final step, you need to add the following to your project:
//!
//! ```ignore
//! include!(concat!(env!("OUT_DIR"), "/wasm_binary.rs"));
//! ```
//!
//! This will include the generated WASM binary as two constants `WASM_BINARY` and `WASM_BINARY_BLOATY`.
//! The former is a compact WASM binary and the latter is not compacted.
//!
//! ## Environment variables
//!
//! By using environment variables, you can configure which WASM binaries are built and how:
//!
//! - `SKIP_WASM_BUILD` - Skips building any WASM binary. This is useful when only native should be recompiled.
//! - `BUILD_DUMMY_WASM_BINARY` - Builds dummy WASM binaries. These dummy binaries are empty and useful
//! for `cargo check` runs.
//! - `WASM_BUILD_TYPE` - Sets the build type for building WASM binaries. Supported values are `release` or `debug`.
//! By default the build type is equal to the build type used by the main build.
//! - `TRIGGER_WASM_BUILD` - Can be set to trigger a WASM build. On subsequent calls the value of the variable
//! needs to change. As WASM builder instructs `cargo` to watch for file changes
//! this environment variable should only be required in certain circumstances.
//!
//! Each project can be skipped individually by using the environment variable `SKIP_PROJECT_NAME_WASM_BUILD`.
//! Where `PROJECT_NAME` needs to be replaced by the name of the cargo project, e.g. `node-runtime` will
//! be `NODE_RUNTIME`.
//!
//! ## Prerequisites:
//!
//! WASM builder requires the following prerequisities for building the WASM binary:
//!
//! - rust nightly + `wasm32-unknown-unknown` toolchain
//! - wasm-gc
//!
use std::{env, fs, path::PathBuf, process::Command};
mod prerequisites;
mod wasm_project;
/// Environment variable that tells us to skip building the WASM binary.
const SKIP_BUILD_ENV: &str = "SKIP_WASM_BUILD";
/// Environment variable to force a certain build type when building the WASM binary.
/// Expects "debug" or "release" as value.
///
/// By default the WASM binary uses the same build type as the main cargo build.
const WASM_BUILD_TYPE_ENV: &str = "WASM_BUILD_TYPE";
/// Build the currently built project as WASM binary.
///
/// The current project is determined by using the `CARGO_MANIFEST_DIR` environment variable.
///
/// `file_name` - The name + path of the file being generated. The file contains the
/// constant `WASM_BINARY`, which contains the built WASM binary.
/// `cargo_manifest` - The path to the `Cargo.toml` of the project that should be built.
pub fn build_project(file_name: &str, cargo_manifest: &str) {
if check_skip_build() {
return;
}
let cargo_manifest = PathBuf::from(cargo_manifest);
if !cargo_manifest.exists() {
create_out_file(
file_name,
format!("compile_error!(\"'{}' does not exists!\")", cargo_manifest.display())
);
return
}
if !cargo_manifest.ends_with("Cargo.toml") {
create_out_file(
file_name,
format!("compile_error!(\"'{}' no valid path to a `Cargo.toml`!\")", cargo_manifest.display())
);
return
}
if let Some(err_msg) = prerequisites::check() {
create_out_file(file_name, format!("compile_error!(\"{}\");", err_msg));
return
}
let (wasm_binary, bloaty) = wasm_project::create_and_compile(&cargo_manifest);
create_out_file(
file_name,
format!(
r#"
pub const WASM_BINARY: &[u8] = include_bytes!("{wasm_binary}");
pub const WASM_BINARY_BLOATY: &[u8] = include_bytes!("{wasm_binary_bloaty}");
"#,
wasm_binary = wasm_binary.wasm_binary_path(),
wasm_binary_bloaty = bloaty.wasm_binary_bloaty_path(),
),
);
}
/// Checks if the build of the WASM binary should be skipped.
fn check_skip_build() -> bool {
env::var(SKIP_BUILD_ENV).is_ok()
}
fn create_out_file(file_name: &str, content: String) {
fs::write(
file_name,
content
).expect("Creating and writing can not fail; qed");
}
/// Get a cargo command that compiles with nightly
fn get_nightly_cargo() -> Command {
if Command::new("rustup").args(&["run", "nightly", "cargo"])
.status().map(|s| s.success()).unwrap_or(false)
{
let mut cmd = Command::new("rustup");
cmd.args(&["run", "nightly", "cargo"]);
cmd
} else {
Command::new("cargo")
}
}
@@ -0,0 +1,89 @@
// Copyright 2019 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
use std::{process::{Command, Stdio}, fs};
use tempfile::tempdir;
/// Checks that all prerequisites are installed.
///
/// # Returns
/// Returns `None` if everything was found and `Some(ERR_MSG)` if something could not be found.
pub fn check() -> Option<&'static str> {
if !check_nightly_installed() {
return Some("Rust nightly not installed, please install it!")
}
if Command::new("wasm-gc").stdout(Stdio::null()).status().map(|s| !s.success()).unwrap_or(true) {
return Some("wasm-gc not installed, please install it!")
}
if !check_wasm_toolchain_installed() {
return Some("Rust WASM toolchain not installed, please install it!")
}
None
}
fn check_nightly_installed() -> bool {
let version = Command::new("cargo")
.arg("--version")
.output()
.map_err(|_| ())
.and_then(|o| String::from_utf8(o.stdout).map_err(|_| ()))
.unwrap_or_default();
let version2 = Command::new("rustup")
.args(&["run", "nightly", "cargo", "--version"])
.output()
.map_err(|_| ())
.and_then(|o| String::from_utf8(o.stdout).map_err(|_| ()))
.unwrap_or_default();
version.contains("-nightly") || version2.contains("-nightly")
}
fn check_wasm_toolchain_installed() -> bool {
let temp = tempdir().expect("Creating temp dir does not fail; qed");
fs::create_dir_all(temp.path().join("src")).expect("Creating src dir does not fail; qed");
let test_file = temp.path().join("src/lib.rs");
let manifest_path = temp.path().join("Cargo.toml");
fs::write(&manifest_path,
r#"
[package]
name = "wasm-test"
version = "1.0.0"
edition = "2018"
[lib]
name = "wasm_test"
crate-type = ["cdylib"]
[workspace]
"#,
).expect("Writing wasm-test manifest does not fail; qed");
fs::write(&test_file, "pub fn test() {}")
.expect("Writing to the test file does not fail; qed");
let manifest_path = manifest_path.display().to_string();
crate::get_nightly_cargo()
.args(&["build", "--target=wasm32-unknown-unknown", "--manifest-path", &manifest_path])
.status()
.map(|s| s.success())
.unwrap_or(false)
}
@@ -0,0 +1,336 @@
// Copyright 2019 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
use std::{fs, path::{Path, PathBuf}, borrow::ToOwned, process::Command, env};
use toml::value::Table;
use build_helper::rerun_if_changed;
use cargo_metadata::MetadataCommand;
use walkdir::WalkDir;
use fs2::FileExt;
/// Holds the path to the bloaty WASM binary.
pub struct WasmBinaryBloaty(PathBuf);
impl WasmBinaryBloaty {
/// Returns the path to the bloaty wasm binary.
pub fn wasm_binary_bloaty_path(&self) -> String {
self.0.display().to_string().replace('\\', "/")
}
}
/// Holds the path to the WASM binary.
pub struct WasmBinary(PathBuf);
impl WasmBinary {
/// Returns the path to the wasm binary.
pub fn wasm_binary_path(&self) -> String {
self.0.display().to_string().replace('\\', "/")
}
}
/// A lock for the WASM workspace.
struct WorkspaceLock(fs::File);
impl WorkspaceLock {
/// Create a new lock
fn new(wasm_workspace_root: &Path) -> Self {
let lock = fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open(wasm_workspace_root.join("wasm_workspace.lock"))
.expect("Opening the lock file does not fail");
lock.lock_exclusive().expect("Locking `wasm_workspace.lock` failed");
WorkspaceLock(lock)
}
}
impl Drop for WorkspaceLock {
fn drop(&mut self) {
let _ = self.0.unlock();
}
}
/// Creates the WASM project, compiles the WASM binary and compacts the WASM binary.
///
/// # Returns
/// The path to the compact WASM binary and the bloaty WASM binary.
pub fn create_and_compile(cargo_manifest: &Path) -> (WasmBinary, WasmBinaryBloaty) {
let wasm_workspace_root = get_wasm_workspace_root();
let wasm_workspace = wasm_workspace_root.join("wbuild");
// Lock the workspace exclusively for us
let _lock = WorkspaceLock::new(&wasm_workspace_root);
let project = create_project(cargo_manifest, &wasm_workspace);
create_wasm_workspace_project(&wasm_workspace);
build_project(&project);
let (wasm_binary, bloaty) = compact_wasm_file(&project, cargo_manifest, &wasm_workspace);
generate_rerun_if_changed_instructions(cargo_manifest, &project, &wasm_workspace);
(wasm_binary, bloaty)
}
/// Find the `Cargo.lock` relative to the `OUT_DIR` environment variable.
///
/// If the `Cargo.lock` cannot be found, we emit a warning and return `None`.
fn find_cargo_lock(cargo_manifest: &Path) -> Option<PathBuf> {
let mut path = build_helper::out_dir();
while path.pop() {
if path.join("Cargo.lock").exists() {
return Some(path.join("Cargo.lock"))
}
}
build_helper::warning!(
"Could not find `Cargo.lock` for `{}`, while searching from `{}`.",
cargo_manifest.display(),
build_helper::out_dir().display()
);
None
}
/// Extract the crate name from the given `Cargo.toml`.
fn get_crate_name(cargo_manifest: &Path) -> String {
let cargo_toml: Table = toml::from_str(
&fs::read_to_string(cargo_manifest).expect("File exists as checked before; qed")
).expect("Cargo manifest is a valid toml file; qed");
let package = cargo_toml
.get("package")
.and_then(|t| t.as_table())
.expect("`package` key exists in valid `Cargo.toml`; qed");
package.get("name").and_then(|p| p.as_str()).map(ToOwned::to_owned).expect("Package name exists; qed")
}
/// Returns the name for the wasm binary.
fn get_wasm_binary_name(cargo_manifest: &Path) -> String {
get_crate_name(cargo_manifest).replace('-', "_")
}
/// Returns the root path of the wasm workspace.
fn get_wasm_workspace_root() -> PathBuf {
let mut out_dir = build_helper::out_dir();
while out_dir.parent().is_some() {
if out_dir.parent().map(|p| p.ends_with("target")).unwrap_or(false) {
return out_dir;
}
out_dir.pop();
}
panic!("Could not find target dir in: {}", build_helper::out_dir().display())
}
fn create_wasm_workspace_project(wasm_workspace: &Path) {
let members = WalkDir::new(wasm_workspace)
.min_depth(1)
.max_depth(1)
.into_iter()
.filter_map(|p| p.ok())
.map(|d| d.into_path())
.filter(|p| p.is_dir() && !p.ends_with("target"))
.filter_map(|p| p.file_name().map(|f| f.to_owned()).and_then(|s| s.into_string().ok()))
.map(|s| format!("\"{}\", ", s))
.collect::<String>();
fs::write(
wasm_workspace.join("Cargo.toml"),
format!(
r#"
[profile.release]
panic = "abort"
lto = true
[profile.dev]
panic = "abort"
[workspace]
members = [ {members} ]
"#,
members = members,
)
).expect("WASM workspace `Cargo.toml` writing can not fail; qed");
}
/// Create the project used to build the wasm binary.
///
/// # Returns
/// The path to the created project.
fn create_project(cargo_manifest: &Path, wasm_workspace: &Path) -> PathBuf {
let crate_name = get_crate_name(cargo_manifest);
let crate_path = cargo_manifest.parent().expect("Parent path exists; qed");
let wasm_binary = get_wasm_binary_name(cargo_manifest);
let project_folder = wasm_workspace.join(&crate_name);
fs::create_dir_all(project_folder.join("src")).expect("Wasm project dir create can not fail; qed");
fs::write(
project_folder.join("Cargo.toml"),
format!(
r#"
[package]
name = "{crate_name}-wasm"
version = "1.0.0"
edition = "2018"
[lib]
name = "{wasm_binary}"
crate-type = ["cdylib"]
[dependencies]
wasm_project = {{ package = "{crate_name}", path = "{crate_path}", default-features = false, features = [ "no_std" ] }}
"#,
crate_name = crate_name,
crate_path = crate_path.display(),
wasm_binary = wasm_binary,
)
).expect("Project `Cargo.toml` writing can not fail; qed");
fs::write(
project_folder.join("src/lib.rs"),
format!(
r#"
#![no_std]
pub use wasm_project::*;
"#
)
).expect("Project `lib.rs` writing can not fail; qed");
if let Some(crate_lock_file) = find_cargo_lock(cargo_manifest) {
// Use the `Cargo.lock` of the main project.
fs::copy(crate_lock_file, wasm_workspace.join("Cargo.lock"))
.expect("Copying the `Cargo.lock` can not fail; qed");
}
project_folder
}
/// Returns if the project should be built as a release.
fn is_release_build() -> bool {
if let Ok(var) = env::var(crate::WASM_BUILD_TYPE_ENV) {
match var.as_str() {
"release" => true,
"debug" => false,
var => panic!(
"Unexpected value for `{}` env variable: {}\nOne of the following are expected: `debug` or `release`.",
crate::WASM_BUILD_TYPE_ENV,
var,
),
}
} else {
!build_helper::debug()
}
}
/// Build the project to create the WASM binary.
fn build_project(project: &Path) {
let manifest_path = project.join("Cargo.toml");
let mut build_cmd = crate::get_nightly_cargo();
build_cmd.args(&["build", "--target=wasm32-unknown-unknown"])
.arg(format!("--manifest-path={}", manifest_path.display()))
.env("RUSTFLAGS", "-C link-arg=--export-table")
// We don't want to call ourselves recursively
.env(crate::SKIP_BUILD_ENV, "");
if is_release_build() {
build_cmd.arg("--release");
};
println!("Executing build command: {:?}", build_cmd);
match build_cmd.status().map(|s| s.success()) {
Ok(true) => {},
_ => panic!("Failed to compile WASM binary"),
}
}
/// Compact the WASM binary using `wasm-gc`. Returns the path to the bloaty WASM binary.
fn compact_wasm_file(
project: &Path,
cargo_manifest: &Path,
wasm_workspace: &Path,
) -> (WasmBinary, WasmBinaryBloaty) {
let target = if is_release_build() { "release" } else { "debug" };
let wasm_binary = get_wasm_binary_name(cargo_manifest);
let wasm_file = wasm_workspace.join("target/wasm32-unknown-unknown")
.join(target)
.join(format!("{}.wasm", wasm_binary));
let wasm_compact_file = project.join(format!("{}.compact.wasm", wasm_binary));
let res = Command::new("wasm-gc")
.arg(&wasm_file)
.arg(&wasm_compact_file)
.status()
.map(|s| s.success());
if !res.unwrap_or(false) {
panic!("Failed to compact generated WASM binary.");
}
(WasmBinary(wasm_compact_file), WasmBinaryBloaty(wasm_file))
}
/// Generate the `rerun-if-changed` instructions for cargo to make sure that the WASM binary is
/// rebuilt when needed.
fn generate_rerun_if_changed_instructions(
cargo_manifest: &Path,
project_folder: &Path,
wasm_workspace: &Path,
) {
// Rerun `build.rs` if the `Cargo.lock` changes
if let Some(cargo_lock) = find_cargo_lock(cargo_manifest) {
rerun_if_changed(cargo_lock);
}
let metadata = MetadataCommand::new()
.manifest_path(project_folder.join("Cargo.toml"))
.exec()
.expect("`cargo metadata` can not fail!");
// Make sure that if any file/folder of a depedency change, we need to rerun the `build.rs`
metadata.packages.into_iter()
.filter(|package| !package.manifest_path.starts_with(wasm_workspace))
.for_each(|package| {
let mut manifest_path = package.manifest_path;
if manifest_path.ends_with("Cargo.toml") {
manifest_path.pop();
}
rerun_if_changed(&manifest_path);
WalkDir::new(manifest_path)
.into_iter()
.filter_map(|p| p.ok())
.for_each(|p| rerun_if_changed(p.path()));
});
// Register our env variables
println!("cargo:rerun-if-env-changed={}", crate::SKIP_BUILD_ENV);
println!("cargo:rerun-if-env-changed={}", crate::WASM_BUILD_TYPE_ENV);
}