Minimal parachain framework part 1 (#113)

* dynamic inclusion threshold calculator

* collators interface

* collation helpers

* initial proposal-creation future

* create proposer when asked to propose

* remove local_availability duty

* statement table tracks includable parachain count

* beginnings of timing future

* finish proposal logic

* remove stray println

* extract shared table to separate module

* change ordering

* includability tracking

* fix doc

* initial changes to parachains module

* initialise dummy block before API calls

* give polkadot control over round proposer based on random seed

* propose only after enough candidates

* flesh out parachains module a bit more

* set_heads

* actually introduce set_heads to runtime

* update block_builder to accept parachains

* split block validity errors from real errors in evaluation

* update WASM runtimes

* polkadot-api methods for parachains additions

* delay evaluation until candidates are ready

* comments

* fix dynamic inclusion with zero initial

* test for includability tracker

* wasm validation of parachain candidates

* move primitives to primitives crate

* remove runtime-std dependency from codec

* adjust doc

* polkadot-parachain-primitives

* kill legacy polkadot-validator crate

* basic-add test chain

* test for basic_add parachain

* move to test-chains dir

* use wasm-build

* new wasm directory layout

* reorganize a bit more

* Fix for rh-minimal-parachain (#141)

* Remove extern "C"

We already encountered such behavior (bug?) in pwasm-std, I believe.

* Fix `panic_fmt` signature by adding `_col`

Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.

* Add linker flags and use wasm-gc in build script

Pass --import-memory to LLD to emit wasm binary with imported memory.

Also use wasm-gc instead of wasm-build.

* Fix effective_max.

I'm not sure why it was the way it was actually.

* Recompile wasm.

* Fix indent

* more basic_add tests

* validate parachain WASM

* produce statements on receiving statements

* tests for reactive statement production

* fix build

* add OOM lang item to runtime-io

* use dynamic_inclusion when evaluating as well

* fix update_includable_count

* remove dead code

* grumbles

* actually defer round_proposer logic

* update wasm

* address a few more grumbles

* grumbles

* update WASM checkins

* remove dependency on tokio-timer
This commit is contained in:
Robert Habermeier
2018-05-25 16:16:01 +02:00
committed by GitHub
parent 59e8adb604
commit c473c0c734
41 changed files with 2769 additions and 872 deletions
+17
View File
@@ -0,0 +1,17 @@
[package]
name = "polkadot-parachain"
version = "0.1.0"
authors = ["Parity Technologies <admin@parity.io>"]
description = "Types and utilities for creating and working with parachains"
[dependencies]
substrate-codec = { path = "../../substrate/codec", default-features = false }
wasmi = { version = "0.1", optional = true }
error-chain = { version = "0.11", optional = true }
[dev-dependencies]
tiny-keccak = "1.4"
[features]
default = ["std"]
std = ["substrate-codec/std", "wasmi", "error-chain"]
+146
View File
@@ -0,0 +1,146 @@
// Copyright 2017 Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Defines primitive types for creating or validating a parachain.
//!
//! When compiled with standard library support, this crate exports a `wasm`
//! module that can be used to validate parachain WASM.
//!
//! ## Parachain WASM
//!
//! Polkadot parachain WASM is in the form of a module which imports a memory
//! instance and exports a function `validate`.
//!
//! `validate` accepts as input two `i32` values, representing a pointer/length pair
//! respectively, that encodes `ValidationParams`.
//!
//! `validate` returns an `i32` which is a pointer to a little-endian 32-bit integer denoting a length.
//! Subtracting the length from the initial pointer will give a new pointer to the actual return data,
//!
//! ASCII-diagram demonstrating the return data format:
//!
//! ```ignore
//! [return data][len (LE-u32)]
//! ^~~returned pointer
//! ```
//!
//! The `load_params` and `write_result` functions provide utilities for setting up
//! a parachain WASM module in Rust.
#![cfg_attr(not(feature = "std"), no_std)]
#![cfg_attr(not(feature = "std"), feature(alloc))]
/// Re-export of substrate-codec.
pub extern crate substrate_codec as codec;
#[cfg(not(feature = "std"))]
extern crate alloc;
#[cfg(feature = "std")]
extern crate core;
#[cfg(feature = "std")]
extern crate wasmi;
#[cfg(feature = "std")]
#[macro_use]
extern crate error_chain;
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
use codec::Slicable;
#[cfg(feature = "std")]
pub mod wasm;
/// Validation parameters for evaluating the parachain validity function.
// TODO: consolidated ingress and balance downloads
#[derive(PartialEq, Eq)]
#[cfg_attr(feature = "std", derive(Debug))]
pub struct ValidationParams {
/// The collation body.
pub block_data: Vec<u8>,
/// Previous head-data.
pub parent_head: Vec<u8>,
}
impl Slicable for ValidationParams {
fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
self.block_data.using_encoded(|s| v.extend(s));
self.parent_head.using_encoded(|s| v.extend(s));
v
}
fn decode<I: codec::Input>(input: &mut I) -> Option<Self> {
Some(ValidationParams {
block_data: Slicable::decode(input)?,
parent_head: Slicable::decode(input)?,
})
}
}
/// The result of parachain validation.
// TODO: egress and balance uploads
#[derive(PartialEq, Eq)]
#[cfg_attr(feature = "std", derive(Debug))]
pub struct ValidationResult {
/// New head data that should be included in the relay chain state.
pub head_data: Vec<u8>
}
impl Slicable for ValidationResult {
fn encode(&self) -> Vec<u8> {
self.head_data.encode()
}
fn decode<I: codec::Input>(input: &mut I) -> Option<Self> {
Some(ValidationResult {
head_data: Slicable::decode(input)?,
})
}
}
/// Load the validation params from memory when implementing a Rust parachain.
///
/// Offset and length must have been provided by the validation
/// function's entry point.
pub unsafe fn load_params(offset: usize, len: usize) -> ValidationParams {
let mut slice = ::core::slice::from_raw_parts(offset as *const u8, len);
ValidationParams::decode(&mut slice).expect("Invalid input data")
}
/// Allocate the validation result in memory, getting the return-pointer back.
///
/// As described in the crate docs, this is a pointer to the appended length
/// of the vector.
pub fn write_result(result: ValidationResult) -> usize {
let mut encoded = result.encode();
let len = encoded.len();
assert!(len <= u32::max_value() as usize, "Len too large for parachain-WASM abi");
(len as u32).using_encoded(|s| encoded.extend(s));
// do not alter `encoded` beyond this point. may reallocate.
let end_ptr = &encoded[len] as *const u8 as usize;
// leak so it doesn't get zeroed.
::core::mem::forget(encoded);
end_ptr
}
+166
View File
@@ -0,0 +1,166 @@
// Copyright 2017 Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! WASM re-execution of a parachain candidate.
//! In the context of relay-chain candidate evaluation, there are some additional
//! steps to ensure that the provided input parameters are correct.
//! Assuming the parameters are correct, this module provides a wrapper around
//! a WASM VM for re-execution of a parachain candidate.
use codec::Slicable;
use wasmi::{self, Module, ModuleInstance, MemoryInstance, MemoryDescriptor, MemoryRef, ModuleImportResolver};
use wasmi::{memory_units, RuntimeValue};
use wasmi::Error as WasmError;
use super::{ValidationParams, ValidationResult};
use std::cell::RefCell;
error_chain! {
types { Error, ErrorKind, ResultExt; }
foreign_links {
Wasm(WasmError);
}
errors {
/// Call data too big. WASM32 only has a 32-bit address space.
ParamsTooLarge(len: usize) {
description("Validation parameters took up too much space to execute in WASM"),
display("Validation parameters took up {} bytes, max allowed by WASM is {}", len, i32::max_value()),
}
/// Bad return data or type.
BadReturn {
description("Validation function returned invalid data."),
display("Validation function returned invalid data."),
}
}
}
struct Resolver {
max_memory: u32, // in pages.
memory: RefCell<Option<MemoryRef>>,
}
impl ModuleImportResolver for Resolver {
fn resolve_memory(
&self,
field_name: &str,
descriptor: &MemoryDescriptor,
) -> Result<MemoryRef, WasmError> {
if field_name == "memory" {
let effective_max = descriptor.maximum().unwrap_or(self.max_memory);
if descriptor.initial() > self.max_memory || effective_max > self.max_memory {
Err(WasmError::Instantiation("Module requested too much memory".to_owned()))
} else {
let mem = MemoryInstance::alloc(
memory_units::Pages(descriptor.initial() as usize),
descriptor.maximum().map(|x| memory_units::Pages(x as usize)),
)?;
*self.memory.borrow_mut() = Some(mem.clone());
Ok(mem)
}
} else {
Err(WasmError::Instantiation("Memory imported under unknown name".to_owned()))
}
}
}
/// Validate a candidate under the given validation code.
///
/// This will fail if the validation code is not a proper parachain validation module.
pub fn validate_candidate(validation_code: &[u8], params: ValidationParams) -> Result<ValidationResult, Error> {
use wasmi::LINEAR_MEMORY_PAGE_SIZE;
// maximum memory in bytes
const MAX_MEM: u32 = 1024 * 1024 * 1024; // 1 GiB
// instantiate the module.
let (module, memory) = {
let module = Module::from_buffer(validation_code)?;
let module_resolver = Resolver {
max_memory: MAX_MEM / LINEAR_MEMORY_PAGE_SIZE.0 as u32,
memory: RefCell::new(None),
};
let module = ModuleInstance::new(
&module,
&wasmi::ImportsBuilder::new().with_resolver("env", &module_resolver),
)?.run_start(&mut wasmi::NopExternals).map_err(WasmError::Trap)?;
let memory = module_resolver.memory.borrow_mut()
.as_ref()
.ok_or_else(|| WasmError::Instantiation("No imported memory instance".to_owned()))?
.clone();
(module, memory)
};
// allocate call data in memory.
let (offset, len) = {
let encoded_call_data = params.encode();
// hard limit from WASM.
if encoded_call_data.len() > i32::max_value() as usize {
bail!(ErrorKind::ParamsTooLarge(encoded_call_data.len()));
}
let call_data_pages = (encoded_call_data.len() / LINEAR_MEMORY_PAGE_SIZE.0) +
(encoded_call_data.len() % LINEAR_MEMORY_PAGE_SIZE.0);
let call_data_pages = wasmi::memory_units::Pages(call_data_pages);
if memory.current_size() < call_data_pages {
memory.grow(call_data_pages - memory.current_size())?;
}
memory.set(0, &encoded_call_data).expect("enough memory allocated just before this; \
copying never fails if memory is large enough; qed");
(0, encoded_call_data.len() as i32)
};
let output = module.invoke_export(
"validate",
&[RuntimeValue::I32(offset), RuntimeValue::I32(len)],
&mut wasmi::NopExternals,
)?;
match output {
Some(RuntimeValue::I32(len_offset)) => {
let len_offset = len_offset as u32;
let mut len_bytes = [0u8; 4];
memory.get_into(len_offset, &mut len_bytes)?;
let len = u32::decode(&mut &len_bytes[..])
.ok_or_else(|| ErrorKind::BadReturn)?;
let return_offset = if len > len_offset {
bail!(ErrorKind::BadReturn);
} else {
len_offset - len
};
// TODO: optimize when `wasmi` lets you inspect memory with a closure.
let raw_return = memory.get(return_offset, len as usize)?;
ValidationResult::decode(&mut &raw_return[..])
.ok_or_else(|| ErrorKind::BadReturn)
.map_err(Into::into)
}
_ => bail!(ErrorKind::BadReturn),
}
}
@@ -0,0 +1,2 @@
target/
Cargo.lock
@@ -0,0 +1,25 @@
[package]
name = "basic_add"
version = "0.1.0"
authors = ["Parity Technologies <admin@parity.io>"]
description = "Test parachain which adds to a number as its state transition"
[lib]
crate-type = ["cdylib"]
[dependencies]
polkadot-parachain = { path = "../../", default-features = false }
wee_alloc = "0.2.0"
tiny-keccak = "1.4"
pwasm-libc = "0.2"
[features]
default = ["std"]
std = ["polkadot-parachain/std"]
[profile.release]
panic = "abort"
lto = true
[workspace]
members = []
@@ -0,0 +1,143 @@
// Copyright 2017 Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Basic parachain that adds a number as part of its state.
#![cfg_attr(not(feature = "std"), no_std)]
#![cfg_attr(not(feature = "std"), feature(alloc, core_intrinsics, global_allocator, lang_items))]
#[cfg(not(feature = "std"))]
extern crate alloc;
extern crate polkadot_parachain as parachain;
extern crate wee_alloc;
extern crate tiny_keccak;
extern crate pwasm_libc;
use parachain::codec::{Slicable, Input};
#[cfg(not(feature = "std"))]
mod wasm;
#[cfg(not(feature = "std"))]
pub use wasm::*;
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
// Define global allocator.
#[cfg(not(feature = "std"))]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
// Head data for this parachain.
#[derive(Default, Clone)]
struct HeadData {
// Block number
number: u64,
// parent block keccak256
parent_hash: [u8; 32],
// hash of post-execution state.
post_state: [u8; 32],
}
impl Slicable for HeadData {
fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
self.number.using_encoded(|s| v.extend(s));
self.parent_hash.using_encoded(|s| v.extend(s));
self.post_state.using_encoded(|s| v.extend(s));
v
}
fn decode<I: Input>(input: &mut I) -> Option<Self> {
Some(HeadData {
number: Slicable::decode(input)?,
parent_hash: Slicable::decode(input)?,
post_state: Slicable::decode(input)?,
})
}
}
// Block data for this parachain.
#[derive(Default, Clone)]
struct BlockData {
// State to begin from.
state: u64,
// Amount to add (overflowing)
add: u64,
}
impl Slicable for BlockData {
fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
self.state.using_encoded(|s| v.extend(s));
self.add.using_encoded(|s| v.extend(s));
v
}
fn decode<I: Input>(input: &mut I) -> Option<Self> {
Some(BlockData {
state: Slicable::decode(input)?,
add: Slicable::decode(input)?,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use parachain::ValidationParams;
const TEST_CODE: &[u8] = include_bytes!("../wasm/test.wasm");
fn hash_state(state: u64) -> [u8; 32] {
::tiny_keccak::keccak256(state.encode().as_slice())
}
fn hash_head(head: &HeadData) -> [u8; 32] {
::tiny_keccak::keccak256(head.encode().as_slice())
}
#[test]
fn execute_good_on_parent() {
let parent_head = HeadData {
number: 0,
parent_hash: [0; 32],
post_state: hash_state(0),
};
let block_data = BlockData {
state: 0,
add: 512,
};
let ret = parachain::wasm::validate_candidate(TEST_CODE, ValidationParams {
parent_head: parent_head.encode(),
block_data: block_data.encode(),
}).unwrap();
let new_head = HeadData::decode(&mut &ret.head_data[..]).unwrap();
assert_eq!(new_head.number, 1);
assert_eq!(new_head.parent_hash, hash_head(&parent_head));
assert_eq!(new_head.post_state, hash_state(512));
}
}
+1
View File
@@ -0,0 +1 @@
src
@@ -0,0 +1,58 @@
// Copyright 2017 Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Defines WASM module logic.
use parachain::{self, ValidationResult};
use parachain::codec::Slicable;
use super::{HeadData, BlockData};
#[lang = "panic_fmt"]
#[no_mangle]
pub extern fn panic_fmt(
_args: ::core::fmt::Arguments,
_file: &'static str,
_line: u32,
_col: u32,
) -> ! {
use core::intrinsics;
unsafe {
intrinsics::abort();
}
}
#[no_mangle]
pub extern fn validate(offset: usize, len: usize) -> usize {
let hash_state = |state: u64| ::tiny_keccak::keccak256(state.encode().as_slice());
let params = unsafe { ::parachain::load_params(offset, len) };
let parent_head = HeadData::decode(&mut &params.parent_head[..])
.expect("invalid parent head format.");
let block_data = BlockData::decode(&mut &params.block_data[..])
.expect("invalid block data format.");
assert_eq!(hash_state(block_data.state), parent_head.post_state, "wrong post-state proof");
let new_state = block_data.state.saturating_add(block_data.add);
let new_head = HeadData {
number: parent_head.number + 1,
parent_hash: ::tiny_keccak::keccak256(&params.parent_head[..]),
post_state: hash_state(new_state),
};
parachain::write_result(ValidationResult { head_data: new_head.encode() })
}
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
set -e
rm -rf ./target
for i in */
do
i=${i%/}
cd $i
# TODO: stop using exact nightly when wee-alloc works on normal nightly.
RUSTFLAGS="-C link-arg=--import-memory" cargo +nightly-2018-03-07 build --target=wasm32-unknown-unknown --release --no-default-features
wasm-gc target/wasm32-unknown-unknown/release/$i.wasm ../../tests/res/$i.wasm
cd ..
done
+170
View File
@@ -0,0 +1,170 @@
// Copyright 2017 Parity Technologies (UK) Ltd.
// This file is part of Polkadot.
// Polkadot is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Polkadot is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! Basic parachain that adds a number as part of its state.
extern crate polkadot_parachain as parachain;
extern crate tiny_keccak;
use parachain::ValidationParams;
use parachain::codec::{Slicable, Input};
// Head data for this parachain.
#[derive(Default, Clone)]
struct HeadData {
// Block number
number: u64,
// parent block keccak256
parent_hash: [u8; 32],
// hash of post-execution state.
post_state: [u8; 32],
}
impl Slicable for HeadData {
fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
self.number.using_encoded(|s| v.extend(s));
self.parent_hash.using_encoded(|s| v.extend(s));
self.post_state.using_encoded(|s| v.extend(s));
v
}
fn decode<I: Input>(input: &mut I) -> Option<Self> {
Some(HeadData {
number: Slicable::decode(input)?,
parent_hash: Slicable::decode(input)?,
post_state: Slicable::decode(input)?,
})
}
}
// Block data for this parachain.
#[derive(Default, Clone)]
struct BlockData {
// State to begin from.
state: u64,
// Amount to add (overflowing)
add: u64,
}
impl Slicable for BlockData {
fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
self.state.using_encoded(|s| v.extend(s));
self.add.using_encoded(|s| v.extend(s));
v
}
fn decode<I: Input>(input: &mut I) -> Option<Self> {
Some(BlockData {
state: Slicable::decode(input)?,
add: Slicable::decode(input)?,
})
}
}
const TEST_CODE: &[u8] = include_bytes!("res/basic_add.wasm");
fn hash_state(state: u64) -> [u8; 32] {
::tiny_keccak::keccak256(state.encode().as_slice())
}
fn hash_head(head: &HeadData) -> [u8; 32] {
::tiny_keccak::keccak256(head.encode().as_slice())
}
#[test]
fn execute_good_on_parent() {
let parent_head = HeadData {
number: 0,
parent_hash: [0; 32],
post_state: hash_state(0),
};
let block_data = BlockData {
state: 0,
add: 512,
};
let ret = parachain::wasm::validate_candidate(TEST_CODE, ValidationParams {
parent_head: parent_head.encode(),
block_data: block_data.encode(),
}).unwrap();
let new_head = HeadData::decode(&mut &ret.head_data[..]).unwrap();
assert_eq!(new_head.number, 1);
assert_eq!(new_head.parent_hash, hash_head(&parent_head));
assert_eq!(new_head.post_state, hash_state(512));
}
#[test]
fn execute_good_chain_on_parent() {
let mut number = 0;
let mut parent_hash = [0; 32];
let mut last_state = 0;
for add in 0..10 {
let parent_head = HeadData {
number,
parent_hash,
post_state: hash_state(last_state),
};
let block_data = BlockData {
state: last_state,
add,
};
let ret = parachain::wasm::validate_candidate(TEST_CODE, ValidationParams {
parent_head: parent_head.encode(),
block_data: block_data.encode(),
}).unwrap();
let new_head = HeadData::decode(&mut &ret.head_data[..]).unwrap();
assert_eq!(new_head.number, number + 1);
assert_eq!(new_head.parent_hash, hash_head(&parent_head));
assert_eq!(new_head.post_state, hash_state(last_state + add));
number += 1;
parent_hash = hash_head(&new_head);
last_state += add;
}
}
#[test]
fn execute_bad_on_parent() {
let parent_head = HeadData {
number: 0,
parent_hash: [0; 32],
post_state: hash_state(0),
};
let block_data = BlockData {
state: 256, // start state is wrong.
add: 256,
};
let _ret = parachain::wasm::validate_candidate(TEST_CODE, ValidationParams {
parent_head: parent_head.encode(),
block_data: block_data.encode(),
}).unwrap_err();
}
Binary file not shown.