Executor Environment parameterization (#6161)

* Re-apply changes without Diener, rebase to the lastest master

* Cache pruning

* Bit-pack InstantiationStrategy

* Move ExecutorParams version inside the structure itself

* Rework runtime API and executor parameters storage

* Pass executor parameters through backing subsystem

* Update Cargo.lock

* Introduce `ExecutorParams` to approval voting subsys

* Introduce `ExecutorParams` to dispute coordinator

* `cargo fmt`

* Simplify requests from backing subsys

* Fix tests

* Replace manual config cloning with `.clone()`

* Move constants to module

* Parametrize executor performing PVF pre-check

* Fix Malus

* Fix test runtime

* Introduce session executor params as a constant defined by session info
pallet

* Use Parity SCALE codec instead of hand-crafted binary encoding

* Get rid of constants; Add docs

* Get rid of constants

* Minor typo

* Fix Malus after rebase

* `cargo fmt`

* Use transparent SCALE encoding instead of explicit

* Clean up

* Get rid of relay parent to session index mapping

* Join environment type and version in a single enum element

* Use default execution parameters if running an old runtime

* `unwrap()` -> `expect()`

* Correct API version

* Constants are back in town

* Use constants for execution environment types

* Artifact separation, first try

* Get rid of explicit version

* PVF execution queue worker separation

* Worker handshake

* Global renaming

* Minor fixes resolving discussions

* Two-stage requesting of executor params to make use of runtime API cache

* Proper error handling in pvf-checker

* Executor params storage bootstrapping

* Propagate migration to v3 network runtimes

* Fix storage versioning

* Ensure `ExecutorParams` serialization determinism; Add comments

* Rename constants to make things a bit more deterministic
Get rid of stale code

* Tidy up a structure of active PVFs

* Minor formatting

* Fix comment

* Add try-runtime hooks

* Add storage version write on upgrade

Co-authored-by: Andronik <write@reusable.software>

* Add pre- and post-upgrade assertions

* Require to specify environment type; Remove redundant `impl`s

* Add `ExecutorParamHash` creation from `H256`

* Fix candidate validation subsys tests

* Return splittable error from executor params request fn

* Revert "Return splittable error from executor params request fn"

This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456.

* Decompose approval voting metrics

* Use more relevant errors

* Minor formatting fix

* Assert a valid environment type instead of checking

* Fix `try-runtime` hooks

* After-merge fixes

* Add migration logs

* Remove dead code

* Fix tests

* Fix tests

* Back to the strongly typed implementation

* Promote strong types to executor interface

* Remove stale comment

* Move executor params to `SessionInfo`: primitives and runtime

* Move executor params to `SessionInfo`: node

* Try to bump primitives and API version

* Get rid of `MallocSizeOf`

* Bump target API version to v4

* Make use of session index already in place

* Back to v3

* Fix all the tests

* Add migrations to all the runtimes

* Make use of existing `SessionInfo` in approval voting subsys

* Rename `TARGET` -> `LOG_TARGET`

* Bump all the primitives to v3

* Fix Rococo ParachainHost API version

* Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes

* Fix nits from discussions; add comments

* Re-evaluate queue logic

* Rework job assignment in execution queue

* Add documentation

* Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching)

* Couple `Pvf` with `ExecutorParams` wherever possible

* Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning

* Fix comment

* Fix CI tests

* Fix clippy warnings

* Address nits from discussions

* Add a placeholder for raw data

* Fix non exhaustive match

* Remove redundant reexports and fix imports

* Keep only necessary semantic features, as discussed

* Rework `RuntimeInfo` to support mock implementation for tests

* Remove unneeded bound

* `cargo fmt`

* Revert "Remove unneeded bound"

This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61.

* Fix PVF host tests

* Fix PVF checker tests

* Fix overseer declarations

* Simplify tests

* `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT`

* Add a unit test for varying executor parameters

* Minor fixes from discussions

* Add prechecking max. memory parameter (see paritytech/srlabs_findings#110)

* Fix and improve a test

* Remove `ExecutionEnvironment` and `RawData`

* New primitives versioning in parachain host API

* `disputes()` implementation for Kusama and Polkadot

* Move `ExecutorParams` from `vstaging` to stable primitives

* Move disputes from `vstaging` to stable implementation

* Fix `try-runtime`

* Fixes after merge

* Move `ExecutorParams` to the bottom of `SessionInfo`

* Revert "Move executor params to `SessionInfo`: primitives and runtime"

This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce.

* Always use fresh activated live hash in pvf precheck
(re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa)

* Fixing tests (broken commit)

* Fix candidate validation tests

* Fix PVF host test

* Minor fixes

* Address discussions

* Restore migration

* Fix `use` to only include what is needed instead of `*`

* Add comment to never touch `DEFAULT_CONFIG`

* Update migration to set default `ExecutorParams` for `dispute_period`
sessions back

* Use `earliest_stored_session` instead of calculations

* Nit

* Add logs

* Treat any runtime error as `NotSupported` again

* Always return default executor params if not available

* Revert "Always return default executor params if not available"

This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30.

* Add paritytech/substrate#9997 workaround

* `cargo fmt`

* Remove migration (again!)

* Bump executor params to API v4 (backport from #6698)

---------

Co-authored-by: Andronik <write@reusable.software>
This commit is contained in:
s0me0ne-unkn0wn
2023-02-15 12:26:09 +01:00
committed by GitHub
parent 7f6b8e6df9
commit dd0a556665
40 changed files with 1243 additions and 330 deletions
+21 -13
View File
@@ -17,6 +17,7 @@
use crate::{error::PrepareError, host::PrepareResultSender, prepare::PrepareStats};
use always_assert::always;
use polkadot_parachain::primitives::ValidationCodeHash;
use polkadot_primitives::vstaging::ExecutorParamsHash;
use std::{
collections::HashMap,
path::{Path, PathBuf},
@@ -37,19 +38,19 @@ impl AsRef<[u8]> for CompiledArtifact {
}
}
/// Identifier of an artifact. Right now it only encodes a code hash of the PVF. But if we get to
/// multiple engine implementations the artifact ID should include the engine type as well.
/// Identifier of an artifact. Encodes a code hash of the PVF and a hash of executor parameter set.
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct ArtifactId {
pub(crate) code_hash: ValidationCodeHash,
pub(crate) executor_params_hash: ExecutorParamsHash,
}
impl ArtifactId {
const PREFIX: &'static str = "wasmtime_";
/// Creates a new artifact ID with the given hash.
pub fn new(code_hash: ValidationCodeHash) -> Self {
Self { code_hash }
pub fn new(code_hash: ValidationCodeHash, executor_params_hash: ExecutorParamsHash) -> Self {
Self { code_hash, executor_params_hash }
}
/// Tries to recover the artifact id from the given file name.
@@ -59,14 +60,18 @@ impl ArtifactId {
use std::str::FromStr as _;
let file_name = file_name.strip_prefix(Self::PREFIX)?;
let code_hash = Hash::from_str(file_name).ok()?.into();
let (code_hash_str, executor_params_hash_str) = file_name.split_once('_')?;
let code_hash = Hash::from_str(code_hash_str).ok()?.into();
let executor_params_hash =
ExecutorParamsHash::from_hash(Hash::from_str(executor_params_hash_str).ok()?);
Some(Self { code_hash })
Some(Self { code_hash, executor_params_hash })
}
/// Returns the expected path to this artifact given the root of the cache.
pub fn path(&self, cache_path: &Path) -> PathBuf {
let file_name = format!("{}{:#x}", Self::PREFIX, self.code_hash);
let file_name =
format!("{}{:#x}_{:#x}", Self::PREFIX, self.code_hash, self.executor_params_hash);
cache_path.join(file_name)
}
}
@@ -214,6 +219,7 @@ impl Artifacts {
#[cfg(test)]
mod tests {
use super::{ArtifactId, Artifacts};
use polkadot_primitives::vstaging::ExecutorParamsHash;
use sp_core::H256;
use std::{path::Path, str::FromStr};
@@ -224,13 +230,16 @@ mod tests {
assert_eq!(
ArtifactId::from_file_name(
"wasmtime_0x0022800000000000000000000000000000000000000000000000000000000000"
"wasmtime_0x0022800000000000000000000000000000000000000000000000000000000000_0x0033900000000000000000000000000000000000000000000000000000000000"
),
Some(ArtifactId::new(
hex_literal::hex![
"0022800000000000000000000000000000000000000000000000000000000000"
]
.into()
.into(),
ExecutorParamsHash::from_hash(sp_core::H256(hex_literal::hex![
"0033900000000000000000000000000000000000000000000000000000000000"
])),
)),
);
}
@@ -240,13 +249,12 @@ mod tests {
let path = Path::new("/test");
let hash =
H256::from_str("1234567890123456789012345678901234567890123456789012345678901234")
.unwrap()
.into();
.unwrap();
assert_eq!(
ArtifactId::new(hash).path(path).to_str(),
ArtifactId::new(hash.into(), ExecutorParamsHash::from_hash(hash)).path(path).to_str(),
Some(
"/test/wasmtime_0x1234567890123456789012345678901234567890123456789012345678901234"
"/test/wasmtime_0x1234567890123456789012345678901234567890123456789012345678901234_0x1234567890123456789012345678901234567890123456789012345678901234"
),
);
}
+149 -38
View File
@@ -30,8 +30,23 @@ use futures::{
stream::{FuturesUnordered, StreamExt as _},
Future, FutureExt,
};
use polkadot_node_primitives::BACKING_EXECUTION_TIMEOUT;
use polkadot_primitives::vstaging::{ExecutorParams, ExecutorParamsHash};
use slotmap::HopSlotMap;
use std::{collections::VecDeque, fmt, path::PathBuf, time::Duration};
use std::{
collections::VecDeque,
fmt,
path::PathBuf,
time::{Duration, Instant},
};
/// The amount of time a job for which the queue does not have a compatible worker may wait in the
/// queue. After that time passes, the queue will kill the first worker which becomes idle to
/// re-spawn a new worker to execute the job immediately.
/// To make any sense and not to break things, the value should be greater than minimal execution
/// timeout in use, and less than the block time.
const MAX_KEEP_WAITING: Duration =
Duration::from_millis(BACKING_EXECUTION_TIMEOUT.as_millis() as u64 * 2);
slotmap::new_key_type! { struct Worker; }
@@ -41,6 +56,7 @@ pub enum ToQueue {
artifact: ArtifactPathId,
execution_timeout: Duration,
params: Vec<u8>,
executor_params: ExecutorParams,
result_tx: ResultSender,
},
}
@@ -49,12 +65,15 @@ struct ExecuteJob {
artifact: ArtifactPathId,
execution_timeout: Duration,
params: Vec<u8>,
executor_params: ExecutorParams,
result_tx: ResultSender,
waiting_since: Instant,
}
struct WorkerData {
idle: Option<IdleWorker>,
handle: WorkerHandle,
executor_params_hash: ExecutorParamsHash,
}
impl fmt::Debug for WorkerData {
@@ -79,7 +98,17 @@ impl Workers {
self.spawn_inflight + self.running.len() < self.capacity
}
fn find_available(&self) -> Option<Worker> {
fn find_available(&self, executor_params_hash: ExecutorParamsHash) -> Option<Worker> {
self.running.iter().find_map(|d| {
if d.1.idle.is_some() && d.1.executor_params_hash == executor_params_hash {
Some(d.0)
} else {
None
}
})
}
fn find_idle(&self) -> Option<Worker> {
self.running
.iter()
.find_map(|d| if d.1.idle.is_some() { Some(d.0) } else { None })
@@ -94,7 +123,7 @@ impl Workers {
}
enum QueueEvent {
Spawn(IdleWorker, WorkerHandle),
Spawn(IdleWorker, WorkerHandle, ExecuteJob),
StartWork(Worker, Outcome, ArtifactId, ResultSender),
}
@@ -154,6 +183,66 @@ impl Queue {
purge_dead(&self.metrics, &mut self.workers).await;
}
}
/// Tries to assign a job in the queue to a worker. If an idle worker is provided, it does its
/// best to find a job with a compatible execution environment unless there are jobs in the
/// queue waiting too long. In that case, it kills an existing idle worker and spawns a new
/// one. It may spawn an additional worker if that is affordable.
/// If all the workers are busy or the queue is empty, it does nothing.
/// Should be called every time a new job arrives to the queue or a job finishes.
fn try_assign_next_job(&mut self, finished_worker: Option<Worker>) {
// New jobs are always pushed to the tail of the queue; the one at its head is always
// the eldest one.
let eldest = if let Some(eldest) = self.queue.get(0) { eldest } else { return };
// By default, we're going to execute the eldest job on any worker slot available, even if
// we have to kill and re-spawn a worker
let mut worker = None;
let mut job_index = 0;
// But if we're not pressed for time, we can try to find a better job-worker pair not
// requiring the expensive kill-spawn operation
if eldest.waiting_since.elapsed() < MAX_KEEP_WAITING {
if let Some(finished_worker) = finished_worker {
if let Some(worker_data) = self.workers.running.get(finished_worker) {
for (i, job) in self.queue.iter().enumerate() {
if worker_data.executor_params_hash == job.executor_params.hash() {
(worker, job_index) = (Some(finished_worker), i);
break
}
}
}
}
}
if worker.is_none() {
// Try to obtain a worker for the job
worker = self.workers.find_available(self.queue[job_index].executor_params.hash());
}
if worker.is_none() {
if let Some(idle) = self.workers.find_idle() {
// No available workers of required type but there are some idle ones of other
// types, have to kill one and re-spawn with the correct type
if self.workers.running.remove(idle).is_some() {
self.metrics.execute_worker().on_retired();
}
}
}
if worker.is_none() && !self.workers.can_afford_one_more() {
// Bad luck, no worker slot can be used to execute the job
return
}
let job = self.queue.remove(job_index).expect("Job is just checked to be in queue; qed");
if let Some(worker) = worker {
assign(self, worker, job);
} else {
spawn_extra_worker(self, job);
}
}
}
async fn purge_dead(metrics: &Metrics, workers: &mut Workers) {
@@ -172,29 +261,30 @@ async fn purge_dead(metrics: &Metrics, workers: &mut Workers) {
}
fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) {
let ToQueue::Enqueue { artifact, execution_timeout, params, result_tx } = to_queue;
let ToQueue::Enqueue { artifact, execution_timeout, params, executor_params, result_tx } =
to_queue;
gum::debug!(
target: LOG_TARGET,
validation_code_hash = ?artifact.id.code_hash,
"enqueueing an artifact for execution",
);
queue.metrics.execute_enqueued();
let job = ExecuteJob { artifact, execution_timeout, params, result_tx };
if let Some(available) = queue.workers.find_available() {
assign(queue, available, job);
} else {
if queue.workers.can_afford_one_more() {
spawn_extra_worker(queue);
}
queue.queue.push_back(job);
}
let job = ExecuteJob {
artifact,
execution_timeout,
params,
executor_params,
result_tx,
waiting_since: Instant::now(),
};
queue.queue.push_back(job);
queue.try_assign_next_job(None);
}
async fn handle_mux(queue: &mut Queue, event: QueueEvent) {
match event {
QueueEvent::Spawn(idle, handle) => {
handle_worker_spawned(queue, idle, handle);
QueueEvent::Spawn(idle, handle, job) => {
handle_worker_spawned(queue, idle, handle, job);
},
QueueEvent::StartWork(worker, outcome, artifact_id, result_tx) => {
handle_job_finish(queue, worker, outcome, artifact_id, result_tx);
@@ -202,16 +292,23 @@ async fn handle_mux(queue: &mut Queue, event: QueueEvent) {
}
}
fn handle_worker_spawned(queue: &mut Queue, idle: IdleWorker, handle: WorkerHandle) {
fn handle_worker_spawned(
queue: &mut Queue,
idle: IdleWorker,
handle: WorkerHandle,
job: ExecuteJob,
) {
queue.metrics.execute_worker().on_spawned();
queue.workers.spawn_inflight -= 1;
let worker = queue.workers.running.insert(WorkerData { idle: Some(idle), handle });
let worker = queue.workers.running.insert(WorkerData {
idle: Some(idle),
handle,
executor_params_hash: job.executor_params.hash(),
});
gum::debug!(target: LOG_TARGET, ?worker, "execute worker spawned");
if let Some(job) = queue.queue.pop_front() {
assign(queue, worker, job);
}
assign(queue, worker, job);
}
/// If there are pending jobs in the queue, schedules the next of them onto the just freed up
@@ -280,42 +377,45 @@ fn handle_job_finish(
if let Some(idle_worker) = idle_worker {
if let Some(data) = queue.workers.running.get_mut(worker) {
data.idle = Some(idle_worker);
if let Some(job) = queue.queue.pop_front() {
assign(queue, worker, job);
}
return queue.try_assign_next_job(Some(worker))
}
} else {
// Note it's possible that the worker was purged already by `purge_dead`
if queue.workers.running.remove(worker).is_some() {
queue.metrics.execute_worker().on_retired();
}
if !queue.queue.is_empty() {
// The worker has died and we still have work we have to do. Request an extra worker.
//
// That can potentially overshoot, but that should be OK.
spawn_extra_worker(queue);
}
}
queue.try_assign_next_job(None);
}
fn spawn_extra_worker(queue: &mut Queue) {
fn spawn_extra_worker(queue: &mut Queue, job: ExecuteJob) {
queue.metrics.execute_worker().on_begin_spawn();
gum::debug!(target: LOG_TARGET, "spawning an extra worker");
queue
.mux
.push(spawn_worker_task(queue.program_path.clone(), queue.spawn_timeout).boxed());
.push(spawn_worker_task(queue.program_path.clone(), job, queue.spawn_timeout).boxed());
queue.workers.spawn_inflight += 1;
}
async fn spawn_worker_task(program_path: PathBuf, spawn_timeout: Duration) -> QueueEvent {
/// Spawns a new worker to execute a pre-assigned job.
/// A worker is never spawned as idle; a job to be executed by the worker has to be determined
/// beforehand. In such a way, a race condition is avoided: during the worker being spawned,
/// another job in the queue, with an incompatible execution environment, may become stale, and
/// the queue would have to kill a newly started worker and spawn another one.
/// Nevertheless, if the worker finishes executing the job, it becomes idle and may be used to execute other jobs with a compatible execution environment.
async fn spawn_worker_task(
program_path: PathBuf,
job: ExecuteJob,
spawn_timeout: Duration,
) -> QueueEvent {
use futures_timer::Delay;
loop {
match super::worker::spawn(&program_path, spawn_timeout).await {
Ok((idle, handle)) => break QueueEvent::Spawn(idle, handle),
match super::worker::spawn(&program_path, job.executor_params.clone(), spawn_timeout).await
{
Ok((idle, handle)) => break QueueEvent::Spawn(idle, handle, job),
Err(err) => {
gum::warn!(target: LOG_TARGET, "failed to spawn an execute worker: {:?}", err);
@@ -328,7 +428,8 @@ async fn spawn_worker_task(program_path: PathBuf, spawn_timeout: Duration) -> Qu
/// Ask the given worker to perform the given job.
///
/// The worker must be running and idle.
/// The worker must be running and idle. The job and the worker must share the same execution
/// environment parameter set.
fn assign(queue: &mut Queue, worker: Worker, job: ExecuteJob) {
gum::debug!(
target: LOG_TARGET,
@@ -337,6 +438,16 @@ fn assign(queue: &mut Queue, worker: Worker, job: ExecuteJob) {
"assigning the execute worker",
);
debug_assert_eq!(
queue
.workers
.running
.get(worker)
.expect("caller must provide existing worker; qed")
.executor_params_hash,
job.executor_params.hash()
);
let idle = queue.workers.claim_idle(worker).expect(
"this caller must supply a worker which is idle and running;
thus claim_idle cannot return None;
+42 -2
View File
@@ -28,7 +28,9 @@ use cpu_time::ProcessTime;
use futures::{pin_mut, select_biased, FutureExt};
use futures_timer::Delay;
use parity_scale_codec::{Decode, Encode};
use polkadot_parachain::primitives::ValidationResult;
use polkadot_primitives::vstaging::ExecutorParams;
use std::{
path::{Path, PathBuf},
sync::{mpsc::channel, Arc},
@@ -37,13 +39,29 @@ use std::{
use tokio::{io, net::UnixStream};
/// Spawns a new worker with the given program path that acts as the worker and the spawn timeout.
/// Sends a handshake message to the worker as soon as it is spawned.
///
/// The program should be able to handle `<program-path> execute-worker <socket-path>` invocation.
pub async fn spawn(
program_path: &Path,
executor_params: ExecutorParams,
spawn_timeout: Duration,
) -> Result<(IdleWorker, WorkerHandle), SpawnErr> {
spawn_with_program_path("execute", program_path, &["execute-worker"], spawn_timeout).await
let (mut idle_worker, worker_handle) =
spawn_with_program_path("execute", program_path, &["execute-worker"], spawn_timeout)
.await?;
send_handshake(&mut idle_worker.stream, Handshake { executor_params })
.await
.map_err(|error| {
gum::warn!(
target: LOG_TARGET,
worker_pid = %idle_worker.pid,
?error,
"failed to send a handshake to the spawned worker",
);
SpawnErr::Handshake
})?;
Ok((idle_worker, worker_handle))
}
/// Outcome of PVF execution.
@@ -159,6 +177,21 @@ pub async fn start_work(
}
}
async fn send_handshake(stream: &mut UnixStream, handshake: Handshake) -> io::Result<()> {
framed_send(stream, &handshake.encode()).await
}
async fn recv_handshake(stream: &mut UnixStream) -> io::Result<Handshake> {
let handshake_enc = framed_recv(stream).await?;
let handshake = Handshake::decode(&mut &handshake_enc[..]).map_err(|_| {
io::Error::new(
io::ErrorKind::Other,
"execute pvf recv_handshake: failed to decode Handshake".to_owned(),
)
})?;
Ok(handshake)
}
async fn send_request(
stream: &mut UnixStream,
artifact_path: &Path,
@@ -203,6 +236,11 @@ async fn recv_response(stream: &mut UnixStream) -> io::Result<Response> {
})
}
#[derive(Encode, Decode)]
struct Handshake {
executor_params: ExecutorParams,
}
#[derive(Encode, Decode)]
pub enum Response {
Ok { result_descriptor: ValidationResult, duration: Duration },
@@ -225,7 +263,9 @@ impl Response {
/// the path to the socket used to communicate with the host.
pub fn worker_entrypoint(socket_path: &str) {
worker_event_loop("execute", socket_path, |rt_handle, mut stream| async move {
let executor = Arc::new(Executor::new().map_err(|e| {
let handshake = recv_handshake(&mut stream).await?;
let executor = Arc::new(Executor::new(handshake.executor_params).map_err(|e| {
io::Error::new(io::ErrorKind::Other, format!("cannot create executor: {}", e))
})?);
+41 -7
View File
@@ -16,6 +16,7 @@
//! Interface to the Substrate Executor
use polkadot_primitives::vstaging::executor_params::{ExecutorParam, ExecutorParams};
use sc_executor_common::{
runtime_blob::RuntimeBlob,
wasm_runtime::{InvokeMethod, WasmModule as _},
@@ -46,7 +47,11 @@ const EXTRA_HEAP_PAGES: u64 = 2048;
/// The number of bytes devoted for the stack during wasm execution of a PVF.
const NATIVE_STACK_MAX: u32 = 256 * 1024 * 1024;
const CONFIG: Config = Config {
// VALUES OF THE DEFAULT CONFIGURATION SHOULD NEVER BE CHANGED
// They are used as base values for the execution environment parametrization.
// To overwrite them, add new ones to `EXECUTOR_PARAMS` in the `session_info` pallet and perform
// a runtime upgrade to make them active.
const DEFAULT_CONFIG: Config = Config {
allow_missing_func_imports: true,
cache_path: None,
semantics: Semantics {
@@ -97,17 +102,42 @@ pub fn prevalidate(code: &[u8]) -> Result<RuntimeBlob, sc_executor_common::error
/// Runs preparation on the given runtime blob. If successful, it returns a serialized compiled
/// artifact which can then be used to pass into `Executor::execute` after writing it to the disk.
pub fn prepare(blob: RuntimeBlob) -> Result<Vec<u8>, sc_executor_common::error::WasmError> {
sc_executor_wasmtime::prepare_runtime_artifact(blob, &CONFIG.semantics)
pub fn prepare(
blob: RuntimeBlob,
executor_params: ExecutorParams,
) -> Result<Vec<u8>, sc_executor_common::error::WasmError> {
let semantics = params_to_wasmtime_semantics(executor_params)
.map_err(|e| sc_executor_common::error::WasmError::Other(e))?;
sc_executor_wasmtime::prepare_runtime_artifact(blob, &semantics)
}
fn params_to_wasmtime_semantics(par: ExecutorParams) -> Result<Semantics, String> {
let mut sem = DEFAULT_CONFIG.semantics.clone();
let mut stack_limit = if let Some(stack_limit) = sem.deterministic_stack_limit.clone() {
stack_limit
} else {
return Err("No default stack limit set".to_owned())
};
for p in par.iter() {
match p {
ExecutorParam::MaxMemorySize(mms) => sem.max_memory_size = Some(*mms as usize),
ExecutorParam::StackLogicalMax(slm) => stack_limit.logical_max = *slm,
ExecutorParam::StackNativeMax(snm) => stack_limit.native_stack_max = *snm,
ExecutorParam::PrecheckingMaxMemory(_) => (), // TODO: Not implemented yet
}
}
sem.deterministic_stack_limit = Some(stack_limit);
Ok(sem)
}
pub struct Executor {
thread_pool: rayon::ThreadPool,
spawner: TaskSpawner,
config: Config,
}
impl Executor {
pub fn new() -> Result<Self, String> {
pub fn new(params: ExecutorParams) -> Result<Self, String> {
// Wasmtime powers the Substrate Executor. It compiles the wasm bytecode into native code.
// That native code does not create any stacks and just reuses the stack of the thread that
// wasmtime was invoked from.
@@ -154,7 +184,10 @@ impl Executor {
let spawner =
TaskSpawner::new().map_err(|e| format!("cannot create task spawner: {}", e))?;
Ok(Self { thread_pool, spawner })
let mut config = DEFAULT_CONFIG.clone();
config.semantics = params_to_wasmtime_semantics(params)?;
Ok(Self { thread_pool, spawner, config })
}
/// Executes the given PVF in the form of a compiled artifact and returns the result of execution
@@ -183,7 +216,7 @@ impl Executor {
s.spawn(move |_| {
// spawn does not return a value, so we need to use a variable to pass the result.
*result = Some(
do_execute(compiled_artifact_path, params, spawner)
do_execute(compiled_artifact_path, self.config.clone(), params, spawner)
.map_err(|err| format!("execute error: {:?}", err)),
);
});
@@ -195,6 +228,7 @@ impl Executor {
unsafe fn do_execute(
compiled_artifact_path: &Path,
config: Config,
params: &[u8],
spawner: impl sp_core::traits::SpawnNamed + 'static,
) -> Result<Vec<u8>, sc_executor_common::error::Error> {
@@ -208,7 +242,7 @@ unsafe fn do_execute(
sc_executor::with_externalities_safe(&mut ext, || {
let runtime = sc_executor_wasmtime::create_runtime_from_artifact::<HostFunctions>(
compiled_artifact_path,
CONFIG,
config,
)?;
runtime.new_instance()?.call(InvokeMethod::Export("validate_block"), params)
})?
+91 -51
View File
@@ -25,7 +25,7 @@ use crate::{
error::PrepareError,
execute,
metrics::Metrics,
prepare, PrepareResult, Priority, Pvf, ValidationError, LOG_TARGET,
prepare, PrepareResult, Priority, PvfWithExecutorParams, ValidationError, LOG_TARGET,
};
use always_assert::never;
use futures::{
@@ -33,6 +33,7 @@ use futures::{
Future, FutureExt, SinkExt, StreamExt,
};
use polkadot_parachain::primitives::ValidationResult;
use polkadot_primitives::vstaging::ExecutorParams;
use std::{
collections::HashMap,
path::{Path, PathBuf},
@@ -83,11 +84,11 @@ impl ValidationHost {
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
pub async fn precheck_pvf(
&mut self,
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
result_tx: PrepareResultSender,
) -> Result<(), String> {
self.to_host_tx
.send(ToHost::PrecheckPvf { pvf, result_tx })
.send(ToHost::PrecheckPvf { pvf_with_params, result_tx })
.await
.map_err(|_| "the inner loop hung up".to_string())
}
@@ -101,7 +102,7 @@ impl ValidationHost {
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
pub async fn execute_pvf(
&mut self,
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
execution_timeout: Duration,
params: Vec<u8>,
priority: Priority,
@@ -109,7 +110,7 @@ impl ValidationHost {
) -> Result<(), String> {
self.to_host_tx
.send(ToHost::ExecutePvf(ExecutePvfInputs {
pvf,
pvf_with_params,
execution_timeout,
params,
priority,
@@ -125,7 +126,10 @@ impl ValidationHost {
/// situations this function should return immediately.
///
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
pub async fn heads_up(&mut self, active_pvfs: Vec<Pvf>) -> Result<(), String> {
pub async fn heads_up(
&mut self,
active_pvfs: Vec<PvfWithExecutorParams>,
) -> Result<(), String> {
self.to_host_tx
.send(ToHost::HeadsUp { active_pvfs })
.await
@@ -134,13 +138,13 @@ impl ValidationHost {
}
enum ToHost {
PrecheckPvf { pvf: Pvf, result_tx: PrepareResultSender },
PrecheckPvf { pvf_with_params: PvfWithExecutorParams, result_tx: PrepareResultSender },
ExecutePvf(ExecutePvfInputs),
HeadsUp { active_pvfs: Vec<Pvf> },
HeadsUp { active_pvfs: Vec<PvfWithExecutorParams> },
}
struct ExecutePvfInputs {
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
execution_timeout: Duration,
params: Vec<u8>,
priority: Priority,
@@ -265,6 +269,7 @@ pub fn start(config: Config, metrics: Metrics) -> (ValidationHost, impl Future<O
struct PendingExecutionRequest {
execution_timeout: Duration,
params: Vec<u8>,
executor_params: ExecutorParams,
result_tx: ResultSender,
}
@@ -279,11 +284,13 @@ impl AwaitingPrepare {
artifact_id: ArtifactId,
execution_timeout: Duration,
params: Vec<u8>,
executor_params: ExecutorParams,
result_tx: ResultSender,
) {
self.0.entry(artifact_id).or_default().push(PendingExecutionRequest {
execution_timeout,
params,
executor_params,
result_tx,
});
}
@@ -420,8 +427,8 @@ async fn handle_to_host(
to_host: ToHost,
) -> Result<(), Fatal> {
match to_host {
ToHost::PrecheckPvf { pvf, result_tx } => {
handle_precheck_pvf(artifacts, prepare_queue, pvf, result_tx).await?;
ToHost::PrecheckPvf { pvf_with_params, result_tx } => {
handle_precheck_pvf(artifacts, prepare_queue, pvf_with_params, result_tx).await?;
},
ToHost::ExecutePvf(inputs) => {
handle_execute_pvf(
@@ -449,10 +456,10 @@ async fn handle_to_host(
async fn handle_precheck_pvf(
artifacts: &mut Artifacts,
prepare_queue: &mut mpsc::Sender<prepare::ToQueue>,
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
result_sender: PrepareResultSender,
) -> Result<(), Fatal> {
let artifact_id = pvf.as_artifact_id();
let artifact_id = pvf_with_params.as_artifact_id();
if let Some(state) = artifacts.artifact_state_mut(&artifact_id) {
match state {
@@ -474,7 +481,7 @@ async fn handle_precheck_pvf(
prepare_queue,
prepare::ToQueue::Enqueue {
priority: Priority::Normal,
pvf,
pvf_with_params,
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
},
)
@@ -500,8 +507,9 @@ async fn handle_execute_pvf(
awaiting_prepare: &mut AwaitingPrepare,
inputs: ExecutePvfInputs,
) -> Result<(), Fatal> {
let ExecutePvfInputs { pvf, execution_timeout, params, priority, result_tx } = inputs;
let artifact_id = pvf.as_artifact_id();
let ExecutePvfInputs { pvf_with_params, execution_timeout, params, priority, result_tx } =
inputs;
let artifact_id = pvf_with_params.as_artifact_id();
if let Some(state) = artifacts.artifact_state_mut(&artifact_id) {
match state {
@@ -515,19 +523,26 @@ async fn handle_execute_pvf(
artifact: ArtifactPathId::new(artifact_id, cache_path),
execution_timeout,
params,
executor_params: pvf_with_params.executor_params(),
result_tx,
},
)
.await?;
},
ArtifactState::Preparing { .. } => {
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
awaiting_prepare.add(
artifact_id,
execution_timeout,
params,
pvf_with_params.executor_params(),
result_tx,
);
},
ArtifactState::FailedToProcess { last_time_failed, num_failures, error } => {
if can_retry_prepare_after_failure(*last_time_failed, *num_failures, error) {
gum::warn!(
target: LOG_TARGET,
?pvf,
?pvf_with_params,
?artifact_id,
?last_time_failed,
%num_failures,
@@ -541,11 +556,12 @@ async fn handle_execute_pvf(
waiting_for_response: Vec::new(),
num_failures: *num_failures,
};
let executor_params = pvf_with_params.executor_params().clone();
send_prepare(
prepare_queue,
prepare::ToQueue::Enqueue {
priority,
pvf,
pvf_with_params,
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
},
)
@@ -553,7 +569,13 @@ async fn handle_execute_pvf(
// Add an execution request that will wait to run after this prepare job has
// finished.
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
awaiting_prepare.add(
artifact_id,
execution_timeout,
params,
executor_params,
result_tx,
);
} else {
let _ = result_tx.send(Err(ValidationError::from(error.clone())));
}
@@ -562,19 +584,20 @@ async fn handle_execute_pvf(
} else {
// Artifact is unknown: register it and enqueue a job with the corresponding priority and
// PVF.
let executor_params = pvf_with_params.executor_params();
artifacts.insert_preparing(artifact_id.clone(), Vec::new());
send_prepare(
prepare_queue,
prepare::ToQueue::Enqueue {
priority,
pvf,
pvf_with_params,
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
},
)
.await?;
// Add an execution request that will wait to run after this prepare job has finished.
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
awaiting_prepare.add(artifact_id, execution_timeout, params, executor_params, result_tx);
}
Ok(())
@@ -583,7 +606,7 @@ async fn handle_execute_pvf(
async fn handle_heads_up(
artifacts: &mut Artifacts,
prepare_queue: &mut mpsc::Sender<prepare::ToQueue>,
active_pvfs: Vec<Pvf>,
active_pvfs: Vec<PvfWithExecutorParams>,
) -> Result<(), Fatal> {
let now = SystemTime::now();
@@ -619,7 +642,7 @@ async fn handle_heads_up(
prepare_queue,
prepare::ToQueue::Enqueue {
priority: Priority::Normal,
pvf: active_pvf,
pvf_with_params: active_pvf,
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
},
)
@@ -635,7 +658,7 @@ async fn handle_heads_up(
prepare_queue,
prepare::ToQueue::Enqueue {
priority: Priority::Normal,
pvf: active_pvf,
pvf_with_params: active_pvf,
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
},
)
@@ -699,7 +722,9 @@ async fn handle_prepare_done(
// It's finally time to dispatch all the execution requests that were waiting for this artifact
// to be prepared.
let pending_requests = awaiting_prepare.take(&artifact_id);
for PendingExecutionRequest { execution_timeout, params, result_tx } in pending_requests {
for PendingExecutionRequest { execution_timeout, params, executor_params, result_tx } in
pending_requests
{
if result_tx.is_canceled() {
// Preparation could've taken quite a bit of time and the requester may be not interested
// in execution anymore, in which case we just skip the request.
@@ -718,6 +743,7 @@ async fn handle_prepare_done(
artifact: ArtifactPathId::new(artifact_id.clone(), cache_path),
execution_timeout,
params,
executor_params,
result_tx,
},
)
@@ -856,7 +882,7 @@ mod tests {
/// Creates a new PVF which artifact id can be uniquely identified by the given number.
fn artifact_id(descriminator: u32) -> ArtifactId {
Pvf::from_discriminator(descriminator).as_artifact_id()
PvfWithExecutorParams::from_discriminator(descriminator).as_artifact_id()
}
fn artifact_path(descriminator: u32) -> PathBuf {
@@ -1065,7 +1091,7 @@ mod tests {
let mut test = builder.build();
let mut host = test.host_handle();
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
let to_sweeper_rx = &mut test.to_sweeper_rx;
run_until(
@@ -1079,7 +1105,7 @@ mod tests {
// Extend TTL for the first artifact and make sure we don't receive another file removal
// request.
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
test.poll_ensure_to_sweeper_is_empty().await;
}
@@ -1090,7 +1116,7 @@ mod tests {
let (result_tx, result_rx_pvf_1_1) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf1".to_vec(),
Priority::Normal,
@@ -1101,7 +1127,7 @@ mod tests {
let (result_tx, result_rx_pvf_1_2) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf1".to_vec(),
Priority::Critical,
@@ -1112,7 +1138,7 @@ mod tests {
let (result_tx, result_rx_pvf_2) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(2),
PvfWithExecutorParams::from_discriminator(2),
TEST_EXECUTION_TIMEOUT,
b"pvf2".to_vec(),
Priority::Normal,
@@ -1190,7 +1216,9 @@ mod tests {
// First, test a simple precheck request.
let (result_tx, result_rx) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
.await
.unwrap();
// The queue received the prepare request.
assert_matches!(
@@ -1214,7 +1242,9 @@ mod tests {
let mut precheck_receivers = Vec::new();
for _ in 0..3 {
let (result_tx, result_rx) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(2), result_tx).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(2), result_tx)
.await
.unwrap();
precheck_receivers.push(result_rx);
}
// Received prepare request.
@@ -1249,7 +1279,7 @@ mod tests {
// Send PVF for the execution and request the prechecking for it.
let (result_tx, result_rx_execute) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf2".to_vec(),
Priority::Critical,
@@ -1264,7 +1294,9 @@ mod tests {
);
let (result_tx, result_rx) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
.await
.unwrap();
// Suppose the preparation failed, the execution queue is empty and both
// "clients" receive their results.
@@ -1286,13 +1318,15 @@ mod tests {
let mut precheck_receivers = Vec::new();
for _ in 0..3 {
let (result_tx, result_rx) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(2), result_tx).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(2), result_tx)
.await
.unwrap();
precheck_receivers.push(result_rx);
}
let (result_tx, _result_rx_execute) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(2),
PvfWithExecutorParams::from_discriminator(2),
TEST_EXECUTION_TIMEOUT,
b"pvf2".to_vec(),
Priority::Critical,
@@ -1332,7 +1366,9 @@ mod tests {
// Submit a precheck request that fails.
let (result_tx, result_rx) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
.await
.unwrap();
// The queue received the prepare request.
assert_matches!(
@@ -1354,7 +1390,9 @@ mod tests {
// Submit another precheck request.
let (result_tx_2, result_rx_2) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(1), result_tx_2).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx_2)
.await
.unwrap();
// Assert the prepare queue is empty.
test.poll_ensure_to_prepare_queue_is_empty().await;
@@ -1368,7 +1406,9 @@ mod tests {
// Submit another precheck request.
let (result_tx_3, result_rx_3) = oneshot::channel();
host.precheck_pvf(Pvf::from_discriminator(1), result_tx_3).await.unwrap();
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx_3)
.await
.unwrap();
// Assert the prepare queue is empty - we do not retry for precheck requests.
test.poll_ensure_to_prepare_queue_is_empty().await;
@@ -1388,7 +1428,7 @@ mod tests {
// Submit a execute request that fails.
let (result_tx, result_rx) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1418,7 +1458,7 @@ mod tests {
// Submit another execute request. We shouldn't try to prepare again, yet.
let (result_tx_2, result_rx_2) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1440,7 +1480,7 @@ mod tests {
// Submit another execute request.
let (result_tx_3, result_rx_3) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1490,7 +1530,7 @@ mod tests {
// Submit an execute request that fails.
let (result_tx, result_rx) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1523,7 +1563,7 @@ mod tests {
// Submit another execute request.
let (result_tx_2, result_rx_2) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1548,7 +1588,7 @@ mod tests {
// Submit another execute request.
let (result_tx_3, result_rx_3) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf".to_vec(),
Priority::Critical,
@@ -1575,7 +1615,7 @@ mod tests {
let mut host = test.host_handle();
// Submit a heads-up request that fails.
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
// The queue received the prepare request.
assert_matches!(
@@ -1592,7 +1632,7 @@ mod tests {
.unwrap();
// Submit another heads-up request.
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
// Assert the prepare queue is empty.
test.poll_ensure_to_prepare_queue_is_empty().await;
@@ -1601,7 +1641,7 @@ mod tests {
futures_timer::Delay::new(PREPARE_FAILURE_COOLDOWN).await;
// Submit another heads-up request.
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
// Assert the prepare queue contains the request.
assert_matches!(
@@ -1617,7 +1657,7 @@ mod tests {
let (result_tx, result_rx) = oneshot::channel();
host.execute_pvf(
Pvf::from_discriminator(1),
PvfWithExecutorParams::from_discriminator(1),
TEST_EXECUTION_TIMEOUT,
b"pvf1".to_vec(),
Priority::Normal,
+1 -1
View File
@@ -110,7 +110,7 @@ pub use sp_tracing;
pub use error::{InvalidCandidate, PrepareError, PrepareResult, ValidationError};
pub use prepare::PrepareStats;
pub use priority::Priority;
pub use pvf::Pvf;
pub use pvf::{Pvf, PvfWithExecutorParams};
pub use host::{start, Config, ValidationHost};
pub use metrics::Metrics;
+15 -4
View File
@@ -25,6 +25,7 @@ use always_assert::never;
use futures::{
channel::mpsc, future::BoxFuture, stream::FuturesUnordered, Future, FutureExt, StreamExt,
};
use polkadot_primitives::vstaging::ExecutorParams;
use slotmap::HopSlotMap;
use std::{
fmt,
@@ -69,6 +70,7 @@ pub enum ToPool {
worker: Worker,
code: Arc<Vec<u8>>,
artifact_path: PathBuf,
executor_params: ExecutorParams,
preparation_timeout: Duration,
},
}
@@ -214,7 +216,7 @@ fn handle_to_pool(
metrics.prepare_worker().on_begin_spawn();
mux.push(spawn_worker_task(program_path.to_owned(), spawn_timeout).boxed());
},
ToPool::StartWork { worker, code, artifact_path, preparation_timeout } => {
ToPool::StartWork { worker, code, artifact_path, executor_params, preparation_timeout } => {
if let Some(data) = spawned.get_mut(worker) {
if let Some(idle) = data.idle.take() {
let preparation_timer = metrics.time_preparation();
@@ -226,6 +228,7 @@ fn handle_to_pool(
code,
cache_path.to_owned(),
artifact_path,
executor_params,
preparation_timeout,
preparation_timer,
)
@@ -275,12 +278,20 @@ async fn start_work_task<Timer>(
code: Arc<Vec<u8>>,
cache_path: PathBuf,
artifact_path: PathBuf,
executor_params: ExecutorParams,
preparation_timeout: Duration,
_preparation_timer: Option<Timer>,
) -> PoolEvent {
let outcome =
worker::start_work(&metrics, idle, code, &cache_path, artifact_path, preparation_timeout)
.await;
let outcome = worker::start_work(
&metrics,
idle,
code,
&cache_path,
artifact_path,
executor_params,
preparation_timeout,
)
.await;
PoolEvent::StartWork(worker, outcome)
}
+61 -28
View File
@@ -17,7 +17,10 @@
//! A queue that handles requests for PVF preparation.
use super::pool::{self, Worker};
use crate::{artifacts::ArtifactId, metrics::Metrics, PrepareResult, Priority, Pvf, LOG_TARGET};
use crate::{
artifacts::ArtifactId, metrics::Metrics, PrepareResult, Priority, PvfWithExecutorParams,
LOG_TARGET,
};
use always_assert::{always, never};
use futures::{channel::mpsc, stream::StreamExt as _, Future, SinkExt};
use std::{
@@ -33,7 +36,11 @@ pub enum ToQueue {
///
/// Note that it is incorrect to enqueue the same PVF again without first receiving the
/// [`FromQueue`] response.
Enqueue { priority: Priority, pvf: Pvf, preparation_timeout: Duration },
Enqueue {
priority: Priority,
pvf_with_params: PvfWithExecutorParams,
preparation_timeout: Duration,
},
}
/// A response from queue.
@@ -78,7 +85,7 @@ slotmap::new_key_type! { pub struct Job; }
struct JobData {
/// The priority of this job. Can be bumped.
priority: Priority,
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
/// The timeout for the preparation job.
preparation_timeout: Duration,
worker: Option<Worker>,
@@ -208,8 +215,8 @@ impl Queue {
async fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) -> Result<(), Fatal> {
match to_queue {
ToQueue::Enqueue { priority, pvf, preparation_timeout } => {
handle_enqueue(queue, priority, pvf, preparation_timeout).await?;
ToQueue::Enqueue { priority, pvf_with_params, preparation_timeout } => {
handle_enqueue(queue, priority, pvf_with_params, preparation_timeout).await?;
},
}
Ok(())
@@ -218,19 +225,19 @@ async fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) -> Result<(), Fat
async fn handle_enqueue(
queue: &mut Queue,
priority: Priority,
pvf: Pvf,
pvf_with_params: PvfWithExecutorParams,
preparation_timeout: Duration,
) -> Result<(), Fatal> {
gum::debug!(
target: LOG_TARGET,
validation_code_hash = ?pvf.code_hash,
validation_code_hash = ?pvf_with_params.code_hash(),
?priority,
?preparation_timeout,
"PVF is enqueued for preparation.",
);
queue.metrics.prepare_enqueued();
let artifact_id = pvf.as_artifact_id();
let artifact_id = pvf_with_params.as_artifact_id();
if never!(
queue.artifact_id_to_job.contains_key(&artifact_id),
"second Enqueue sent for a known artifact"
@@ -247,7 +254,10 @@ async fn handle_enqueue(
return Ok(())
}
let job = queue.jobs.insert(JobData { priority, pvf, preparation_timeout, worker: None });
let job =
queue
.jobs
.insert(JobData { priority, pvf_with_params, preparation_timeout, worker: None });
queue.artifact_id_to_job.insert(artifact_id, job);
if let Some(available) = find_idle_worker(queue) {
@@ -338,7 +348,7 @@ async fn handle_worker_concluded(
// this can't be None;
// qed.
let job_data = never_none!(queue.jobs.remove(job));
let artifact_id = job_data.pvf.as_artifact_id();
let artifact_id = job_data.pvf_with_params.as_artifact_id();
queue.artifact_id_to_job.remove(&artifact_id);
@@ -424,7 +434,7 @@ async fn spawn_extra_worker(queue: &mut Queue, critical: bool) -> Result<(), Fat
async fn assign(queue: &mut Queue, worker: Worker, job: Job) -> Result<(), Fatal> {
let job_data = &mut queue.jobs[job];
let artifact_id = job_data.pvf.as_artifact_id();
let artifact_id = job_data.pvf_with_params.as_artifact_id();
let artifact_path = artifact_id.path(&queue.cache_path);
job_data.worker = Some(worker);
@@ -435,8 +445,9 @@ async fn assign(queue: &mut Queue, worker: Worker, job: Job) -> Result<(), Fatal
&mut queue.to_pool_tx,
pool::ToPool::StartWork {
worker,
code: job_data.pvf.code.clone(),
code: job_data.pvf_with_params.code(),
artifact_path,
executor_params: job_data.pvf_with_params.executor_params(),
preparation_timeout: job_data.preparation_timeout,
},
)
@@ -503,8 +514,8 @@ mod tests {
use std::task::Poll;
/// Creates a new PVF which artifact id can be uniquely identified by the given number.
fn pvf(descriminator: u32) -> Pvf {
Pvf::from_discriminator(descriminator)
fn pvf_with_params(descriminator: u32) -> PvfWithExecutorParams {
PvfWithExecutorParams::from_discriminator(descriminator)
}
async fn run_until<R>(
@@ -613,7 +624,7 @@ mod tests {
test.send_queue(ToQueue::Enqueue {
priority: Priority::Normal,
pvf: pvf(1),
pvf_with_params: pvf_with_params(1),
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
});
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
@@ -626,7 +637,10 @@ mod tests {
result: Ok(PrepareStats::default()),
});
assert_eq!(test.poll_and_recv_from_queue().await.artifact_id, pvf(1).as_artifact_id());
assert_eq!(
test.poll_and_recv_from_queue().await.artifact_id,
pvf_with_params(1).as_artifact_id()
);
}
#[tokio::test]
@@ -635,12 +649,20 @@ mod tests {
let priority = Priority::Normal;
let preparation_timeout = PRECHECK_PREPARATION_TIMEOUT;
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(1), preparation_timeout });
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(2), preparation_timeout });
test.send_queue(ToQueue::Enqueue {
priority,
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
preparation_timeout,
});
test.send_queue(ToQueue::Enqueue {
priority,
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
preparation_timeout,
});
// Start a non-precheck preparation for this one.
test.send_queue(ToQueue::Enqueue {
priority,
pvf: pvf(3),
pvf_with_params: PvfWithExecutorParams::from_discriminator(3),
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
});
@@ -669,7 +691,7 @@ mod tests {
// Enqueue a critical job.
test.send_queue(ToQueue::Enqueue {
priority: Priority::Critical,
pvf: pvf(4),
pvf_with_params: PvfWithExecutorParams::from_discriminator(4),
preparation_timeout,
});
@@ -685,7 +707,7 @@ mod tests {
test.send_queue(ToQueue::Enqueue {
priority: Priority::Normal,
pvf: pvf(1),
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
preparation_timeout,
});
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
@@ -696,7 +718,7 @@ mod tests {
// Enqueue a critical job, which warrants spawning over the soft limit.
test.send_queue(ToQueue::Enqueue {
priority: Priority::Critical,
pvf: pvf(2),
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
preparation_timeout,
});
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
@@ -722,12 +744,20 @@ mod tests {
let priority = Priority::Normal;
let preparation_timeout = PRECHECK_PREPARATION_TIMEOUT;
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(1), preparation_timeout });
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(2), preparation_timeout });
test.send_queue(ToQueue::Enqueue {
priority,
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
preparation_timeout,
});
test.send_queue(ToQueue::Enqueue {
priority,
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
preparation_timeout,
});
// Start a non-precheck preparation for this one.
test.send_queue(ToQueue::Enqueue {
priority,
pvf: pvf(3),
pvf_with_params: PvfWithExecutorParams::from_discriminator(3),
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
});
@@ -753,7 +783,10 @@ mod tests {
// Since there is still work, the queue requested one extra worker to spawn to handle the
// remaining enqueued work items.
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
assert_eq!(test.poll_and_recv_from_queue().await.artifact_id, pvf(1).as_artifact_id());
assert_eq!(
test.poll_and_recv_from_queue().await.artifact_id,
pvf_with_params(1).as_artifact_id()
);
}
#[tokio::test]
@@ -762,7 +795,7 @@ mod tests {
test.send_queue(ToQueue::Enqueue {
priority: Priority::Normal,
pvf: pvf(1),
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
});
@@ -787,7 +820,7 @@ mod tests {
test.send_queue(ToQueue::Enqueue {
priority: Priority::Normal,
pvf: pvf(1),
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
});
+26 -7
View File
@@ -34,6 +34,7 @@ use crate::{
use cpu_time::ProcessTime;
use futures::{pin_mut, select_biased, FutureExt};
use parity_scale_codec::{Decode, Encode};
use polkadot_primitives::vstaging::ExecutorParams;
use sp_core::hexdisplay::HexDisplay;
use std::{
panic,
@@ -85,6 +86,7 @@ pub async fn start_work(
code: Arc<Vec<u8>>,
cache_path: &Path,
artifact_path: PathBuf,
executor_params: ExecutorParams,
preparation_timeout: Duration,
) -> Outcome {
let IdleWorker { stream, pid } = worker;
@@ -97,7 +99,9 @@ pub async fn start_work(
);
with_tmp_file(stream, pid, cache_path, |tmp_file, mut stream| async move {
if let Err(err) = send_request(&mut stream, code, &tmp_file, preparation_timeout).await {
if let Err(err) =
send_request(&mut stream, code, &tmp_file, &executor_params, preparation_timeout).await
{
gum::warn!(
target: LOG_TARGET,
worker_pid = %pid,
@@ -271,15 +275,19 @@ async fn send_request(
stream: &mut UnixStream,
code: Arc<Vec<u8>>,
tmp_file: &Path,
executor_params: &ExecutorParams,
preparation_timeout: Duration,
) -> io::Result<()> {
framed_send(stream, &code).await?;
framed_send(stream, path_to_bytes(tmp_file)).await?;
framed_send(stream, &executor_params.encode()).await?;
framed_send(stream, &preparation_timeout.encode()).await?;
Ok(())
}
async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf, Duration)> {
async fn recv_request(
stream: &mut UnixStream,
) -> io::Result<(Vec<u8>, PathBuf, ExecutorParams, Duration)> {
let code = framed_recv(stream).await?;
let tmp_file = framed_recv(stream).await?;
let tmp_file = bytes_to_path(&tmp_file).ok_or_else(|| {
@@ -288,6 +296,13 @@ async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf,
"prepare pvf recv_request: non utf-8 artifact path".to_string(),
)
})?;
let executor_params_enc = framed_recv(stream).await?;
let executor_params = ExecutorParams::decode(&mut &executor_params_enc[..]).map_err(|_| {
io::Error::new(
io::ErrorKind::Other,
"prepare pvf recv_request: failed to decode ExecutorParams".to_string(),
)
})?;
let preparation_timeout = framed_recv(stream).await?;
let preparation_timeout = Duration::decode(&mut &preparation_timeout[..]).map_err(|e| {
io::Error::new(
@@ -295,7 +310,7 @@ async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf,
format!("prepare pvf recv_request: failed to decode duration: {:?}", e),
)
})?;
Ok((code, tmp_file, preparation_timeout))
Ok((code, tmp_file, executor_params, preparation_timeout))
}
async fn send_response(stream: &mut UnixStream, result: PrepareResult) -> io::Result<()> {
@@ -347,7 +362,8 @@ pub fn worker_entrypoint(socket_path: &str) {
worker_event_loop("prepare", socket_path, |rt_handle, mut stream| async move {
loop {
let worker_pid = std::process::id();
let (code, dest, preparation_timeout) = recv_request(&mut stream).await?;
let (code, dest, executor_params, preparation_timeout) =
recv_request(&mut stream).await?;
gum::debug!(
target: LOG_TARGET,
%worker_pid,
@@ -372,7 +388,7 @@ pub fn worker_entrypoint(socket_path: &str) {
// Spawn another thread for preparation.
let prepare_fut = rt_handle
.spawn_blocking(move || {
let result = prepare_artifact(&code);
let result = prepare_artifact(&code, executor_params);
// Get the `ru_maxrss` stat. If supported, call getrusage for the thread.
#[cfg(target_os = "linux")]
@@ -454,14 +470,17 @@ pub fn worker_entrypoint(socket_path: &str) {
});
}
fn prepare_artifact(code: &[u8]) -> Result<CompiledArtifact, PrepareError> {
fn prepare_artifact(
code: &[u8],
executor_params: ExecutorParams,
) -> Result<CompiledArtifact, PrepareError> {
panic::catch_unwind(|| {
let blob = match crate::executor_intf::prevalidate(code) {
Err(err) => return Err(PrepareError::Prevalidation(format!("{:?}", err))),
Ok(b) => b,
};
match crate::executor_intf::prepare(blob) {
match crate::executor_intf::prepare(blob, executor_params) {
Ok(compiled_artifact) => Ok(CompiledArtifact::new(compiled_artifact)),
Err(err) => Err(PrepareError::Preparation(format!("{:?}", err))),
}
+41 -2
View File
@@ -16,6 +16,7 @@
use crate::artifacts::ArtifactId;
use polkadot_parachain::primitives::ValidationCodeHash;
use polkadot_primitives::vstaging::ExecutorParams;
use sp_core::blake2_256;
use std::{fmt, sync::Arc};
@@ -48,9 +49,47 @@ impl Pvf {
let descriminator_buf = num.to_le_bytes().to_vec();
Pvf::from_code(descriminator_buf)
}
}
/// Returns the artifact ID that corresponds to this PVF.
/// Coupling PVF code with executor params
#[derive(Debug, Clone)]
pub struct PvfWithExecutorParams {
pvf: Pvf,
executor_params: Arc<ExecutorParams>,
}
impl PvfWithExecutorParams {
/// Creates a new PVF-ExecutorParams pair structure
pub fn new(pvf: Pvf, executor_params: ExecutorParams) -> Self {
Self { pvf, executor_params: Arc::new(executor_params) }
}
/// Returns artifact ID that corresponds to the PVF with given executor params
pub(crate) fn as_artifact_id(&self) -> ArtifactId {
ArtifactId::new(self.code_hash)
ArtifactId::new(self.pvf.code_hash, self.executor_params.hash())
}
/// Returns validation code hash for the PVF
pub(crate) fn code_hash(&self) -> ValidationCodeHash {
self.pvf.code_hash
}
/// Returns PVF code
pub(crate) fn code(&self) -> Arc<Vec<u8>> {
self.pvf.code.clone()
}
/// Returns executor params
pub(crate) fn executor_params(&self) -> ExecutorParams {
(*self.executor_params).clone()
}
/// Creates a structure for tests
#[cfg(test)]
pub(crate) fn from_discriminator(num: u32) -> Self {
Self {
pvf: Pvf::from_discriminator(num),
executor_params: Arc::new(ExecutorParams::default()),
}
}
}
+4 -2
View File
@@ -19,6 +19,8 @@
//! N.B. This is not guarded with some feature flag. Overexposing items here may affect the final
//! artifact even for production builds.
use polkadot_primitives::vstaging::ExecutorParams;
pub mod worker_common {
pub use crate::worker_common::{spawn_with_program_path, SpawnErr};
}
@@ -35,12 +37,12 @@ pub fn validate_candidate(
.expect("Decompressing code failed");
let blob = prevalidate(&code)?;
let artifact = prepare(blob)?;
let artifact = prepare(blob, ExecutorParams::default())?;
let tmpdir = tempfile::tempdir()?;
let artifact_path = tmpdir.path().join("blob");
std::fs::write(&artifact_path, &artifact)?;
let executor = Executor::new()?;
let executor = Executor::new(ExecutorParams::default())?;
let result = unsafe {
// SAFETY: This is trivially safe since the artifact is obtained by calling `prepare`
// and is written into a temporary directory in an unmodified state.
@@ -251,6 +251,8 @@ pub enum SpawnErr {
ProcessSpawn,
/// The deadline allotted for the worker spawning and connecting to the socket has elapsed.
AcceptTimeout,
/// Failed to send handshake after successful spawning was signaled
Handshake,
}
/// This is a representation of a potentially running worker. Drop it and the process will be killed.