mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-07-18 10:35:42 +00:00
Executor Environment parameterization (#6161)
* Re-apply changes without Diener, rebase to the lastest master * Cache pruning * Bit-pack InstantiationStrategy * Move ExecutorParams version inside the structure itself * Rework runtime API and executor parameters storage * Pass executor parameters through backing subsystem * Update Cargo.lock * Introduce `ExecutorParams` to approval voting subsys * Introduce `ExecutorParams` to dispute coordinator * `cargo fmt` * Simplify requests from backing subsys * Fix tests * Replace manual config cloning with `.clone()` * Move constants to module * Parametrize executor performing PVF pre-check * Fix Malus * Fix test runtime * Introduce session executor params as a constant defined by session info pallet * Use Parity SCALE codec instead of hand-crafted binary encoding * Get rid of constants; Add docs * Get rid of constants * Minor typo * Fix Malus after rebase * `cargo fmt` * Use transparent SCALE encoding instead of explicit * Clean up * Get rid of relay parent to session index mapping * Join environment type and version in a single enum element * Use default execution parameters if running an old runtime * `unwrap()` -> `expect()` * Correct API version * Constants are back in town * Use constants for execution environment types * Artifact separation, first try * Get rid of explicit version * PVF execution queue worker separation * Worker handshake * Global renaming * Minor fixes resolving discussions * Two-stage requesting of executor params to make use of runtime API cache * Proper error handling in pvf-checker * Executor params storage bootstrapping * Propagate migration to v3 network runtimes * Fix storage versioning * Ensure `ExecutorParams` serialization determinism; Add comments * Rename constants to make things a bit more deterministic Get rid of stale code * Tidy up a structure of active PVFs * Minor formatting * Fix comment * Add try-runtime hooks * Add storage version write on upgrade Co-authored-by: Andronik <write@reusable.software> * Add pre- and post-upgrade assertions * Require to specify environment type; Remove redundant `impl`s * Add `ExecutorParamHash` creation from `H256` * Fix candidate validation subsys tests * Return splittable error from executor params request fn * Revert "Return splittable error from executor params request fn" This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456. * Decompose approval voting metrics * Use more relevant errors * Minor formatting fix * Assert a valid environment type instead of checking * Fix `try-runtime` hooks * After-merge fixes * Add migration logs * Remove dead code * Fix tests * Fix tests * Back to the strongly typed implementation * Promote strong types to executor interface * Remove stale comment * Move executor params to `SessionInfo`: primitives and runtime * Move executor params to `SessionInfo`: node * Try to bump primitives and API version * Get rid of `MallocSizeOf` * Bump target API version to v4 * Make use of session index already in place * Back to v3 * Fix all the tests * Add migrations to all the runtimes * Make use of existing `SessionInfo` in approval voting subsys * Rename `TARGET` -> `LOG_TARGET` * Bump all the primitives to v3 * Fix Rococo ParachainHost API version * Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes * Fix nits from discussions; add comments * Re-evaluate queue logic * Rework job assignment in execution queue * Add documentation * Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching) * Couple `Pvf` with `ExecutorParams` wherever possible * Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning * Fix comment * Fix CI tests * Fix clippy warnings * Address nits from discussions * Add a placeholder for raw data * Fix non exhaustive match * Remove redundant reexports and fix imports * Keep only necessary semantic features, as discussed * Rework `RuntimeInfo` to support mock implementation for tests * Remove unneeded bound * `cargo fmt` * Revert "Remove unneeded bound" This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61. * Fix PVF host tests * Fix PVF checker tests * Fix overseer declarations * Simplify tests * `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT` * Add a unit test for varying executor parameters * Minor fixes from discussions * Add prechecking max. memory parameter (see paritytech/srlabs_findings#110) * Fix and improve a test * Remove `ExecutionEnvironment` and `RawData` * New primitives versioning in parachain host API * `disputes()` implementation for Kusama and Polkadot * Move `ExecutorParams` from `vstaging` to stable primitives * Move disputes from `vstaging` to stable implementation * Fix `try-runtime` * Fixes after merge * Move `ExecutorParams` to the bottom of `SessionInfo` * Revert "Move executor params to `SessionInfo`: primitives and runtime" This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce. * Always use fresh activated live hash in pvf precheck (re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa) * Fixing tests (broken commit) * Fix candidate validation tests * Fix PVF host test * Minor fixes * Address discussions * Restore migration * Fix `use` to only include what is needed instead of `*` * Add comment to never touch `DEFAULT_CONFIG` * Update migration to set default `ExecutorParams` for `dispute_period` sessions back * Use `earliest_stored_session` instead of calculations * Nit * Add logs * Treat any runtime error as `NotSupported` again * Always return default executor params if not available * Revert "Always return default executor params if not available" This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30. * Add paritytech/substrate#9997 workaround * `cargo fmt` * Remove migration (again!) * Bump executor params to API v4 (backport from #6698) --------- Co-authored-by: Andronik <write@reusable.software>
This commit is contained in:
@@ -17,6 +17,7 @@
|
||||
use crate::{error::PrepareError, host::PrepareResultSender, prepare::PrepareStats};
|
||||
use always_assert::always;
|
||||
use polkadot_parachain::primitives::ValidationCodeHash;
|
||||
use polkadot_primitives::vstaging::ExecutorParamsHash;
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
path::{Path, PathBuf},
|
||||
@@ -37,19 +38,19 @@ impl AsRef<[u8]> for CompiledArtifact {
|
||||
}
|
||||
}
|
||||
|
||||
/// Identifier of an artifact. Right now it only encodes a code hash of the PVF. But if we get to
|
||||
/// multiple engine implementations the artifact ID should include the engine type as well.
|
||||
/// Identifier of an artifact. Encodes a code hash of the PVF and a hash of executor parameter set.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
pub struct ArtifactId {
|
||||
pub(crate) code_hash: ValidationCodeHash,
|
||||
pub(crate) executor_params_hash: ExecutorParamsHash,
|
||||
}
|
||||
|
||||
impl ArtifactId {
|
||||
const PREFIX: &'static str = "wasmtime_";
|
||||
|
||||
/// Creates a new artifact ID with the given hash.
|
||||
pub fn new(code_hash: ValidationCodeHash) -> Self {
|
||||
Self { code_hash }
|
||||
pub fn new(code_hash: ValidationCodeHash, executor_params_hash: ExecutorParamsHash) -> Self {
|
||||
Self { code_hash, executor_params_hash }
|
||||
}
|
||||
|
||||
/// Tries to recover the artifact id from the given file name.
|
||||
@@ -59,14 +60,18 @@ impl ArtifactId {
|
||||
use std::str::FromStr as _;
|
||||
|
||||
let file_name = file_name.strip_prefix(Self::PREFIX)?;
|
||||
let code_hash = Hash::from_str(file_name).ok()?.into();
|
||||
let (code_hash_str, executor_params_hash_str) = file_name.split_once('_')?;
|
||||
let code_hash = Hash::from_str(code_hash_str).ok()?.into();
|
||||
let executor_params_hash =
|
||||
ExecutorParamsHash::from_hash(Hash::from_str(executor_params_hash_str).ok()?);
|
||||
|
||||
Some(Self { code_hash })
|
||||
Some(Self { code_hash, executor_params_hash })
|
||||
}
|
||||
|
||||
/// Returns the expected path to this artifact given the root of the cache.
|
||||
pub fn path(&self, cache_path: &Path) -> PathBuf {
|
||||
let file_name = format!("{}{:#x}", Self::PREFIX, self.code_hash);
|
||||
let file_name =
|
||||
format!("{}{:#x}_{:#x}", Self::PREFIX, self.code_hash, self.executor_params_hash);
|
||||
cache_path.join(file_name)
|
||||
}
|
||||
}
|
||||
@@ -214,6 +219,7 @@ impl Artifacts {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ArtifactId, Artifacts};
|
||||
use polkadot_primitives::vstaging::ExecutorParamsHash;
|
||||
use sp_core::H256;
|
||||
use std::{path::Path, str::FromStr};
|
||||
|
||||
@@ -224,13 +230,16 @@ mod tests {
|
||||
|
||||
assert_eq!(
|
||||
ArtifactId::from_file_name(
|
||||
"wasmtime_0x0022800000000000000000000000000000000000000000000000000000000000"
|
||||
"wasmtime_0x0022800000000000000000000000000000000000000000000000000000000000_0x0033900000000000000000000000000000000000000000000000000000000000"
|
||||
),
|
||||
Some(ArtifactId::new(
|
||||
hex_literal::hex![
|
||||
"0022800000000000000000000000000000000000000000000000000000000000"
|
||||
]
|
||||
.into()
|
||||
.into(),
|
||||
ExecutorParamsHash::from_hash(sp_core::H256(hex_literal::hex![
|
||||
"0033900000000000000000000000000000000000000000000000000000000000"
|
||||
])),
|
||||
)),
|
||||
);
|
||||
}
|
||||
@@ -240,13 +249,12 @@ mod tests {
|
||||
let path = Path::new("/test");
|
||||
let hash =
|
||||
H256::from_str("1234567890123456789012345678901234567890123456789012345678901234")
|
||||
.unwrap()
|
||||
.into();
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
ArtifactId::new(hash).path(path).to_str(),
|
||||
ArtifactId::new(hash.into(), ExecutorParamsHash::from_hash(hash)).path(path).to_str(),
|
||||
Some(
|
||||
"/test/wasmtime_0x1234567890123456789012345678901234567890123456789012345678901234"
|
||||
"/test/wasmtime_0x1234567890123456789012345678901234567890123456789012345678901234_0x1234567890123456789012345678901234567890123456789012345678901234"
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -30,8 +30,23 @@ use futures::{
|
||||
stream::{FuturesUnordered, StreamExt as _},
|
||||
Future, FutureExt,
|
||||
};
|
||||
use polkadot_node_primitives::BACKING_EXECUTION_TIMEOUT;
|
||||
use polkadot_primitives::vstaging::{ExecutorParams, ExecutorParamsHash};
|
||||
use slotmap::HopSlotMap;
|
||||
use std::{collections::VecDeque, fmt, path::PathBuf, time::Duration};
|
||||
use std::{
|
||||
collections::VecDeque,
|
||||
fmt,
|
||||
path::PathBuf,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
/// The amount of time a job for which the queue does not have a compatible worker may wait in the
|
||||
/// queue. After that time passes, the queue will kill the first worker which becomes idle to
|
||||
/// re-spawn a new worker to execute the job immediately.
|
||||
/// To make any sense and not to break things, the value should be greater than minimal execution
|
||||
/// timeout in use, and less than the block time.
|
||||
const MAX_KEEP_WAITING: Duration =
|
||||
Duration::from_millis(BACKING_EXECUTION_TIMEOUT.as_millis() as u64 * 2);
|
||||
|
||||
slotmap::new_key_type! { struct Worker; }
|
||||
|
||||
@@ -41,6 +56,7 @@ pub enum ToQueue {
|
||||
artifact: ArtifactPathId,
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
executor_params: ExecutorParams,
|
||||
result_tx: ResultSender,
|
||||
},
|
||||
}
|
||||
@@ -49,12 +65,15 @@ struct ExecuteJob {
|
||||
artifact: ArtifactPathId,
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
executor_params: ExecutorParams,
|
||||
result_tx: ResultSender,
|
||||
waiting_since: Instant,
|
||||
}
|
||||
|
||||
struct WorkerData {
|
||||
idle: Option<IdleWorker>,
|
||||
handle: WorkerHandle,
|
||||
executor_params_hash: ExecutorParamsHash,
|
||||
}
|
||||
|
||||
impl fmt::Debug for WorkerData {
|
||||
@@ -79,7 +98,17 @@ impl Workers {
|
||||
self.spawn_inflight + self.running.len() < self.capacity
|
||||
}
|
||||
|
||||
fn find_available(&self) -> Option<Worker> {
|
||||
fn find_available(&self, executor_params_hash: ExecutorParamsHash) -> Option<Worker> {
|
||||
self.running.iter().find_map(|d| {
|
||||
if d.1.idle.is_some() && d.1.executor_params_hash == executor_params_hash {
|
||||
Some(d.0)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn find_idle(&self) -> Option<Worker> {
|
||||
self.running
|
||||
.iter()
|
||||
.find_map(|d| if d.1.idle.is_some() { Some(d.0) } else { None })
|
||||
@@ -94,7 +123,7 @@ impl Workers {
|
||||
}
|
||||
|
||||
enum QueueEvent {
|
||||
Spawn(IdleWorker, WorkerHandle),
|
||||
Spawn(IdleWorker, WorkerHandle, ExecuteJob),
|
||||
StartWork(Worker, Outcome, ArtifactId, ResultSender),
|
||||
}
|
||||
|
||||
@@ -154,6 +183,66 @@ impl Queue {
|
||||
purge_dead(&self.metrics, &mut self.workers).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Tries to assign a job in the queue to a worker. If an idle worker is provided, it does its
|
||||
/// best to find a job with a compatible execution environment unless there are jobs in the
|
||||
/// queue waiting too long. In that case, it kills an existing idle worker and spawns a new
|
||||
/// one. It may spawn an additional worker if that is affordable.
|
||||
/// If all the workers are busy or the queue is empty, it does nothing.
|
||||
/// Should be called every time a new job arrives to the queue or a job finishes.
|
||||
fn try_assign_next_job(&mut self, finished_worker: Option<Worker>) {
|
||||
// New jobs are always pushed to the tail of the queue; the one at its head is always
|
||||
// the eldest one.
|
||||
let eldest = if let Some(eldest) = self.queue.get(0) { eldest } else { return };
|
||||
|
||||
// By default, we're going to execute the eldest job on any worker slot available, even if
|
||||
// we have to kill and re-spawn a worker
|
||||
let mut worker = None;
|
||||
let mut job_index = 0;
|
||||
|
||||
// But if we're not pressed for time, we can try to find a better job-worker pair not
|
||||
// requiring the expensive kill-spawn operation
|
||||
if eldest.waiting_since.elapsed() < MAX_KEEP_WAITING {
|
||||
if let Some(finished_worker) = finished_worker {
|
||||
if let Some(worker_data) = self.workers.running.get(finished_worker) {
|
||||
for (i, job) in self.queue.iter().enumerate() {
|
||||
if worker_data.executor_params_hash == job.executor_params.hash() {
|
||||
(worker, job_index) = (Some(finished_worker), i);
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if worker.is_none() {
|
||||
// Try to obtain a worker for the job
|
||||
worker = self.workers.find_available(self.queue[job_index].executor_params.hash());
|
||||
}
|
||||
|
||||
if worker.is_none() {
|
||||
if let Some(idle) = self.workers.find_idle() {
|
||||
// No available workers of required type but there are some idle ones of other
|
||||
// types, have to kill one and re-spawn with the correct type
|
||||
if self.workers.running.remove(idle).is_some() {
|
||||
self.metrics.execute_worker().on_retired();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if worker.is_none() && !self.workers.can_afford_one_more() {
|
||||
// Bad luck, no worker slot can be used to execute the job
|
||||
return
|
||||
}
|
||||
|
||||
let job = self.queue.remove(job_index).expect("Job is just checked to be in queue; qed");
|
||||
|
||||
if let Some(worker) = worker {
|
||||
assign(self, worker, job);
|
||||
} else {
|
||||
spawn_extra_worker(self, job);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn purge_dead(metrics: &Metrics, workers: &mut Workers) {
|
||||
@@ -172,29 +261,30 @@ async fn purge_dead(metrics: &Metrics, workers: &mut Workers) {
|
||||
}
|
||||
|
||||
fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) {
|
||||
let ToQueue::Enqueue { artifact, execution_timeout, params, result_tx } = to_queue;
|
||||
let ToQueue::Enqueue { artifact, execution_timeout, params, executor_params, result_tx } =
|
||||
to_queue;
|
||||
gum::debug!(
|
||||
target: LOG_TARGET,
|
||||
validation_code_hash = ?artifact.id.code_hash,
|
||||
"enqueueing an artifact for execution",
|
||||
);
|
||||
queue.metrics.execute_enqueued();
|
||||
let job = ExecuteJob { artifact, execution_timeout, params, result_tx };
|
||||
|
||||
if let Some(available) = queue.workers.find_available() {
|
||||
assign(queue, available, job);
|
||||
} else {
|
||||
if queue.workers.can_afford_one_more() {
|
||||
spawn_extra_worker(queue);
|
||||
}
|
||||
queue.queue.push_back(job);
|
||||
}
|
||||
let job = ExecuteJob {
|
||||
artifact,
|
||||
execution_timeout,
|
||||
params,
|
||||
executor_params,
|
||||
result_tx,
|
||||
waiting_since: Instant::now(),
|
||||
};
|
||||
queue.queue.push_back(job);
|
||||
queue.try_assign_next_job(None);
|
||||
}
|
||||
|
||||
async fn handle_mux(queue: &mut Queue, event: QueueEvent) {
|
||||
match event {
|
||||
QueueEvent::Spawn(idle, handle) => {
|
||||
handle_worker_spawned(queue, idle, handle);
|
||||
QueueEvent::Spawn(idle, handle, job) => {
|
||||
handle_worker_spawned(queue, idle, handle, job);
|
||||
},
|
||||
QueueEvent::StartWork(worker, outcome, artifact_id, result_tx) => {
|
||||
handle_job_finish(queue, worker, outcome, artifact_id, result_tx);
|
||||
@@ -202,16 +292,23 @@ async fn handle_mux(queue: &mut Queue, event: QueueEvent) {
|
||||
}
|
||||
}
|
||||
|
||||
fn handle_worker_spawned(queue: &mut Queue, idle: IdleWorker, handle: WorkerHandle) {
|
||||
fn handle_worker_spawned(
|
||||
queue: &mut Queue,
|
||||
idle: IdleWorker,
|
||||
handle: WorkerHandle,
|
||||
job: ExecuteJob,
|
||||
) {
|
||||
queue.metrics.execute_worker().on_spawned();
|
||||
queue.workers.spawn_inflight -= 1;
|
||||
let worker = queue.workers.running.insert(WorkerData { idle: Some(idle), handle });
|
||||
let worker = queue.workers.running.insert(WorkerData {
|
||||
idle: Some(idle),
|
||||
handle,
|
||||
executor_params_hash: job.executor_params.hash(),
|
||||
});
|
||||
|
||||
gum::debug!(target: LOG_TARGET, ?worker, "execute worker spawned");
|
||||
|
||||
if let Some(job) = queue.queue.pop_front() {
|
||||
assign(queue, worker, job);
|
||||
}
|
||||
assign(queue, worker, job);
|
||||
}
|
||||
|
||||
/// If there are pending jobs in the queue, schedules the next of them onto the just freed up
|
||||
@@ -280,42 +377,45 @@ fn handle_job_finish(
|
||||
if let Some(idle_worker) = idle_worker {
|
||||
if let Some(data) = queue.workers.running.get_mut(worker) {
|
||||
data.idle = Some(idle_worker);
|
||||
|
||||
if let Some(job) = queue.queue.pop_front() {
|
||||
assign(queue, worker, job);
|
||||
}
|
||||
return queue.try_assign_next_job(Some(worker))
|
||||
}
|
||||
} else {
|
||||
// Note it's possible that the worker was purged already by `purge_dead`
|
||||
if queue.workers.running.remove(worker).is_some() {
|
||||
queue.metrics.execute_worker().on_retired();
|
||||
}
|
||||
|
||||
if !queue.queue.is_empty() {
|
||||
// The worker has died and we still have work we have to do. Request an extra worker.
|
||||
//
|
||||
// That can potentially overshoot, but that should be OK.
|
||||
spawn_extra_worker(queue);
|
||||
}
|
||||
}
|
||||
|
||||
queue.try_assign_next_job(None);
|
||||
}
|
||||
|
||||
fn spawn_extra_worker(queue: &mut Queue) {
|
||||
fn spawn_extra_worker(queue: &mut Queue, job: ExecuteJob) {
|
||||
queue.metrics.execute_worker().on_begin_spawn();
|
||||
gum::debug!(target: LOG_TARGET, "spawning an extra worker");
|
||||
|
||||
queue
|
||||
.mux
|
||||
.push(spawn_worker_task(queue.program_path.clone(), queue.spawn_timeout).boxed());
|
||||
.push(spawn_worker_task(queue.program_path.clone(), job, queue.spawn_timeout).boxed());
|
||||
queue.workers.spawn_inflight += 1;
|
||||
}
|
||||
|
||||
async fn spawn_worker_task(program_path: PathBuf, spawn_timeout: Duration) -> QueueEvent {
|
||||
/// Spawns a new worker to execute a pre-assigned job.
|
||||
/// A worker is never spawned as idle; a job to be executed by the worker has to be determined
|
||||
/// beforehand. In such a way, a race condition is avoided: during the worker being spawned,
|
||||
/// another job in the queue, with an incompatible execution environment, may become stale, and
|
||||
/// the queue would have to kill a newly started worker and spawn another one.
|
||||
/// Nevertheless, if the worker finishes executing the job, it becomes idle and may be used to execute other jobs with a compatible execution environment.
|
||||
async fn spawn_worker_task(
|
||||
program_path: PathBuf,
|
||||
job: ExecuteJob,
|
||||
spawn_timeout: Duration,
|
||||
) -> QueueEvent {
|
||||
use futures_timer::Delay;
|
||||
|
||||
loop {
|
||||
match super::worker::spawn(&program_path, spawn_timeout).await {
|
||||
Ok((idle, handle)) => break QueueEvent::Spawn(idle, handle),
|
||||
match super::worker::spawn(&program_path, job.executor_params.clone(), spawn_timeout).await
|
||||
{
|
||||
Ok((idle, handle)) => break QueueEvent::Spawn(idle, handle, job),
|
||||
Err(err) => {
|
||||
gum::warn!(target: LOG_TARGET, "failed to spawn an execute worker: {:?}", err);
|
||||
|
||||
@@ -328,7 +428,8 @@ async fn spawn_worker_task(program_path: PathBuf, spawn_timeout: Duration) -> Qu
|
||||
|
||||
/// Ask the given worker to perform the given job.
|
||||
///
|
||||
/// The worker must be running and idle.
|
||||
/// The worker must be running and idle. The job and the worker must share the same execution
|
||||
/// environment parameter set.
|
||||
fn assign(queue: &mut Queue, worker: Worker, job: ExecuteJob) {
|
||||
gum::debug!(
|
||||
target: LOG_TARGET,
|
||||
@@ -337,6 +438,16 @@ fn assign(queue: &mut Queue, worker: Worker, job: ExecuteJob) {
|
||||
"assigning the execute worker",
|
||||
);
|
||||
|
||||
debug_assert_eq!(
|
||||
queue
|
||||
.workers
|
||||
.running
|
||||
.get(worker)
|
||||
.expect("caller must provide existing worker; qed")
|
||||
.executor_params_hash,
|
||||
job.executor_params.hash()
|
||||
);
|
||||
|
||||
let idle = queue.workers.claim_idle(worker).expect(
|
||||
"this caller must supply a worker which is idle and running;
|
||||
thus claim_idle cannot return None;
|
||||
|
||||
@@ -28,7 +28,9 @@ use cpu_time::ProcessTime;
|
||||
use futures::{pin_mut, select_biased, FutureExt};
|
||||
use futures_timer::Delay;
|
||||
use parity_scale_codec::{Decode, Encode};
|
||||
|
||||
use polkadot_parachain::primitives::ValidationResult;
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::{mpsc::channel, Arc},
|
||||
@@ -37,13 +39,29 @@ use std::{
|
||||
use tokio::{io, net::UnixStream};
|
||||
|
||||
/// Spawns a new worker with the given program path that acts as the worker and the spawn timeout.
|
||||
/// Sends a handshake message to the worker as soon as it is spawned.
|
||||
///
|
||||
/// The program should be able to handle `<program-path> execute-worker <socket-path>` invocation.
|
||||
pub async fn spawn(
|
||||
program_path: &Path,
|
||||
executor_params: ExecutorParams,
|
||||
spawn_timeout: Duration,
|
||||
) -> Result<(IdleWorker, WorkerHandle), SpawnErr> {
|
||||
spawn_with_program_path("execute", program_path, &["execute-worker"], spawn_timeout).await
|
||||
let (mut idle_worker, worker_handle) =
|
||||
spawn_with_program_path("execute", program_path, &["execute-worker"], spawn_timeout)
|
||||
.await?;
|
||||
send_handshake(&mut idle_worker.stream, Handshake { executor_params })
|
||||
.await
|
||||
.map_err(|error| {
|
||||
gum::warn!(
|
||||
target: LOG_TARGET,
|
||||
worker_pid = %idle_worker.pid,
|
||||
?error,
|
||||
"failed to send a handshake to the spawned worker",
|
||||
);
|
||||
SpawnErr::Handshake
|
||||
})?;
|
||||
Ok((idle_worker, worker_handle))
|
||||
}
|
||||
|
||||
/// Outcome of PVF execution.
|
||||
@@ -159,6 +177,21 @@ pub async fn start_work(
|
||||
}
|
||||
}
|
||||
|
||||
async fn send_handshake(stream: &mut UnixStream, handshake: Handshake) -> io::Result<()> {
|
||||
framed_send(stream, &handshake.encode()).await
|
||||
}
|
||||
|
||||
async fn recv_handshake(stream: &mut UnixStream) -> io::Result<Handshake> {
|
||||
let handshake_enc = framed_recv(stream).await?;
|
||||
let handshake = Handshake::decode(&mut &handshake_enc[..]).map_err(|_| {
|
||||
io::Error::new(
|
||||
io::ErrorKind::Other,
|
||||
"execute pvf recv_handshake: failed to decode Handshake".to_owned(),
|
||||
)
|
||||
})?;
|
||||
Ok(handshake)
|
||||
}
|
||||
|
||||
async fn send_request(
|
||||
stream: &mut UnixStream,
|
||||
artifact_path: &Path,
|
||||
@@ -203,6 +236,11 @@ async fn recv_response(stream: &mut UnixStream) -> io::Result<Response> {
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Encode, Decode)]
|
||||
struct Handshake {
|
||||
executor_params: ExecutorParams,
|
||||
}
|
||||
|
||||
#[derive(Encode, Decode)]
|
||||
pub enum Response {
|
||||
Ok { result_descriptor: ValidationResult, duration: Duration },
|
||||
@@ -225,7 +263,9 @@ impl Response {
|
||||
/// the path to the socket used to communicate with the host.
|
||||
pub fn worker_entrypoint(socket_path: &str) {
|
||||
worker_event_loop("execute", socket_path, |rt_handle, mut stream| async move {
|
||||
let executor = Arc::new(Executor::new().map_err(|e| {
|
||||
let handshake = recv_handshake(&mut stream).await?;
|
||||
|
||||
let executor = Arc::new(Executor::new(handshake.executor_params).map_err(|e| {
|
||||
io::Error::new(io::ErrorKind::Other, format!("cannot create executor: {}", e))
|
||||
})?);
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
//! Interface to the Substrate Executor
|
||||
|
||||
use polkadot_primitives::vstaging::executor_params::{ExecutorParam, ExecutorParams};
|
||||
use sc_executor_common::{
|
||||
runtime_blob::RuntimeBlob,
|
||||
wasm_runtime::{InvokeMethod, WasmModule as _},
|
||||
@@ -46,7 +47,11 @@ const EXTRA_HEAP_PAGES: u64 = 2048;
|
||||
/// The number of bytes devoted for the stack during wasm execution of a PVF.
|
||||
const NATIVE_STACK_MAX: u32 = 256 * 1024 * 1024;
|
||||
|
||||
const CONFIG: Config = Config {
|
||||
// VALUES OF THE DEFAULT CONFIGURATION SHOULD NEVER BE CHANGED
|
||||
// They are used as base values for the execution environment parametrization.
|
||||
// To overwrite them, add new ones to `EXECUTOR_PARAMS` in the `session_info` pallet and perform
|
||||
// a runtime upgrade to make them active.
|
||||
const DEFAULT_CONFIG: Config = Config {
|
||||
allow_missing_func_imports: true,
|
||||
cache_path: None,
|
||||
semantics: Semantics {
|
||||
@@ -97,17 +102,42 @@ pub fn prevalidate(code: &[u8]) -> Result<RuntimeBlob, sc_executor_common::error
|
||||
|
||||
/// Runs preparation on the given runtime blob. If successful, it returns a serialized compiled
|
||||
/// artifact which can then be used to pass into `Executor::execute` after writing it to the disk.
|
||||
pub fn prepare(blob: RuntimeBlob) -> Result<Vec<u8>, sc_executor_common::error::WasmError> {
|
||||
sc_executor_wasmtime::prepare_runtime_artifact(blob, &CONFIG.semantics)
|
||||
pub fn prepare(
|
||||
blob: RuntimeBlob,
|
||||
executor_params: ExecutorParams,
|
||||
) -> Result<Vec<u8>, sc_executor_common::error::WasmError> {
|
||||
let semantics = params_to_wasmtime_semantics(executor_params)
|
||||
.map_err(|e| sc_executor_common::error::WasmError::Other(e))?;
|
||||
sc_executor_wasmtime::prepare_runtime_artifact(blob, &semantics)
|
||||
}
|
||||
|
||||
fn params_to_wasmtime_semantics(par: ExecutorParams) -> Result<Semantics, String> {
|
||||
let mut sem = DEFAULT_CONFIG.semantics.clone();
|
||||
let mut stack_limit = if let Some(stack_limit) = sem.deterministic_stack_limit.clone() {
|
||||
stack_limit
|
||||
} else {
|
||||
return Err("No default stack limit set".to_owned())
|
||||
};
|
||||
for p in par.iter() {
|
||||
match p {
|
||||
ExecutorParam::MaxMemorySize(mms) => sem.max_memory_size = Some(*mms as usize),
|
||||
ExecutorParam::StackLogicalMax(slm) => stack_limit.logical_max = *slm,
|
||||
ExecutorParam::StackNativeMax(snm) => stack_limit.native_stack_max = *snm,
|
||||
ExecutorParam::PrecheckingMaxMemory(_) => (), // TODO: Not implemented yet
|
||||
}
|
||||
}
|
||||
sem.deterministic_stack_limit = Some(stack_limit);
|
||||
Ok(sem)
|
||||
}
|
||||
|
||||
pub struct Executor {
|
||||
thread_pool: rayon::ThreadPool,
|
||||
spawner: TaskSpawner,
|
||||
config: Config,
|
||||
}
|
||||
|
||||
impl Executor {
|
||||
pub fn new() -> Result<Self, String> {
|
||||
pub fn new(params: ExecutorParams) -> Result<Self, String> {
|
||||
// Wasmtime powers the Substrate Executor. It compiles the wasm bytecode into native code.
|
||||
// That native code does not create any stacks and just reuses the stack of the thread that
|
||||
// wasmtime was invoked from.
|
||||
@@ -154,7 +184,10 @@ impl Executor {
|
||||
let spawner =
|
||||
TaskSpawner::new().map_err(|e| format!("cannot create task spawner: {}", e))?;
|
||||
|
||||
Ok(Self { thread_pool, spawner })
|
||||
let mut config = DEFAULT_CONFIG.clone();
|
||||
config.semantics = params_to_wasmtime_semantics(params)?;
|
||||
|
||||
Ok(Self { thread_pool, spawner, config })
|
||||
}
|
||||
|
||||
/// Executes the given PVF in the form of a compiled artifact and returns the result of execution
|
||||
@@ -183,7 +216,7 @@ impl Executor {
|
||||
s.spawn(move |_| {
|
||||
// spawn does not return a value, so we need to use a variable to pass the result.
|
||||
*result = Some(
|
||||
do_execute(compiled_artifact_path, params, spawner)
|
||||
do_execute(compiled_artifact_path, self.config.clone(), params, spawner)
|
||||
.map_err(|err| format!("execute error: {:?}", err)),
|
||||
);
|
||||
});
|
||||
@@ -195,6 +228,7 @@ impl Executor {
|
||||
|
||||
unsafe fn do_execute(
|
||||
compiled_artifact_path: &Path,
|
||||
config: Config,
|
||||
params: &[u8],
|
||||
spawner: impl sp_core::traits::SpawnNamed + 'static,
|
||||
) -> Result<Vec<u8>, sc_executor_common::error::Error> {
|
||||
@@ -208,7 +242,7 @@ unsafe fn do_execute(
|
||||
sc_executor::with_externalities_safe(&mut ext, || {
|
||||
let runtime = sc_executor_wasmtime::create_runtime_from_artifact::<HostFunctions>(
|
||||
compiled_artifact_path,
|
||||
CONFIG,
|
||||
config,
|
||||
)?;
|
||||
runtime.new_instance()?.call(InvokeMethod::Export("validate_block"), params)
|
||||
})?
|
||||
|
||||
@@ -25,7 +25,7 @@ use crate::{
|
||||
error::PrepareError,
|
||||
execute,
|
||||
metrics::Metrics,
|
||||
prepare, PrepareResult, Priority, Pvf, ValidationError, LOG_TARGET,
|
||||
prepare, PrepareResult, Priority, PvfWithExecutorParams, ValidationError, LOG_TARGET,
|
||||
};
|
||||
use always_assert::never;
|
||||
use futures::{
|
||||
@@ -33,6 +33,7 @@ use futures::{
|
||||
Future, FutureExt, SinkExt, StreamExt,
|
||||
};
|
||||
use polkadot_parachain::primitives::ValidationResult;
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
path::{Path, PathBuf},
|
||||
@@ -83,11 +84,11 @@ impl ValidationHost {
|
||||
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
|
||||
pub async fn precheck_pvf(
|
||||
&mut self,
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
result_tx: PrepareResultSender,
|
||||
) -> Result<(), String> {
|
||||
self.to_host_tx
|
||||
.send(ToHost::PrecheckPvf { pvf, result_tx })
|
||||
.send(ToHost::PrecheckPvf { pvf_with_params, result_tx })
|
||||
.await
|
||||
.map_err(|_| "the inner loop hung up".to_string())
|
||||
}
|
||||
@@ -101,7 +102,7 @@ impl ValidationHost {
|
||||
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
|
||||
pub async fn execute_pvf(
|
||||
&mut self,
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
priority: Priority,
|
||||
@@ -109,7 +110,7 @@ impl ValidationHost {
|
||||
) -> Result<(), String> {
|
||||
self.to_host_tx
|
||||
.send(ToHost::ExecutePvf(ExecutePvfInputs {
|
||||
pvf,
|
||||
pvf_with_params,
|
||||
execution_timeout,
|
||||
params,
|
||||
priority,
|
||||
@@ -125,7 +126,10 @@ impl ValidationHost {
|
||||
/// situations this function should return immediately.
|
||||
///
|
||||
/// Returns an error if the request cannot be sent to the validation host, i.e. if it shut down.
|
||||
pub async fn heads_up(&mut self, active_pvfs: Vec<Pvf>) -> Result<(), String> {
|
||||
pub async fn heads_up(
|
||||
&mut self,
|
||||
active_pvfs: Vec<PvfWithExecutorParams>,
|
||||
) -> Result<(), String> {
|
||||
self.to_host_tx
|
||||
.send(ToHost::HeadsUp { active_pvfs })
|
||||
.await
|
||||
@@ -134,13 +138,13 @@ impl ValidationHost {
|
||||
}
|
||||
|
||||
enum ToHost {
|
||||
PrecheckPvf { pvf: Pvf, result_tx: PrepareResultSender },
|
||||
PrecheckPvf { pvf_with_params: PvfWithExecutorParams, result_tx: PrepareResultSender },
|
||||
ExecutePvf(ExecutePvfInputs),
|
||||
HeadsUp { active_pvfs: Vec<Pvf> },
|
||||
HeadsUp { active_pvfs: Vec<PvfWithExecutorParams> },
|
||||
}
|
||||
|
||||
struct ExecutePvfInputs {
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
priority: Priority,
|
||||
@@ -265,6 +269,7 @@ pub fn start(config: Config, metrics: Metrics) -> (ValidationHost, impl Future<O
|
||||
struct PendingExecutionRequest {
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
executor_params: ExecutorParams,
|
||||
result_tx: ResultSender,
|
||||
}
|
||||
|
||||
@@ -279,11 +284,13 @@ impl AwaitingPrepare {
|
||||
artifact_id: ArtifactId,
|
||||
execution_timeout: Duration,
|
||||
params: Vec<u8>,
|
||||
executor_params: ExecutorParams,
|
||||
result_tx: ResultSender,
|
||||
) {
|
||||
self.0.entry(artifact_id).or_default().push(PendingExecutionRequest {
|
||||
execution_timeout,
|
||||
params,
|
||||
executor_params,
|
||||
result_tx,
|
||||
});
|
||||
}
|
||||
@@ -420,8 +427,8 @@ async fn handle_to_host(
|
||||
to_host: ToHost,
|
||||
) -> Result<(), Fatal> {
|
||||
match to_host {
|
||||
ToHost::PrecheckPvf { pvf, result_tx } => {
|
||||
handle_precheck_pvf(artifacts, prepare_queue, pvf, result_tx).await?;
|
||||
ToHost::PrecheckPvf { pvf_with_params, result_tx } => {
|
||||
handle_precheck_pvf(artifacts, prepare_queue, pvf_with_params, result_tx).await?;
|
||||
},
|
||||
ToHost::ExecutePvf(inputs) => {
|
||||
handle_execute_pvf(
|
||||
@@ -449,10 +456,10 @@ async fn handle_to_host(
|
||||
async fn handle_precheck_pvf(
|
||||
artifacts: &mut Artifacts,
|
||||
prepare_queue: &mut mpsc::Sender<prepare::ToQueue>,
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
result_sender: PrepareResultSender,
|
||||
) -> Result<(), Fatal> {
|
||||
let artifact_id = pvf.as_artifact_id();
|
||||
let artifact_id = pvf_with_params.as_artifact_id();
|
||||
|
||||
if let Some(state) = artifacts.artifact_state_mut(&artifact_id) {
|
||||
match state {
|
||||
@@ -474,7 +481,7 @@ async fn handle_precheck_pvf(
|
||||
prepare_queue,
|
||||
prepare::ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf,
|
||||
pvf_with_params,
|
||||
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
|
||||
},
|
||||
)
|
||||
@@ -500,8 +507,9 @@ async fn handle_execute_pvf(
|
||||
awaiting_prepare: &mut AwaitingPrepare,
|
||||
inputs: ExecutePvfInputs,
|
||||
) -> Result<(), Fatal> {
|
||||
let ExecutePvfInputs { pvf, execution_timeout, params, priority, result_tx } = inputs;
|
||||
let artifact_id = pvf.as_artifact_id();
|
||||
let ExecutePvfInputs { pvf_with_params, execution_timeout, params, priority, result_tx } =
|
||||
inputs;
|
||||
let artifact_id = pvf_with_params.as_artifact_id();
|
||||
|
||||
if let Some(state) = artifacts.artifact_state_mut(&artifact_id) {
|
||||
match state {
|
||||
@@ -515,19 +523,26 @@ async fn handle_execute_pvf(
|
||||
artifact: ArtifactPathId::new(artifact_id, cache_path),
|
||||
execution_timeout,
|
||||
params,
|
||||
executor_params: pvf_with_params.executor_params(),
|
||||
result_tx,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
},
|
||||
ArtifactState::Preparing { .. } => {
|
||||
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
|
||||
awaiting_prepare.add(
|
||||
artifact_id,
|
||||
execution_timeout,
|
||||
params,
|
||||
pvf_with_params.executor_params(),
|
||||
result_tx,
|
||||
);
|
||||
},
|
||||
ArtifactState::FailedToProcess { last_time_failed, num_failures, error } => {
|
||||
if can_retry_prepare_after_failure(*last_time_failed, *num_failures, error) {
|
||||
gum::warn!(
|
||||
target: LOG_TARGET,
|
||||
?pvf,
|
||||
?pvf_with_params,
|
||||
?artifact_id,
|
||||
?last_time_failed,
|
||||
%num_failures,
|
||||
@@ -541,11 +556,12 @@ async fn handle_execute_pvf(
|
||||
waiting_for_response: Vec::new(),
|
||||
num_failures: *num_failures,
|
||||
};
|
||||
let executor_params = pvf_with_params.executor_params().clone();
|
||||
send_prepare(
|
||||
prepare_queue,
|
||||
prepare::ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf,
|
||||
pvf_with_params,
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
},
|
||||
)
|
||||
@@ -553,7 +569,13 @@ async fn handle_execute_pvf(
|
||||
|
||||
// Add an execution request that will wait to run after this prepare job has
|
||||
// finished.
|
||||
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
|
||||
awaiting_prepare.add(
|
||||
artifact_id,
|
||||
execution_timeout,
|
||||
params,
|
||||
executor_params,
|
||||
result_tx,
|
||||
);
|
||||
} else {
|
||||
let _ = result_tx.send(Err(ValidationError::from(error.clone())));
|
||||
}
|
||||
@@ -562,19 +584,20 @@ async fn handle_execute_pvf(
|
||||
} else {
|
||||
// Artifact is unknown: register it and enqueue a job with the corresponding priority and
|
||||
// PVF.
|
||||
let executor_params = pvf_with_params.executor_params();
|
||||
artifacts.insert_preparing(artifact_id.clone(), Vec::new());
|
||||
send_prepare(
|
||||
prepare_queue,
|
||||
prepare::ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf,
|
||||
pvf_with_params,
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Add an execution request that will wait to run after this prepare job has finished.
|
||||
awaiting_prepare.add(artifact_id, execution_timeout, params, result_tx);
|
||||
awaiting_prepare.add(artifact_id, execution_timeout, params, executor_params, result_tx);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
@@ -583,7 +606,7 @@ async fn handle_execute_pvf(
|
||||
async fn handle_heads_up(
|
||||
artifacts: &mut Artifacts,
|
||||
prepare_queue: &mut mpsc::Sender<prepare::ToQueue>,
|
||||
active_pvfs: Vec<Pvf>,
|
||||
active_pvfs: Vec<PvfWithExecutorParams>,
|
||||
) -> Result<(), Fatal> {
|
||||
let now = SystemTime::now();
|
||||
|
||||
@@ -619,7 +642,7 @@ async fn handle_heads_up(
|
||||
prepare_queue,
|
||||
prepare::ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: active_pvf,
|
||||
pvf_with_params: active_pvf,
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
},
|
||||
)
|
||||
@@ -635,7 +658,7 @@ async fn handle_heads_up(
|
||||
prepare_queue,
|
||||
prepare::ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: active_pvf,
|
||||
pvf_with_params: active_pvf,
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
},
|
||||
)
|
||||
@@ -699,7 +722,9 @@ async fn handle_prepare_done(
|
||||
// It's finally time to dispatch all the execution requests that were waiting for this artifact
|
||||
// to be prepared.
|
||||
let pending_requests = awaiting_prepare.take(&artifact_id);
|
||||
for PendingExecutionRequest { execution_timeout, params, result_tx } in pending_requests {
|
||||
for PendingExecutionRequest { execution_timeout, params, executor_params, result_tx } in
|
||||
pending_requests
|
||||
{
|
||||
if result_tx.is_canceled() {
|
||||
// Preparation could've taken quite a bit of time and the requester may be not interested
|
||||
// in execution anymore, in which case we just skip the request.
|
||||
@@ -718,6 +743,7 @@ async fn handle_prepare_done(
|
||||
artifact: ArtifactPathId::new(artifact_id.clone(), cache_path),
|
||||
execution_timeout,
|
||||
params,
|
||||
executor_params,
|
||||
result_tx,
|
||||
},
|
||||
)
|
||||
@@ -856,7 +882,7 @@ mod tests {
|
||||
|
||||
/// Creates a new PVF which artifact id can be uniquely identified by the given number.
|
||||
fn artifact_id(descriminator: u32) -> ArtifactId {
|
||||
Pvf::from_discriminator(descriminator).as_artifact_id()
|
||||
PvfWithExecutorParams::from_discriminator(descriminator).as_artifact_id()
|
||||
}
|
||||
|
||||
fn artifact_path(descriminator: u32) -> PathBuf {
|
||||
@@ -1065,7 +1091,7 @@ mod tests {
|
||||
let mut test = builder.build();
|
||||
let mut host = test.host_handle();
|
||||
|
||||
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
|
||||
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
|
||||
|
||||
let to_sweeper_rx = &mut test.to_sweeper_rx;
|
||||
run_until(
|
||||
@@ -1079,7 +1105,7 @@ mod tests {
|
||||
|
||||
// Extend TTL for the first artifact and make sure we don't receive another file removal
|
||||
// request.
|
||||
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
|
||||
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
|
||||
test.poll_ensure_to_sweeper_is_empty().await;
|
||||
}
|
||||
|
||||
@@ -1090,7 +1116,7 @@ mod tests {
|
||||
|
||||
let (result_tx, result_rx_pvf_1_1) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf1".to_vec(),
|
||||
Priority::Normal,
|
||||
@@ -1101,7 +1127,7 @@ mod tests {
|
||||
|
||||
let (result_tx, result_rx_pvf_1_2) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf1".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1112,7 +1138,7 @@ mod tests {
|
||||
|
||||
let (result_tx, result_rx_pvf_2) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(2),
|
||||
PvfWithExecutorParams::from_discriminator(2),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf2".to_vec(),
|
||||
Priority::Normal,
|
||||
@@ -1190,7 +1216,9 @@ mod tests {
|
||||
|
||||
// First, test a simple precheck request.
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// The queue received the prepare request.
|
||||
assert_matches!(
|
||||
@@ -1214,7 +1242,9 @@ mod tests {
|
||||
let mut precheck_receivers = Vec::new();
|
||||
for _ in 0..3 {
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(2), result_tx).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(2), result_tx)
|
||||
.await
|
||||
.unwrap();
|
||||
precheck_receivers.push(result_rx);
|
||||
}
|
||||
// Received prepare request.
|
||||
@@ -1249,7 +1279,7 @@ mod tests {
|
||||
// Send PVF for the execution and request the prechecking for it.
|
||||
let (result_tx, result_rx_execute) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf2".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1264,7 +1294,9 @@ mod tests {
|
||||
);
|
||||
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Suppose the preparation failed, the execution queue is empty and both
|
||||
// "clients" receive their results.
|
||||
@@ -1286,13 +1318,15 @@ mod tests {
|
||||
let mut precheck_receivers = Vec::new();
|
||||
for _ in 0..3 {
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(2), result_tx).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(2), result_tx)
|
||||
.await
|
||||
.unwrap();
|
||||
precheck_receivers.push(result_rx);
|
||||
}
|
||||
|
||||
let (result_tx, _result_rx_execute) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(2),
|
||||
PvfWithExecutorParams::from_discriminator(2),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf2".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1332,7 +1366,9 @@ mod tests {
|
||||
|
||||
// Submit a precheck request that fails.
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(1), result_tx).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// The queue received the prepare request.
|
||||
assert_matches!(
|
||||
@@ -1354,7 +1390,9 @@ mod tests {
|
||||
|
||||
// Submit another precheck request.
|
||||
let (result_tx_2, result_rx_2) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(1), result_tx_2).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx_2)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Assert the prepare queue is empty.
|
||||
test.poll_ensure_to_prepare_queue_is_empty().await;
|
||||
@@ -1368,7 +1406,9 @@ mod tests {
|
||||
|
||||
// Submit another precheck request.
|
||||
let (result_tx_3, result_rx_3) = oneshot::channel();
|
||||
host.precheck_pvf(Pvf::from_discriminator(1), result_tx_3).await.unwrap();
|
||||
host.precheck_pvf(PvfWithExecutorParams::from_discriminator(1), result_tx_3)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Assert the prepare queue is empty - we do not retry for precheck requests.
|
||||
test.poll_ensure_to_prepare_queue_is_empty().await;
|
||||
@@ -1388,7 +1428,7 @@ mod tests {
|
||||
// Submit a execute request that fails.
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1418,7 +1458,7 @@ mod tests {
|
||||
// Submit another execute request. We shouldn't try to prepare again, yet.
|
||||
let (result_tx_2, result_rx_2) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1440,7 +1480,7 @@ mod tests {
|
||||
// Submit another execute request.
|
||||
let (result_tx_3, result_rx_3) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1490,7 +1530,7 @@ mod tests {
|
||||
// Submit an execute request that fails.
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1523,7 +1563,7 @@ mod tests {
|
||||
// Submit another execute request.
|
||||
let (result_tx_2, result_rx_2) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1548,7 +1588,7 @@ mod tests {
|
||||
// Submit another execute request.
|
||||
let (result_tx_3, result_rx_3) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf".to_vec(),
|
||||
Priority::Critical,
|
||||
@@ -1575,7 +1615,7 @@ mod tests {
|
||||
let mut host = test.host_handle();
|
||||
|
||||
// Submit a heads-up request that fails.
|
||||
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
|
||||
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
|
||||
|
||||
// The queue received the prepare request.
|
||||
assert_matches!(
|
||||
@@ -1592,7 +1632,7 @@ mod tests {
|
||||
.unwrap();
|
||||
|
||||
// Submit another heads-up request.
|
||||
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
|
||||
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
|
||||
|
||||
// Assert the prepare queue is empty.
|
||||
test.poll_ensure_to_prepare_queue_is_empty().await;
|
||||
@@ -1601,7 +1641,7 @@ mod tests {
|
||||
futures_timer::Delay::new(PREPARE_FAILURE_COOLDOWN).await;
|
||||
|
||||
// Submit another heads-up request.
|
||||
host.heads_up(vec![Pvf::from_discriminator(1)]).await.unwrap();
|
||||
host.heads_up(vec![PvfWithExecutorParams::from_discriminator(1)]).await.unwrap();
|
||||
|
||||
// Assert the prepare queue contains the request.
|
||||
assert_matches!(
|
||||
@@ -1617,7 +1657,7 @@ mod tests {
|
||||
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
host.execute_pvf(
|
||||
Pvf::from_discriminator(1),
|
||||
PvfWithExecutorParams::from_discriminator(1),
|
||||
TEST_EXECUTION_TIMEOUT,
|
||||
b"pvf1".to_vec(),
|
||||
Priority::Normal,
|
||||
|
||||
@@ -110,7 +110,7 @@ pub use sp_tracing;
|
||||
pub use error::{InvalidCandidate, PrepareError, PrepareResult, ValidationError};
|
||||
pub use prepare::PrepareStats;
|
||||
pub use priority::Priority;
|
||||
pub use pvf::Pvf;
|
||||
pub use pvf::{Pvf, PvfWithExecutorParams};
|
||||
|
||||
pub use host::{start, Config, ValidationHost};
|
||||
pub use metrics::Metrics;
|
||||
|
||||
@@ -25,6 +25,7 @@ use always_assert::never;
|
||||
use futures::{
|
||||
channel::mpsc, future::BoxFuture, stream::FuturesUnordered, Future, FutureExt, StreamExt,
|
||||
};
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
use slotmap::HopSlotMap;
|
||||
use std::{
|
||||
fmt,
|
||||
@@ -69,6 +70,7 @@ pub enum ToPool {
|
||||
worker: Worker,
|
||||
code: Arc<Vec<u8>>,
|
||||
artifact_path: PathBuf,
|
||||
executor_params: ExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
},
|
||||
}
|
||||
@@ -214,7 +216,7 @@ fn handle_to_pool(
|
||||
metrics.prepare_worker().on_begin_spawn();
|
||||
mux.push(spawn_worker_task(program_path.to_owned(), spawn_timeout).boxed());
|
||||
},
|
||||
ToPool::StartWork { worker, code, artifact_path, preparation_timeout } => {
|
||||
ToPool::StartWork { worker, code, artifact_path, executor_params, preparation_timeout } => {
|
||||
if let Some(data) = spawned.get_mut(worker) {
|
||||
if let Some(idle) = data.idle.take() {
|
||||
let preparation_timer = metrics.time_preparation();
|
||||
@@ -226,6 +228,7 @@ fn handle_to_pool(
|
||||
code,
|
||||
cache_path.to_owned(),
|
||||
artifact_path,
|
||||
executor_params,
|
||||
preparation_timeout,
|
||||
preparation_timer,
|
||||
)
|
||||
@@ -275,12 +278,20 @@ async fn start_work_task<Timer>(
|
||||
code: Arc<Vec<u8>>,
|
||||
cache_path: PathBuf,
|
||||
artifact_path: PathBuf,
|
||||
executor_params: ExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
_preparation_timer: Option<Timer>,
|
||||
) -> PoolEvent {
|
||||
let outcome =
|
||||
worker::start_work(&metrics, idle, code, &cache_path, artifact_path, preparation_timeout)
|
||||
.await;
|
||||
let outcome = worker::start_work(
|
||||
&metrics,
|
||||
idle,
|
||||
code,
|
||||
&cache_path,
|
||||
artifact_path,
|
||||
executor_params,
|
||||
preparation_timeout,
|
||||
)
|
||||
.await;
|
||||
PoolEvent::StartWork(worker, outcome)
|
||||
}
|
||||
|
||||
|
||||
@@ -17,7 +17,10 @@
|
||||
//! A queue that handles requests for PVF preparation.
|
||||
|
||||
use super::pool::{self, Worker};
|
||||
use crate::{artifacts::ArtifactId, metrics::Metrics, PrepareResult, Priority, Pvf, LOG_TARGET};
|
||||
use crate::{
|
||||
artifacts::ArtifactId, metrics::Metrics, PrepareResult, Priority, PvfWithExecutorParams,
|
||||
LOG_TARGET,
|
||||
};
|
||||
use always_assert::{always, never};
|
||||
use futures::{channel::mpsc, stream::StreamExt as _, Future, SinkExt};
|
||||
use std::{
|
||||
@@ -33,7 +36,11 @@ pub enum ToQueue {
|
||||
///
|
||||
/// Note that it is incorrect to enqueue the same PVF again without first receiving the
|
||||
/// [`FromQueue`] response.
|
||||
Enqueue { priority: Priority, pvf: Pvf, preparation_timeout: Duration },
|
||||
Enqueue {
|
||||
priority: Priority,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
},
|
||||
}
|
||||
|
||||
/// A response from queue.
|
||||
@@ -78,7 +85,7 @@ slotmap::new_key_type! { pub struct Job; }
|
||||
struct JobData {
|
||||
/// The priority of this job. Can be bumped.
|
||||
priority: Priority,
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
/// The timeout for the preparation job.
|
||||
preparation_timeout: Duration,
|
||||
worker: Option<Worker>,
|
||||
@@ -208,8 +215,8 @@ impl Queue {
|
||||
|
||||
async fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) -> Result<(), Fatal> {
|
||||
match to_queue {
|
||||
ToQueue::Enqueue { priority, pvf, preparation_timeout } => {
|
||||
handle_enqueue(queue, priority, pvf, preparation_timeout).await?;
|
||||
ToQueue::Enqueue { priority, pvf_with_params, preparation_timeout } => {
|
||||
handle_enqueue(queue, priority, pvf_with_params, preparation_timeout).await?;
|
||||
},
|
||||
}
|
||||
Ok(())
|
||||
@@ -218,19 +225,19 @@ async fn handle_to_queue(queue: &mut Queue, to_queue: ToQueue) -> Result<(), Fat
|
||||
async fn handle_enqueue(
|
||||
queue: &mut Queue,
|
||||
priority: Priority,
|
||||
pvf: Pvf,
|
||||
pvf_with_params: PvfWithExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
) -> Result<(), Fatal> {
|
||||
gum::debug!(
|
||||
target: LOG_TARGET,
|
||||
validation_code_hash = ?pvf.code_hash,
|
||||
validation_code_hash = ?pvf_with_params.code_hash(),
|
||||
?priority,
|
||||
?preparation_timeout,
|
||||
"PVF is enqueued for preparation.",
|
||||
);
|
||||
queue.metrics.prepare_enqueued();
|
||||
|
||||
let artifact_id = pvf.as_artifact_id();
|
||||
let artifact_id = pvf_with_params.as_artifact_id();
|
||||
if never!(
|
||||
queue.artifact_id_to_job.contains_key(&artifact_id),
|
||||
"second Enqueue sent for a known artifact"
|
||||
@@ -247,7 +254,10 @@ async fn handle_enqueue(
|
||||
return Ok(())
|
||||
}
|
||||
|
||||
let job = queue.jobs.insert(JobData { priority, pvf, preparation_timeout, worker: None });
|
||||
let job =
|
||||
queue
|
||||
.jobs
|
||||
.insert(JobData { priority, pvf_with_params, preparation_timeout, worker: None });
|
||||
queue.artifact_id_to_job.insert(artifact_id, job);
|
||||
|
||||
if let Some(available) = find_idle_worker(queue) {
|
||||
@@ -338,7 +348,7 @@ async fn handle_worker_concluded(
|
||||
// this can't be None;
|
||||
// qed.
|
||||
let job_data = never_none!(queue.jobs.remove(job));
|
||||
let artifact_id = job_data.pvf.as_artifact_id();
|
||||
let artifact_id = job_data.pvf_with_params.as_artifact_id();
|
||||
|
||||
queue.artifact_id_to_job.remove(&artifact_id);
|
||||
|
||||
@@ -424,7 +434,7 @@ async fn spawn_extra_worker(queue: &mut Queue, critical: bool) -> Result<(), Fat
|
||||
async fn assign(queue: &mut Queue, worker: Worker, job: Job) -> Result<(), Fatal> {
|
||||
let job_data = &mut queue.jobs[job];
|
||||
|
||||
let artifact_id = job_data.pvf.as_artifact_id();
|
||||
let artifact_id = job_data.pvf_with_params.as_artifact_id();
|
||||
let artifact_path = artifact_id.path(&queue.cache_path);
|
||||
|
||||
job_data.worker = Some(worker);
|
||||
@@ -435,8 +445,9 @@ async fn assign(queue: &mut Queue, worker: Worker, job: Job) -> Result<(), Fatal
|
||||
&mut queue.to_pool_tx,
|
||||
pool::ToPool::StartWork {
|
||||
worker,
|
||||
code: job_data.pvf.code.clone(),
|
||||
code: job_data.pvf_with_params.code(),
|
||||
artifact_path,
|
||||
executor_params: job_data.pvf_with_params.executor_params(),
|
||||
preparation_timeout: job_data.preparation_timeout,
|
||||
},
|
||||
)
|
||||
@@ -503,8 +514,8 @@ mod tests {
|
||||
use std::task::Poll;
|
||||
|
||||
/// Creates a new PVF which artifact id can be uniquely identified by the given number.
|
||||
fn pvf(descriminator: u32) -> Pvf {
|
||||
Pvf::from_discriminator(descriminator)
|
||||
fn pvf_with_params(descriminator: u32) -> PvfWithExecutorParams {
|
||||
PvfWithExecutorParams::from_discriminator(descriminator)
|
||||
}
|
||||
|
||||
async fn run_until<R>(
|
||||
@@ -613,7 +624,7 @@ mod tests {
|
||||
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: pvf(1),
|
||||
pvf_with_params: pvf_with_params(1),
|
||||
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
|
||||
});
|
||||
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
|
||||
@@ -626,7 +637,10 @@ mod tests {
|
||||
result: Ok(PrepareStats::default()),
|
||||
});
|
||||
|
||||
assert_eq!(test.poll_and_recv_from_queue().await.artifact_id, pvf(1).as_artifact_id());
|
||||
assert_eq!(
|
||||
test.poll_and_recv_from_queue().await.artifact_id,
|
||||
pvf_with_params(1).as_artifact_id()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -635,12 +649,20 @@ mod tests {
|
||||
|
||||
let priority = Priority::Normal;
|
||||
let preparation_timeout = PRECHECK_PREPARATION_TIMEOUT;
|
||||
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(1), preparation_timeout });
|
||||
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(2), preparation_timeout });
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
|
||||
preparation_timeout,
|
||||
});
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
|
||||
preparation_timeout,
|
||||
});
|
||||
// Start a non-precheck preparation for this one.
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf: pvf(3),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(3),
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
});
|
||||
|
||||
@@ -669,7 +691,7 @@ mod tests {
|
||||
// Enqueue a critical job.
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Critical,
|
||||
pvf: pvf(4),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(4),
|
||||
preparation_timeout,
|
||||
});
|
||||
|
||||
@@ -685,7 +707,7 @@ mod tests {
|
||||
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: pvf(1),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
|
||||
preparation_timeout,
|
||||
});
|
||||
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
|
||||
@@ -696,7 +718,7 @@ mod tests {
|
||||
// Enqueue a critical job, which warrants spawning over the soft limit.
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Critical,
|
||||
pvf: pvf(2),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
|
||||
preparation_timeout,
|
||||
});
|
||||
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
|
||||
@@ -722,12 +744,20 @@ mod tests {
|
||||
|
||||
let priority = Priority::Normal;
|
||||
let preparation_timeout = PRECHECK_PREPARATION_TIMEOUT;
|
||||
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(1), preparation_timeout });
|
||||
test.send_queue(ToQueue::Enqueue { priority, pvf: pvf(2), preparation_timeout });
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
|
||||
preparation_timeout,
|
||||
});
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(2),
|
||||
preparation_timeout,
|
||||
});
|
||||
// Start a non-precheck preparation for this one.
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority,
|
||||
pvf: pvf(3),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(3),
|
||||
preparation_timeout: LENIENT_PREPARATION_TIMEOUT,
|
||||
});
|
||||
|
||||
@@ -753,7 +783,10 @@ mod tests {
|
||||
// Since there is still work, the queue requested one extra worker to spawn to handle the
|
||||
// remaining enqueued work items.
|
||||
assert_eq!(test.poll_and_recv_to_pool().await, pool::ToPool::Spawn);
|
||||
assert_eq!(test.poll_and_recv_from_queue().await.artifact_id, pvf(1).as_artifact_id());
|
||||
assert_eq!(
|
||||
test.poll_and_recv_from_queue().await.artifact_id,
|
||||
pvf_with_params(1).as_artifact_id()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -762,7 +795,7 @@ mod tests {
|
||||
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: pvf(1),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
|
||||
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
|
||||
});
|
||||
|
||||
@@ -787,7 +820,7 @@ mod tests {
|
||||
|
||||
test.send_queue(ToQueue::Enqueue {
|
||||
priority: Priority::Normal,
|
||||
pvf: pvf(1),
|
||||
pvf_with_params: PvfWithExecutorParams::from_discriminator(1),
|
||||
preparation_timeout: PRECHECK_PREPARATION_TIMEOUT,
|
||||
});
|
||||
|
||||
|
||||
@@ -34,6 +34,7 @@ use crate::{
|
||||
use cpu_time::ProcessTime;
|
||||
use futures::{pin_mut, select_biased, FutureExt};
|
||||
use parity_scale_codec::{Decode, Encode};
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
use sp_core::hexdisplay::HexDisplay;
|
||||
use std::{
|
||||
panic,
|
||||
@@ -85,6 +86,7 @@ pub async fn start_work(
|
||||
code: Arc<Vec<u8>>,
|
||||
cache_path: &Path,
|
||||
artifact_path: PathBuf,
|
||||
executor_params: ExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
) -> Outcome {
|
||||
let IdleWorker { stream, pid } = worker;
|
||||
@@ -97,7 +99,9 @@ pub async fn start_work(
|
||||
);
|
||||
|
||||
with_tmp_file(stream, pid, cache_path, |tmp_file, mut stream| async move {
|
||||
if let Err(err) = send_request(&mut stream, code, &tmp_file, preparation_timeout).await {
|
||||
if let Err(err) =
|
||||
send_request(&mut stream, code, &tmp_file, &executor_params, preparation_timeout).await
|
||||
{
|
||||
gum::warn!(
|
||||
target: LOG_TARGET,
|
||||
worker_pid = %pid,
|
||||
@@ -271,15 +275,19 @@ async fn send_request(
|
||||
stream: &mut UnixStream,
|
||||
code: Arc<Vec<u8>>,
|
||||
tmp_file: &Path,
|
||||
executor_params: &ExecutorParams,
|
||||
preparation_timeout: Duration,
|
||||
) -> io::Result<()> {
|
||||
framed_send(stream, &code).await?;
|
||||
framed_send(stream, path_to_bytes(tmp_file)).await?;
|
||||
framed_send(stream, &executor_params.encode()).await?;
|
||||
framed_send(stream, &preparation_timeout.encode()).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf, Duration)> {
|
||||
async fn recv_request(
|
||||
stream: &mut UnixStream,
|
||||
) -> io::Result<(Vec<u8>, PathBuf, ExecutorParams, Duration)> {
|
||||
let code = framed_recv(stream).await?;
|
||||
let tmp_file = framed_recv(stream).await?;
|
||||
let tmp_file = bytes_to_path(&tmp_file).ok_or_else(|| {
|
||||
@@ -288,6 +296,13 @@ async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf,
|
||||
"prepare pvf recv_request: non utf-8 artifact path".to_string(),
|
||||
)
|
||||
})?;
|
||||
let executor_params_enc = framed_recv(stream).await?;
|
||||
let executor_params = ExecutorParams::decode(&mut &executor_params_enc[..]).map_err(|_| {
|
||||
io::Error::new(
|
||||
io::ErrorKind::Other,
|
||||
"prepare pvf recv_request: failed to decode ExecutorParams".to_string(),
|
||||
)
|
||||
})?;
|
||||
let preparation_timeout = framed_recv(stream).await?;
|
||||
let preparation_timeout = Duration::decode(&mut &preparation_timeout[..]).map_err(|e| {
|
||||
io::Error::new(
|
||||
@@ -295,7 +310,7 @@ async fn recv_request(stream: &mut UnixStream) -> io::Result<(Vec<u8>, PathBuf,
|
||||
format!("prepare pvf recv_request: failed to decode duration: {:?}", e),
|
||||
)
|
||||
})?;
|
||||
Ok((code, tmp_file, preparation_timeout))
|
||||
Ok((code, tmp_file, executor_params, preparation_timeout))
|
||||
}
|
||||
|
||||
async fn send_response(stream: &mut UnixStream, result: PrepareResult) -> io::Result<()> {
|
||||
@@ -347,7 +362,8 @@ pub fn worker_entrypoint(socket_path: &str) {
|
||||
worker_event_loop("prepare", socket_path, |rt_handle, mut stream| async move {
|
||||
loop {
|
||||
let worker_pid = std::process::id();
|
||||
let (code, dest, preparation_timeout) = recv_request(&mut stream).await?;
|
||||
let (code, dest, executor_params, preparation_timeout) =
|
||||
recv_request(&mut stream).await?;
|
||||
gum::debug!(
|
||||
target: LOG_TARGET,
|
||||
%worker_pid,
|
||||
@@ -372,7 +388,7 @@ pub fn worker_entrypoint(socket_path: &str) {
|
||||
// Spawn another thread for preparation.
|
||||
let prepare_fut = rt_handle
|
||||
.spawn_blocking(move || {
|
||||
let result = prepare_artifact(&code);
|
||||
let result = prepare_artifact(&code, executor_params);
|
||||
|
||||
// Get the `ru_maxrss` stat. If supported, call getrusage for the thread.
|
||||
#[cfg(target_os = "linux")]
|
||||
@@ -454,14 +470,17 @@ pub fn worker_entrypoint(socket_path: &str) {
|
||||
});
|
||||
}
|
||||
|
||||
fn prepare_artifact(code: &[u8]) -> Result<CompiledArtifact, PrepareError> {
|
||||
fn prepare_artifact(
|
||||
code: &[u8],
|
||||
executor_params: ExecutorParams,
|
||||
) -> Result<CompiledArtifact, PrepareError> {
|
||||
panic::catch_unwind(|| {
|
||||
let blob = match crate::executor_intf::prevalidate(code) {
|
||||
Err(err) => return Err(PrepareError::Prevalidation(format!("{:?}", err))),
|
||||
Ok(b) => b,
|
||||
};
|
||||
|
||||
match crate::executor_intf::prepare(blob) {
|
||||
match crate::executor_intf::prepare(blob, executor_params) {
|
||||
Ok(compiled_artifact) => Ok(CompiledArtifact::new(compiled_artifact)),
|
||||
Err(err) => Err(PrepareError::Preparation(format!("{:?}", err))),
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
use crate::artifacts::ArtifactId;
|
||||
use polkadot_parachain::primitives::ValidationCodeHash;
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
use sp_core::blake2_256;
|
||||
use std::{fmt, sync::Arc};
|
||||
|
||||
@@ -48,9 +49,47 @@ impl Pvf {
|
||||
let descriminator_buf = num.to_le_bytes().to_vec();
|
||||
Pvf::from_code(descriminator_buf)
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the artifact ID that corresponds to this PVF.
|
||||
/// Coupling PVF code with executor params
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PvfWithExecutorParams {
|
||||
pvf: Pvf,
|
||||
executor_params: Arc<ExecutorParams>,
|
||||
}
|
||||
|
||||
impl PvfWithExecutorParams {
|
||||
/// Creates a new PVF-ExecutorParams pair structure
|
||||
pub fn new(pvf: Pvf, executor_params: ExecutorParams) -> Self {
|
||||
Self { pvf, executor_params: Arc::new(executor_params) }
|
||||
}
|
||||
|
||||
/// Returns artifact ID that corresponds to the PVF with given executor params
|
||||
pub(crate) fn as_artifact_id(&self) -> ArtifactId {
|
||||
ArtifactId::new(self.code_hash)
|
||||
ArtifactId::new(self.pvf.code_hash, self.executor_params.hash())
|
||||
}
|
||||
|
||||
/// Returns validation code hash for the PVF
|
||||
pub(crate) fn code_hash(&self) -> ValidationCodeHash {
|
||||
self.pvf.code_hash
|
||||
}
|
||||
|
||||
/// Returns PVF code
|
||||
pub(crate) fn code(&self) -> Arc<Vec<u8>> {
|
||||
self.pvf.code.clone()
|
||||
}
|
||||
|
||||
/// Returns executor params
|
||||
pub(crate) fn executor_params(&self) -> ExecutorParams {
|
||||
(*self.executor_params).clone()
|
||||
}
|
||||
|
||||
/// Creates a structure for tests
|
||||
#[cfg(test)]
|
||||
pub(crate) fn from_discriminator(num: u32) -> Self {
|
||||
Self {
|
||||
pvf: Pvf::from_discriminator(num),
|
||||
executor_params: Arc::new(ExecutorParams::default()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,6 +19,8 @@
|
||||
//! N.B. This is not guarded with some feature flag. Overexposing items here may affect the final
|
||||
//! artifact even for production builds.
|
||||
|
||||
use polkadot_primitives::vstaging::ExecutorParams;
|
||||
|
||||
pub mod worker_common {
|
||||
pub use crate::worker_common::{spawn_with_program_path, SpawnErr};
|
||||
}
|
||||
@@ -35,12 +37,12 @@ pub fn validate_candidate(
|
||||
.expect("Decompressing code failed");
|
||||
|
||||
let blob = prevalidate(&code)?;
|
||||
let artifact = prepare(blob)?;
|
||||
let artifact = prepare(blob, ExecutorParams::default())?;
|
||||
let tmpdir = tempfile::tempdir()?;
|
||||
let artifact_path = tmpdir.path().join("blob");
|
||||
std::fs::write(&artifact_path, &artifact)?;
|
||||
|
||||
let executor = Executor::new()?;
|
||||
let executor = Executor::new(ExecutorParams::default())?;
|
||||
let result = unsafe {
|
||||
// SAFETY: This is trivially safe since the artifact is obtained by calling `prepare`
|
||||
// and is written into a temporary directory in an unmodified state.
|
||||
|
||||
@@ -251,6 +251,8 @@ pub enum SpawnErr {
|
||||
ProcessSpawn,
|
||||
/// The deadline allotted for the worker spawning and connecting to the socket has elapsed.
|
||||
AcceptTimeout,
|
||||
/// Failed to send handshake after successful spawning was signaled
|
||||
Handshake,
|
||||
}
|
||||
|
||||
/// This is a representation of a potentially running worker. Drop it and the process will be killed.
|
||||
|
||||
Reference in New Issue
Block a user