* pre-checking: Reject failed PVFs
* paras: immediately reject any PVF that cannot reach a supermajority
* Make the `quorum` reject condition a bit more clear semantically
* Add comment
* Update implementer's guide
* Update a link
Not related to the rest of the PR, but I randomly noticed and fixed this.
* Update runtime/parachains/src/paras/tests.rs
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
* Remove unneeded loop
* Log PVF retries using `info!`
* Change retry logs to `warn!` and add preparation failure log
* Log PVF execution failure
* Clarify why we reject failed PVFs
* Fix PVF reject runtime benchmarks
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
* Replace async-std with tokio in PVF subsystem
* Rework workers to use `select!` instead of a mutex
The improvement in code readability is more important than the thread overhead.
* Remove unnecessary `fuse`
* Add explanation for `expect()`
* Update node/core/pvf/src/worker_common.rs
Co-authored-by: Bastian Köcher <info@kchr.de>
* Update node/core/pvf/src/worker_common.rs
Co-authored-by: Bastian Köcher <info@kchr.de>
* Address some review comments
* Shutdown tokio runtime
* Run cargo fmt
* Add a small note about retries
* Fix up merge
* Rework `cpu_time_monitor_loop` to return when other thread finishes
* Add error string to PrepareError::IoErr variant
* Log when artifacts fail to prepare
* Fix `cpu_time_monitor_loop`; fix test
* Fix text
* Fix a couple of potential minor data races.
First data race was due to logging in the CPU monitor thread even if the
job (other thread) finished. It can technically finish before or after the log.
Maybe best would be to move this log to the `select!`s, where we are guaranteed
to have chosen the timed-out branch, although there would be a bit of
duplication.
Also, it was possible for this thread to complete before we executed
`finished_tx.send` in the other thread, which would trigger an error as the
receiver has already been dropped. And right now, such a spurious error from
`send` would be returned even if the job otherwise succeeded.
* Update Cargo.lock
Co-authored-by: Bastian Köcher <info@kchr.de>
* PVF preparation: do not conflate errors
+ Adds some more granularity to the prepare errors.
+ Better distinguish whether errors occur on the host side or the worker.
+ Do not kill the worker if the error happened on the host side.
+ Do not retry preparation if the error was `Panic`.
+ Removes unnecessary indirection with `Selected` type.
* Add missing docs, resolve TODOs
* Address review comments and remove TODOs
* Fix error in CI
* Undo unnecessary change
* Update couple of comments
* Don't return error for stream shutdown
* Update node/core/pvf/src/worker_common.rs
* rust 1.64 enables workspace properties
* add edition, repository and authors.
* of course, update the version in one place.
Co-authored-by: Andronik <write@reusable.software>
* Put in skeleton logic for CPU-time-preparation
Still needed:
- Flesh out logic
- Refactor some spots
- Tests
* Continue filling in logic for prepare worker CPU time changes
* Fix compiler errors
* Update lenience factor
* Fix some clippy lints for PVF module
* Fix compilation errors
* Address some review comments
* Add logging
* Add another log
* Address some review comments; change Mutex to AtomicBool
* Refactor handling response bytes
* Add CPU clock timeout logic for execute jobs
* Properly handle AtomicBool flag
* Use `Ordering::Relaxed`
* Refactor thread coordination logic
* Fix bug
* Add some timing information to execute tests
* Add section about the mitigation to the IG
* minor: Change more `Ordering`s to `Relaxed`
* candidate-validation: Fix build errors
* Add clippy config and remove .cargo from gitignore
* first fixes
* Clippyfied
* Add clippy CI job
* comment out rusty-cachier
* minor
* fix ci
* remove DAG from check-dependent-project
* add DAG to clippy
Co-authored-by: alvicsam <alvicsam@gmail.com>
* westend: update transaction version
* polkadot: update transaction version
* kusama: update transaction version
* Bump spec_version to 9330
* bump versions to 0.9.33
* Add PVF module documentation
TODO (once the PRs land):
- [ ] Document executor parametrization.
- [ ] Document CPU time measurement of timeouts.
* Update node/core/pvf/src/lib.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Clarify meaning of PVF acronym
* Move PVF doc to implementer's guide
* Clean up implementer's guide a bit
* Add page for PVF types
* pvf: Better separation between crate docs and implementer's guide
* ci: Add "prevalidating" to the dictionary
* ig: Remove types/chain.md
The types contained therein did not exist and the file was not referenced
anywhere.
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Fix a couple of typos
* Retry failed PVF execution
PVF execution that fails due to AmbiguousWorkerDeath should be retried once.
This should reduce the occurrence of failures due to transient conditions.
Closes#6195
* Address a couple of nits
* Write tests; refactor (add `validate_candidate_with_retry`)
* Update node/core/candidate-validation/src/lib.rs
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
* Rename timeout consts and timeout parameter; bump leniency
* Update implementor's guide with info about PVFs
* Make glossary a bit easier to read
* Add a note to LENIENT_PREPARATION_TIMEOUT
* Remove PVF-specific section from glossary
* Fix some typos
* Bump crate versions
* Bump spec_version to 9280 for kusama
* Bump spec_version to 9280 for polkadot
* Bump spec_version to 9280 for rococo
* Bump spec_version to 9280 for westend
* update Cargo.lock
Co-authored-by: parity-processbot <>
The PVF host is designed to avoid spawning tasks to minimize knowledge
of outer code. Using `async_std::task::spawn` (or Tokio's counterpart)
deemed unacceptable, `SpawnNamed` undesirable. Instead there is only one
task returned that is spawned by the candidate-validation subsystem.
The tasks from the sub-components are polled by that root task.
However, the way the tasks are bundled was incorrect. There was a giant
select that was polling those tasks. Particularly, that implies that as soon as
one of the arms of that select goes into await those sub-tasks stop
getting polled. This is a recipe for a deadlock which indeed happened
here.
Specifically, the deadlock happened during sending messages to the
execute queue by calling
[`send_execute`](https://github.com/paritytech/polkadot/blob/a68d9be35656dcd96e378fd9dd3d613af754d48a/node/core/pvf/src/host.rs#L601).
When the channel to the queue reaches the capacity, the control flow is
suspended until the queue handles those messages. Since this code is
essentially reached from [one of the select
arms](https://github.com/paritytech/polkadot/blob/a68d9be35656dcd96e378fd9dd3d613af754d48a/node/core/pvf/src/host.rs#L371),
the queue won't be given the control and thus no further progress can be
made.
This problem is solved by bundling the tasks one level higher instead,
by `selecting` over those long-running tasks.
We also stop treating returning from those long-running tasks as error
conditions, since that can happen during legit shutdown.
* Rename to BagError
* Additional parameter for 'revert' command
* Set aux revert param to None
* Align to changes in how the WASM executor is configured in `substrate`
* update lockfile for {"substrate"}
* update lockfile for {"substrate"}
* Update substrate
* Update substrate
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: parity-processbot <>
Marcin S