Commit Graph
36 Commits
Author SHA1 Message Date
Marcin Sandparity-processbot 082d9e1d6e Companion for Polkadot#7337 (#2929)
* Companion for Polkadot#7337

Companion for paritytech/polkadot#7337

* Remove unnecessary items relating to PVF workers

* Remove `dont_use_external_workers` parameter

* Update Cargo.lock

* update lockfile for {"polkadot", "substrate"}

* Update Cargo.lock

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>
2023-08-01 08:32:20 +00:00
Marcin Sandparity-processbot 972814a2be Companion for Polkadot 7253 (#2621)
* Companion for Polkadot #7253

Polkadot PR: https://github.com/paritytech/polkadot/pull/7253

* Update Cargo.lock

* Update Cargo.lock

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>
2023-05-25 21:07:56 +00:00
Marcin S 2decceef76 Executor: Add create_runtime_from_artifact_bytes (#14184) 2023-05-22 11:43:51 -04:00
Marcin Sandparity-processbot 457d1724df [Polkadot Companion] for 7101 (#2470)
* [Polkadot Companion] for 7101

PR: https://github.com/paritytech/polkadot/pull/7101

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>
2023-04-21 11:04:39 +00:00
Marcin S bfd43dbce5 WASM executor: add OutputExceedsBounds variant to Error (#13841)
* WASM executor: add `OutputExceedsBounds` variant to `Error`

Previously this was a `WasmError`, which is intended for runtime construction errors. However this
led to confusion as output-exceeds-bounds occurs due to execution of `validate_block`.

* Fix warning
2023-04-07 10:35:41 +00:00
Marcin Sandparity-processbot 9691acacd4 Polkadot companion #6603: Use a BoundedVec in ValidationResult (#2161)
* Convert message `Vec`s into `BoundedVec`s

* cargo fmt

* Fix missing conversions in collator

* Fix the fix

* Fix the fix to the fix

* Fix tests

* Fix^4

* Avoid use of unwrap

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: parity-processbot <>
2023-02-16 15:50:00 +00:00
Marcin S d5a7991429 Use a BoundedVec in ValidationResult (#6603)
* Use a `BoundedVec` in `ValidationResult`

> Use a `BoundedVec` for `upward_messages` and `horizontal_messages` in order to
> limit the number of individual messages/memory allocations right at decoding
> time. The reason for this is that the `ValidationResult` may contain a code
> upgrade (including a full PVF binary), so the total size limit can't be set
> too low and this limit will still allow several millions of upward messages,
> which will (due to the memory allocator overhead) already have a
> non-negligible memory footprint in decoded form.

* List all fields when hashing so we don't miss one

* Define types for  `BoundedVec`s of messages

* Fix test compile errors

* Depend on `bounded-collections` 0.1.4 (fixes allocation issue)

* Fix compilation issue

* Derive `Hash` instead of manual `impl`

* Avoid use of unwrap
2023-02-16 15:08:56 +00:00
Marcin S fd70d01274 Refactor PVF preparation memory stats (#6693)
* Refactor PVF preparation memory stats

The original purpose of this change was to gate metrics that are unsupported by
some systems behind conditional compilation directives (#[cfg]); see
https://github.com/paritytech/polkadot/pull/6675#discussion_r1099996209.

Then I started doing some random cleanups and simplifications and got a bit
carried away. 🙈 The code should be overall tidier than before.

Changes:
- Don't register unsupported metrics (e.g. `max_rss` on non-Linux systems)
- Introduce `PrepareStats` struct as an abstraction over the `Ok` values of
  `PrepareResult`. It is cleaner, and can be easily modified in the future.
- Other small changes

* Minor fixes to comments

* Fix compile errors

* Try to fix some Linux errors

* Mep

* Fix candidate-validation tests

* Update docstring
2023-02-14 15:48:57 +00:00
Marcin S e004d5a5ed Fix dependabot labels (#6674) 2023-02-07 10:32:14 +00:00
Marcin S f317115b99 pvf: Log memory metrics from preparation (#6565)
* Add getrusage and memory tracker for precheck preparation

* Log memory stats metrics after prechecking

* Fix tests

* Try to fix errors (linux-only so I'm relying on CI here)

* Try to fix CI

* Add module docs for `prepare/memory_stats.rs`; fix CI error

* Report memory stats for all preparation jobs

* Use `RUSAGE_SELF` instead of `RUSAGE_THREAD`

Not sure why I did that -- was a brainfart on my end.

* Revert last commit (RUSAGE_THREAD is correct)

* Use exponential buckets

* Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS

* Increase poll interval

* Revert "Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS"

This reverts commit becf7a815409ab530fc61370abffcd1b97b9a777.
2023-02-06 11:17:21 +00:00
Marcin S 0d79b7853c Minor: Update output validity tests (#13190)
* Minor: Update output validity tests

Quick follow-up to https://github.com/paritytech/substrate/pull/13183.

Mainly, I wanted to double check that the `test_return_max_memory_offset` test doesn't pass just
because the output length is 0.

I also:

- Organized these tests into a module.
- Added a comment explaining why we don't use the `wasm_export_functions` macro.

* Update test based on review comment
2023-02-02 13:41:40 -05:00
Marcin SandBastian Köcher e8288005f7 Fix potential huge allocation as a result of validate_block output (#13183)
* Fix potential huge allocation as a result of `validate_block` output

* Address review comments; add more tests

* Update client/executor/wasmtime/src/runtime.rs

* Remove unnecessary comments

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-20 10:19:30 +00:00
Marcin Sands0me0ne-unkn0wn 2efa3bab98 pre-checking: Reject failed PVFs (#6492)
* pre-checking: Reject failed PVFs

* paras: immediately reject any PVF that cannot reach a supermajority

* Make the `quorum` reject condition a bit more clear semantically

* Add comment

* Update implementer's guide

* Update a link

Not related to the rest of the PR, but I randomly noticed and fixed this.

* Update runtime/parachains/src/paras/tests.rs

Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>

* Remove unneeded loop

* Log PVF retries using `info!`

* Change retry logs to `warn!` and add preparation failure log

* Log PVF execution failure

* Clarify why we reject failed PVFs

* Fix PVF reject runtime benchmarks

Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
2023-01-12 09:24:42 +00:00
Marcin S ae74d33a93 pvf: Fix missing execution request when retrying preparation (#6537)
* pvf: Add checks for result sender when retrying preparation in tests

* pvf: Fix missing execution request when retrying preparation

* Update comment
2023-01-11 20:01:43 +00:00
Marcin S 3cf18f11ac Minor fixes (#6533)
* Minor fixes

* Fix compile errors
2023-01-10 17:59:12 +00:00
Marcin SandBastian Köcher 44fd95661c Replace async-std with tokio in PVF subsystem (#6419)
* Replace async-std with tokio in PVF subsystem

* Rework workers to use `select!` instead of a mutex

The improvement in code readability is more important than the thread overhead.

* Remove unnecessary `fuse`

* Add explanation for `expect()`

* Update node/core/pvf/src/worker_common.rs

Co-authored-by: Bastian Köcher <info@kchr.de>

* Update node/core/pvf/src/worker_common.rs

Co-authored-by: Bastian Köcher <info@kchr.de>

* Address some review comments

* Shutdown tokio runtime

* Run cargo fmt

* Add a small note about retries

* Fix up merge

* Rework `cpu_time_monitor_loop` to return when other thread finishes

* Add error string to PrepareError::IoErr variant

* Log when artifacts fail to prepare

* Fix `cpu_time_monitor_loop`; fix test

* Fix text

* Fix a couple of potential minor data races.

First data race was due to logging in the CPU monitor thread even if the
job (other thread) finished. It can technically finish before or after the log.

Maybe best would be to move this log to the `select!`s, where we are guaranteed
to have chosen the timed-out branch, although there would be a bit of
duplication.

Also, it was possible for this thread to complete before we executed
`finished_tx.send` in the other thread, which would trigger an error as the
receiver has already been dropped. And right now, such a spurious error from
`send` would be returned even if the job otherwise succeeded.

* Update Cargo.lock

Co-authored-by: Bastian Köcher <info@kchr.de>
2023-01-10 10:51:13 +01:00
Marcin S 715e98268a Use polkadot-node-metrics where possible (#6484)
See https://github.com/paritytech/polkadot/issues/6015. (Doesn't close yet,
there might be more we can do.)
2023-01-06 15:01:23 +00:00
Marcin S f0106b30fa Log PVF retries (#6504) 2023-01-04 22:18:41 +00:00
Marcin S e0a0475a05 PVF preparation: do not conflate errors (#6384)
* PVF preparation: do not conflate errors

+ Adds some more granularity to the prepare errors.
+ Better distinguish whether errors occur on the host side or the worker.
+ Do not kill the worker if the error happened on the host side.
+ Do not retry preparation if the error was `Panic`.
+ Removes unnecessary indirection with `Selected` type.

* Add missing docs, resolve TODOs

* Address review comments and remove TODOs

* Fix error in CI

* Undo unnecessary change

* Update couple of comments

* Don't return error for stream shutdown

* Update node/core/pvf/src/worker_common.rs
2022-12-20 08:32:12 -05:00
Marcin S ff9c01d585 Remove parity-util-mem (#1958)
Fully removes the `parity-util-mem` dependency. This just required upgrading to the latest versions
of the polkadot crates.
2022-12-06 17:05:33 -05:00
Marcin S ab090ab7d5 Let the PVF host kill the worker on timeout (#6381)
* Let the PVF host kill the worker on timeout

* Fix comment

* Fix inaccurate comments; add missing return statement

* Fix a comment

* Fix comment
2022-12-06 13:03:18 -05:00
Marcin Sparity-processbotAndronik
3f951cce16 Companion for paritytech/substrate#12795 (#6374)
* Begin removing `parity-util-mem`; remove `collect_memory_stats`

* Update some dependencies that were using `parity-util-mem`

* Remove `trie-memory-tracker` feature

* Update Cargo.lock

* Update `kvdb-shared-tests`

* Add back jemalloc

* Add missing license header

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
Co-authored-by: Andronik <write@reusable.software>
2022-12-06 16:41:00 +00:00
Marcin Sandordian aa21e56744 Remove mem_info and references to parity-util-mem (#12795)
* Remove mem_info and some references to parity-util-mem

* [Draft] Finish removing references to `parity-util-mem`

* Upgrade dependencies

* Update scripts/ci/deny.toml

Co-authored-by: ordian <write@reusable.software>

* Fix Cargo.lock (remove unwanted dependency changes)

* Removed unused argument

* Run cargo fmt (didn't have pre-commit set up)

* Fix some CI errors

* Fix another CI error

* Remove unused dependency

Co-authored-by: ordian <write@reusable.software>
2022-12-06 14:55:10 +00:00
Marcin S 28a4e90912 Use CPU clock timeout for PVF jobs (#6282)
* Put in skeleton logic for CPU-time-preparation

Still needed:
- Flesh out logic
- Refactor some spots
- Tests

* Continue filling in logic for prepare worker CPU time changes

* Fix compiler errors

* Update lenience factor

* Fix some clippy lints for PVF module

* Fix compilation errors

* Address some review comments

* Add logging

* Add another log

* Address some review comments; change Mutex to AtomicBool

* Refactor handling response bytes

* Add CPU clock timeout logic for execute jobs

* Properly handle AtomicBool flag

* Use `Ordering::Relaxed`

* Refactor thread coordination logic

* Fix bug

* Add some timing information to execute tests

* Add section about the mitigation to the IG

* minor: Change more `Ordering`s to `Relaxed`

* candidate-validation: Fix build errors
2022-11-30 13:17:31 +01:00
Marcin S b5d5a7ee4f Remove parity-util-mem from runtime-api cache (#6367) 2022-11-29 15:28:44 +00:00
Marcin S 186a572907 Provide some more granular metrics for polkadot_pvf_execution_time (#6346) 2022-11-28 10:26:17 -05:00
Marcin S 01f8154110 ig: Fix description of execution retry delay (#6342)
The delay is 3s and not 1s. I removed the reference to a specific number of
seconds as it may be too specific for a high-level description.
2022-11-24 08:06:56 -05:00
Marcin SandAndrei Sandu 1dec2433ae Add PVF module documentation (#6293)
* Add PVF module documentation

TODO (once the PRs land):

- [ ] Document executor parametrization.

- [ ] Document CPU time measurement of timeouts.

* Update node/core/pvf/src/lib.rs

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* Clarify meaning of PVF acronym

* Move PVF doc to implementer's guide

* Clean up implementer's guide a bit

* Add page for PVF types

* pvf: Better separation between crate docs and implementer's guide

* ci: Add "prevalidating" to the dictionary

* ig: Remove types/chain.md

The types contained therein did not exist and the file was not referenced
anywhere.

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2022-11-23 08:20:25 -05:00
Marcin S d53513ff66 Fixes "for loop over an Option" warnings (#6291)
Was seeing these warnings when running `cargo check --all`:

```
warning: for loop over an `Option`. This is more readably written as an `if let` statement
    --> node/core/approval-voting/src/lib.rs:1147:21
     |
1147 |             for activated in update.activated {
     |                              ^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
     |
1147 |             while let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~~~~         ~~~
help: consider using `if let` to clear intent
     |
1147 |             if let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~         ~~~
```

My guess is that `activated` used to be a SmallVec or similar, as is
`deactivated`. It was changed to an `Option`, the `for` still compiled (it's
technically correct, just weird), and the compiler didn't catch it until now.
2022-11-15 09:58:26 -05:00
6d7f33e612 Retry failed PVF execution (AmbiguousWorkerDeath) (#6235)
* Fix a couple of typos

* Retry failed PVF execution

PVF execution that fails due to AmbiguousWorkerDeath should be retried once.
This should reduce the occurrence of failures due to transient conditions.

Closes #6195

* Address a couple of nits

* Write tests; refactor (add `validate_candidate_with_retry`)

* Update node/core/candidate-validation/src/lib.rs

Co-authored-by: Andronik <write@reusable.software>

Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
2022-11-08 15:36:36 -05:00
Marcin S 63c6f184cb Retry failed PVF prepare jobs (#6213) 2022-11-08 13:30:14 +00:00
Marcin S 6cc91f9187 Add a last change footer to the implementers guide (#6216)
* Add a `last change` footer to the implementers guide

Some of the newcomers were noticing outdated pages in the implementer's guide.
This idea came up as a heuristic for how up-to-date an individual page is.

* Update `build-implementers-guide` CI job
2022-11-08 13:06:02 +00:00
Marcin S 1f8219767e PVF timeouts follow-up (#6151)
* Rename timeout consts and timeout parameter; bump leniency

* Update implementor's guide with info about PVFs

* Make glossary a bit easier to read

* Add a note to LENIENT_PREPARATION_TIMEOUT

* Remove PVF-specific section from glossary

* Fix some typos
2022-11-01 10:59:53 -04:00
Marcin Sandparity-processbot d4e3501ec2 Make some fixes to logging in PVF subsystem (#6180)
* Log exit status code for workers

* Make log for execute job conclusion match prepare job conclusion

Trace log for conclusion of prepare job:

```rs
gum::debug!(
	target: LOG_TARGET,
	validation_code_hash = ?artifact_id.code_hash,
	?worker,
	?rip,
	"prepare worker concluded",
);
```

Co-authored-by: parity-processbot <>
2022-10-26 17:49:51 +02:00
Marcin S aff57ef19d Add missing prerequisite in README for implementers guide (#6181) 2022-10-24 06:56:41 +00:00
Marcin S 17730b85be Separate preparation timeouts for PVF prechecking and execution (#6139)
* Add some documentation

* Add `compilation_timeout` parameter for PVF preparation job

* Update buckets in prometheus metrics

* Update prepare/queue tests

* Update pvf-prechecking overview in implementer docs

* Fix some CI checks
2022-10-13 11:00:57 +00:00