* PVF: Remove `rayon` and some uses of `tokio`
1. We were using `rayon` to spawn a superfluous thread to do execution, so it was removed.
2. We were using `rayon` to set a threadpool-specific thread stack size, and AFAIK we couldn't do that with `tokio` (it's possible [per-runtime](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.thread_stack_size) but not per-thread). Since we want to remove `tokio` from the workers [anyway](https://github.com/paritytech/polkadot/issues/7117), I changed it to spawn threads with the `std::thread` API instead of `tokio`.[^1]
[^1]: NOTE: This PR does not totally remove the `tokio` dependency just yet.
3. Since `std::thread` API is not async, we could no longer `select!` on the threads as futures, so the `select!` was changed to a naive loop.
4. The order of thread selection was flipped to make (3) sound (see note in code).
I left some TODO's related to panics which I'm going to address soon as part of https://github.com/paritytech/polkadot/issues/7045.
* PVF: Vote invalid on panics in execution thread (after a retry)
Also make sure we kill the worker process on panic errors and internal errors to
potentially clear any error states independent of the candidate.
* Address a couple of TODOs
Addresses a couple of follow-up TODOs from
https://github.com/paritytech/polkadot/pull/7153.
* Add some documentation to implementer's guide
* Fix compile error
* Fix compile errors
* Fix compile error
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Address comments + couple other changes (see message)
- Measure the CPU time in the prepare thread, so the observed time is not
affected by any delays in joining on the thread.
- Measure the full CPU time in the execute thread.
* Implement proper thread synchronization
Use condvars i.e. `Arc::new((Mutex::new(true), Condvar::new()))` as per the std
docs.
Considered also using a condvar to signal the CPU thread to end, in place of an
mpsc channel. This was not done because `Condvar::wait_timeout_while` is
documented as being imprecise, and `mpsc::Receiver::recv_timeout` is not
documented as such. Also, we would need a separate condvar, to avoid this case:
the worker thread finishes its job, notifies the condvar, the CPU thread returns
first, and we join on it and not the worker thread. So it was simpler to leave
this part as is.
* Catch panics in threads so we always notify condvar
* Use `WaitOutcome` enum instead of bool condition variable
* Fix retry timeouts to depend on exec timeout kind
* Address review comments
* Make the API for condvars in workers nicer
* Add a doc
* Use condvar for memory stats thread
* Small refactor
* Enumerate internal validation errors in an enum
* Fix comment
* Add a log
* Fix test
* Update variant naming
* Address a missed TODO
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
There is a deny(clippy::dbg_macro) in the crate root, so newer
Clippy fails here since tests use dbg.
But dbg in tests are fine IMHO.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* test that successfully verify self-generated proof
* verify proof result on all unpaused nodes
* tighten interface: reuse connection
* generate the proof on arbitrary node
* s/stub-name/validator
seems this doesn't pass some regex
* rename script
* add basic mmr-leaves test
* documentation formatting
* check lower bound on mmr leaves
* test beefy rpc: finalized heads match
* test mmr api: stateless proofs
* set lower bound on number of leaves
* change leaves in proof generation
* remove TODOs
* cleanup: consistently ignore zndsl name arg
* refactor: simplify returns
Co-authored-by: Serban Iorga <serban@parity.io>
* lax finalized head test
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* fixup! refactor: simplify returns
* refactor out getApis
Co-authored-by: Serban Iorga <serban@parity.io>
* split out paused validator from group
* refactor: don't ignore node arg
don't randomize selection of proof generating / finalized head
retrieving node, but use the node arg for this.
* only check min block height, not relative
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* verify finalized heads share same canonical chain
picks the node with the highest finalized head, gets its blockchain
headers since genesis, and verifies that all other nodes' finalized
heads are in said blockchain.
* fixup! split out paused validator from group
Co-authored-by: Serban Iorga <serban@parity.io>
---------
Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* companion for #13384
* adjust parsing RPC address from process output
* update rpc cli
* update lockfile for {"substrate"}
* bump zombienet v1.3.48
* bump zombienet version
* allow zombienet-tests-misc-upgrade-node to fail
* add comment and issue link to allowed_failure
* grumbles: disable failed job
* disabled the correct test
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Javier Viola <javier@parity.io>
* add swapped event to registrar
* check swapped event on integration test
* check for swapped event in paras_registrar swap tests
* cargo +nightly fmt --all
* add test for para to para swap
* make paraId event consistent
* cargo +nightly fmt
Bastian Köcher