* Refactor PVF preparation memory stats
The original purpose of this change was to gate metrics that are unsupported by
some systems behind conditional compilation directives (#[cfg]); see
https://github.com/paritytech/polkadot/pull/6675#discussion_r1099996209.
Then I started doing some random cleanups and simplifications and got a bit
carried away. 🙈 The code should be overall tidier than before.
Changes:
- Don't register unsupported metrics (e.g. `max_rss` on non-Linux systems)
- Introduce `PrepareStats` struct as an abstraction over the `Ok` values of
`PrepareResult`. It is cleaner, and can be easily modified in the future.
- Other small changes
* Minor fixes to comments
* Fix compile errors
* Try to fix some Linux errors
* Mep
* Fix candidate-validation tests
* Update docstring
* WIP
* Fix templates
- fix conditions to fetch misc PRs
- render the misc section only if required
- add html comment to help debugging
- add a new API section to T2 labels
* Introduce jemalloc-stats feature flag
* remove unneeded space
* Update node/overseer/src/lib.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* Update Cargo.toml
Co-authored-by: Marcin S. <marcin@bytedude.com>
* revert making tikv-jemallocator depend on jemalloc-stats
* conditionally import memory_stats instead of using dead_code
* fix test via expllicit import
* Add jemalloc-stats feature to crates, propagate it from root
* Apply `jemalloc-stats` feature to prepare mem stats; small refactor
* effect changes recommended on PR
* Update node/overseer/src/metrics.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* fix compile error on in pipeline for linux. missing import
* Update node/overseer/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* revert to defining collect_memory_stats inline
---------
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Add getrusage and memory tracker for precheck preparation
* Log memory stats metrics after prechecking
* Fix tests
* Try to fix errors (linux-only so I'm relying on CI here)
* Try to fix CI
* Add module docs for `prepare/memory_stats.rs`; fix CI error
* Report memory stats for all preparation jobs
* Use `RUSAGE_SELF` instead of `RUSAGE_THREAD`
Not sure why I did that -- was a brainfart on my end.
* Revert last commit (RUSAGE_THREAD is correct)
* Use exponential buckets
* Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS
* Increase poll interval
* Revert "Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS"
This reverts commit becf7a815409ab530fc61370abffcd1b97b9a777.
* Introduce a job scanning and ensuring there are licenses
* Showcase a red test
* Add missing licenses
* Cleanup
* Extend the check
* Add missing licenses
* CI trigger
* Update beefy-merkle-tree dep to merkle-tree
* update merkle-tree to binary-merkle-tree
* update substrate refs
---------
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* disputes/slashing: slash only backers for ForInvalid
* add an assertion in mock impl
* fix tests
* do not slash backers on onconcluded disputes
* slash an intersection of backers and losers
* zombienet/disputes: check for offence only for invalid disputes
* add backing votes to disputes bench builder
* Update runtime/parachains/src/builder.rs
* Brad implementers guide revisions 2 (#6239)
* Add disputes subsystems fix
* Updated dispute approval vote import reasoning
* Improved wording of my changes
* Resolving issues brought up in comments
* Update disputes prioritisation in `dispute-coordinator` (#6130)
* Scraper processes CandidateBacked events
* Change definition of best-effort
* Fix `dispute-coordinator` tests
* Unit test for dispute filtering
* Clarification comment
* Add tests
* Fix logic
If a dispute is not backed, not included and not confirmed we
don't participate but we do import votes.
* Add metrics for refrained participations
* Revert "Add tests"
This reverts commit 7b8391a087922ced942cde9cd2b50ff3f633efc0.
* Revert "Unit test for dispute filtering"
This reverts commit 92ba5fe678214ab360306313a33c781338e600a0.
* fix dispute-coordinator tests
* Fix scraping
* new tests
* Small fixes in guide
* Apply suggestions from code review
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Fix some comments and remove a pointless test
* Code review feedback
* Clarification comment in tests
* Some tests
* Reference counted `CandidateHash` in scraper
* Proper handling for Backed and Included candidates in scraper
Backed candidates which are not included should be kept for a
predetermined window of finalized blocks. E.g. if a candidate is backed
but not included in block 2, and the window size is 2, the same
candidate should be cleaned after block 4 is finalized.
Add reference counting for candidates in scraper. A candidate can be
added on multiple block heights so we have to make sure we don't clean
it prematurely from the scraper.
Add tests.
* Update comments in tests
* Guide update
* Fix cleanup logic for `backed_candidates_by_block_number`
* Simplify cleanup
* Make spellcheck happy
* Update tests
* Extract candidate backing logic in separate struct
* Code review feedback
* Treat backed and included candidates in the same fashion
* Update some comments
* Small improvements in test
* spell check
* Fix some more comments
* clean -> prune
* Code review feedback
* Reword comment
* spelling
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* approval-voting: remove redundant validation check (#6266)
* approval-voting: remove a redundant check
* candidate-validation: remove unreachable check
* remove fill_block (#6200)
Co-authored-by: parity-processbot <>
* fix a compilation warning (#6279)
Fixes#6277.
* Only report concluded if there is an actual dispute. (#6270)
* Only report concluded if there is an actual dispute.
Hence no "non"-disputes will be added to disputes anymore.
* Fix redundant check.
* Test for no onesided disputes.
Co-authored-by: eskimor <eskimor@no-such-url.com>
* [ci] fix buildah image (#6281)
* Revert special casing of Kusama for grandpa rounds. (#6217)
Co-authored-by: eskimor <eskimor@no-such-url.com>
* Fixes "for loop over an `Option`" warnings (#6291)
Was seeing these warnings when running `cargo check --all`:
```
warning: for loop over an `Option`. This is more readably written as an `if let` statement
--> node/core/approval-voting/src/lib.rs:1147:21
|
1147 | for activated in update.activated {
| ^^^^^^^^^^^^^^^^
|
= note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
|
1147 | while let Some(activated) = update.activated {
| ~~~~~~~~~~~~~~~ ~~~
help: consider using `if let` to clear intent
|
1147 | if let Some(activated) = update.activated {
| ~~~~~~~~~~~~ ~~~
```
My guess is that `activated` used to be a SmallVec or similar, as is
`deactivated`. It was changed to an `Option`, the `for` still compiled (it's
technically correct, just weird), and the compiler didn't catch it until now.
* companion for #12599 (#6290)
* companion for #12599
* update Cargo.lock
* use cargo path instead of diener
* update lockfile for {"substrate"}
Co-authored-by: parity-processbot <>
* remove the runtime check and test
* append keys on past-session slashing
* runtime/disputes: allow importing backing votes after explicit for
* explicit MaliciousBacker error and a test
* update an outdated comment
* Revert "update an outdated comment"
This reverts commit 7c4c3f5a848f16e2b61435e981d814f00333ed41.
* Revert "remove the runtime check and test"
This reverts commit a5bff0c75e77effb5b7d3a1691de1b14bcdbd648.
* incremental punishment post conclusion + test
* punish backers post FOR vote
* remove unnecessary lifetime annotation
* add a comment to zombinet test
* typo
* fmt
* post merge test fixes
* fix test after changes in master
* address review nits
---------
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Sergej Sakac <73715684+Szegoo@users.noreply.github.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Marcin S <marcin@bytedude.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* Update benchmark's successful origin api
* rustfmt
* unwrap for indirect origin dep
* replace unwrap by expect with a message
---------
Co-authored-by: parity-processbot <>
* polkadot: clear migrations included in 0.9.37
* rococo: clear migrations included in 0.9.37
* westend: clear migrations included in 0.9.37
* kusama: clear migrations included in 0.9.37
* remove unused import: xcm_config::CheckAccount
Marcin S