We wanted to change niceness to accomodate the fact that some of the
preparation tasks are low priority. For example, when a node sees that
there is a new para was onboarded the node may start preparing right
away. Since all other activities are more important, such as network I/O
or validation of the backed candidates and preparation of the
immediatelly needed PVFs.
However, it turned out that this approach does not work: generally
non-root processes can only decrease niceness and they cannot increase
it to the previous value, as was assumed by the code.
Apart from that, https://github.com/paritytech/polkadot/pull/4123
assumes all PVFs are prepared in the same way. Specifically, that if a
PVF preparation failed before, then PVF pre-checking will also report
that it was failed, even though it could happen that preparation failed
due to being low-priority. In order to avoid such cases, we decided to
simplify the whole preparation model. Preparation under low priority
does not work well with that.
Closes https://github.com/paritytech/polkadot/issues/4520
* Companion PR for removing Prometheus metrics prefix
* Was missing some metrics
* Fix missing renames
* Fix test
* Fixes
* Update test
* Update Substrate
* Second time
* remove prefix from intergration test for zombienet
* update zombienet image
* Update Substrate
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Javier Viola <pepoviola@gmail.com>
* remove Default from CandidateHash
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: fmt
* remove backed candidate default
* Partial migration away from CandidateReceipt::default
* Remove more CandidateReceipt defaults
* fmt
* Mostly remove CommittedCandidateReceipt default usage
* Remove CommittedCandidateReceipt
* Remove more Defaults from polakdot primitives v1 + fmt
* Remove more Default from polkadot primites v1
* WIP trying to get overseer example + tests to compile
* feat: add primitives test helpers
* reduce deps of helper
* update primitive helpers
* make candidate validation compile
* fixup cargo lock
* make av-store compile
* fixup disputes coordinator tests
* test: fixup backing
* test: fixup approval voting
* fixup bitfield signing
* test: fixup runtime-api
* test: fixup availability dist
* foxi[ pverseer test]
* remove some Defaults, remove bounds from `dummy`
All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.
* it's a test helper, so always use std
* test: fixup parachains runtime tests
Excluding benches.
* fix keyring
* fix paras runtime properly, no more default
* Remove fn dummy() usage from approval voting
* Move TestCandidateBuilder out of av store to test helpers
* Make candidate validation tests pass
* Make most dispute coirdinator tests pass
* Make provisioner tests work
* Make availability recovery tests work with test helpers
* Update polkadot-collator-protocol tests
* Update statement distribution tests
* Update polkadot overseer examples and tests
* Derive default for validation code so we don't break unrelated things
* Make para runtime test pass (no bench)
* Some more work
* chore: cargo fmt
* cargo fix
* avoid some Default::default
* fixup dispute coordinator test
* remove unused crate deps
* remove Default::default wherever possible, replace by dummy_* for the most part
* chore: cargo fmt
* Remove some warnings
* Remove CommittedCandidateReceipt dummy
* Remove CandidateReceipt dummy
* Remove CandidateDescriptor dummy
* Remove commented out code
* Fix para runtime tests
* chore: nightly
* Some updates to the builder
* Dynamically adjust mock head data size
* Make dispute cooridinator tests work
* Fix test candidate_backing_reorders_votes work
* +nightly-2021-10-29 fmt
* Spelling and remove a default use in builder
* Various clean up
* More small updates
* fmt
* More small updates
* Doc comments for test helpers
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs
* Update lib.rs
* review comments
* fix warnings
* fix test by using correct candidate receipt relay parent
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
* test/malus: craft the first maliciously disputing actor
* initial draft
* Add Dockerfile and instructions how to use it to build malus image locally
* Forgot one flag for the build cmd
* we are not docker specific, we are happy to use any containerruntime
* shuffle things around
* add initial tera based integration test
* chores
* fixins
* simple setup to start
* other samples (WIP)
* add Docker version with cargo-chef
* update substarte and small change of orders in commands in the container file
* metrics one
* fmt
* minor
* fixin
* fix metric names
* -d
* add open gauge
* fmt
* spellcheck
* fix test
* adjust to changed error messages
* refactor, more malus impls
* more malus changes
* foo
* minor cleanup
* suggest garbage candidate
* chore
* fix suggest garabge malus
* malus: back garbage candidate
* cargo lock
* re-introduce metrics
* chore: cargo fmt
* undoe 1.54.0 output, CI uses 1.53.0 rustc
* update location of js types
* Fix trybuild
* add tag to image name also; this will be replaced in the prod version
* Tests fixed
* add some fix me
* add dockerfile for ci
* Add docker file for malus for ci
* use variables in .toml file
* add chnages for malus test
* some fixes
* some more fixes
* Update .gitlab-ci.yml
* add local build for polkadot and malus
* some fixes
* enable disputes feature in CI
* ok, ok
* rename: MsgFilter -> MessageInterceptor
* remove TODO that would not have worked
* intercept
* refactor
* fix README and containers
* fix
* chore: cargo fmt
* avoid some more malus-$VARIANT references
* fix argument order
* chore: add about
* Update sanity check in relay chain selection
* fix order, add dispute-unavailable-block malus
* fixup: avoid underflow issue
* it's all u32
* fix conditional use
* Revert "it's all u32"
This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.
* Revert "fixup: avoid underflow issue"
This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.
* Revert "Update sanity check in relay chain selection"
This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.
* update the malus bin
* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some FIXME reminders
* update path to index.js
* Update .gitlab-ci.yml
* Update node/malus/integrationtests/0001-dispute-valid-block.toml
* try 1: make malus test run
* chore: cargo fmt
* temporary fix
* use subcommand syntax from latest gurke
* cargo +nightly fmt
* add collator to a a test
* docs: add env vars to README
* update ci to run dispute-valid-block test
* needs the polkadot image
* Fix path for nodejs container
* post merge fix
* download proper dir containg configs for malus test
* update the malus ci job
* rm a whitespace
* temp build for malus
* use correct build command for temp malus
* remove subcommands for now
* set max validators per core in the default HostConfig
* tabs
* update beefy
* fixup
* fixup II
* make one variant compile
* make other variants compile
* revert changes to chain_spec
* fmt
* build malus image from polkadot-test-malus again
* revert unrelated changes
* try fixing build-malus job
* Revert "remove subcommands for now"
This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.
* try fixing build-malus job II
* MVP working dispute-ancestor
* renames
* fix PVF execution on malus
* fix test
* fix typo
* fmt
* checkmate
* try something
* make it actually work
* some tweaks to 01 feature test
* fmt
* sleep a bit more
* complete wococoization
* some tweaks to 01 feature test
* typo fix
* use correct metric names
* fix
* ffs
* .
* try some rearrangement
* Attempt to wait till initial node bootstrap in test
* Fix test syntax
* Run malus tests with v2 script
* Proper symlink created
* simnet v14
* add zombienet tests
* add zombie net test - draft
* add more tests to dispute suite
* add within to fix tests
* replace test directory and start test migration
* migrate all the tests
* add timeout to tests
* reduce debug
* make easy to test in dev
* set appropriated debug
* use image from ci
* fix config for test
* set images from ci
* fix config
* add COLIMAGE env
* tweek tests
* reduce debug
* typo
* wip, migrate old test to zombie-net
* adjunt test config for zombie-net
* run mauls 0001 test only
* clean and setup smoke-test in zombie-net
* add extra time to assertinons
* clean code to merge and improve README
* add info to access logs
* improved readme
* merge master and resolve conflicts
* Update zombienet_tests/README.md
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* clean and consolidate zombienet name
* change runner in gitlab
* add comment explain why we use wococo
* change tag for runner
* remove unused tests
* remove dup Dockerfile and update description
* fmt
* fix compilation post-merge
* fmt
* cut me Some slack
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* Mostly notes.
* Better error messages.
* Introduce Fatal/NonFatal + drop back channel participation
- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.
* Better error messages.
* Utility function for receiving `CandidateEvent`s.
* Ordering module typechecks.
* cargo fmt
* Prepare spam slots module.
* Implement SpamSlots mechanism.
* Implement queues.
* cargo fmt
* Participation.
* Participation taking shape.
* Finish participation.
* cargo fmt
* Cleanup.
* WIP: Cleanup + Integration.
* Make `RollingSessionWindow` initialized by default.
* Make approval voting typecheck.
* Get rid of lazy_static & fix approval voting tests
* Move `SessionWindowSize` to node primitives.
* Implement dispute coordinator initialization.
* cargo fmt
* Make queues return error instead of boolean.
* Initialized: WIP
* Introduce chain api for getting finalized block.
* Fix ordering to only prune candidates on finalized events.
* Pruning of old sessions in spam slots.
* New import logic.
* Make everything typecheck.
* Fix warnings.
* Get rid of obsolete dispute-participation.
* Fixes.
* Add back accidentelly deleted Cargo.lock
* Deliver disputes in an ordered fashion.
* Add module docs for errors
* Use type synonym.
* hidden docs.
* Fix overseer tests.
* Ordering provider taking `CandidateReceipt`.
... To be kicked on one next commit.
* Fix ordering to use relay_parent
as included block is not unique per candidate.
* Add comment in ordering.rs.
* Take care of duplicate entries in queues.
* Better spam slots.
* Review remarks + docs.
* Fix db tests.
* Participation tests.
* Also scrape votes on first leaf for good measure.
* Make tests typecheck.
* Spelling.
* Only participate in actual disputes, not on every import.
* Don't account backing votes to spam slots.
* Fix more tests.
* Don't participate if we don't have keys.
* Fix tests, typos and warnings.
* Fix merge error.
* Spelling fixes.
* Add missing docs.
* Queue tests.
* More tests.
* Add metrics + don't short circuit import.
* Basic test for ordering provider.
* Import fix.
* Remove dead link.
* One more dead link.
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* Substrate companion: Authority discovery multiple peer ids
Authority discovery before had a fixed mapping from `PeerId` to
`AuthorityId`. This wasn't correct, as a `PeerId` can actually map to
multiple `AuthorityId`s. The linked Substrate pr fixes this.
https://github.com/paritytech/substrate/pull/10259
* Update node/network/availability-distribution/src/requester/mod.rs
* Update node/network/collator-protocol/src/validator_side/mod.rs
* Update node/network/statement-distribution/src/tests.rs
* Update guide
* Adapt to Substrate pr
* Update Substrate
Sergei Shulepov