* pvf-precheck: Add `sign` in subsystem-util
Right now, most of operations that sign stuff in polkadot protocol are
handled by a very convenient tool - `Signed`. However `Signed` assumes
that whatever is signed is anchored to some `parent_hash` which works
for most cases, but does not work for others.
One instance of such a case is pre-checking (#3211). There validators
submit signed votes on-chain. A vote is valid for the entire session. If
we were to use `Signed` we would have to root a vote in some block of
that session and during vote verification check that this block is
indeed within the session. This is especially annoying since we agreed
to use unsigned extrinsics to submit votes and we need to make the
unsigned extrinsic validation as slim as possible.
(FWIW, the definition of a pre-checking vote can be seen in the next
diff in the stack)
That's the reason why we opted-out from using `Signed` for pre-checking
and decided to go with the manual signing approach. Almost every piece
of machinery is in place except for signing which is presented in this
PR.
* pvf-precheck: Add `PvfCheckStatement` to polkadot-primitives
This is an insubstantial PR that just unlocks PRs down the line. This PR
is a part of #3211.
Regarding the `PvfCheckStatement` struct itself: this is a structure
that will be used to convert from/into the binary representation and
ultimately will be used to sign and submit votes onto chain.
Right now, most of operations that sign stuff in polkadot protocol are
handled by a very convenient tool - `Signed`. However `Signed` assumes
that whatever is signed is anchored to some `parent_hash` which works
for most cases, but does not work for others.
One instance of such a case is pre-checking (#3211). There validators
submit signed votes on-chain. A vote is valid for the entire session. If
we were to use `Signed` we would have to root a vote in some block of
that session and during vote verification check that this block is
indeed within the session. This is especially annoying since we agreed
to use unsigned extrinsics to submit votes and we need to make the
unsigned extrinsic validation as slim as possible.
(FWIW, the definition of a pre-checking vote can be seen in the next
diff in the stack)
That's the reason why we opted-out from using `Signed` for pre-checking
and decided to go with the manual signing approach. Almost every piece
of machinery is in place except for signing which is presented in this
PR.
* minor: assure conditions match
This simplifies visual integrity checks that an overseer is connected
when it has to be.
* fix: avoid printing a misleading log in case of the disabled disputes feature
* chore: comments
* add expressive types for the selection algorithm
* rococo-runtime: Switch to latest `construct_runtime!` syntax
Besides that it fixes pallet macro errors in other crates that popped up
because of this switch.
* FMT
Closes https://github.com/paritytech/polkadot/issues/4293
This PR changes the way how we treat a certain subset of PVF preparation
errors. Specifically, now only the deterministic errors are treated as
invalid candidates. That is, the errors that are easily
attributable to either the the PVF contents or the wasmtime code, but
not e.g. I/O errors that could be triggered by the OS (insufficient
memory, disk failure, too much load, etc). The latter are treated as
internal errors and thus do not trigger the disputes.
* Impose new restrictions on paras init and cleanup
For upcoming PVF pre-checking feature we will need to impose a couple of
new restrictions for:
- `schedule_para_initialize`.
- `schedule_para_cleanup`.
Specifically, for the former we do not want to allow registration of
wasm blob that is empty, i.e. 0 bytes. While that currently already
does not make a lot of sense, it allows us to simplify the PVF
pre-checking logic: if this PR is deployed before the following changes
for PVF prechecking then we can be sure that no paras onboarding have to
have to go through the PVF pre-checking. In case, we deploy it
altogether this property will allow us to distingush paras that came in
before PVF pre-checking.
For `schedule_para_cleanup` we do not want to allow offboarding of paras
that are undergoing the upgrade process. While this is not a harsh
restriction this change allows us to avoid making the PVF prechecking
more complicated than it has to be.
* Add a test for schedule_para_initialize
* Link to `ParaLifecycle::is_stable` in docs.
* `schedule_para_{init,cleanup}` docs
Now they link to their original declarations in the pallet for more
details.
* test/malus: craft the first maliciously disputing actor
* initial draft
* Add Dockerfile and instructions how to use it to build malus image locally
* Forgot one flag for the build cmd
* we are not docker specific, we are happy to use any containerruntime
* shuffle things around
* add initial tera based integration test
* chores
* fixins
* simple setup to start
* other samples (WIP)
* add Docker version with cargo-chef
* update substarte and small change of orders in commands in the container file
* metrics one
* fmt
* minor
* fixin
* fix metric names
* -d
* add open gauge
* fmt
* spellcheck
* fix test
* adjust to changed error messages
* refactor, more malus impls
* more malus changes
* foo
* minor cleanup
* suggest garbage candidate
* chore
* fix suggest garabge malus
* malus: back garbage candidate
* cargo lock
* re-introduce metrics
* chore: cargo fmt
* undoe 1.54.0 output, CI uses 1.53.0 rustc
* update location of js types
* Fix trybuild
* add tag to image name also; this will be replaced in the prod version
* Tests fixed
* add some fix me
* add dockerfile for ci
* Add docker file for malus for ci
* use variables in .toml file
* add chnages for malus test
* some fixes
* some more fixes
* Update .gitlab-ci.yml
* add local build for polkadot and malus
* some fixes
* enable disputes feature in CI
* ok, ok
* rename: MsgFilter -> MessageInterceptor
* remove TODO that would not have worked
* intercept
* refactor
* fix README and containers
* fix
* chore: cargo fmt
* avoid some more malus-$VARIANT references
* fix argument order
* chore: add about
* Update sanity check in relay chain selection
* fix order, add dispute-unavailable-block malus
* fixup: avoid underflow issue
* it's all u32
* fix conditional use
* Revert "it's all u32"
This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.
* Revert "fixup: avoid underflow issue"
This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.
* Revert "Update sanity check in relay chain selection"
This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.
* update the malus bin
* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some FIXME reminders
* update path to index.js
* Update .gitlab-ci.yml
* Update node/malus/integrationtests/0001-dispute-valid-block.toml
* try 1: make malus test run
* chore: cargo fmt
* temporary fix
* use subcommand syntax from latest gurke
* cargo +nightly fmt
* add collator to a a test
* docs: add env vars to README
* update ci to run dispute-valid-block test
* needs the polkadot image
* Fix path for nodejs container
* post merge fix
* download proper dir containg configs for malus test
* update the malus ci job
* rm a whitespace
* temp build for malus
* use correct build command for temp malus
* remove subcommands for now
* set max validators per core in the default HostConfig
* tabs
* update beefy
* fixup
* fixup II
* make one variant compile
* make other variants compile
* revert changes to chain_spec
* fmt
* build malus image from polkadot-test-malus again
* revert unrelated changes
* try fixing build-malus job
* Revert "remove subcommands for now"
This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.
* try fixing build-malus job II
* MVP working dispute-ancestor
* renames
* fix PVF execution on malus
* fix test
* fix typo
* fmt
* checkmate
* try something
* make it actually work
* some tweaks to 01 feature test
* fmt
* sleep a bit more
* complete wococoization
* some tweaks to 01 feature test
* typo fix
* use correct metric names
* fix
* ffs
* .
* try some rearrangement
* Attempt to wait till initial node bootstrap in test
* Fix test syntax
* Run malus tests with v2 script
* Proper symlink created
* simnet v14
* add zombienet tests
* add zombie net test - draft
* add more tests to dispute suite
* add within to fix tests
* replace test directory and start test migration
* migrate all the tests
* add timeout to tests
* reduce debug
* make easy to test in dev
* set appropriated debug
* use image from ci
* fix config for test
* set images from ci
* fix config
* add COLIMAGE env
* tweek tests
* reduce debug
* typo
* wip, migrate old test to zombie-net
* adjunt test config for zombie-net
* run mauls 0001 test only
* clean and setup smoke-test in zombie-net
* add extra time to assertinons
* clean code to merge and improve README
* add info to access logs
* improved readme
* merge master and resolve conflicts
* Update zombienet_tests/README.md
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* clean and consolidate zombienet name
* change runner in gitlab
* add comment explain why we use wococo
* change tag for runner
* remove unused tests
* remove dup Dockerfile and update description
* fmt
* fix compilation post-merge
* fmt
* cut me Some slack
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* impl prefered items
Closes#4330
* do not stop attempting to select, just because one did not fit
* doc
* prefered -> preferred
* missing usage of the preferred indices
* sigh
* shuffle is not available for chacha
* remove duplicate weight addition
* ref vs no ref
* Mostly notes.
* Better error messages.
* Introduce Fatal/NonFatal + drop back channel participation
- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.
* Better error messages.
* Utility function for receiving `CandidateEvent`s.
* Ordering module typechecks.
* cargo fmt
* Prepare spam slots module.
* Implement SpamSlots mechanism.
* Implement queues.
* cargo fmt
* Participation.
* Participation taking shape.
* Finish participation.
* cargo fmt
* Cleanup.
* WIP: Cleanup + Integration.
* Make `RollingSessionWindow` initialized by default.
* Make approval voting typecheck.
* Get rid of lazy_static & fix approval voting tests
* Move `SessionWindowSize` to node primitives.
* Implement dispute coordinator initialization.
* cargo fmt
* Make queues return error instead of boolean.
* Initialized: WIP
* Introduce chain api for getting finalized block.
* Fix ordering to only prune candidates on finalized events.
* Pruning of old sessions in spam slots.
* New import logic.
* Make everything typecheck.
* Fix warnings.
* Get rid of obsolete dispute-participation.
* Fixes.
* Add back accidentelly deleted Cargo.lock
* Deliver disputes in an ordered fashion.
* Add module docs for errors
* Use type synonym.
* hidden docs.
* Fix overseer tests.
* Ordering provider taking `CandidateReceipt`.
... To be kicked on one next commit.
* Fix ordering to use relay_parent
as included block is not unique per candidate.
* Add comment in ordering.rs.
* Take care of duplicate entries in queues.
* Better spam slots.
* Review remarks + docs.
* Fix db tests.
* Participation tests.
* Also scrape votes on first leaf for good measure.
* Make tests typecheck.
* Spelling.
* Only participate in actual disputes, not on every import.
* Don't account backing votes to spam slots.
* Fix more tests.
* Don't participate if we don't have keys.
* Fix tests, typos and warnings.
* Fix merge error.
* Spelling fixes.
* Add missing docs.
* Queue tests.
* More tests.
* Add metrics + don't short circuit import.
* Basic test for ordering provider.
* Import fix.
* Remove dead link.
* One more dead link.
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
antonio-dropulic