Fixes https://github.com/paritytech/polkadot-sdk/issues/3144
Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229
### Summary
Some preparations for Runtime to support elastic scaling, guarded by
config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
introduces a per-candidate `CoreIndex` but does it in a hacky way to
avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
and networking protocols.
#### Including `CoreIndex` in `BackedCandidate`
If the `ElasticScalingMVP` feature bit is enabled then
`BackedCandidate::validator_indices` is extended by 8 bits.
The value stored in these bits represents the assumed core index for the
candidate.
It is temporary solution which works by creating a mapping from
`BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
discovered by checking in which validator group the validator that
signed the statement is.
TODO:
- [x] fix tests
- [x] add new tests
- [x] Bump runtime API for Kusama, so we have that node features thing!
-> https://github.com/polkadot-fellows/runtimes/pull/194
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: alindima <alin@parity.io>
closes https://github.com/paritytech/parity-bridges-common/issues/2796
This partially reverts the #2439 - there are some changes (unrelated to
CI) that we still want to keep. The reason of that removal is that with
async backing enabled for Rococo AH (and for other chains in the near
future), we see a lot of issues there (because we run `14` nodes +
additional standalone process within a same container and it causes a
lot of timeouts). There's no way known to me to fix it right now, so
we're removing those tests hopefully temporarily to keep CI green
Brridges zombienet tests are non-standard - zombienet currently missing
multiple relay chains support (see e.g.
https://github.com/paritytech/zombienet/pull/796), so we need to go live
with two relay networks, their parachains + custom test runner (which
e.g. doesn't shutdown net when its tests are finished and instead waits
for both networks tests to complete). So we are stuck with native
zombienet provider => this PR is an attempt to gather everything in a
single docker container and run tests there ~Draft, because it is far
from finishing - what I want now is to see how it works on CI~
<del>PR custom review is deprecated. Now `review-bot` performs these
functions.</del>
PR removes unused ci jobs and adjusts zombienet jobs for merge queues
cc @Bullrich
Part of https://github.com/paritytech/polkadot-sdk/issues/2787
This is an initial PR that adds some basic BEEFY checks to the warp sync
zombienet tests. To be more specific, it does the following:
- Changes the snapshot used by the warp sync zombienet tests to one
built from an updated version of the kitchensink runtime, that supports
BEEFY
- Adds some basic BEEFY checks to the warp sync zombienet tests
- Deduplicates some params of the warp sync zombienet tests, making them
easier to extend
Closes#1591.
The purpose of this PR is filter out backing statements from the network
signed by disabled validators. This is just an optimization, since we
will do filtering in the runtime in #1863 to avoid nodes to filter
garbage out at block production time.
- [x] Ensure it's ok to fiddle with the mask of manifests
- [x] Write more unit tests
- [x] Test locally
- [x] simple zombienet test
- [x] PRDoc
---------
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Remove old version for `cli_args`, since this was fixed in the latest
version of zombienet and the `latest` version of polkadot introduce the
new flag `--insecure-validator-i-know-what-i-do`.
Fix jobs like
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4726174
Thx!
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,
## Overall idea
When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.
This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:
- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.
## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.
## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor @burdges, let me know what you think.
- [x] More and more testing.
- [x] Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742
- [x] Final versi burn-in before merging
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
**Overview:**
Adding an extra malus variant focusing on disputing finalized blocks. It
will:
- wrap around approval-voting
- listen to `OverseerSignal::BlockFinalized` and when encountered start
a dispute for the `dispute_offset`th ancestor
- simply pass through all other messages and signals
Add zombienet tests testing various edgecases:
- disputing freshly finalized blocks
- disputing stale finalized blocks
- disputing eagerly pruned finalized blocks (might be separate PR)
**TODO:**
- [x] Register new malus variant
- [x] Simple pass through wrapper (approval-voting)
- [x] Simple network definition
- [x] Listen to block finalizations
- [x] Fetch ancestor hash
- [x] Fetch session index
- [x] Fetch candidate
- [x] Construct and send dispute message
- [x] zndsl test 1 checking that disputes on fresh finalizations resolve
valid Closes#1365
- [x] zndsl test 2 checking that disputes for too old finalized blocks
are not possible Closes#1364
- [ ] zndsl test 3 checking that disputes for candidates with eagerly
pruned relay parent state are handled correctly #1359 (deferred to a
separate PR)
- [x] Unit tests for new malus variant (testing cli etc)
- [x] Clean/streamline error handling
- [ ] ~~Ensure it tests properly on session boundaries~~
---------
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Add `build-push-image-polkadot-debug` job to needs since we changed to
use the polkadot-debug image from the current branch for cumulus test
and we need to be sure that the image is ready. Fix issues like
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4481059
cc: @bkchr
Co-authored-by: Bastian Köcher <git@kchr.de>
PR prepares CI to the GitHub Merge Queues. All github actions that were
running in PR adjusted so they can run in the merge queues. Zombienet
jobs will do nothing during PRs but they will run during merge queues.
Jobs that will be skipped during PR:
- all zombienet jobs
- all publish docker jobs
Jobs that will be skipped during merge queue:
- check-labels
- check-prdoc
- pr-custom-review
- review trigger
cc https://github.com/paritytech/ci_cd/issues/862
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_**
This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.
There are still a few things that need to be done:
- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802
- [x] Fix existing approval-distribution and approval-voting tests
- [x] Fix bitfield-distribution and statement-distribution tests
- [x] Fix network bridge tests
- [x] Implement todos in the code
- [x] Add tests to cover new code
- [x] Update metrics
- [x] Remove the approval distribution aggression levels: TBD PR
- [x] Parachains DB migration
- [x] Test network protocol upgrade on Versi
- [x] Versi Load test
- [x] Add Zombienet test
- [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
- [x] Fix AcceptedDuplicate
- [x] Fix DB migration so that we can still keep old data.
- [x] Final Versi burn in
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
- Async-backing related primitives are stable `primitives::v6`
- Async-backing API is now part of `api_version(7)`
- It's enabled on Rococo and Westend runtimes
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Add logic to check if the `BUILD_RELEASE_VERSION` tag is available in
docker registry, if not calculate the previous version to use as
secondary image. This fix the issue in test using the `secondary image`
and bumping the `NODE_VERSION`. (e.g #1495)
* zombienet: use test-parachain image for the slashing test
* use the right image
* try polkadot-parachain image
* try naming collator alice 🙈
* add needed job for the pipeline
* fix user id in polkadot-parachain-debug image
* small tweaks to the test
* another small tweak
* yet another small tweak
* bump zombienet version
---------
Co-authored-by: Javier Viola <javier@parity.io>
Andrei Sandu