Alexandru Gheorghe
a84dd0dba5
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Zombienet tests
The content of this directory is meant to be used by Parity's private CI/CD infrastructure with private tools. At the moment those tools are still early stage of development and we don't know if / when they will available for public use.
Contents of this directory
parachains At the moment this directory only have one test related to parachains: /parachains-smoke-test, that check
the parachain registration and the block height.
Resources
Running tests locally
To run any test locally use the native provider (zombienet test -p native ...) you need first build the binaries. They
are:
adder-collator->polkadot/target/testnet/adder-collatormalus->polkadot/target/testnet/maluspolkadot->polkadot/target/testnet/polkadot,polkadot/target/testnet/polkadot-prepare-worker,polkadot/target/testnet/polkadot-execute-workerpolkadot-collator->cumulus/target/release/polkadot-parachainundying-collator->polkadot/target/testnet/undying-collator
To build them use:
adder-collator->cargo build --profile testnet -p test-parachain-adder-collatorundying-collator->cargo build --profile testnet -p test-parachain-undying-collatormalus->cargo build --profile testnet -p polkadot-test-maluspolkadot(in the Polkadot repo) andpolkadot-collator(in Cumulus repo) ->cargo build --profile testnet
One solution is to use the .set_env file (from this directory) and fill the CUSTOM_PATHS before source it to patch
the PATH of your system to find the binaries you just built.
E.g.:
$ cat .set_env
(...)
# by the order of this array
CUSTOM_PATHS=(
"~/polkadot/target/release"
"~/polkadot/target/testnet"
"~/cumulus/target/release"
)
(...)
source .set_env
Then you have your PATH customized and ready to run zombienet. NOTE: You should need to do this ones per
terminal session, since we are patching the PATH and re-exporting. Or you can also source this file in your
.bashrc file to get executed automatically in each new session.
Example:
You can run a test locally by executing:
zombienet test -p native 0001-parachains-pvf.zndsl
Questions / permissions
Ping in element Javier (@javier:matrix.parity.io) to ask questions or grant permission to run the test from your local
setup.