introduce malus + zombienet based integration tests (#4131)

* test/malus: craft the first maliciously disputing actor

* initial draft

* Add Dockerfile and instructions how to use it to build malus image locally

* Forgot one flag for the build cmd

* we are not docker specific, we are happy to use any containerruntime

* shuffle things around

* add initial tera based integration test

* chores

* fixins

* simple setup to start

* other samples (WIP)

* add Docker version with cargo-chef

* update substarte and small change of orders in commands in the container file

* metrics one

* fmt

* minor

* fixin

* fix metric names

* -d

* add open gauge

* fmt

* spellcheck

* fix test

* adjust to changed error messages

* refactor, more malus impls

* more malus changes

* foo

* minor cleanup

* suggest garbage candidate

* chore

* fix suggest garabge malus

* malus: back garbage candidate

* cargo lock

* re-introduce metrics

* chore: cargo fmt

* undoe 1.54.0 output, CI uses 1.53.0 rustc

* update location of js types

* Fix trybuild

* add tag to image name also; this will be replaced in the prod version

* Tests fixed

* add some fix me

* add dockerfile for ci

* Add docker file for malus for ci

* use variables in .toml file

* add chnages for malus test

* some fixes

* some more fixes

* Update .gitlab-ci.yml

* add local build for polkadot and malus

* some fixes

* enable disputes feature in CI

* ok, ok

* rename: MsgFilter -> MessageInterceptor

* remove TODO that would not have worked

* intercept

* refactor

* fix README and containers

* fix

* chore: cargo fmt

* avoid some more malus-$VARIANT references

* fix argument order

* chore: add about

* Update sanity check in relay chain selection

* fix order, add dispute-unavailable-block malus

* fixup: avoid underflow issue

* it's all u32

* fix conditional use

* Revert "it's all u32"

This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.

* Revert "fixup: avoid underflow issue"

This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.

* Revert "Update sanity check in relay chain selection"

This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.

* update the malus bin

* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature

Co-authored-by: Andronik Ordian <write@reusable.software>

* add some FIXME reminders

* update path to index.js

* Update .gitlab-ci.yml

* Update node/malus/integrationtests/0001-dispute-valid-block.toml

* try 1: make malus test run

* chore: cargo fmt

* temporary fix

* use subcommand syntax from latest gurke

* cargo +nightly fmt

* add collator to a a test

* docs: add env vars to README

* update ci to run dispute-valid-block test

* needs the polkadot image

* Fix path for nodejs container

* post merge fix

* download proper dir  containg configs for malus test

* update the malus ci job

* rm a whitespace

* temp build for malus

* use correct build command for temp malus

* remove subcommands for now

* set max validators per core in the default HostConfig

* tabs

* update beefy

* fixup

* fixup II

* make one variant compile

* make other variants compile

* revert changes to chain_spec

* fmt

* build malus image from polkadot-test-malus again

* revert unrelated changes

* try fixing build-malus job

* Revert "remove subcommands for now"

This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.

* try fixing build-malus job II

* MVP working dispute-ancestor

* renames

* fix PVF execution on malus

* fix test

* fix typo

* fmt

* checkmate

* try something

* make it actually work

* some tweaks to 01 feature test

* fmt

* sleep a bit more

* complete wococoization

* some tweaks to 01 feature test

* typo fix

* use correct metric names

* fix

* ffs

* .

* try some rearrangement

* Attempt to wait till initial node bootstrap in test

* Fix test syntax

* Run malus tests with v2 script

* Proper symlink created

* simnet v14

* add zombienet tests

* add zombie net test - draft

* add more tests to dispute suite

* add within to fix tests

* replace test directory and start test migration

* migrate all the tests

* add timeout to tests

* reduce debug

* make easy to test in dev

* set appropriated debug

* use image from ci

* fix config for test

* set images from ci

* fix config

* add COLIMAGE env

* tweek tests

* reduce debug

* typo

* wip, migrate old test to zombie-net

* adjunt test config for zombie-net

* run mauls 0001 test only

* clean and setup smoke-test in zombie-net

* add extra time to assertinons

* clean code to merge and improve README

* add info to access logs

* improved readme

* merge master and resolve conflicts

* Update zombienet_tests/README.md

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* clean and consolidate zombienet name

* change runner in gitlab

* add comment explain why we use wococo

* change tag for runner

* remove unused tests

* remove dup Dockerfile and update description

* fmt

* fix compilation post-merge

* fmt

* cut me Some slack

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>

polkadot/node/malus/Cargo.toml

@@ -1,11 +1,3 @@
-[lib]
-name = "malus"
-path = "src/lib.rs"
-
-[[bin]]
-name = "malus-variant-a"
-path = "src/variant-a.rs"
-
 [package]
 name = "polkadot-test-malus"
 description = "Misbehaving nodes for local testnets, system and Simnet tests."
@@ -16,17 +8,34 @@ edition = "2018"
 readme = "README.md"
 publish = false
 
+[[bin]]
+name = "malus"
+path = "src/malus.rs"
+
 [dependencies]
 polkadot-cli = { path = "../../cli", default-features = false, features = [ "cli", "malus" ]  }
 polkadot-node-subsystem = { path = "../subsystem" }
 polkadot-node-subsystem-util = { path = "../subsystem-util" }
+polkadot-node-subsystem-types = { path = "../subsystem-types" }
+polkadot-node-core-dispute-coordinator = { path = "../core/dispute-coordinator" }
 polkadot-node-core-candidate-validation = { path = "../core/candidate-validation" }
+polkadot-node-core-backing = { path = "../core/backing" }
+polkadot-node-primitives = { path = "../primitives" }
+polkadot-primitives = { path = "../../primitives" }
 polkadot-node-core-pvf = { path = "../core/pvf" }
 parity-util-mem = { version = "0.10.0", default-features = false, features = ["jemalloc-global"] }
 color-eyre = { version = "0.5.11", default-features = false }
 assert_matches = "1.5"
 structopt = "0.3.25"
 async-trait = "0.1.51"
+sp-keystore = { git = "https://github.com/paritytech/substrate", branch = "master" }
+futures = "0.3.16"
+futures-timer = "3.0.2"
+tracing = "0.1.26"
+
+[features]
+default = [] # we do not enable disputes by default to avoid feature leak
+disputes = ["polkadot-cli/disputes"]
 
 [dev-dependencies]
 polkadot-node-subsystem-test-helpers = { path = "../subsystem-test-helpers" }