Update SimNet test to the new format (#4051)

* simnet-v10 update simnet image version * update config * update path to js scripts * use current branch to download config for tests * use current branch to download config for tests * temp use simnet-v10 * update also the toml file * Rework test according to CIHelper format * Call cihelper script for running tests * Switch to v12 * Version 13 * simnet v14 * Readme extended Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
2026-05-30 21:01:02 +00:00 · 2021-11-03 11:34:26 +01:00
parent 07389ebab9
commit 05bfc53878
8 changed files with 108 additions and 138 deletions
@@ -31,6 +31,8 @@ variables:
  VAULT_SERVER_URL:                "https://vault.parity-mgmt-vault.parity.io"
  VAULT_AUTH_PATH:                 "gitlab-parity-io-jwt"
  VAULT_AUTH_ROLE:                 "cicd_gitlab_parity_${CI_PROJECT_NAME}"
+  SIMNET_IMAGE:                    "europe-west3-docker.pkg.dev/parity-simnet/simnet-images/simnet:v14"
+  PIPELINE_SCRIPTS_TAG:            "v0.1"

 default:
  cache:                           {}
@@ -566,8 +568,12 @@ deploy-polkasync-kusama:

 simnet-tests:
  stage:                           deploy
-  image:                           "docker.io/paritytech/simnet:${SIMNET_REF}"
+  image:                           "${SIMNET_IMAGE}"
  <<:                              *kubernetes-env
+  variables:
+    # FIXME: change branch to "master" after merging this PR
+    # GH_DIR:                        "https://github.com/paritytech/polkadot/tree/master/simnet_tests"
+    GH_DIR:                        "https://github.com/paritytech/polkadot/tree/simnet-v10/simnet_tests"
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_COMMIT_REF_NAME == "master"
@@ -584,17 +590,19 @@ simnet-tests:

  # SIMNET_REF is a gitlab variable
  before_script:
-    - echo "Simnet Tests Config
-        ${SIMNET_IMAGE_NAME} ${SIMNET_REF}
-        ${PARACHAINS_IMAGE_NAME} ${PARACHAINS_IMAGE_TAG}
-        ${COLLATOR_IMAGE_NAME} ${COLLATOR_IMAGE_TAG}"
+    - echo "Simnet Tests Config"
+    - echo "${SIMNET_IMAGE}"
+    - echo "${PARACHAINS_IMAGE_NAME} ${PARACHAINS_IMAGE_TAG}"
+    - echo "${COLLATOR_IMAGE_NAME} ${COLLATOR_IMAGE_TAG}"
+    - echo "${GH_DIR}"
  script:
-    - /home/nonroot/simnet/gurke/scripts/run-test-environment-manager.sh
-        --github-remote-dir="https://github.com/paritytech/polkadot/tree/master/simnet_tests"
-        --config="simnet_tests/configs/simple_rococo_testnet.toml"
-        --image="${PARACHAINS_IMAGE_NAME}:${PARACHAINS_IMAGE_TAG}"
-        --image-2="${COLLATOR_IMAGE_NAME}:${COLLATOR_IMAGE_TAG}"
-        --image-simnet="docker.io/paritytech/simnet:${SIMNET_REF}"
+    - /home/nonroot/simnet/scripts/run-test-environment-manager-v2.sh
+        --github-remote-dir="${GH_DIR}"
+        --tag=smoketest
+        --image="PARACHAINSIMAGE=${PARACHAINS_IMAGE_NAME}:${PARACHAINS_IMAGE_TAG}"
+        --image="SYNTHIMAGE=${PARACHAINS_IMAGE_NAME}:${PARACHAINS_IMAGE_TAG}"
+        --image="COLIMAGE=${COLLATOR_IMAGE_NAME}:${COLLATOR_IMAGE_TAG}"
+        --image="SCRIPTSIMAGE=${SIMNET_IMAGE}"
  allow_failure:                   true
  retry: 2
  tags:
@@ -1,4 +1,4 @@
-# Simulation tests, or high level integration tests
+# Simulation tests, or end-to-end tests

 _The content of this directory is meant to be used by Parity's private CI/CD
 infrastructure with private tools. At the moment those tools are still early
@@ -7,32 +7,77 @@ public use._

 ## Contents of this directory

-`configs` directory contains config files in toml format that describe how to
-configure the simulation network that you want to launch.
+This directory contains different test suits, everyone one of them contains the set of test cases.
+Every test suits is defined by its definition file test_suit_description.toml. More information about
+structure of test suits and test cases may be found in [SimNet repository](https://gitlab.parity.io/parity/simnet/-/tree/master/ci_helper).

-`tests` directory contains [Cucumber](https://cucumber.io/) files. Those are
-Behavior-Driven Development test files that describe tests in plain English.
-Under the hood there are assertions that specific metrics should have specific
-values.
+Every test case deploys a test network, using toml config file, and runs the test,
+using a test scenario, written in [Cucumber](https://cucumber.io/).

-At the moment we have only one test for parachains: `/parachains.features`
-This test uses a JS script that we added to Simnet image and it's launched
-by this step in the cucumber file:
-`Then launch 'node' with parameters '--unhandled-rejections=strict /usr/local/bin/simnet_scripts test_parachain ./configs/adder.json ws://localhost:11222 100 10'`
+These test suits are run by Polkadot CI in different pipelines, for every commit in PR, for commit into master etc.
+It's the responsibility of the test's developer to provide the correct tag for their test, in order to let CI know, when
+this test case should be run. For the baseline the existing tags from the existing tests may be used. If these tags are not
+sufficient, the new tag may be created. But CI team should be aware of this tag and condition, when this test case should be run.

-`run_test.sh` is an entry point for running all tests in the folder.
-Any setup required for tests (but cannot be done in configs) is performed
-here. The main script's responsibility is to run [Gurke](https://github.com/paritytech/gurke)
-with passed parameters.
-In order to use this script locally, you need to install
+In order to run a test case locally, you need to install
 [Gurke](https://github.com/paritytech/gurke)
 Once you have access to a kubernetes cluster (meaning you can do `kubectl get pods`)
-you can run this script with no arguments, like `./run_test.sh` and tests should run.
+you can use Gurke in order to deploy a chain and run the test (see gurke's manual for the commands).
 Kubernetes cluster can be local, spawned with
 [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
 or an instance living in the
 [cloud](https://github.com/paritytech/gurke/blob/main/docs/How-to-setup-access-to-gke-k8s-cluster.md)

+## How to add new test cases
+New test case may be added either into the existing test suit or with creation of the new test suit.
+In any case it's better to create the test and run it locally first, using Gurke (see above).
+- In order to add the test case into the existing test suit, the new element (test case) should be added into [[test-cases]] array in test_suit_description.toml of this test suit. The example:
+
+```
+# The existing test case
+[[test-cases]]
+tags = ["all", "smoketest"]
+chain-config = "configs/default_local_testnet.toml"
+scenarios = ["tests/001-smoketest.feature"]
+
+# The new test case
+[[test-cases]]
+tags = ["all", "load"]
+allowed-to-fail = true
+chain-config = "configs/default_local_testnet.toml"
+scenarios = ["tests/002-loadtest.feature"]
+```
+(See note about test case's tags above).
+
+- In order to create a new test suit for the test case, new folder with test suit description file (test_suit_description.toml) should be created. The exact name is mandatory, CI traverses all subfolfders of simnet_tests directory and looks for this file, in order to build the list of test suits. In this description file the general information about the test suit and array of the test cases should be specified. The example of test_suit_description.toml file with some verbose comments:
+```
+name = "Name of the test suit"
+description = "General information about the suit"
+# It is the path to the setup script, that may be needed for the test suit.
+# You can perform in this script the actions, that are required for the chain deployment, but have to be performed before
+# spawing the chain. Like pre-generation of seeds. This script is run by CI before spawing the chain in the cluster
+setup-script="setup_script.sh"
+# If the config of the test case requires some custom Docker images, the names for these images should be listed in this section
+# CI has to provide all these images in the format image_name = some_value in order to run test cases from this suit
+# Obviosuly CI should be aware, if the new custom image is added.
+required-images = [
+	"SYNTHIMAGE",
+	"COLIMAGE",
+	"SCRIPTSIMAGE",
+	"PARACHAINSIMAGE"
+]
+
+# Array of the test cases
+# Every elements should be started with [[test-cases]]
+[[test-cases]]
+# See tags information above
+tags = ["all", "smoketest"]
+# The config, that will be used in order to spawn the chain
+chain-config = "configs/simple_rococo_testnet.toml"
+# The array of the scenarios, that will be run on the deployed chain
+scenarios = ["tests/parachains.feature"]
+```
+
 ### [Here is link to barcamp presentation of Simnet](https://www.crowdcast.io/e/ph49xu01)

 ### [Here is link to the Simnet repo, hosted on private gitlab](https://gitlab.parity.io/parity/simnet/-/tree/master)
@@ -2,8 +2,10 @@
 timeout = 300

 [settings.defaults]
-command = "/usr/local/bin/polkadot"
+image = "{{get_env(name="SYNTHIMAGE") | safe }}"
+command = "polkadot"
 chain-spec = "rococo-local.json"
+chain-name = "rococo-local"
 timeout = 300

 [init_nodes.chainspec]
@@ -23,10 +25,14 @@ timeout = 300

 [nodes.alice]
 validator = true
+image = "{{get_env(name="PARACHAINSIMAGE") | safe }}"
+command = "polkadot"
 extra-args = ["--alice"]

 [nodes.bob]
 validator = true
+image = "{{get_env(name="PARACHAINSIMAGE") | safe }}"
+command = "polkadot"
 extra-args = ["--bob"]

 [nodes.collator01]
@@ -38,7 +44,7 @@ image = "{{get_env(name="SCRIPTSIMAGE") | safe }}"
 command-with-args = """
 cd simnet_scripts;
 npm run build;
-node /usr/local/bin/simnet_scripts register_parachain /cfg/genesis-wasm /cfg/genesis-state 100 true ws://bootnode:9944;
+node /usr/local/bin/simnet_scripts/dist/index.js register_parachain /cfg/genesis-wasm /cfg/genesis-state 100 true ws://bootnode:9944;
 tail -f /dev/null
 """
 copy-files = [
@@ -0,0 +1,13 @@
+name = "Parachains test"
+description = "Smoke test for parachains, that deploy a simple collator and register parachain"
+required-images = [
+	"SYNTHIMAGE",
+	"COLIMAGE",
+	"SCRIPTSIMAGE",
+	"PARACHAINSIMAGE"
+]
+
+[[test-cases]]
+tags = ["all", "smoketest"]
+chain-config = "configs/simple_rococo_testnet.toml"
+scenarios = ["tests/parachains.feature"]
@@ -0,0 +1,6 @@
+Feature: ParaTesting
+
+  Scenario: spawn parachains network and check parachains
+    Given a test network
+    Then sleep 200 seconds
+    Then launch 'node' with parameters '/usr/local/bin/simnet_scripts/dist/index.js test_parachain /usr/local/bin/simnet_scripts/type_defs/adder.json ws://localhost:11222 100 10'
@@ -1,102 +0,0 @@
-#!/bin/bash
-
-### ARGS FOR THIS SCRIPT ###
-# ./${SCRIPT_NAME} NAMESPACE IMAGE LOG_PATH
-# All args have default values, specify args to override
-# e.g: ./${SCRIPT_NAME} radu-test parity/substrate:some_feature /var/log/gurke 
-
-set -eou pipefail
-SCRIPT_NAME="$0"
-SCRIPT_PATH=$(dirname "${SCRIPT_NAME}")     # relative
-SCRIPT_PATH=$(cd "${SCRIPT_PATH}" && pwd)   # absolutized and normalized
-
-function random_string {
-   head -1 <(fold -w 30  <(tr -dc 'a-z0-9' < /dev/urandom))
- }
-
-NAMESPACE=${1:-gurke-"$(random_string)"-runtest}
-IMAGE=${2:-"docker.io/paritypr/synth-wave:master"}
-LOG_PATH=${3:-"${SCRIPT_PATH}/logs"}
-COLIMAGE=${4:-"docker.io/paritypr/colander:master"}
-SCRIPTSIMAGE=${5:-"docker.io/paritytech/simnet:latest"}
-
-mkdir -p "${SCRIPT_PATH}"/logs
-
-echo "Running tests in namespace: ${NAMESPACE}"
-echo "Testing image: ${IMAGE}"
-echo "Storing scripts logs to: ${LOG_PATH}"
-echo "Colator image is ${COLIMAGE}"
-echo "SCRIPTSIMAGE image is ${SCRIPTSIMAGE}"
-
-
-function forward_port {
-
-  # RUN_IN_CONTAINER is env var that is set in the dockerfile
-  # use the -v operator to explicitly test if a variable is set
-  if  [[ ! -v RUN_IN_CONTAINER  ]]  ; then 
-    if is_port_forward_running ; then
-      kill_previous_job
-    fi
-  fi
-  start_forwading_job 
-}
-
-FORWARD_GREP_FILTER='kubectl.*[p]ort-forward.*svc/rpc.*11222'
-
-function is_port_forward_running {
-  # shellcheck disable=SC2009
-  ps aux | grep -qE "${FORWARD_GREP_FILTER}" 
-}
-
-function kill_previous_job {
-  # shellcheck disable=SC2009
-  job_pid=$(ps aux | grep -E "${FORWARD_GREP_FILTER}" | awk '{ print $2 }')
-  echo  "INFO Killed forwading port 9944 into bootnode"
-  kill "${job_pid}"
-}
-
-function start_forwading_job {
-  kubectl -n "${NAMESPACE}" \
-          expose pod bootnode \
-          --name=rpc \
-          --type=NodePort \
-          --target-port=9944 \
-          --port=9944
-  kubectl -n "${NAMESPACE}" \
-          port-forward svc/rpc 11222:9944 &> "${LOG_PATH}/forward-${NAMESPACE}.log" &
-  sleep 2
-  echo "INFO Started forwading port 9944 into bootnode"
-}
-
-function update_api {
-  pwd
-  cd "${SCRIPT_PATH}"/../../simnet_scripts/
-  npm run build
-  cd -
-  pwd
-}
-
-echo "INFO: Checking if namespace has no pods"
-kubectl -n "${NAMESPACE}" get pods
-
-export NAMESPACE="${NAMESPACE}"
-export COLIMAGE="${COLIMAGE}"
-export SYNTHIMAGE="${IMAGE}"
-export SCRIPTSIMAGE="${SCRIPTSIMAGE}"
-
-cd "${SCRIPT_PATH}"
-
-set -x  # echo the commands to stdout
-gurke  spawn --config "${SCRIPT_PATH}"/configs/simple_rococo_testnet.toml \
-            -n "${NAMESPACE}" \
-            --image "${IMAGE}"
-
-echo "INFO: Checking if pods launched correctly"
-kubectl -n "${NAMESPACE}" get pods -o wide
-echo "INFO: Updating Polkadot JS API"
-update_api
-forward_port
-
-# Run tests
-gurke test "${NAMESPACE}" "${SCRIPT_PATH}"/tests  --log-path "${LOG_PATH}"
-
@@ -1,6 +0,0 @@
-Feature: ParaTesting
-
-  Scenario: spawn parachains network and check parachains
-    Given a test network
-    Then sleep 200 seconds
-    Then launch 'node' with parameters '--unhandled-rejections=strict /usr/local/bin/simnet_scripts test_parachain ./configs/adder.json ws://localhost:11222 100 10'