pezkuwichain
2093647fea
- Add 72 rebrand workflow files (polkadot→pezkuwi, substrate→bizinikiwi, cumulus→pezcumulus) - Add GitHub actions, issue templates, and configs - Removed unnecessary workflows (fork-sync, gitspiegel, upstream-tracker, sync-templates, backport) - Renamed zombienet test files to match new naming convention
84 lines
3.3 KiB
Markdown
84 lines
3.3 KiB
Markdown
# Zombienet Flaky Tests
|
|
|
|
This document explains how to manage flaky or temporarily disabled zombienet tests in the Pezkuwi SDK repository.
|
|
|
|
## Overview
|
|
|
|
The `.github/zombienet-flaky-tests` file contains a list of zombienet tests that are currently disabled due to flakiness or known issues. These tests are automatically skipped during CI runs but are tracked for future re-enabling.
|
|
|
|
## File Format
|
|
|
|
Each line in the `zombienet-flaky-tests` file follows this format:
|
|
|
|
```
|
|
<test-job-name>:<issue-number>
|
|
```
|
|
|
|
**Example:**
|
|
```
|
|
zombienet-pezkuwi-functional-0014-chunk-fetching-network-compatibility:9980
|
|
zombienet-pezcumulus-0009-elastic_scaling_pov_recovery:8986
|
|
```
|
|
|
|
- **test-job-name**: The exact job name as defined in the zombienet test YAML files
|
|
- **issue-number**: GitHub issue number tracking the flaky test (for reference and follow-up)
|
|
|
|
## How It Works
|
|
|
|
1. **Test Discovery**: The zombienet workflows read the test definitions from:
|
|
- `.github/zombienet-tests/zombienet_pezkuwi_tests.yml`
|
|
- `.github/zombienet-tests/zombienet_cumulus_tests.yml`
|
|
- `.github/zombienet-tests/zombienet_substrate_tests.yml`
|
|
- `.github/zombienet-tests/zombienet_teyrchain-template_tests.yml`
|
|
|
|
2. **Filtering**: During the preflight job, tests listed in `zombienet-flaky-tests` are filtered out from the test matrix.
|
|
|
|
3. **Execution**: Only non-flaky tests are executed in the CI pipeline.
|
|
|
|
## Adding a Flaky Test
|
|
|
|
If you encounter a flaky test that needs to be temporarily disabled:
|
|
|
|
1. **Create or find a GitHub issue** tracking the flaky behavior
|
|
2. **Add an entry** to `.github/zombienet-flaky-tests`:
|
|
```
|
|
zombienet-<suite>-<test-name>:<issue-number>
|
|
```
|
|
3. **Commit and push** the change
|
|
4. The CI will automatically validate that:
|
|
- The entry follows the correct format
|
|
- The referenced GitHub issue exists
|
|
- (Warning if the issue is closed)
|
|
5. The test will be automatically skipped in subsequent CI runs
|
|
|
|
## Re-enabling a Test
|
|
|
|
Once a flaky test has been fixed:
|
|
|
|
1. **Verify the fix** by running the test locally or in a test branch
|
|
2. **Remove the entry** from `.github/zombienet-flaky-tests`
|
|
3. **Close the associated GitHub issue** (or update it with the fix)
|
|
4. **Commit and push** the change
|
|
5. The test will be automatically included in subsequent CI runs
|
|
|
|
## Validation
|
|
|
|
The `.github/zombienet-flaky-tests` file is automatically validated in CI whenever it's modified. The validation checks:
|
|
|
|
- **Format**: Each entry must follow the `<test-name>:<issue-number>` format
|
|
- **Issue vs PR**: The referenced number must be a GitHub Issue, not a Pull Request
|
|
- **Issue existence**: The referenced GitHub issue must exist in the repository
|
|
- **Issue state**: A warning is shown if the referenced issue is closed (suggesting the entry might be outdated)
|
|
|
|
The validation workflow runs on pull requests that modify:
|
|
- `.github/zombienet-flaky-tests`
|
|
- `.github/scripts/check-zombienet-flaky-tests.sh`
|
|
- `.github/workflows/check-zombienet-flaky-tests.yml`
|
|
|
|
## Monitoring
|
|
|
|
- The number of currently disabled tests is displayed in the CI logs during zombienet test runs
|
|
- You can view the current list at: [`.github/zombienet-flaky-tests`](./zombienet-flaky-tests)
|
|
- Each disabled test should have an associated GitHub issue for tracking
|
|
- The validation script can be run locally: `.github/scripts/check-zombienet-flaky-tests.sh .github/zombienet-flaky-tests`
|