* Use a new approach to make automatic-crate-publishing uninterruptible
It's not neccessary to dynamically change the `interruptible` setting
for *all* jobs as the old approach does; gitlab already considers a
pipeline uninterruptible as soon as a single uninterruptible job has
started (cf https://docs.gitlab.com/ee/ci/yaml/#interruptible).
IMO this approach is more readable, as it avoids dynamically loading the
.defaults section from different files based on import conditions; the
logic is now shorter and entirely contained in the main .gitlab-ci.yml.
* Make triggered multi-project pipelines uninterruptible
Currently those jobs also get included in multi-project pipelines such
as https://gitlab.parity.io/parity/mirrors/scripts/-/pipelines/255546,
even when the description hasn't actually changed, because `changes:`
conditions always evaluate to true on non-branch pipelines.
* fix fetch refspec
* assume base is master if the api call fails
* merge master into pr branch rather than the other way around; same
effect but imo cleaner and more readable
* don't re-fetch pr branch, we want to test the version our pipeline is
running on
* timestamp gitlab ci job outputs
Based on previous work by @alvicsam in #13047.
* inline timestamp script
Some of our jobs don't check out the substrate repo.
* include .timestamp in pipelines overriding the default before_script
Still not including it in the zombienet jobs, they have their own
timestamping anyway.
* move timestamp.yml to shared pipeline repo
https://gitlab.parity.io/parity/infrastructure/ci_cd/shared
* [ci] Deduplicate variables: sections in pipeline specs
The prettier yaml parser doesn't like these.
* [ci] provide git clean filter to format pipeline specs
* [ci] Reformat pipeline specs with prettier
* zombienet validators warp sync draft
Sketch of warp-sync test for validators.
Not tested.
Follow-up of: #12769
* yet another warp-sync scenario added
- all validators are synced from DB,
- some full nodes are synced from DB,
- full-node is warp-synced
* fixes
* fixes
* missing files
* path fixed
---------
Co-authored-by: parity-processbot <>
* CI: Rewrite `check-each-crate` in python
This is a much better readable version of the script and it should also work on Macos and not
siltently fail ;)
* Fix dumb bugs and print everything to stderr
* Don't buffer Python output
* 🤦
* 🤦🤦
* Use check all until we have more macos runners
* Update scripts/ci/gitlab/check-each-crate.py
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update scripts/ci/gitlab/pipeline/test.yml
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* more improvements for the crate publishing pipeline
* move default definitions to the publish-crates script
* add script to check the crate publishing pipeline at the start
* fix yaml references
* move more variables to .crates-publishing-pipeline
* separate .crates-publishing-pipeline from .crates-publishing-variables
* clean up redundant and unused code
* avoid unintentionally canceling the scheduled crate publishing job
because publish-crates and publish-crates-manual share the resource group
"crates-publishing", any instance of publish-crates-manual cancels a running
instance of publish-crates, as demonstrated by
https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2212179
a workaround for that unintended interaction is to avoid creating instances of
publish-crates-manual and instead require pipelines to be triggered manually by
checking $CI_JOB_MANUAL == "true"
* check manual pipelines by $CI_PIPELINE_SOURCE instead of $CI_JOB_MANUAL
* make crate-publishing pipelines uninterruptible
* use conditional includes to work around interruptible limitations
* organize comments
* remove interruptible from common pipeline
* wip: check include
* wip: check include
* fix include
* fix include
* fix include
* fix yaml
* fix yaml
* remove shared common-pipeline
* wip: retry common-pipeline
* move .default-template to .gitlab-ci.yml
* fix the pipeline
add comments
* fix default-pipeline.yml
* revert publish-crates-manual to when: manual
* move "needs:" back to publish-crates
* avoid manual repetition
* improve previous commit
* try to avoid manual repetition
* fix indentation
* minor adjustments
* move defaults to top of .gitlab-ci.yml
* fix positioning on default in the diff
* comments
* indentation
* Apply suggestions from code review
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
* zombienet: warp-sync integration test added
* spelling
* Readme corrected
* dir name updated
* Check second phase of warp sync
* zombienet pipeline enable + naive test network
* zombienet stage added
* paritypr/substrate-debug image added for zombienet testing
* debugs added
* debugs added
* buildah problem fixed
* rollback
* runner tag
* test name corrected
* dir renamed (regex problem)
* common code clean up
* common code clean up
* fix
* warp sync test improvements
* full sha used
as short is too short (https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2051228#L38)
* disable tracing for nodes
* COMMON_USER -> DOCKERIO_USER
* refs reworked
* paritypr/substrate image used
* DOCKERIO -> DOCKER
* generate-ws-db toml cleanup
* improvements
* fix
* raw chain spec used
* zombienet v1.3.18 used
* zombienet: warp sync test enabled
* chain-spec path corrected
* log parsing improved
Checking if log does not container error or verification failed messages
* warp sync test: removed validators
* fix
* review remarks applied
* dir test name changed: 0000_block_building -> 0000-block-building
* transaction finalized test added
* transaction finalized test: error handling improved
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* Explicitly touch `version.rs` to invalidate the related cache
* zombienet add logs as artifacts
* Revert "Explicitly touch `version.rs` to invalidate the related cache"
This reverts commit 9d00ccfe897a280581156c281961a32665dba6d5.
* file naming changed
Co-authored-by: parity-processbot <>
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
* implement crate publishing from CI
* fix indentation
* use resource_group for job exclusivity
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
* correct publish = false
* Remove YAML anchors as GitLab's `extends:` doesn't need it
* Temporarily force cache upload for the new jobs
* Revert `RUSTY_CACHIER_FORCE_UPLOAD`
* pin libp2p-tcp=0.37.0 for sc-telemetry
* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"
This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.
* always collect generated crates
* increase timeout for publish-crates-template
* Force upload the new job cache again
* Revert "Force upload the new job cache again"
This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.
* reformat
* improve timeout explanation
* s/usual/average
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
* sort crates before splitting them into groups
this is useful so that crates always get routed to a specific group for a given version of the source code, which means that jobs for each batch can be reliably retried individually
* more verbose output
* misc improvements
* put uniq after sort
uniq filters by adjacent lines
* shellcheck
* rm useless backlashes
* handle edge case of no crates detected
* check all crates individually
It's relevant to check workspace crates individually because otherwise their compilation problems
due to feature misconfigurations won't be caught, as exemplified by
https://github.com/paritytech/substrate/issues/12705
* adapt to lack of multiple macos runners
https://github.com/paritytech/substrate/pull/12709#discussion_r1022868752
* fix cancel-pipeline-cargo-check-each-crate-macos
* fix cargo-check-each-crate-macos again
* time command execution
* fix YAML anchors
* add explanation for rounding division
* ensure the minimum of one crate per group
* collect artifacts for pipeline stopper
* revert hardcoded crates_per_group
* re-add crates_per_group=1
* construct_runtime: Fix generation of types behind features
With the recent addition of supporting features in `construct_runtime!` there was a bug overseen.
The `AllPalletsWithSystem` etc type declarations would be declared twice when a certain was enabled.
The problem was that in the macro we didn't feature gate the types that should be declared when
there is no feature enabled. This pull request now takes care of feature gating this type behind
`all(#( not(feature) ))`. So, these types will only be enabled if no of the configured features is enabled.
* Fix tests
* FMT
Mira Ressel