* zombienet validators warp sync draft
Sketch of warp-sync test for validators.
Not tested.
Follow-up of: #12769
* yet another warp-sync scenario added
- all validators are synced from DB,
- some full nodes are synced from DB,
- full-node is warp-synced
* fixes
* fixes
* missing files
* path fixed
---------
Co-authored-by: parity-processbot <>
* CI: Rewrite `check-each-crate` in python
This is a much better readable version of the script and it should also work on Macos and not
siltently fail ;)
* Fix dumb bugs and print everything to stderr
* Don't buffer Python output
* 🤦
* 🤦🤦
* Use check all until we have more macos runners
* Update scripts/ci/gitlab/check-each-crate.py
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update scripts/ci/gitlab/pipeline/test.yml
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* more improvements for the crate publishing pipeline
* move default definitions to the publish-crates script
* add script to check the crate publishing pipeline at the start
* fix yaml references
* move more variables to .crates-publishing-pipeline
* separate .crates-publishing-pipeline from .crates-publishing-variables
* clean up redundant and unused code
* avoid unintentionally canceling the scheduled crate publishing job
because publish-crates and publish-crates-manual share the resource group
"crates-publishing", any instance of publish-crates-manual cancels a running
instance of publish-crates, as demonstrated by
https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2212179
a workaround for that unintended interaction is to avoid creating instances of
publish-crates-manual and instead require pipelines to be triggered manually by
checking $CI_JOB_MANUAL == "true"
* check manual pipelines by $CI_PIPELINE_SOURCE instead of $CI_JOB_MANUAL
* make crate-publishing pipelines uninterruptible
* use conditional includes to work around interruptible limitations
* organize comments
* remove interruptible from common pipeline
* wip: check include
* wip: check include
* fix include
* fix include
* fix include
* fix yaml
* fix yaml
* remove shared common-pipeline
* wip: retry common-pipeline
* move .default-template to .gitlab-ci.yml
* fix the pipeline
add comments
* fix default-pipeline.yml
* revert publish-crates-manual to when: manual
* move "needs:" back to publish-crates
* avoid manual repetition
* improve previous commit
* try to avoid manual repetition
* fix indentation
* minor adjustments
* move defaults to top of .gitlab-ci.yml
* fix positioning on default in the diff
* comments
* indentation
* Apply suggestions from code review
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
* zombienet: warp-sync integration test added
* spelling
* Readme corrected
* dir name updated
* Check second phase of warp sync
* zombienet pipeline enable + naive test network
* zombienet stage added
* paritypr/substrate-debug image added for zombienet testing
* debugs added
* debugs added
* buildah problem fixed
* rollback
* runner tag
* test name corrected
* dir renamed (regex problem)
* common code clean up
* common code clean up
* fix
* warp sync test improvements
* full sha used
as short is too short (https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2051228#L38)
* disable tracing for nodes
* COMMON_USER -> DOCKERIO_USER
* refs reworked
* paritypr/substrate image used
* DOCKERIO -> DOCKER
* generate-ws-db toml cleanup
* improvements
* fix
* raw chain spec used
* zombienet v1.3.18 used
* zombienet: warp sync test enabled
* chain-spec path corrected
* log parsing improved
Checking if log does not container error or verification failed messages
* warp sync test: removed validators
* fix
* review remarks applied
* dir test name changed: 0000_block_building -> 0000-block-building
* transaction finalized test added
* transaction finalized test: error handling improved
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* Explicitly touch `version.rs` to invalidate the related cache
* zombienet add logs as artifacts
* Revert "Explicitly touch `version.rs` to invalidate the related cache"
This reverts commit 9d00ccfe897a280581156c281961a32665dba6d5.
* file naming changed
Co-authored-by: parity-processbot <>
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
* implement crate publishing from CI
* fix indentation
* use resource_group for job exclusivity
ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions
* correct publish = false
* Remove YAML anchors as GitLab's `extends:` doesn't need it
* Temporarily force cache upload for the new jobs
* Revert `RUSTY_CACHIER_FORCE_UPLOAD`
* pin libp2p-tcp=0.37.0 for sc-telemetry
* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"
This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.
* always collect generated crates
* increase timeout for publish-crates-template
* Force upload the new job cache again
* Revert "Force upload the new job cache again"
This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.
* reformat
* improve timeout explanation
* s/usual/average
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
* sort crates before splitting them into groups
this is useful so that crates always get routed to a specific group for a given version of the source code, which means that jobs for each batch can be reliably retried individually
* more verbose output
* misc improvements
* put uniq after sort
uniq filters by adjacent lines
* shellcheck
* rm useless backlashes
* handle edge case of no crates detected
* check all crates individually
It's relevant to check workspace crates individually because otherwise their compilation problems
due to feature misconfigurations won't be caught, as exemplified by
https://github.com/paritytech/substrate/issues/12705
* adapt to lack of multiple macos runners
https://github.com/paritytech/substrate/pull/12709#discussion_r1022868752
* fix cancel-pipeline-cargo-check-each-crate-macos
* fix cargo-check-each-crate-macos again
* time command execution
* fix YAML anchors
* add explanation for rounding division
* ensure the minimum of one crate per group
* collect artifacts for pipeline stopper
* revert hardcoded crates_per_group
* re-add crates_per_group=1
* construct_runtime: Fix generation of types behind features
With the recent addition of supporting features in `construct_runtime!` there was a bug overseen.
The `AllPalletsWithSystem` etc type declarations would be declared twice when a certain was enabled.
The problem was that in the macro we didn't feature gate the types that should be declared when
there is no feature enabled. This pull request now takes care of feature gating this type behind
`all(#( not(feature) ))`. So, these types will only be enabled if no of the configured features is enabled.
* Fix tests
* FMT
* Fix benchmarks and adds CI to test them
Instead of waiting for benchmarks failing in Polkadot CI, we also can just test them in Substrate :P
* Do not overflow
* re-enable check-dependent-cumulus
* temporary: use handle-extra-dependencies
* temporary: trim CI
* CI: include build stage
* CI: include test stage
* CI: include test stage
* Revert "temporary: trim CI"
This reverts commit dcf4ae8d842bc445a065c7ccdc3b6a603034faa4.
* CI: fix weird revert
* Revert "temporary: use handle-extra-dependencies"
This reverts commit bc0dc0f21f10284a23f66fdd8509ca6df89f2586.
* CI undebug
Co-authored-by: TriplEight <denis.pisarev@parity.io>
* Introduce `rusty-cachier`
* Return LF at the end of file
* Use `entrypoint` to `unshare(1)` into a new mount namespace
* Use `rusty-cachier`-provided absolute path for `CARGO_TARGET_DIR` everywhere
* Debug single `build-rustdoc` job
* CI: debug
* CI: debug
* CI: debug
* `unshare(1)` is no longer needed
* CI: remove debug
* Revert "Debug single `build-rustdoc` job"
* Formatiing
* Update scripts/ci/gitlab/pipeline/build.yml
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
Sergejs Kostjucenko