Commit Graph

94 Commits

Author SHA1 Message Date
Liam Aharon a98bfc1b38 remove --locked from companion check command (#14690) 2023-08-01 13:07:53 +10:00
Chevdor 0d5a48dcc9 Update Matrix and secrets (#14646)
* WIP

* Fix usage of the s3krit/walking-tag-action action

* Fix notification jobs

* Fix notification content

---------

Co-authored-by: parity-processbot <>
2023-07-28 13:06:13 +00:00
Liam Aharon fbc0e40a5a ci: check runtime migrations (#14618)
* poc: check polkadot runtime upgrade

* typo

* try nested extends

* add checks for other networks

* .test-refs-no-trigger-prs-only

* Update scripts/ci/gitlab/pipeline/build.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

* Update scripts/ci/gitlab/pipeline/build.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

* Update scripts/ci/gitlab/pipeline/build.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

* Update scripts/ci/gitlab/pipeline/build.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

---------

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
2023-07-28 01:11:32 +10:00
Kian Paimani 5b268d4417 minor updates to the substrate doc crate (#14614) 2023-07-23 14:32:05 -07:00
Kian Paimani 817c97d65d add doc-only substrate entry point crate (#14581)
* add doc-only substrate entry point crate

* document a few more things

* add more

* fix width

* Update primitives/io/src/lib.rs

Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>

* add link

* update cargo toml file

* fix sp-io docs

* improve

* small update

* add license

* satisfy license job

* add a line about FRAME

* CI happy now

* make CI more happy

* Let the check run for the whole workspace

* Forward the substrate node again as default run

* update binary names

* upate verison test

* Fix fix fix

* Fix

* rename to substrate-node in more places

* Revert "rename to substrate-node in more places"

This reverts commit 66960f84a1b6f1f7c638b4040e28e9fbabb8adf5.

* fix

* Fix build pipeline

* Fix properly plus add some docs

---------

Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
2023-07-19 18:36:50 +00:00
Bastian Köcher 5eb816d7a6 Removal of execution strategies (#14387)
* Start

* More work!

* Moar

* More changes

* More fixes

* More worrk

* More fixes

* More fixes to make it compile

* Adds `NoOffchainStorage`

* Pass the extensions

* Small basti making small progress

* Fix merge errors and remove `ExecutionContext`

* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`

Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.

* Fix compilation

* Register the global extensions inside runtime api instance

* Fixes

* Fix `generate_initial_session_keys` by passing the keystore extension

* Fix the grandpa tests

* Fix more tests

* Fix more tests

* Don't set any heap pages if there isn't an override

* Fix small fallout

* FMT

* Fix tests

* More tests

* Offchain worker custom extensions

* More fixes

* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Fixes

* Fixes

* Set offchain transaction pool in BABE before using it in the runtime

* Add the `offchain_tx_pool` to Grandpa as well

* Fix the nodes

* Print some error when using the old warnings

* Fix merge issues

* Fix compilation

* Rename `babe_link`

* Rename to `offchain_tx_pool_factory`

* Cleanup

* FMT

* Fix benchmark name

* Fix `try-runtime`

* Remove `--execution` CLI args

* Make clippy happy

* Forward bls functions

* Fix docs

* Update UI tests

* Update client/api/src/execution_extensions.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/cli/src/params/import_params.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Pass the offchain storage to the MMR RPC

* Update client/api/src/execution_extensions.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Review comments

* Fixes

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-07-11 14:21:38 +00:00
Bastian Köcher a2b01c061b CI: Do not fail on junit upload failure (#14545)
* CI: Do not fail on junit upload failure

* fix yaml for test-linux-stable

---------

Co-authored-by: alvicsam <alvicsam@gmail.com>
2023-07-11 11:26:44 +02:00
Liam Aharon cfe5e8a8df run tests with --experimental flag (#14502) 2023-07-06 21:35:55 +00:00
Bastian Köcher 53f4b4df76 Run all tests with try-runtime feature enabled (#14396)
* Run all tests with `try-runtime` feature enabled

There are some tests that are guarded behind the `try-runtime` feature which are currently not being
executed. This pull requests changes this and enables the feature for the test run.

* Add some docs

* Rename variable
2023-06-16 12:16:19 +02:00
Mira Ressel b78473d28a cargo-hfuzz ci: specify manifest patches via cargo --config (#14372)
This is less brittle than locally editing Cargo.toml, which breaks when
our devs submit PRs such as #13992 that also temporarily add a patch
section.
2023-06-14 13:54:58 +02:00
Oleg Plakida 982db4309c Change unit test upload step (#14124)
Co-authored-by: parity-processbot <>
2023-05-16 11:24:54 +01:00
Bastian Köcher a8cd277d1c Re-enable node-cli tests in CI and fix them (#14093)
With the recent change of re-running flaky tests this should be fine!
2023-05-08 16:19:38 +02:00
Oleg Plakida 71a77c7054 Add flawky tests report (#13654) 2023-05-05 13:16:52 +01:00
Bastian Köcher e2547f5064 Improve handling of unset StorageVersion (#13417)
* Improve handling of unset `StorageVersion`

When a user is forgetting to set the storage version in a pallet and calls
`current_storage_version` to compare it against the `on_chain_storage_version` it will now fail to
compile the code. Before the pallet macro just returned `StorageVersion::default()` for
`current_storage_version` leading to potential issues with migrations. Besides that it also checks
in `post_upgrade` that the pallet storage version was upgraded and thus, no migration was missed.

* Use correct `Cargo.lock`

* Fixes

* Fix test

* Update frame/support/test/tests/pallet.rs

* Ensure we don't set a storage version when the pallet is missing the attribute

* Fix merge conflict

* Update frame/support/procedural/src/pallet/expand/hooks.rs

Co-authored-by: Roman Useinov <roman.useinov@gmail.com>

* Update frame/support/procedural/src/pallet/expand/hooks.rs

Co-authored-by: Roman Useinov <roman.useinov@gmail.com>

* Fix compilation

* Do not run everything with `try-runtime`

* Fix test

* Apply suggestions from code review

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix `no-metadata-docs`

---------

Co-authored-by: Roman Useinov <roman.useinov@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>
2023-05-04 21:18:53 +02:00
Vladimir Istyufeev 93165bc4d2 CI: cargo-check-benches: don't merge in master if base ref isn't master (#14071) 2023-05-04 14:32:16 +04:00
Mira Ressel 45a8647607 Don't run check-crates-publishing job on prs (#14064)
* Restore `check-crates-publishing-pipeline` job

This job was errorneously deleted in lieu of the
`check-crates-publishing` job.

This reverts commit 202836cb47.

* Don't run `check-crate-publishing` job on prs.

Originally attempted in #14044. It's been broken for a while and we're
not sure yet how to move forward in the longer term.
2023-05-03 12:47:22 +02:00
Vladimir Istyufeev bedb497d55 CI: migrate to Google Cloud (#13994) 2023-05-02 13:34:14 +04:00
Alexander Samusev b46a512e1a [ci] Add message to cargo-deny (#14001)
* [ci] Add message to cargo-deny

* fix cargo-deny-licenses
2023-04-25 10:15:57 +02:00
Alexander Samusev e9ff897511 [ci] Update buildah command and version (#13989) 2023-04-24 17:33:21 +02:00
Mira Ressel 9544ec765d replace unused cargo-deny check with a new cargo-deny-licenses job (#13956)
* ci: remove the cargo-deny job

It's been broken and disabled for quite a while, and per internal
discussion there doesn't seem to be any interest in fixing it and
actually using the job output for anything.

* ci: add new cargo-deny-licenses job

It'll run on all PRs and fail if external dependencies with unsuitable
licenses are detected.
2023-04-20 14:03:05 +02:00
Mira Ressel 040011a056 ci: drop skip-if-draft job (#13961)
It's been effectively disabled for half a year already, and according to
internal discussion we don't want it back.
2023-04-20 14:02:40 +02:00
Mira Ressel 378f68a6d1 Fail dependant checks on cargo check warnings (#13922)
* Fail dependant checks on cargo check warnings

* Adjust feature flags for check-dependent-polkadot
2023-04-14 14:16:17 +02:00
Mira Ressel 8953feaeb2 Refactor the automatic-crate-publishing logic, make triggered pipelines uninterruptible as well (#13908)
* Use a new approach to make automatic-crate-publishing uninterruptible

It's not neccessary to dynamically change the `interruptible` setting
for *all* jobs as the old approach does; gitlab already considers a
pipeline uninterruptible as soon as a single uninterruptible job has
started (cf https://docs.gitlab.com/ee/ci/yaml/#interruptible).

IMO this approach is more readable, as it avoids dynamically loading the
.defaults section from different files based on import conditions; the
logic is now shorter and entirely contained in the main .gitlab-ci.yml.

* Make triggered multi-project pipelines uninterruptible
2023-04-13 20:17:12 +02:00
Mira Ressel 84c9e2f63a invoke clippy with --locked (#13882) 2023-04-12 12:05:11 +02:00
Mira Ressel 4e32ae1fbe use stable rust toolchain in ci (#13830)
cargo-fmt stays on the nightly pipeline; our fmt config uses a heap of
unstable features.
2023-04-11 17:02:50 +02:00
Vladimir Istyufeev 9c9492476c CI: rephrase RUSTFLAGS (#13735) 2023-03-28 15:41:10 +04:00
Vladimir Istyufeev 551367d4cb CI: Investigate why RUSTFLAGS differs on master and PRs (#13686) 2023-03-28 04:00:18 +04:00
Mira Ressel 6213161e33 add a new ci job to fuzz sp-arithmetic (#13673)
as requested in https://github.com/paritytech/ci_cd/issues/761
2023-03-27 07:23:53 +02:00
Mira Ressel 3d40d08bf8 only push docker descriptions on master branch pipelines (#13664)
Currently those jobs also get included in multi-project pipelines such
as https://gitlab.parity.io/parity/mirrors/scripts/-/pipelines/255546,
even when the description hasn't actually changed, because `changes:`
conditions always evaluate to true on non-branch pipelines.
2023-03-21 14:06:55 +02:00
Sergejs Kostjucenko 36fab3268c fix publish docker description ci job (#13663)
* fix publish docker description ci job

* remove publish-refs from docker description jobs
2023-03-21 14:05:26 +02:00
Mira Ressel b117dbdf08 refactor cargo-check-benches logic
* fix fetch refspec
* assume base is master if the api call fails
* merge master into pr branch rather than the other way around; same
  effect but imo cleaner and more readable
* don't re-fetch pr branch, we want to test the version our pipeline is
  running on
2023-03-16 15:57:05 +01:00
Mira Ressel 3a9e37c0a7 fix BASE lookup in cargo-check-benches 2023-03-16 15:57:05 +01:00
Mira Ressel 8945c1bd42 timestamp gitlab ci job outputs (#13605)
* timestamp gitlab ci job outputs

Based on previous work by @alvicsam in #13047.

* inline timestamp script

Some of our jobs don't check out the substrate repo.

* include .timestamp in pipelines overriding the default before_script

Still not including it in the zombienet jobs, they have their own
timestamping anyway.

* move timestamp.yml to shared pipeline repo

https://gitlab.parity.io/parity/infrastructure/ci_cd/shared
2023-03-15 16:37:34 +00:00
Mira Ressel 84638524e7 Automatically format ci pipeline specs with prettier (#13441)
* [ci] Deduplicate variables: sections in pipeline specs

The prettier yaml parser doesn't like these.

* [ci] provide git clean filter to format pipeline specs

* [ci] Reformat pipeline specs with prettier
2023-02-24 15:02:30 +00:00
Sergejs Kostjucenko 2568c6d48b Move rules into common place (#13428) 2023-02-23 16:53:03 +02:00
Sergejs Kostjucenko 0669b8bb05 fix docker description publishing (#13399) 2023-02-16 18:41:10 +02:00
Sergejs Kostjucenko 4ff906860c fix image name (#13398) 2023-02-16 14:37:59 +02:00
Alexander Samusev bcd6484cac [ci] Add dependencies for publish-crates (#13397)
* [ci] Add dependencies for publishsing crates

* apply review comments

* fix needs
2023-02-16 13:10:40 +01:00
Alexander Samusev d5e1fcc471 [ci] Move publish-crates-locally to publish stage (#13395) 2023-02-16 12:09:32 +01:00
Sergejs Kostjucenko 3c0d8fe149 CI add subkey docker image description and CI job to publish it to the hub.docker.com (#13392)
* Add subkey docker image description and ci job to publish it

* Add link to docs

* fix formatting

* fix ci
2023-02-15 22:35:16 +02:00
Michal Kucharczyk cad4982afd zombienet: warp-sync tests for validators (#13078)
* zombienet validators warp sync draft

Sketch of warp-sync test for validators.
Not tested.

Follow-up of: #12769

* yet another warp-sync scenario added

- all validators are synced from DB,
- some full nodes are synced from DB,
- full-node is warp-synced

* fixes

* fixes

* missing files

* path fixed

---------

Co-authored-by: parity-processbot <>
2023-02-07 10:28:18 +01:00
Bastian Köcher 1da71f0226 CI: Rewrite check-each-crate in python (#13238)
* CI: Rewrite `check-each-crate` in python

This is a much better readable version of the script and it should also work on Macos and not
siltently fail ;)

* Fix dumb bugs and print everything to stderr

* Don't buffer Python output

* 🤦

* 🤦 🤦

* Use check all until we have more macos runners

* Update scripts/ci/gitlab/check-each-crate.py

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update scripts/ci/gitlab/pipeline/test.yml

Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-01-25 22:49:03 +00:00
Michal Kucharczyk d85fd6b527 zobmienet tests are not supposed to fail (#13015)
* zobmienet tests are not supposed to fail

* Update scripts/ci/gitlab/pipeline/zombienet.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
2023-01-18 16:26:12 +01:00
JP 21141f5d4c More improvements for the crate publishing pipeline (#13153)
* more improvements for the crate publishing pipeline

* move default definitions to the publish-crates script

* add script to check the crate publishing pipeline at the start

* fix yaml references

* move more variables to .crates-publishing-pipeline

* separate .crates-publishing-pipeline from .crates-publishing-variables

* clean up redundant and unused code
2023-01-16 14:03:46 +00:00
Vladimir Istyufeev 3af93b8a3a CI: Code mark to request a pipeline failure (#13139) 2023-01-13 20:21:44 +04:00
JP 406e8ca417 remove update-node-template (#13100)
will be moved to polkadot's pipeline

ref https://github.com/paritytech/release-engineering/issues/142
2023-01-11 09:03:24 +00:00
JP 72f086b795 Avoid unintentionally canceling the scheduled crate publishing job (#13088)
* avoid unintentionally canceling the scheduled crate publishing job

because publish-crates and publish-crates-manual share the resource group
"crates-publishing", any instance of publish-crates-manual cancels a running
instance of publish-crates, as demonstrated by
https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2212179

a workaround for that unintended interaction is to avoid creating instances of
publish-crates-manual and instead require pipelines to be triggered manually by
checking $CI_JOB_MANUAL == "true"

* check manual pipelines by $CI_PIPELINE_SOURCE instead of $CI_JOB_MANUAL

* make crate-publishing pipelines uninterruptible

* use conditional includes to work around interruptible limitations

* organize comments

* remove interruptible from common pipeline

* wip: check include

* wip: check include

* fix include

* fix include

* fix include

* fix yaml

* fix yaml

* remove shared common-pipeline

* wip: retry common-pipeline

* move .default-template to .gitlab-ci.yml

* fix the pipeline

add comments

* fix default-pipeline.yml

* revert publish-crates-manual to when: manual

* move "needs:" back to publish-crates

* avoid manual repetition

* improve previous commit

* try to avoid manual repetition

* fix indentation

* minor adjustments

* move defaults to top of .gitlab-ci.yml

* fix positioning on default in the diff

* comments

* indentation

* Apply suggestions from code review

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
2023-01-10 12:01:44 +00:00
JP 3cce86c02c change crates publishing to be scheduled (#13002) 2022-12-22 12:21:30 +00:00
João Paulo Silva de Souza a5cab922a5 Set WASM_BUILD_WORKSPACE_HINT before starting CI jobs (#12816)
* set WASM_BUILD_WORKSPACE_HINT

this enables wasm-builder to find the Cargo.lock from within generated packages

* wip: try out https://github.com/paritytech/pipeline-scripts/pull/90

* wip: fixup

* wip

* revert ci skip

* set up WASM_BUILD_WORKSPACE_HINT for .kubernetes-env as well

* Update scripts/ci/gitlab/pipeline/publish.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
2022-12-16 12:29:55 +00:00
João Paulo Silva de Souza 79acac7909 increase timeout for publishing (#12934)
necessary when lots of crates need to be published, as demonstrated by https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2158667
2022-12-15 12:19:50 +00:00