Commit Graph

83 Commits

Author SHA1 Message Date
Bastian Köcher 9d75f3e3e7 Fix benchmarks and adds CI to test them (#12068)
* Fix benchmarks and adds CI to test them

Instead of waiting for benchmarks failing in Polkadot CI, we also can just test them in Substrate :P

* Do not overflow
2022-08-19 13:07:14 +02:00
Alexander Samusev db2d66e59b [ci] Improve cancel-pipeline job (#12008)
* [WIP][ci] Improve cancel-pipeline job

* fix job name

* test that fail works

* debug cancel-pipeline

* remove artifacts-false from cancel-pipeline jobs

* split cancel pipeline jobs

* fail test-linux-stable 2/3

* fail test-linux-stable 3/3

* fail cargo-check-benches 1/2

* fail cargo-check-benches 2/2

* fail test-linux-stable-int

* fail cargo-check-subkey

* fail check-tracing

* fail check-tracing

* fix pipeline
2022-08-12 11:03:27 +02:00
Nikos Kontakis 103f770e75 Rename node-runtime to node-kitchensink-runtime (#11930)
* Rename node=runtime to kithensink-runtime

* Undo md formatting
2022-08-02 15:25:52 +00:00
Alexander Samusev 2e8249dc17 [ci] chores: remove cargo install nextest (#11961) 2022-08-02 15:01:15 +02:00
Mak 135c584358 Integrate automatic update of substrate-node-template (#11931)
* Integrate automatic update of substrate-node-template

* Update scripts/ci/gitlab/pipeline/publish.yml

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>

Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
2022-07-29 12:16:30 +00:00
Alexander Samusev 151c5d3fd9 [ci] remove cargo-check-nixos job (#11873)
* [ci] remove cargo-check-nixos job

* remove shell.nix
2022-07-26 11:37:54 +02:00
Alexander Samusev 3753d70e45 [ci] fix node-bench-regression-guard job (#11901)
* [Do not merge] [ci] debug node-bench-regression-guard

* debug

* fix artifacts path

* add debug job

* debug

* debug

* fix job, return pipeline
2022-07-25 11:15:47 +02:00
Alexander Samusev c3812f409c [ci] Generate rustdocs without dependencies (#11885) 2022-07-21 15:35:18 +02:00
Alexander Samusev ed346d97c5 [ci] improvments to make pipeline faster (#11829)
* [DO NOT MERGE] test-linux-stable parallel on 3 ci nodes

* add debug message

* adjust rusty-cachier

* empty commit

* move test-linux-stable to test.yml

* make cargo-check-benches and test-wasmer-sandbox parallel

* fix comment

* Update scripts/ci/gitlab/pipeline/test.yml

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

* Update scripts/ci/gitlab/pipeline/test.yml

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

* if to case

* use case instead if in cargo-check-benches

* format

* add comments to output

* add comment

* add quotes

* Update scripts/ci/gitlab/pipeline/test.yml

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

Co-authored-by: parity-processbot <>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
2022-07-15 10:44:46 +02:00
Vlad b357ecd467 Explain why rusty-cachier is put first (#11740) 2022-06-23 17:54:15 +03:00
Vlad 2d9f187364 Put rusty-cachier before PR merge into master for cargo-check-benches job (#11737) 2022-06-22 19:00:00 +03:00
João Paulo Silva de Souza 94a7e278fa check-dependent-cumulus should only be executed for PRs (#11693)
the script executed by check-dependent-cumulus only works for PRs, as shown in https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1630771#L87, which comes from https://github.com/paritytech/pipeline-scripts/blob/3ad10ddc0d985ef5326974a1143229c6429befab/check_dependent_project.sh#L443
2022-06-20 12:18:06 +00:00
Oliver Tale-Yazdi eb4be02155 Expose Benchmarking Component Ranges (#11545)
* Add component ranges to benchmarking

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Adding component ranges to templates

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* tweak script to reduce diff

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_identity --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/identity/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2022-06-03 20:43:54 +00:00
Alexander Samusev 98f6d8e96c [ci] use cargo nextest instead cargo test in test-linux-stable (#11576)
* [DO NOT MERGE] Experimenting with nextest

* enable jobs

* enable stages

* add comment

* create test-frame-support job
2022-06-02 21:18:59 +02:00
João Paulo Silva de Souza 3977de095b reactivate check-dependent-cumulus (#11506)
* re-enable check-dependent-cumulus

* temporary: use handle-extra-dependencies

* temporary: trim CI

* CI: include build stage

* CI: include test stage

* CI: include test stage

* Revert "temporary: trim CI"

This reverts commit dcf4ae8d842bc445a065c7ccdc3b6a603034faa4.

* CI: fix weird revert

* Revert "temporary: use handle-extra-dependencies"

This reverts commit bc0dc0f21f10284a23f66fdd8509ca6df89f2586.

* CI undebug

Co-authored-by: TriplEight <denis.pisarev@parity.io>
2022-06-02 16:15:32 +02:00
Vlad a084dd684f CI: fix build-subkey-macos build job (#11573)
* CI: fix `build-subkey-macos` build job

* CI: use full path for the `CARGO_TARGET_DIR` default value
2022-06-02 11:48:12 +03:00
Vlad 7d8477b48d rusty-cachier pipeline impovements and fixes (#11572) 2022-06-01 19:48:53 +03:00
Bastian Köcher bb6bbab687 Fix all warnings when building for wasm (#11569)
* Fix all warnings when building for wasm

Besides that it also enables warnings as errors for wasm builds in the CI.

* FMT

* Make clippy happy
2022-06-01 13:03:21 +00:00
Vlad a4fdcb9a06 Don't limit test-linux-stable-int job output (#11560) 2022-06-01 14:28:24 +03:00
Vlad c91c1c793e Introduce rusty-cachier (#11462)
* Introduce `rusty-cachier`

* Return LF at the end of file

* Use `entrypoint` to `unshare(1)` into a new mount namespace

* Use `rusty-cachier`-provided absolute path for `CARGO_TARGET_DIR` everywhere

* Debug single `build-rustdoc` job

* CI: debug

* CI: debug

* CI: debug

* `unshare(1)` is no longer needed

* CI: remove debug

* Revert "Debug single `build-rustdoc` job"

* Formatiing

* Update scripts/ci/gitlab/pipeline/build.yml

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
2022-05-31 17:10:04 +02:00
Sergejs Kostjucenko 6198a5fb06 add rule to the ci job (#11511) 2022-05-24 13:10:13 +03:00
Shawn Tabrizi d9f33dc7e9 Create Script to Run All Benchmarks (#11493)
* Create run_all_benchmarks.sh

* Update run_all_benchmarks.sh

* Update run_all_benchmarks.sh

* Review fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update scripts/run_all_benchmarks.sh

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* typo

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add default for $1

* Typo

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update run_all_benchmarks.sh

* new weights on benchmarking machine

* prefer `--chain=dev`

* fix compile

* fix command

* fmt

* dont use square brackets

* Extend doc

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove +nightly

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add error file an run execute everything optimistically

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-05-23 16:56:42 +00:00
João Paulo Silva de Souza 9dd9a98eff disable check-dependent-cumulus (#11450) 2022-05-18 11:35:15 +00:00
Alexander Samusev ce77cb1735 [ci] Adjust job order in pipeline test stage with Gitlab DAG (#11442)
* [Do Not Merge] Test gitlab DAG in pipeline

* add jobs for pipeline cancel

* add check-tracing to cancel-pipeline
2022-05-18 13:14:25 +02:00
Alexander Samusev 52d9bf24cc [ci] Divide .gitlab-ci.yml into several files (#11333)
* rebase

* update Cargo.lock

* revert Cargo.lock

* fix Cargo.lock from 11384
2022-05-11 11:01:02 +02:00
Shawn Tabrizi 7d5202341b Basic TOML Lint (#11348)
* basic lint

* lint ordering
2022-05-04 13:38:54 +00:00
Sergejs Kostjucenko 46891e849f Move scripts used in CI to the new location (#11008)
Move scripts used in CI to the new location - **./scripts/ci/**

* Move github scripts

* Move more files

* Move ci scripts and fix dependencies

* Update docs/node-template-release.md

Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>

* Remove Cargo.lock

* Apply suggestions from code review

Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

* Make more paths uniform

Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
2022-03-14 10:42:34 +02:00
Benjamin Kampmann 327e4ad4ac do not check unleash on every PR, only master and tags (#5054)
* do not check unleash on every PR, only master and tags

* move scripts folder

* add signed-tag check to CI

* remove publish-to-crates-io dependencies

Co-authored-by: s3krit <pugh@s3kr.it>
2020-02-25 20:02:57 +01:00
s3krit 1e3e6a75f9 Add gitlab job for publishing draft releases (#5009)
Idea is much the same as we currently do on polkadot - When a new tag is pushed (that fits our pattern for tags, e.g. v1.2.3), find a list of labelled changes and generate a changelog based on that. Create a draft release on Github and post about it on Matrix.
2020-02-21 13:08:45 +01:00
Benjamin Kampmann 60e5011c72 Reorganising the repository - external renames and moves (#4074)
* Adding first rough ouline of the repository structure

* Remove old CI stuff

* add title

* formatting fixes

* move node-exits job's script to scripts dir

* Move docs into subdir

* move to bin

* move maintainence scripts, configs and helpers into its own dir

* add .local to ignore

* move core->client

* start up 'test' area

* move test client

* move test runtime

* make test move compile

* Add dependencies rule enforcement.

* Fix indexing.

* Update docs to reflect latest changes

* Moving /srml->/paint

* update docs

* move client/sr-* -> primitives/

* clean old readme

* remove old broken code in rhd

* update lock

* Step 1.

* starting to untangle client

* Fix after merge.

* start splitting out client interfaces

* move children and blockchain interfaces

* Move trie and state-machine to primitives.

* Fix WASM builds.

* fixing broken imports

* more interface moves

* move backend and light to interfaces

* move CallExecutor

* move cli off client

* moving around more interfaces

* re-add consensus crates into the mix

* fix subkey path

* relieve client from executor

* starting to pull out client from grandpa

* move is_decendent_of out of client

* grandpa still depends on client directly

* lemme tests pass

* rename srml->paint

* Make it compile.

* rename interfaces->client-api

* Move keyring to primitives.

* fixup libp2p dep

* fix broken use

* allow dependency enforcement to fail

* move fork-tree

* Moving wasm-builder

* make env

* move build-script-utils

* fixup broken crate depdencies and names

* fix imports for authority discovery

* fix typo

* update cargo.lock

* fixing imports

* Fix paths and add missing crates

* re-add missing crates
2019-11-14 21:51:17 +01:00
Yuanchao Sun 9515d34aee Also replace paths in [build-dependencies] (#4039)
* Also replace paths in [build-dependencies]

* Update scripts/node-template-release/src/main.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Slightly reduce memory usage

* Update scripts/node-template-release/src/main.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
2019-11-07 11:48:30 +01:00
Ashley 934d7aac1c Remove sr-arithmetic/fuzzer from workspace to fix windows builds (#3872)
* Remove sr-arithmetic/fuzzer from workspace to fix windows builds

* Remove sr-arithmetic/fuzzer from check_runtime.sh
2019-10-22 13:24:58 +02:00
Andrew Dirksen 470b62366f Change manual dependency on wasm-gc executable to an automatic cargo dependency. (#3854)
Improves user experience.
2019-10-19 10:38:19 +02:00
André Silva 6e5fe1a557 deps: update clap and structopt (#3809) 2019-10-12 18:31:49 +01:00
Bastian Köcher 875fa73728 Fix build-only-wasm script (#3536) 2019-09-03 14:13:32 +01:00
Bastian Köcher 4ff97bd856 Make it easier to just rebuild the WASM files (#3510)
* Adds `WASM_TARGET_DIRECTORY` env variable to `wasm-builder`

* Create the `wasm-builder-runner`'s in a common workspace

* Make `wasm-builder` trigger less rebuilds

* Version up

* Adds script for building only the WASM files

* Apply suggestions from code review

Co-Authored-By: André Silva <andre.beat@gmail.com>
Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>
2019-08-29 09:06:51 +02:00
Bastian Köcher d81df14391 Make node-template-release create a workspace (#3484) 2019-08-27 07:43:10 +02:00
ddorgan 6815ce59fe Fix artifact location required for flaming fir deployments (#3268) 2019-07-31 17:49:04 +01:00
TriplEight d98f3c6023 WIP: Subkey docker image (#3183)
* WIP to release subkey im docker

* debug run with publishing

* debug build

* debug build 2

* easier sync

* fix the template

* fix typo
2019-07-26 08:40:53 +08:00
Max Inden ab761f7a09 scripts/sentry-node: Ensure release mode and reserved connection for A (#3077)
Ensure all nodes in the sentry setup run in release mode. In addition
configure sentry-a as a reserved node for validator-a.
2019-07-10 10:15:23 +01:00
Max Inden 6639ab339a Introduce flag to enable sentry nodes to participate in grandpa gossip (#3018)
Given the following situation: A validator 'A' is not supposed to be
connected to the public internet to protect it from e.g. a DoS attack.
Instead it connects to a sentry node 'sentry-A' which connects to the
public internet. Validator 'B' can reach validator 'A' via sentry node
'sentry-A' and vice versa.

A sentry node needs to participate in the grandpa gossip without
becoming a validator itself in order to forward these message to its
validator. This commit adds a new command line flag (`--grandpa-voter`)
forcing a node to participate in the grandpa voting process even though
no `--key` was specified. Due to the fact that it does not have a key,
it does not become a validator in the network.

In order to simulate the above situation this commit also adds a Docker
Compose file (`scripts/sentry-node/docker-compose.yml`) with further
documentation.
2019-07-09 18:35:37 +02:00
Bastian Köcher ae80c05232 Fix node-template-release and make cargo doc faster (#3019) 2019-07-04 16:26:03 +02:00
Bastian Köcher fe08221479 Build WASM binaries as part of cargo build (#2868)
* Introduce `wasm-builder` and `wasm-builder-runner` to retire `build.sh`

Make use of `wasm-builder` in `test-runtime`.

* Add build script and remove the wasm project

* Port `node-runtime` to new wasm-builder

* Make `substrate-executor` tests work with `wasm-builder`

* Move `node-template` to `wasm-builder`

* Remove `build.sh` :)

* Remove the last include_bytes

* Adds the missing build.rs files

* Remove `build.sh` from CI

* Debug CI

* Make it work in CI

* CI attempt 3

* Make `substrate-runtime-test` compile on stable

* Ahhh, some missed `include_bytes!`

* AHH

* Add suggestions

* Improve search for `Cargo.lock` and don't panic if it is not found

* Searching from manifest path was no good idea

* Make the `wasm-builder` source better configurable

* Expose the bloaty wasm binary as well

* Make sure to rerun WASM recompilation on changes in dependencies

* Introduce new `WASM_BUILD_TYPE` env and make sure to call `build.rs` on
changes to env variables

* Remove `build.sh` from READMEs

* Rename the projects

* Fixes CI

* Update lock file

* Fixes merge-conflict

* Apply suggestions from code review

Co-Authored-By: TriplEight <denis.pisarev@parity.io>

* Try to make windows happy

* Replace all back slashes in paths with slashes

* Apply suggestions from code review

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Use cargo from `CARGO` env variable

* Fix compilation

* Use `rustup` for running the nightly build

* Make individual projects skipable

* Fix compilation

* Fixes compilation

* Build all WASM projects in one workspace

* Replace more back slashes!

* Remove `inlcude_bytes!`

* Adds some documentation

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* More review comments

* Update `Cargo.lock`

* Set license

* Apply suggestions from code review

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* More review comments + adds `TRIGGER_WASM_BUILD` env

* Fix doc tests

* Increase version + update README

* Switch crates.io version of `wasm-builder`

* Update README

* Switch to released version of `wasm-builder-runner`
2019-07-04 11:34:06 +02:00
David Craven b2622b611e Don't label PR's with gotissues. (#2993) 2019-07-02 12:42:58 +02:00
Lio李欧 66ae3a57ff Passthru build.sh/script.sh command line arguments (#2837) 2019-06-11 16:00:27 +02:00
DemiMarie-parity 418d4a0ce0 Add digest subtypes and update dependencies (#2734)
* Add different digest types for different parts of the code

* Add script to update dependencies

* Remove an old changelog

* `Inherent` → `PreRuntime`
2019-06-06 22:16:59 +02:00
Pierre Krieger f210557211 Require --locked when building runtime (#2813)
* Require --locked when building runtime

* Update locks

* Do it in a different way

* Accidentally reverted Cargo.lock

* pass on arguments in build.sh scripts
2019-06-06 15:53:22 +02:00
DemiMarie-parity c7d1204ce5 Introduce inherent digests (#2466)
* Introduce inherent digests

* Implement inherent digests

* fix silly error

* Implementation of inherent digests in BABE

All tests pass. There are still limitations:

1. The runtime strips out inherent digests, so BABE must re-add them.
2. The test runtime checks that it can re-compute all digests.  It
   can’t, so I had to comment out that test.

* Fix compilation and seal import

Seals were not imported correctly: the pre-digest was imported twice,
instead of both it and the seal being imported.  Also, other parts of
the code did not compile due to incomplete refactoring.

* Remove bogus assertion

* Fix testsuite compilation

* Remove unused import

* Fix compiler diagnostics

* Add inherent digest parameters to block constructors

This enforces that inherent digests are added first.

* Fixup Cargo.lock

* Fix build errors

* Re-add an incorrectly removed import

* Bump primitive-types version

* Update Cargo.lock

* Refactoring

* Use inherent digests for AuRa

They do reach the runtime, but get stripped.  I have not figured out
where.

* Fix compilation errors

* Fix compilation errors due to incorrect types

* Fix whitespace

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Add preamble

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix silly compile error

* Refactor pre-digest finding code into a separate function

* Remove unwanted assertion

It is too likely to bring down the entire blockchain.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Use `find_pre_digest` after runtime, too

Also, use `Member` trait rather than rolling our own requirements.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix various warnings

mostly due to upgrading the dependency on `error_chain`.

* Pre-digests nearly complete

This nearly completes the implementation of pre-runtime digests.

* `Seal2` → `Seal` and fix test suite

* Try to fix the storage error

* Try to fix storage (again)

* Fix tests

* Hopefully finish pre-runtime digests

The key is to pass *only* the pre-runtime digests to the runtime.  The
others must be stripped out by `initialize_block`.

* Fix silly typo

* Fix another silly mistake

* Remove unnecessary filtering of BABE pre-digests

We no longer get duplicate BABE pre-digests, so if they appear, the
header should be rejected outright.

* Update Cargo.lock files

* Reformatting

* Fix silly typo in inherent digest code

Also, revert `error.rs` files that contained calls to the `error_chain!`
macro.

* Try to keep the runtime from stripping pre-digests

Currently runs into the “Storage root must match that calculated”
assertion.

* Don’t compute storage root until storage changes are done.

Also, fix a compilation error.

* Fix compile-time error

* Fix compilation errors

* Fix more compile errors

* Hopefully it compiles this time…

* Fix compilation and add docs

* Prevent BABE from adding duplicate pre-runtime digests

Found by comparing with the AuRa code.  I also did some refactoring.

* Respond to review and fix some warnings

* Delete some dead code introduced earlier

* More dead code goes away

* `ref mut` → `&mut`

* Respond to review and fix some warnings

* Fix compilation error

* Remove unneeded `HashT` type parameter

Suggested-by: Robert Habermeier <robert@parity.io>

* Remove spurious #[allow(deprecated)]

* Document inherent digest parameter to `build_block`

* Delete `Simple` trait

It wasn’t needed

* delete wrongly added files

* Fix trait bounds

* Digest serialization tests

I also did some reformatting and cleanup.

* Apply suggestions from code review

Reformatting

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Swap two arguments to `propose` and `propose_with`

Also, remove some needless unsafe code.

* Remove bogus `#![allow(deprecated)]` annotations

With the removal of the deprecated `Seal` variant, these are not needed.

* Add a missing `#[allow(deprecated)]` in the AuRa tests

* Fix silly compile error

* Fix silly compiler error

RLS did not tell me that I hadn’t fixed `babe/lib.rs`, so I missed it.

* Fixes made automatically by Cargo
2019-05-29 14:13:38 +02:00
TriplEight 672b72f44a some prettiness, fix dockerfile's metadata (#2682) 2019-05-27 14:06:38 +02:00
Tomasz Drwięga c162fc5ff1 Expunge error-chain (feat. tomaka) (#2662)
* Remove error_chain

* Expunge error-chain from rpc and service.

* Expunge from transaction pool.

* Expunge from node/cli

* Expunge from keystore.

* Remove some boilerplate.

* Fix remaining stuff.

* Improve on deprecation message.

* Fix issues.

* Fix trnsaction pool tests.

* Fix the rest.

* Fix borked merge.

* Update lock
2019-05-24 10:35:31 +01:00