Commit Graph

136 Commits

Author SHA1 Message Date
Pierre Krieger 7c7ad18d4e Update the service tasks Grafana dashboard (#7038) 2020-09-08 09:59:37 +00:00
Max Inden f04afd596b .maintain/monitoring/alerting-rules: Add fd alert (#6946)
Alert on high file descriptor allocation.
2020-08-24 15:37:07 +02:00
Alexander Popiak bcc1b7c146 add runtime migrations to release notes/changelog (#6875) 2020-08-13 16:54:05 +00:00
dependabot[bot] 5a81eb86c4 Bump elliptic from 6.5.2 to 6.5.3 in /.maintain/chaostest (#6791)
Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.2 to 6.5.3.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](https://github.com/indutny/elliptic/compare/v6.5.2...v6.5.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-08-03 10:52:43 +02:00
Pierre Krieger 63bd1d8346 Add the Substrate Service Tasks dashboard (#6665) 2020-07-27 09:34:02 +00:00
HarryHong 36e312088e [CI]Chaostest suite initiation (#5793)
* Initiate chaostest cli test suite: singlenodeheight on one dev node
    Added chaostest stages in CI
    Added new docker/k8s resources and environments to CI
    Added new chaos-only tag to gitlab-ci.yml

* Update .maintain/chaostest/src/commands/singlenodeheight/index.js

Co-authored-by: Max Inden <mail@max-inden.de>

* change nameSpace to namespace(one word)

* update chaos ci job to match template

* rename build-pr ci stage to docker [chaos:basic]

* test gitlab-ci [chaos:basic]

* Update .gitlab-ci.yml

* add new build-chaos-only condition

* add *default-vars to singlenodeheight [chaos:basic]

* change build-only to build-rules on substrate jobs [chaos:basic]

* test and change when:on_success to when:always [chaos:basic]

* resolve conflicts and test [chaos:basic]

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>
2020-07-23 12:53:55 +02:00
gabriel klawitter 7e72a658c7 fix check polkadot companion status check (#6688) 2020-07-20 11:05:27 +02:00
Max Inden fe9c01fc68 .maintain/monitoring/alerting-rules: Remove HighCPUUsage alert (#6648)
The `HighCPUUsage` alert is based on the `cpu_usage_percentage` metric.
Instead of exposing the overall CPU usage in percent, the metric exposes
the per core usage summed over all cores.

This commit removes the alert for two reasons:

1. Substrate itself does not expose the core count and thus one can not
alert based on the `cpu_usage_percentage` metric.

2. Alerting based on CPU usage is generic and not specific to Substrate
or Blockchains. Thus any CPU usage alert suffice.
2020-07-17 07:43:57 +00:00
Pierre Krieger 84d607b5ff Update substrate-networking Grafana dashboard (#6649) 2020-07-16 16:45:12 +00:00
Cecile Tonglet 2827637103 Remove polkadot companion detection from url anywhere (#6570) 2020-07-16 11:01:18 +02:00
s3krit f957281008 [CI] Fix check_polkadot_companion_status.sh (#6631)
* check changes_requested and approved separately

* handle checking more than 1 CHANGES_REQUESTED review
2020-07-15 13:30:18 +02:00
André Silva e3ba163714 ci: fix merge on polkadot companion job (#6574) 2020-07-03 21:07:11 +02:00
Cecile Tonglet 2ddbbd7c08 Remove polkadot companion detection from branch name (#6565)
* Initial commit

Forked at: f54614e256
Parent branch: origin/master

* Remove polkadot companion detection from branch name

Even though it was nice it was also error prone as there were no indication whatsoever on the PR
that a polkadot companion branch exists.
2020-07-03 17:29:08 +02:00
pscott 540ae1c161 Update SubstrateCli to return String (#6550)
* Update SubstrateCli to return String

* Add default implementation for executable_name()

* Use display instead of PathBuf

* Get file_name in default impl of executable_name

* Remove String::from and use .into()

* Use default impl for executable_name()

* Use .as_str() and remove useless .to_string()

* Update only sp-io when running companion build

* Remove unneeded update of sp-io in CI

Co-authored-by: Cecile Tonglet <cecile@parity.io>
2020-07-02 13:34:56 +02:00
Max Inden 585ea531a3 .maintain/monitoring/alerting-rules: Adjust transaction queue size alert (#6426)
The transaction queue size alert has been firing with a constant 10
transactions in the queue. While maybe problematic those 10 transactions
don't need to be the same across scrape intervals.

Instead of alerting with a size above 10, alert based on two things:

1. Monotonically increasing queue size

2. Upper limit queue size reached
2020-07-01 10:31:56 +02:00
Benjamin Kampmann f6ebd5f87d Releasing rc4 – Rhinoceros (#6515)
Co-authored-by: Martin Pugh <pugh@s3kr.it>
2020-06-25 23:18:43 +02:00
Max Inden 2cbefce35b .maintain/sentry-node: Remove UI and update Prometheus target (#6473)
Remove burden on user to build polkadot-js apps Docker image locally in
order to get started.

Update Prometheus config fixing target name.
2020-06-25 11:25:28 +02:00
Max Inden fe76ebd548 .maintain/monitoring: Add alerting rule tests (#6343)
* .maintain/monitoring: Add alerting rule tests

* .maintain/monitoring/alerting-rules/alerting-rules.yaml: Break lines

* .gitlab-ci.yml: Add promtool rule testing step
2020-06-19 08:31:42 +02:00
ddorgan 1d364404e7 Update s3 artifact url (#6399) 2020-06-18 10:48:34 +01:00
Gavin Wood 5f991ae9b8 Allow Sudo to do anything (#6375)
* All Sudo to do anything.

* Rename old labels.
2020-06-17 12:22:57 +02:00
Pierre Krieger 0787b189fa Use /dns/ instead of /dns4/ (#6369) 2020-06-16 22:49:44 +02:00
s3krit 14ee618df0 [CI] Add label enforcement (#6365)
* Add label enforcement

* fix .gitlab-ci.yml

* update check_labels.sh
2020-06-16 15:50:50 +02:00
Alexander Theißen 50604fb445 Fix check-line-width CI script (#6326)
* Compare lines to the hash that the PR branched off from

* Use git merge-base to determine common ancestor

* Fixup
2020-06-12 13:47:14 +02:00
Kian Paimani 8a8b4f99c3 Rename all the election operations (#6245)
* Rename and move sp-phragmen

* More renames for equalise

* Update main module doc

* Fix line width

* Line width
2020-06-05 17:33:13 +02:00
Pierre Krieger 9cac359f44 Add a Substrate networking Grafana dashboard template (#6171)
* Add a substrate-networking grafana dashboard

* Capitalize data source

* Do changes
2020-06-01 10:46:34 +02:00
Max Inden aa95c596e6 .maintain/monitoring: Add an initial set of Prometheus alerting rules (#6095)
Create a place to collaborate on Prometheus alerting rules for
Substrate starting with a basic set of rules covering:

- Resource usage
- Block production
- Block finalization
- Transaction queue
- Networking
- ... Others
2020-05-21 16:26:29 +02:00
gabriel klawitter 27250681bf ci: github api use token for pull requests (#6037) 2020-05-15 11:34:16 +05:30
André Silva a975796d76 ci: fix merge on polkadot companion job (#6020) 2020-05-14 01:01:57 +02:00
Alexander Theißen 8110205820 Fix gitlab line-width CI check (#5994)
* Run script in strict mode

* Add proper seperator between revision and file

* Fix copy paste error

* Do not repeat limit number in error text

* Fix bad revision error

* Do not mask pipe errors

* Fix typo

* Remove unnecessary ... syntax

* Do not fetch all commits of master

* Fetching one commit is enough
2020-05-13 10:47:27 +02:00
Bastian Köcher 05cb63dd70 Merge Substrate master before checking Polkadot companion (#5916)
* Merge Substrate master before checking Polkadot companion

* Also merge polkadot/master into the companion pr branch

* Set email and user name
2020-05-07 12:50:25 +02:00
s3krit 2d73ccd65b [CI] Add check for draft PRs (#5800)
* Add check for draft PRs

* Add necessary gitlab-ci boilerplate

* fix github API url (oops!)

* Add github action to retrigger pipeline

* fix workflow trigger

* rename trigger job

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-04-28 17:55:09 +02:00
s3krit d974df243a Increase check-gitlab-pipeline timeout (#5774) 2020-04-24 17:42:49 +02:00
Max Inden d742e88e79 client/authority-discovery: Allow to be run by sentry node (#5568)
* client/authority-discovery: Allow to be run by sentry node

When run as a sentry node, the authority discovery module does not
publish any addresses to the dht, but still discovers validators and
sentry nodes of validators.

* client/authority-discovery/src/lib: Wrap lines at 100 characters

* client/authority-discovery: Remove TODO and unused import

* client/authority-discovery: Pass role to new unit tests

* client/authority-discovery: Apply suggestions

Co-Authored-By: André Silva <123550+andresilva@users.noreply.github.com>

* bin/node/cli/src/service: Use expressions instead of statements

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-04-16 18:11:26 +01:00
Max Inden 61d64e2ca1 .maintain/sentry-node: Add monitoring to docker-compose stack (#5321)
* Substrate Dashboard example

* Improve README

* Update README_dashboard.md

* Add screenshots

* Minor fix

* Minor fix, image link

* .maintain/sentry-node: Add monitoring to docker-compose stack

With this patch a user can run the following fully configured and
monitored setup with a single command:

`docker-compose -f .maintain/sentry-node/docker-compose.yml up`

- 2 validators in two different network namespaces, connected via one
sentry node.

- Polkadot-js/apps to connect to one of the nodes above.

- Prometheus scraping the 3 Substrate nodes.

- Grafana displaying data from Prometheus with community dashboards

* .maintain/monitoring/grafana: Change default datasource name

* .maintain/monitoring/grafana: Add metric namespace option

* .maintain/monitoring/grafana: Remove `host` metric from most metrics

* .maintain/monitoring/grafana: Remove underscore from metric_namespace

* .maintain/monitoring: Use `instance` label instead of `hostname`

To identify a scrape target, one should use `instance` and not
`hostname` as multiple targets might run on the same node.

See https://prometheus.io/docs/concepts/jobs_instances/ for details.

* .maintain/monitoring: Introduce instance variable

* .maintain/monitoring/grafana: Rename substrate_block_height_number

* .maitain/monitoring/grafana: Use instance instead of host in legend

* .maintain/monitoring: Remove node exporter dependency

* .maintain/sentry-node/prometheus: Simplify configuration

* .maintain/monitoring/grafana: Update README and remove images

* .maintain/sentry-node: Improve docs

* .maintain/monitoring/grafana: Use metric_namespace template variable

* Use --sentry from v0.7.29 instead of a reserved-node

* .maintain/sentry-node: Revert sentry-a using validator-b as bootnode

Co-authored-by: DerFredy - @derfredy:matrix.org <derfredy@gmail.com>
Co-authored-by: david <davidd@custom.home>
2020-04-14 16:08:09 +02:00
s3krit 98b2f46a89 CI: Release - monitor gitlab pipeline (#5460)
* Initial commit of gitlab pipeline monitoring

* increase max job time

* Improve debug output

* s/PR/issue/

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-04-09 12:58:28 +02:00
Cecile Tonglet 3da069e359 CLI: refactoring: remove Options from sc_service::Configuration's fields (#5271)
* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd38b40fbc420b02c1f8e61d5b2b1a4e434.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <seunlanlege@gmail.com>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 5906776953392c02beac6bc0bf50f8cbe1a12a01.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b7f3d732be001141afee210fe46a1ef47.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c0550164449e6eb2d3bacb04c750c714fcea.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5d2e1c0dc49cdff8608d59fc40fc59338.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba345c0d79f0d3b5b071933d95474d0614.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a4f1efd86e94c1214420a12e7a4be0099.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-04-07 11:38:07 +02:00
gabriel klawitter 14505471ee ci: check_polkadot: check for companion (#5429) 2020-04-03 16:42:18 +02:00
cheme 334e9333a1 Companion PR ci sed issue. (#5508)
* ci: fix echo sed failure

Co-authored-by: gabriel <gabriel@parity.io>
2020-04-03 17:51:21 +05:30
Cecile Tonglet 72239adda1 check_polkadot.sh: use the branch matching substrate's branch if possible (#5468)
* check_polkadot.sh: use the branch matching substrate's branch if possible

* convert to grep

* update comment
2020-04-01 19:52:34 +05:30
gabriel klawitter 2008d3601a ci: check_polkadot_companion_status: don't verify mergeable_state (#5473) 2020-03-31 19:33:35 +02:00
gabriel klawitter b8eb094c7f Ci check companion (#5419)
* add post-build-test stage

* check status

* fix busybox shell

* renaming for consistency
2020-03-31 10:55:12 +02:00
pscott 408455f8bc Build for only one target for docs.rs (#5427)
* Add docs.rs metadata to all cargo.toml files

* Remove docs.rs metada in substrate's cargo.toml
2020-03-30 09:46:30 +02:00
gabriel klawitter 15df85a000 ci: check_polkadot: look for arbitrary link to polkadot pr eventually (#5415) 2020-03-26 16:40:53 +01:00
gabriel klawitter eb135ccb56 ci: check_polkadot: accept linking to polkadot pr (#5410) 2020-03-26 12:27:55 +01:00
Denis Pisarev 06525e2b47 add cargo-deny job (#5347)
* add cargo-deny job

* feat(ci): add cargo-deny CI job, checks registries, licenses, dependencies dupes and advisories
2020-03-24 18:07:28 +01:00
gabriel klawitter 75123f051e check polkadot for companion pull requests (#5262)
* check_polkadot: move to external script

* check_polkadot: check for companion pr

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2020-03-24 18:29:11 +05:30
Chevdor 86410de227 Add more generated utils to the image (#5305) 2020-03-18 23:04:32 +01:00
Benjamin Kampmann 25e25a83b9 Releasing 2.0.0-alpha.4 (#5279)
* Version bump

* Split generate_changelog.sh into separate script

Can be run in the format `generate_changelog.sh $previous_version $version`.

* remove early exit from publish_draft_release.sh

* adding changelog

* ci: change last_github_release to also find pre-releases

Co-authored-by: Martin Pugh <pugh@s3kr.it>
2020-03-18 14:37:21 +01:00
Max Inden 733d486814 Revert "Substrate Dashboard example (#5284)" (#5293)
This reverts commit 082b66434e.
2020-03-18 11:13:58 +01:00
gabriel klawitter 14c76cea4c ci: check_runtime: fetch release tag iii (#5292) 2020-03-18 15:38:29 +05:30