Commit Graph

4 Commits

Author SHA1 Message Date
Max Inden 51c0d27aa1 .maintain/monitoring: Normalize alerting rules (#7232)
* .maintain/monitoring: Normalize alerting rules

- Start alert names with their component and end with the describing
adjective.

- Describe alert duration in `message` with `for more than` across all
alerts.

* .maintain/monitoring: Fix alert tests
2020-09-30 08:48:48 +00:00
Max Inden fe9c01fc68 .maintain/monitoring/alerting-rules: Remove HighCPUUsage alert (#6648)
The `HighCPUUsage` alert is based on the `cpu_usage_percentage` metric.
Instead of exposing the overall CPU usage in percent, the metric exposes
the per core usage summed over all cores.

This commit removes the alert for two reasons:

1. Substrate itself does not expose the core count and thus one can not
alert based on the `cpu_usage_percentage` metric.

2. Alerting based on CPU usage is generic and not specific to Substrate
or Blockchains. Thus any CPU usage alert suffice.
2020-07-17 07:43:57 +00:00
Max Inden 585ea531a3 .maintain/monitoring/alerting-rules: Adjust transaction queue size alert (#6426)
The transaction queue size alert has been firing with a constant 10
transactions in the queue. While maybe problematic those 10 transactions
don't need to be the same across scrape intervals.

Instead of alerting with a size above 10, alert based on two things:

1. Monotonically increasing queue size

2. Upper limit queue size reached
2020-07-01 10:31:56 +02:00
Max Inden fe76ebd548 .maintain/monitoring: Add alerting rule tests (#6343)
* .maintain/monitoring: Add alerting rule tests

* .maintain/monitoring/alerting-rules/alerting-rules.yaml: Break lines

* .gitlab-ci.yml: Add promtool rule testing step
2020-06-19 08:31:42 +02:00