* .maintain/monitoring: Add alert when continuous task ends
Through the `polkadot_tasks_ended_total` Prometheus metric one can tell
when a task ended. Use this metric to alert when specific
known-to-be-continuous tasks end on a node.
* .maintain/monitoring: Don't hard-code task names
* .maintain/monitoring: Normalize alerting rules
- Start alert names with their component and end with the describing
adjective.
- Describe alert duration in `message` with `for more than` across all
alerts.
* .maintain/monitoring: Fix alert tests
The `HighCPUUsage` alert is based on the `cpu_usage_percentage` metric.
Instead of exposing the overall CPU usage in percent, the metric exposes
the per core usage summed over all cores.
This commit removes the alert for two reasons:
1. Substrate itself does not expose the core count and thus one can not
alert based on the `cpu_usage_percentage` metric.
2. Alerting based on CPU usage is generic and not specific to Substrate
or Blockchains. Thus any CPU usage alert suffice.
The transaction queue size alert has been firing with a constant 10
transactions in the queue. While maybe problematic those 10 transactions
don't need to be the same across scrape intervals.
Instead of alerting with a size above 10, alert based on two things:
1. Monotonically increasing queue size
2. Upper limit queue size reached
Create a place to collaborate on Prometheus alerting rules for
Substrate starting with a basic set of rules covering:
- Resource usage
- Block production
- Block finalization
- Transaction queue
- Networking
- ... Others
* Substrate Dashboard example
* Improve README
* Update README_dashboard.md
* Add screenshots
* Minor fix
* Minor fix, image link
* .maintain/sentry-node: Add monitoring to docker-compose stack
With this patch a user can run the following fully configured and
monitored setup with a single command:
`docker-compose -f .maintain/sentry-node/docker-compose.yml up`
- 2 validators in two different network namespaces, connected via one
sentry node.
- Polkadot-js/apps to connect to one of the nodes above.
- Prometheus scraping the 3 Substrate nodes.
- Grafana displaying data from Prometheus with community dashboards
* .maintain/monitoring/grafana: Change default datasource name
* .maintain/monitoring/grafana: Add metric namespace option
* .maintain/monitoring/grafana: Remove `host` metric from most metrics
* .maintain/monitoring/grafana: Remove underscore from metric_namespace
* .maintain/monitoring: Use `instance` label instead of `hostname`
To identify a scrape target, one should use `instance` and not
`hostname` as multiple targets might run on the same node.
See https://prometheus.io/docs/concepts/jobs_instances/ for details.
* .maintain/monitoring: Introduce instance variable
* .maintain/monitoring/grafana: Rename substrate_block_height_number
* .maitain/monitoring/grafana: Use instance instead of host in legend
* .maintain/monitoring: Remove node exporter dependency
* .maintain/sentry-node/prometheus: Simplify configuration
* .maintain/monitoring/grafana: Update README and remove images
* .maintain/sentry-node: Improve docs
* .maintain/monitoring/grafana: Use metric_namespace template variable
* Use --sentry from v0.7.29 instead of a reserved-node
* .maintain/sentry-node: Revert sentry-a using validator-b as bootnode
Co-authored-by: DerFredy - @derfredy:matrix.org <derfredy@gmail.com>
Co-authored-by: david <davidd@custom.home>
Pierre Krieger