Prometheus uses nail alarm

Prometheus's own email alerts may be submerged in daily emails. Therefore, it is recommended to configure another alarm channel of webhook class.

For flexibility, we can write webhook ourselves, for example, we can send alarm notifications of different rules to different recipients, alarm silence for certain periods of time, and so on.

If the requirement is not high, we can also use the domestic third-party alarm webhook (such as nail dingtalk, Becha bearychat)

Here I will take nails as an example:

There are open source projects on github that can be used directly

Address: https://github.com/timonwong/prometheus-webhook-dingtalk

Download the binary package: prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

Tar xf prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz-C / usr/local/

Cd / usr/local/prometheus-webhook-dingtalk-0.3.0.linux-amd64

. / prometheus-webhook-dingtalk-- ding.profile= "ops_dingding= https://oapi.dingtalk.com/robot/sendaccess_token=xxxxxxxxxxxxxxxxxxxxxx" foreground launch process

TIPS: I got this token when I built a new robot in dingding. You can search it yourself to see how to add a nailing robot. I skipped it here.

And then, we need to get to alertmanager.

Global:

Resolve_timeout: 5m

Receivers:

-name: 'dingding.webhook'

Webhook_configs:

-url: 'server address of http://${prometheus-webhook-dingtalk}: 8060 server.

Send_resolved: true

Route:

Group_by: ['cluster','alertname','job']

Group_wait: 30s

Group_interval: 1m

Repeat_interval: 5h

Receiver: 'dingding.webhook'

Routes:

-match:

Severity: critical

Receiver: 'dingding.webhook'

Group_wait: 10s

Group_interval: 15s

Repeat_interval: 5m

-match:

Severity: warning

Receiver: 'dingding.webhook'

-match:

Alertname: Watchdog

Repeat_interval: 1h

Then, reload the configuration of alertmanager to take effect.