Feature: Alertmanager Watchdog notification

[hrabalvojta]

Hello,

is there any possibility to add function to cover watchdog alert? It would be nice to known monitoring is down.

[alexbakker]

Could you explain what you mean with "watchdog alert"? How does one configure this in Alertmanager? And which changes would you like to see in alertmanager-ntfy to support this?

[hrabalvojta]

How would you monitor if monitoring is working, without adding additional services? Alertmanager has dead man switch. Alert which is always firing called "Watchdog". When this alert is not firing. It means Prometheus is down.

It uses inverted logic. Alert firing good. Resolved bad.

At moment I am using this in kube-prometheus-stack. If you look at routes I am sending everything except Watchdog.

alertmanager:
  config:
    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 1m
      repeat_interval: 12h
      receiver: "ntfy"
      routes:
        - receiver: "ntfy"
          continue: true
          matchers:
            - alertname != "Watchdog"
        - receiver: "keephq"
          continue: true
          matchers:
            - alertname != "Watchdog"
        - receiver: "null"
          continue: true
          matchers:
            - alertname = "Watchdog"
    receivers:
      - name: "null"
      - name: "ntfy"
        webhook_configs:
          - url: https://alertmanager-ntfy.example.com/hook
            send_resolved: true
            http_config:
              basic_auth:
                username: xxxxxxxxxxxxx
                password: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
      - name: "keephq"
        webhook_configs:
          - url: "https://keephq.example.com/alerts/event/prometheus?provider_id=315f747a902245e48bed814700f026f4"
            send_resolved: true
            http_config:
              basic_auth:
                username: xxxxxxxxxxxxx
                password: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

Ideal was to implement "some logic" for alert "Watchdog". Something in style I did not receive this alert for x minutes. Prometheus is down.

[hrabalvojta]

How to configure alertmanager to send everything. # If removing matcher does not work. You can not equal empty

alertmanager:
  config:
    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 1m
      repeat_interval: 12h
      receiver: "ntfy"
      routes:
        - receiver: "ntfy"
          continue: true
        - receiver: "ntfy-2"
          continue: true
          matchers:
            - alertname != ""