Telegram can't parse entities after message truncation with parse_mode="HTML"

[iuhenio]

What did you do?
Used Telegram integration with parse_mode="HTML"

What did you expect to see?
Alert notification in Telegram chat

What did you see instead? Under which circumstances?
Alert notification wasn't sent to chat

Environment

• System information:

  Linux 5.4.0-109-generic x86_64

• Alertmanager version:

  alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17)
  build user: root@265f14f5c6fc
  build date: 20220325-09:31:33
  go version: go1.17.8
  platform: linux/amd64

• Prometheus version:

  prometheus, version 2.34.0 (branch: HEAD, revision: 881111fec4332c33094a6fb2680c71fffc427275)
  build user: root@121ad7ea5487
  build date: 20220315-15:18:00
  go version: go1.17.8
  platform: linux/amd64

• Alertmanager configuration file:

global:
  resolve_timeout: 3m
templates:
- '/etc/alertmanager/templates/*.tmpl'
receivers:
- name: alerts-test1
  telegram_configs:
  - api_url: https://api.telegram.org
    bot_token: 1657974151:BBBO9Iggg_Tif04Tg9asdfSuR83niyNOSHE  # has been changed for security reasons
    chat_id: -10075044612354    # has been changed for security reasons
    disable_notifications: false
    message: '{{ template "telegram.html.message" . }}'
    parse_mode: HTML
    send_resolved: true

route:
  group_by:
  - alertname
  group_interval: 30s
  group_wait: 15s
  receiver: alerts-test1
  repeat_interval: 1d

• telegram.html.message template:

{{ define "telegram.html.message" }}
{{ range .Alerts }}
{{ if eq .Status "firing"}}🔥 <b>{{ .Labels.alertname }}</b> 🔥{{ else }}✅ <b>{{ .Labels.alertname }}</b> ✅{{ end }}
<b>Labels:</b>{{ range $key, $value := .Labels }}{{ if ne $key "alertname" }}
    <b>{{ $key }}</b>: <i>{{ $value }}</i>{{ end }}{{ end }}
<b>Annotations:</b>{{ range $key, $value := .Annotations }}
    <b>{{ $key }}</b>: <i>{{ $value }}</i>{{ end }}
{{ end }}
{{ end }}

• Prometheus configuration file:
  Isn't relevant to the issue

• Logs:

May 16 11:21:59 focal-1 alertmanager[30906]: ts=2022-05-16T08:21:59.430Z caller=telegram.go:72 level=debug integration=telegram msg="truncated message" truncated_message="\n\n🔥 <b>ExporterDown</b> 🔥\n
<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.10</i>\n    <b>environment</b>: <i>testing</i>\n    <b>instance</b>: <i>192.168.56.10:9100</i>\n    <b>job</b>: <i>node_exporter</i>\n    <b>label10</b>
: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label12</b>: <i>valuevaluevaluevaluevaluevaluevaluevalu
evaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label2</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label3</b>: <i>valuev
aluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</
i>\n    <b>label6</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label7</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label8</b>: <i>valuevaluevaluevalue
valuevaluevaluevaluevaluevalue</i>\n    <b>label9</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>severity</b>: <i>critical</i>\n<b>Annotations:</b>\n    <b>description</b>: <i>Ca
n't get data from exporter</i>\n    <b>space_filler</b>: <i>Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Te
st Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test</i>\n    <b>summary</b>:
 <i>Exporter down</i>\n\n🔥 <b>ExporterDown</b> 🔥\n<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.20</i>\n    <b>environment</b>: <i>testing</i>\n    <b>instance</b>: <i>192.168.56.20:9100</i>\n    
<b>job</b>: <i>node_exporter</i>\n    <b>label10</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>lab
el12</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label2</b>: <i>valuevaluevaluevaluevaluevalueva
luevaluevaluevalue</i>\n    <b>label3</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>
valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label6</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label7</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluev
alue</i>\n    <b>label8</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label9</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>severity</b>: <i>critical</i>
\n<b>Annotations:</b>\n    <b>description</b>: <i>Can't get data from exporter</i>\n    <b>space_filler</b>: <i>Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test 
Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Te
st Test Test Test Test Test</i>\n    <b>summary</b>: <i>Exporter down</i>\n\n🔥 <b>ExporterDown</b> 🔥\n<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.30</i>\n    <b>environment</b>: <i>testing</i>\n
    <b>instance</b>: <i>192.168.56.30:9100</i>\n    <b>job</b>: <i>node_exporter</i>\n    <b>label10</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevalu
evaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label12</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n  
  <b>label2</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label3</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluev
aluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>valuevaluevaluevaluevaluevaluevaluevalu..."

May 16 11:21:59 focal-1 alertmanager[30906]: ts=2022-05-16T08:21:59.526Z caller=notify.go:732 level=warn component=dispatcher receiver=alerts-test1 integration=telegram[0] msg="Notify attempt failed, wi
ll retry later" attempts=1 err="telegram: Bad Request: can't parse entities: Can't find end tag corresponding to start tag i (400)"

[petshopjke1337]

Hi, having same issue here. Did anyone find a workaround or was able to solve this? @metalmatze @timmilesdw

[compudaster]

Hello. I had the same problem. Try changing the parsing method to Markdown.

https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

Change "<b>" and "</b>" in the template to **
Change "<i>" and "</i>" to *

[1014670860]

having same issue here, need help

[petshopjke1337]

Hello. I had the same problem. Try changing the parsing method to Markdown.

https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

Change "" and "" in the template to ** Change "" and "" to *
It's not really an option for us. Markdown is buggy and lack some functions.
It's been over a year, i guess telegram integration is not that popular...

[pando85]

I used this workaround until we have a better solution:

        {{- $alertsLen := (len .Alerts) -}}
        {{ range .Alerts }}

            {{- if .Annotations.description }}
    _{{ .Annotations.description }}_
            {{- end }}
            {{- if .Annotations.message }}
    _{{ .Annotations.message }}_
            {{- end }}
            {{- if le $alertsLen 10 }}
    [Prometheus]({{ .GeneratorURL }})
            {{- end }}
        {{ end }}
    {{- end }}

[Shaked]

I am experiencing the same issue. This is not related directly to HTML but rather to Telegram's message size limitation. Once you reach the max size of the message's length, the message is truncated by alertmanager, see:

alertmanager/notify/telegram/telegram.go

Lines 82 to 87 in 14cbe63

	messageText, truncated := notify.TruncateInRunes(tmpl(n.conf.Message), maxMessageLenRunes)
	if truncated {
	level.Warn(n.logger).Log("msg", "Truncated message", "alert", key, "max_runes", maxMessageLenRunes)
	}

Now this might cause an HTML tag or Markdown symbol to be sent unclosed which will cause this error.

Honestly, I'm not sure how this can be fixed properly, it might be better to catch this error and react with a stripped text or even sending an .html/.md file to at least make sure that the alert arrives to its destination.

EDIT:

It might be possible to add another configuration next to group_by, like max_alerts_in_group and allow_bulk which will make sure that each message contains its alerts and N messages are sent when they pass Telegram's limitation.

[segator]

can be configured so it send a message per alert?

[mhdan]

still having the same issue, the max_alerts_in_group and allow_bulk can pass the telegram limitation, adding theses options will save us

[victorryakh]

+1 👍 Still highly relevant. Especially when using cAdvisor, whose alerts contain a large number of labels, so the message is truncated and it breaks HTML structure

[KonstantinNosov]

Just ran into the same issue and developed a template for telegram notifications with following logic

• build full notification message with all alerts
• if it is too big fall back to less verbose single alert format (no labels)
• if even with less verbose format the message appears too big - remove some alerts to fit the size limit (alerts are removed as a whole so telegram does not complain about broken markdown) and add notice about truncation to the message

I use MarkdownV2 as parse_mode, but the same approach may be used for HTML

{{- define "telegram.alert_list" -}}
  {{- $maxLength := 3900 -}}
  {{- $message := "" -}}
  {{- $message_is_truncated := false -}}
  {{- $message_no_labels := "" -}}
  {{- $message_no_labels_is_truncated := false -}}

  {{- range .Alerts -}}
    {{- $escape_regex := "([-_*\\[\\]()~`>#+=|{}.!])" -}}
    
    {{- /* Build alert text */ -}}
    {{- $alertText := ""}}
    {{- if eq .Status "firing" }}
      {{- $alertText = "🔥 "}}
    {{- end -}}
    {{- if eq .Status "resolved" }}
      {{- $alertText = "✅ "}}
    {{- end -}}

    {{- $title := .Labels.alertname | reReplaceAll  $escape_regex "\\$1" -}}
    {{- if .Annotations.summary}}
      {{- $title = printf "%s \\- %s" $title (.Annotations.summary | reReplaceAll  $escape_regex "\\$1")  -}}
    {{- end }}

    {{- $alertText = printf "%s***\\<%s\\> %s***\n" $alertText ( or .Labels.severity "no severity") $title -}}
    {{- if .Annotations.description}}
      {{- $alertText = printf "%s_%s_\n" $alertText ( .Annotations.description | reReplaceAll $escape_regex "\\$1" ) -}}
    {{- end }}
    
    {{- if .GeneratorURL -}}
      {{- $alertText = printf "%s[Source](%s)" $alertText (.GeneratorURL) -}}
    {{- end }}

    {{- if .Annotations.runbook_url -}}
      {{- $alertText = printf "%s [Runbook](%s)" $alertText (.Annotations.runbook_url) -}}
    {{- end -}}

    {{- /* Store alert text without labels in case all alerts would not fit message length limit  */ -}}
    {{- $message_tmp := "" -}}
    {{- if $message_no_labels  -}}
      {{- $message_tmp = printf "%s\n\n%s" $message_no_labels $alertText -}}
    {{- else -}}
      {{- $message_tmp = printf "%s" $alertText -}}
    {{- end -}} 

    {{- if $message_tmp | len | gt $maxLength  -}}
      {{- $message_no_labels = $message_tmp -}}
    {{- else -}}
      {{- $message_no_labels_is_truncated = true -}}
    {{- end -}}

    {{- /* Add labels to alert text */ -}}
    {{- range .Labels.SortedPairs -}}
      {{- if (eq .Name "alertname" "description" "summary" "runbook_url" "severity" "prometheus" ) | not -}}
      {{- $alertText = printf "%s\n🏷 %s: `%s`" $alertText (.Name | reReplaceAll $escape_regex "\\$1" ) .Value -}}
      {{- end -}}
    {{- end }}

    {{- $message_tmp := "" -}}
    {{- if $message  -}}
      {{- $message_tmp = printf "%s\n\n%s" $message $alertText -}}
    {{- else -}}
      {{- $message_tmp = printf "%s" $alertText -}}
    {{- end -}}
    
    {{- if $message_tmp | len | gt $maxLength  -}}
      {{- $message = $message_tmp -}}
    {{- else -}}
      {{- $message_is_truncated = true -}}
    {{- end -}}
    
  {{- end -}}

  {{- $message =  printf "%s\n" $message -}}
  {{- if $message_is_truncated -}}
    {{- $message =  printf "%s\nMessage is truncated to fit telegram API limit" $message -}}
  {{- end -}}

  {{- $message_no_labels =  printf "%s\n" $message_no_labels -}}
  {{- if $message_no_labels_is_truncated -}}
    {{- $message_no_labels =  printf "%s\nMessage is truncated to fit telegram API limit" $message_no_labels -}}
  {{- end -}}

  {{- if $message_is_truncated -}}
    {{- $message_no_labels -}}
  {{- else -}}
    {{- $message -}}
  {{- end -}}
{{- end -}}

{{- define "telegram.message" -}}
🔥  firing: {{ len .Alerts.Firing }} ✅ resolved: {{ len .Alerts.Resolved }}

{{ template "telegram.alert_list" . }}
[See On AlertManager]({{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }})
\#alerts
{{- end -}}

Here is how it looks in telegram:

UPD: wrote a blog post with template updated to output firing alerts first and more inline comments