For slack, [FIRING] contains [RESOLVED] alert

Question

In alertmanager, I set slack and mail on the receiver and compared how they look.

The results are as follows.

mail

3alerts for

[2] Firing

alert1

alert2

[1] Resolved

alert3

slack

[FIRING:2]

alert1

alert3 ★

alert2

As you can see, the client cannot confirm that alert3 has become RESOLVED. What configuration files are needed to configure slack notifications in the same way as email notifications?

The current settings are as follows.

config.yml

  send_resolved: true
  text: |-
    <!channel>
    {{ range .Alerts }}
      *Alert:* {{ .Labels.alertname }} - `{{ .Labels.severity }}`
      *Summary:* {{ .Annotations.summary }}
      *Description:* {{ .Annotations.description }}
      *Details:*
      {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
      {{ end }}
    {{ end }}

We look forward to your answer.

Is there any news on this?

Janiis
– Janiis

2023-06-14 11:42:27 +00:00
Commented Jun 14, 2023 at 11:42 — Janiis
– Janiis, Commented Jun 14, 2023 at 11:42

Stepan Tsybulski · Accepted Answer · 2020-01-06 09:54:54Z

1

I think you configured it correctly. send_resolved: true must do the trick.

We're using slack & email notifications and send_resolved is working fine for both these targets. Please, have a look at Alertmanager's logs. Probably, there was a network issue..

UPDATE

Another reason can be that you updated your alermanager's setting after the initial deployment so you expect that new setting will apply dynamically with the help of prometheus-alertmanager-configmap-reloadcontainer but it often happens that this reload is failing by some reason so what you can do is to delete your alermanager's pod, wait once a new pod is started and see whether it solved your issue or not.

edited Jan 6, 2020 at 9:54

answered Dec 29, 2019 at 23:04

Stepan Tsybulski

1,2119 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

peropreox Over a year ago

No, this issue is always reproduced. Which logs should I look at? Can you share the file name and path?

Stepan Tsybulski Over a year ago

What you have to do is just perform logs command on your Alertmanager's pod: kubectl logs -f [name_of_your_alertmanager_pod] -c prometheus-alertmanager. In case some attempt of email sending failed you will see the reason there.

James Pei · Accepted Answer · 2021-12-30 10:06:08Z

0

It seems the behaviour of alertmanager, when there are multi-alerts OF THE SAME GROUP/TYPE, only after all alert is resoved, the status will become "resolved" or it will remain "firing".

answered Dec 30, 2021 at 10:06

James Pei

1512 silver badges13 bronze badges

Collectives™ on Stack Overflow

For slack, [FIRING] contains [RESOLVED] alert

mail

slack

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

mail

slack

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related