Alerting is a key concept to remediating issues that arise in the systems we monitor. In order to alert effectively, it's helpful to understand the concepts available in Alertmanager and how they are actually executed within the code. In this talk, I'll give a brief, high-level description of the wondrous journey an alert takes, from its triggering in Prometheus to the email in your mailbox. Then, I'll break down in-depth how alerts are ingested, grouped, and processed, including high availability mode. At the end of this exciting trip you'll understand how your configuration affects Alertmanager, what an alerting pipeline is, and why you sometimes get two emails for the same alert.
Speakers: Stuart Nelson (SoundCloud)