PromCon 2018

Richard opens the conference and gives an overview of the state of Prometheus.

The pillars of observability have long been accepted as key components of any microservice-in-production. But what about those new products - those new features - that have yet to be released? Properly instrumenting and leveraging metrics at this stage is perhaps even more crucial; when a product is yet to be released, identifying and addressing early bugs is critical.

Alerting is a key concept to remediating issues that arise in the systems we monitor. In order to alert effectively, it's helpful to understand the concepts available in Alertmanager and how they are actually executed within the code. In this talk, I'll give a brief, high-level description of the wondrous journey an alert takes, from its triggering in Prometheus to the email in your mailbox. Then, I'll break down in-depth how alerts are ingested, grouped, and processed, including high availability mode. At the end of this exciting trip you'll understand how your configuration affects Alertmanager, what an alerting pipeline is, and why you sometimes get two emails for the same alert.

While there are plenty of readily available metrics for monitoring the Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:

Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configured monitoring "packages" hard to build. Simultaneously, it's becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

Autoscaling in Kubernetes used to be an inconsistent concept, however using the new metrics APIs defined by Kubernetes SIG Instrumentation the monitoring system of choice can be used. As it turns out, Prometheus is a popular system being used alongside Kubernetes, and the community has already developed a custom-metrics-api adapter to be used for Prometheus. This means, we can now perform autoscaling on the cluster and application level with metrics collected by Prometheus.

Relabeling is a powerful feature of Prometheus but it is also quite specific to the project. This talk will explain and demonstrate relabeling, with some real use cases. After the presentation, you will understand relabeling and stop to copy/paste relabeling rules you don't understand.

Series cardinality can be a large foot-gun that can take out a Prometheus instance quickly. It can have similar impacts to downstream systems when using remote write. Finding the offending series and recovering from such an event can cause a large downtime for your Prometheus installation. In this talk I will show an implementation of a sidecar to Prometheus that automatically detects and squelches series whose cardinality is growing quickly. Using a combination of recording rules, dynamic metric relabel configs, and alerting, we can detect, throttle, and alert on any explosion in progress.

The Prometheus Monitoring system has been thriving for several years. Along with its powerful data model, operational simplicity and reliability have been a key factor in its success. However, some questions were still largely unaddressed to this day. How can we store historical data at the order of petabytes in a reliable and cost-efficient way? Can we do so without sacrificing responsive query times? And what about a global view of all our metrics and transparent handling of HA setups?

The new open source Flux query engine for InfluxDB is designed to be decoupled from storage. This makes it possible to query many different data sources, including Prometheus, with the new engine. The engine also decouples the representation of the query language from the execution representation. This makes it possible to add support for additional languages, like PromQL.

In this panel, we will explore some of the emerging and competing approaches to Prometheus scalable long-term storage.

By definition, SREs are responsible for the reliability of sites, but what if they don't own any sites themselves? Within ING, the largest bank of the Netherlands, BizDevOps teams are autonomous and responsible for building and running their services. In theory, that could make the existence of SRE obsolete, right? How can you improve availability for end customers in an environment of engineers with full service ownership? How to convince without the power of intervention? How to improve without being blameful?

Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We've built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I'll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We show how PromQL and Alertmanager can be used to detect abnormalities in LAN and WiFi networks, using various Prometheus exporters to take data from WLAN controllers and SNMP. We use Unifi, Grafana and of course Prometheus + Alertmanager, along with a few exporters.

Prometheus has established a de-facto standard for cloud-native metric exposition. OpenMetrics aims to achieve true interoperability, and to bridge the gap towards other types of monitoring.

Users have been looking for a better understanding of how Prometheus monitoring and other commercial monitoring tools compare and contrast when it comes to Docker and Kubernetes monitoring. Are they enemies? Lovers? Twins separated at birth? Let's go there. This talk will discuss the Prometheus ecosystem from a vendor perspective.

Prometheus is capable of and used for monitoring large infrastructures, from corporate networks to world-spanning CDNs. But while performing excellent on setups of large scales, it is also extremely well suited for small setups due to its simplicity and modularity both in operation as well in usage.

Over the past 6 months at Fastly we've migrated away from our legacy monitoring systems and have deployed Prometheus as our primary system for infrastructure and application monitoring.

We will show you how we are trying to make Prometheus more stable by running automated benchmarking for risky PRs and before every release.

Prometheus client libraries are notably different from most other options in the space. In order to get the best insights into your applications it helps to know how they are designed, and why they are designed that way. This talk will look at how client libraries are structured, how that makes them easy to use, some tips for instrumentation, and why you should use them even if you aren't using Prometheus.

A panel of Prometheus maintainers will answer questions about Prometheus, alternating between audience questions and presubmitted ones. Submit your questions in advance here!.

Richard will close the conference with a few parting words.