PromCon EU 2019

Thinking of replacing your existing monitoring solution with Prometheus? In this talk we will show you how Runtastic moved its monitoring and alerting stack from Nagios to Prometheus. We'll show you how we managed to do this, how monitoring and alerting works with Prometheus and what challenges we were facing.

Grafana is a wonderful tool for visualizing Prometheus data. Getting started with Grafana is simple: login, create a dashboard, add some graphs, and call it a day. Come back later, add some graphs, edit some graphs, hit save and it's all live! Flash forward a few months or years and your Grafana instance probably has tens or hundreds of dashboards. Many of them present similar data in slightly inconsistent ways. You can see who last edited a dashboard and why, but it's hard to visualize the changes they made. There's no good way to review changes before they go live like you would for any other code you write. One row on your favorite dashboard is ever so slightly taller than the others and it makes you twitch every time you notice. At DigitalOcean we've solved these problems by storing our dashboard definitions in git, making them human-readable with grafonnet, and updating them via continuous integration. In this talk we'll describe our setup, show how it enables component re-use between dashboards, and demonstrate how it improves the ergonomics of managing Grafana dashboards at scale.

Prometheus, at first look, can appear very simple and at the same time very complicated, in this talk will be illustrate how a small/medium IT company can approach the deployment of the prometheus stack (Prometheus, Grafana, alertmanager and exporters), start from the local testing environment and ansible roles, to the targets labeling and exporters configurations. Will be also outlined the benefits that Prometheus can give to a company in terms of metrics visibility, the easiness to plug in new systems for generic metrics collections and the knowledge sharing, without necessary be used for monitoring dynamic and complex systems.

PromQL can feel intimidating at first for those new to the Prometheus ecosystem, but it is an essential skill to learn when starting out monitoring and alerting in the Prometheus ecosystem. In this beginner-focused, demo-driven talk Ian Billett will walk through the thought processes, mental models and coping strategies he uses to tame PromQL and effectively query metrics data. We will demystify instant vector selectors from range vector selectors, understand how aggregate functions work, and, time-permitting, even touch on some PromQL engine internals.

Prometheus offers lots of features and flexibility which often leaves newcomers with lots of open questions: Which service discovery to use? How to secure access to metrics? Which architecture to implement? In order to provide some input back to the community, this talk will outline how we run Prometheus at HUK-COBURG in order to provide a multi-tenant turn-key monitoring experience in a security-focused environment. It covers basic decisions like scrape intervals and service discovery and extends to the implementation of encrypted and authenticated connections to monitoring targets using sshified and team-aware access to Prometheus/Grafana access via prometheus-filter-proxy.

Getting someone to know that a problem exists is the main reason why we run monitoring infrastructure. Prometheus and AlertManager are very good at it but requirements vary from person to person. In this presentation, Simon Pasquier will introduce the Prometheus alerting philosophy. Then he will discuss AlertManager's good practices and pitfalls. Finally he will show advanced use cases.

GitLab.com’s monolithic Rails application experiences high week-on-week traffic growth. To ensure availability, GitLab’s Infrastructure team track and plan ahead in order to avoid hitting capacity limits in the application, whether these limits be CPU, database connection pools, memory, storage or any number of other finite resources. Hitting these limits could result in hours, or days, of degraded service while workarounds are put in place. With this in mind, the team set about building a set of tools on top of Prometheus recording rules and alerts to provide them with the information they need to be sufficiently forewarned, up to a month in advance, of potential resource saturation issues. If you’ve ever felt that you’re reactively responding to resource saturation issues, this session will provide practical examples of how we’re building a framework for resource planning into our SRE team workflow. We’ll be presenting our open-source solution and explaining how it works for us.

The cloud runs on servers, and those live in datacenters. Most everything in datacenters speaks ModBus/TCP; SNMP being considered too modern. Lessons learned and problems overcome along with some nice pictures.

Remote write storage is a hot topic, several systems existed for a few years (Cortex, M3DB, InfluxDB) and some systems have been born in the past few months (Thanos receive, VictoriaMetrics). Each system has it's own architecture and different trade-offs. In this talk, Alejandro will present adidas findings on experiments to benchmark various remote write open source storage systems such as Cortex, M3DB, Thanos, VictoriaMetrics and others. The experiments try to benchmark scalability and elasticity of each storage system.

Prometheus's remote storage API allows PromQL evaluation to work with stores other than TSDB, and with data that may not have originated via native Prometheus scraping. PromQL has expectations about the data served by the API, which must be met in order to give a "compatible" user experience with native Prometheus. I'll present my teams experience marrying an existing distributed time series database, originally designed around storing downsampled statsd-style metrics, with a PromQL evaluation engine. I'll cover topics including: downsampling, counter aggregation, series staleness, and others.

Speakers: Bartlomiej Plotka and Tom Wilkie

This talk goes over how our micro-services infrastructure is set up at Prezi, how it used to be monitored with convoluted and unreliable solutions, how we introduced Prometheus to all of our services, what we learned, where we failed, how it improved our engineers' life and what it enables us to do now. We recently spent 8 months redesigning our entire monitoring infrastructure, replacing 10 years of unreliable organically grown solutions to a uniform and highly available Prometheus-based platform. Our services being mostly written in Django/Python, we spent a lot of time getting in right and contributed back to the community (see https://engineering.prezi.com/prometheus-unchained-331b7ab8737, https://github.com/prezi/django-exporter). This was also the opportunity to revise our alerting philosophy (using the famous "My Philosophy on Alerting") and had to re-educate all of our engineers on how to alert and how to make great dashboards (or not making any and just using generic alerts and dashboards). A key point was to measure the impact of these changes by counting the alerts and how engineer react to them (was the alert actionable?). Based on that, we can see how well our practices are performing and react if necessary.

Spending time in the Prometheus community quickly instills a fear of high cardinality metrics; however, cardinality provides immense value for certain workloads. Learn when to use high cardinality metrics, how to reduce unnecessary cardinality, and how to find the sources of excessive cardinality in your data.

Emerging Nodeless Kubernetes solutions like virtual-kubelet and Kiyot simplify public cloud k8s node management, save cost, and improve multi-tenant security. This talk explores how Prometheus can be used to monitor Nodeless Kubernetes workloads on public cloud. Demo included.

Jaeger is an OSS distributed tracing solution, also part of the CNCF. Standalone, it can add immense value but when coupled with Prometheus, there is a lot more to gain. We have leveraged Prometheus and Jaeger together to make Cortex 10x faster and we will be sharing our journey and key learnings. With Prometheus, we have our RED dashboards that highlight which services are slow, and we then use Jaeger to drill down and investigate which functions are taking how long, if we’re making too many calls, if we could batch calls together, etc. We then verify the impact using Jaeger and Prometheus after rolling out the changes. Without our RED dashboards, it would be shooting in the dark to find out which spans/services we should look at in Jaeger. Usually the slowness occurs in only one cluster. To enable filtering of traces to only that cluster/env, we enrich the traces with the same metadata that prometheus has. We were also seeing very slow queries once in a while that weren’t very obvious when using only dashboards. We then started logging the trace ID in our request logs. We picked the requests that took too long, and used their trace IDs to probe for issues with that particular request. While this approach works, you need a solid log aggregation system, and even then it’s not exactly easy to find the interesting traces (longest/shortest requests). We will show you how Prometheus helps by providing us exemplars that can let us directly jump to the right traces from the latency dashboards. We will end the talk by demo-ing everything discussed.

One of the reasons we collect metrics is to be able to alert on them. This presentation will introduce you some concepts of PromQL, prometheus and alertmanager to highly improve the quality and reliability of your alerts. This talk will cover different topic, including: - Reducing flapping alerts - Hysteresis - "Time of the day" based alerting - Computed thresholds with data history

The OpenMetrics format intends to standardise metric exposition, making it easy for both those developing and operating systems to monitor them. It is however a new format. Will it be supported by your monitoring system? Will you need to rewrite your existing instrumentation? What's needed to transition? What about 3rd party systems you don't control? How does this differ and expand, and improve on the existing Prometheus format? This session will cover all of these questions.

Since a major revamp of the Prometheus storage engine as part of the Prometheus 2.0 release, there have been a lot of enhancements and feature additions in TSDB.

For many years, most Ruby applications couldn't use Prometheus, because the Ruby client didn't support pre-fork servers. It turns out this wasn't solved for such a long time because it's a surprisingly hard problem. Many have tried to solve this with different approaches, but we found the best one to be the simplest. Let me show you the dark arts of inter-process communication, and how we ended up fully supporting metrics in Ruby.

Representing distributions in a metrics-based monitoring system is both important and hard. Doing it right unlocks many powerful use cases. What does Prometheus offer in this regard? And what are the pitfalls with the current state of the art? To help understand the present, this talk will also shed light on the past. How have the Histogram and Summary metric types become what they are today with all their weal and woe? Finally, let's have a look at the rich body of research happening in the area, and how it might benefit future versions of Prometheus.