Walter Heck

Designing for Failure

While we all work very hard to build high-available, fault-tolerant and resillient applications and infrastructures the end-goal is currently often something along the lines of loosly-coupled/microservices with zero-downtime in mind. Upgrades are tied to CI/CD pipelines and we should be sipping pina coladas on the beach. Time to unleash the Chaos Monkey, because that is what Netflix does, so we should try it as well.

Now, the backend DB failed. The middleware application is returning errors, and your frontend is showing a fancy 5xx.

While each layer is able to scale independently or fail-over to another region, even a simple timeout @ the DB can cause a cascading failure.

The application is designed to work, not designed to recover from failure.

Designing for failure applies to both software development and infrastructure architecture, and I'd like to talk about a couple of points to highlight this paradigm.

Please note that this talk replaces one entitled "Introduction to Metal³" that was due to have been given by Stephen Benjamin, who has sent his apologies but is now unable to attend.

Terraform seems to be the configuration management of the cloud: a tool that allows us to define our infrastructure as code so it becomes automatable and testable. It suffers from a bunch of issues though. These issues have been encountered and solved before in other tools like puppet, but the story very much resembles that one. This session will draw some parallels between early puppet and early terraform days and explain what we can do to make these changes in a better way.

If you need to manage more then a reasonably sized infrastructure in today's day and age you will want to automate things as much as possible. In order to do that properly, you'll need to use tools that help you manage your infrastructure. This poses a problem: before you can manage your infrastructure, you now need to manage the infrastructure you will use to manage your infrastructure. The interesting bit is that if you use industry best practices, a lot of that management infra will look the same or similar across organisations. So why do we need to reinvent the weel over and over?

This talk will present OpsTheater, a stack of tools used to manage an infrastructure made up of current industry best practices (Puppet, GitLab, Icinga, ELK stack). OpsTheater is a fully open source project that was started in late 2015 and is growing to be a great set of tools that takes away the work we do over and over: managing the infrastructure to manage our infrastructure. We will look at the problems OpsTheater solves, the state of the project and what you can do with it today. If you want to help, we'll show where you an go and what you can do. If time permits, a demo will be added showing some of the things we can do with OpsTheater.

This talk will discuss the implementation of puppet across 15000 servers in a very traditional enterprise organisation. This involved 20 teams of people new to puppet. On top of that 70% of servers were Windows, making this quite an interesting challenge. The talk will outline the experiences in onboarding puppet in this enterprise environment. Things that worked, things that didn't work and everything in between.

Walter Heck is one of the co-organisers of config management camp, co-chair of the config management devroom at FOSDEM and an avid promoter of Open Source software.