Alban Crequy

Until now, you could define seccomp policies in Kubernetes to allow or deny system calls but not much more. The new Seccomp Notify feature in Linux 5.9 will enable more complex policies and the ability to write your own agents to handle new use cases in Kubernetes. Attend to find out why and how!

Seccomp is a feature of the Linux kernel that allows to filter the system calls that a process is allowed to execute. This is commonly used by containers as a way to improve the isolation between the container and the host. Both container runtime runc and Kubernetes allow users to define a Seccomp policy via the OCI Runtime Specification and the PodSpec respectively.

Seccomp recently grew a new feature called the Seccomp Notify in Linux 5.0 and improved in Linux 5.9. This allows a seccomp policy not only to take an immediate decision on whether to allow or deny a system call, but also to defer the control to an external process that I called the Seccomp Agent. The Seccomp agent can decide to block the system call, let it continue, or, up to some extent, execute the system call on behalf of the container. This allows new use cases like running privileged workloads in a safer way and some unprivileged container builds setups.

In this talk, I will present the Seccomp Notify feature and the architecture in runc that makes use of it. I will describe the current status of this feature in Kubernetes. I will demonstrate a couple of use cases in Kubernetes and show how easy it is to build your own seccomp agent in Golang to support new use cases. The audience can expect mentions of pidfd_getfd, the addfd ioctl, and more.

I will present Inspektor Gadget and traceloop, a tracing tool to trace system calls in cgroups or in containers using BPF and overwritable ring buffers.

Many people use the “strace” tool to synchronously trace system calls using ptrace. Traceloop similarly traces system calls but asynchronously in the background, using BPF and tracing per cgroup. I’ll show how it can be integrated with systemd and with Kubernetes via Inspektor Gadget.

Traceloop's traces are recorded in a fast, in-memory, overwritable ring buffer like a flight recorder. As opposed to “strace”, the tracing could be permanently enabled on systemd services or Kubernetes pods and inspected in case of a crash. This is like a always-on “strace in the past”.

Traceloop uses BPF through the gobpf library. Several new features have been added in gobpf for the needs of traceloop: support for overwritable ring buffers and swapping buffers when the userspace utility dumps the buffer.

I will show how to use BPFtrace to trace both function calls in your programs and kernel functions. Then, I will show how such a low-level tracing tool can be used in high-level environments like Kubernetes clusters with kubectl-trace.

kubectl-trace (https://github.com/fntlnz/kubectl-trace) schedules bpftrace programs on your kubernetes cluster using the kubectl.

BPFtrace (https://github.com/iovisor/bpftrace) is a new tracing tool announced in October 2018. You can write one-liner commands in your shell using a high-level tracing language inspired by awk and C, and predecessor tracers such as DTrace and SystemTap. Behind the scenes, it compiles this high-level language with LLVM into eBPF bytecode (extended Berkeley Packet Filter) available in recent Linux kernels. It allows you to trace different layers of the system: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints.

It can be used to trace containers, for example running on Kubernetes, the most popular container orchestration tool. BPFtrace is used as a base tool by kubectl-trace to build more complex tracing tools on Kubernetes.

Container engines like Docker and rkt download container images containing applications to be executed. With Kubernetes – a container orchestration system – updates to a new version of an application can be propagated on the nodes automatically. When the new version has only a few changes compared to the previous versions, this results in downloading the same content again, wasting network bandwidth. With casync – a content-addressable data synchronization tool – we could do better.

I’ll start by introducing the current container image distribution mechanisms used in the Docker Registry and in the App Container Image Discovery protocol. I’ll quickly mention other ideas around saving network bandwidth, like bittorrent in quay.io. Then, I'll present my experiments with casync (Content-Addressable Data Synchronization Tool) and show how much the network usage can be reduced in a Kubernetes cluster.

Testing applications is important, as shows the rise of continuous integration and automated testing. In this talk, I will focus on one area of testing that is difficult to automate: poor network connectivity. Developers usually work within reliable networking conditions so they might not notice issues that arise in other networking conditions. I will explain how traffic control on Linux can help to simulate various network connectivity. I will run a demo with Weave Scope showing how an application running in Kubernetes behaves when changing network parameters.

Linux containers as a virtualization solution are becoming more prevalent. An example of a container solution is rkt, whose development started end of 2014. Under the hood, container software on Linux uses the same cgroups and namespaces API. Both container software and kernel code for containers continue to be developed.

Alban will first describe the Linux API that make containers possible: namespaces and cgroups. Then he will explain how rkt containers use this Linux API.