Frank Ch. Eigler

Debugging data is a necessary evil. It is necessary for running debuggers in situ, some tracing tools, or for coredump analysis. It is evil because it is big - potentially many times the size of the binaries. Therefore, software distributions have conflicting needs to generate & keep this data but not burden everyone with its storage.

We will review some degrees of freedom for debugging data distribution, across compiled languages and OS distributions, identifying some of the best practices. We will identify the remaining shortcomings that necessitate exploring yet another way of making debuginfo data available needed.

We will present the elfutils debuginfo-server, where a web service offers a lightweight, build-id-indexed lookup of debuginfo-related data on demand. This service is designed to run on a nearby host, private or shared within teams, or even by OS distributions. Clients built into various debugging type tools will be demonstrated.

We present a live-patching technique based on systemtap's programmable in-situ instrumentation. These patches are limited to modifying data instead of code, but are often sufficient to put a bandage on a bleeding security vulnerability - or even a plain bug.

Some security vulnerabilities can't be patched right away. Maybe the vendor hasn't provided an update; maybe the service can't be restarted; maybe the software is private, abandoned, or unchangeable.

Do not despair! Systemtap, a programmable system introspection tool, is customarily used to trace and profile. But its toolkit includes instruments to poke too - to change state. It turns out that this is enough to work around many vulnerabilities. Since systemtap scripts can be run against a live system, we can protect against exploits without a restart.

We will present a set of approaches and applicability criteria, depending on the nature of the vulnerability and its exploits. Some problems can be surgically corrected; others require killing the process before it turns to crime.