Akihiro Suda

It wasn't easy to reproduce the same container image from its Dockerfile, due to changes in timestamps and "aptgettable" package versions. This lack of reproducibility has been a threat to the trustworthiness of container images and binary artifacts built inside containers.

In this talk, Akihiro Suda will introduce the current work being done to enable reproducible builds in the Dockerfile ecosystem.

This talk will consist of two parts:

The first part will explain the current status of implementing the SOURCE_DATE_EPOCH specification [1] in BuildKit [2] for deterministic timestamps of rootfs files and OCI metadata.

The second part will introduce a new tool called repro-get [3] which can be used to deterministically install a specific snapshot of apt, dnf, apk, and pacman packages. The packages are fetched by their SHA256 hash from various file providers including HTTP(S) sites, local filesystems, OCI registries, and even IPFS. The repro-get tool is expected to be used for containers in conjunction with the SOURCE_DATE_EPOCH work, but it can also be useful in non-container environments.

• [1] https://reproducible-builds.org/specs/source-date-epoch/
• [2] https://github.com/moby/buildkit
• [3] https://github.com/reproducible-containers/repro-get

Setting up multi-cluster and multi-cloud container networking for dev environments has been too harder than it needs to be. NoRouter is a novel instant networking stack to overcome this difficulty, by transferring IP packets over stdio streams (aka "shell connections"). NoRouter works with any container, any VM, and any baremetal machine, on anywhere, as long as the shell connection is available from your laptop, e.g. kubectl exec, docker exec, or ssh.

NoRouter heavily focuses on dev UX and does not need any setup other than deploying a single binary and writing a short YAML manifest. And yet it works even without the root privilege.

More info: https://norouter.io/docs/

The biggest problem of the OCI Image Spec is that a container cannot be started until all the tarball layers are downloaded, even though more than 90% of the tarball contents are often unneeded for the actual workload.

This session will show state-of-the-art alternative image formats, which allow runtime implementations to start a container without waiting for all its image contents to be locally available.

Especially, this session will put focus on CRFS/stargz and its implementation status in containerd (https://github.com/containerd/containerd/issues/3731). The plan for BuildKit integration will be shown as well.

Kubernetes supports several security mechanisms such as Seccomp, Apparmor, SELinux, and runAsUser for protecting the hosts from container-breakout attacks. However, these mechanisms are not sufficient for the security demand because Kubelet and CRI/OCI runtimes require the root privileges on the hosts, and these components are seriously bug-prone. The dependency on the root privileges has been also problematic for promoting Kubernetes to the HPC world, where users are often disallowed to install software as the root.

In this talk, Akihiro and Giuseppe will show the community’s ongoing work for making Kubernetes deployable and runnable as a non-root user, by using User Namespaces. The main topics of discussion will be UID/GID mapping, unprivileged Copy-on-Write filesystems, Usermode networking (Slirp), and Cgroups.

Developing and maintaining distributed systems like Hadoop is difficult. The difficulty comes from many factors, but we believe that one of the most important reasons is lacking of a good debugger for bugs specific to distributed systems. (e.g., non-deterministic hardware faults, message ordering, ..)

In the talk, we will show Earthquake, our open-source debugging framework for distributed systems. Earthquakes permutes Ethernet packets, Filesystem events, Java/C function calls, and injected faults in various orders so as to control non-determinism in the cluster. Basically, Earthquake permutes events in a random order, but the user can write his/her own state exploration policy (in Go language) for finding deep bugs efficiently. Earthquake also controls non-determinism of the thread interleaving by calling sched_setattr(2) with randomized parameters.

We will also share our successful stories about testing some Hadoop components with Earthquake. For ZooKeeper, we found a distributed race condition bug which decreases availability of a ZooKeeper cluster. We also reproduced a known ZooKeeper bug that no one had successfully reproduced for 2 years, and analyzed its cause. For YARN, we found a disk-fault tolerance bug that inappropriately marks faulty node as healthy. We also found bugs of non-Hadoop softwares, such as etcd.

With Earthquake, you can also test your real distibuted systems without any modification.