Christian Kniep

While containerization revolutionized the delivery and execution of software, it introduces new challenges as the usual practice with one big software file-system with a subsequent module load to rule all environments is not feasable with containers. The talk introduces the 'HPC Container Conformance' Project which aims to provide guidance on how to build container images and how to annotate them so that end-users and system admins can integrate them in their workflows. The talk also introduces the MetaHub Registry, an OCI compliant container registry to serve environment/hardware specific images and reduce overall complexity of herding all the container images. By using profiles when logging in MetaHub is aware of the context the image is going to be executed and picks the right image from previously pushed/configured container images. Centralising the logic of picking the right software variant (module load) within the registry enables a series of enhancements to the user experience and a reduction in complexity which allows users and admins to focus on their science.

This short talk will disect the container ecosystem for HPC in four segments and discusses what to look out for, what is already settled and how to navigate containers in 2022.

The Container ecosystem spans from spawning a process into an isolated and constrained region of the kernel at bottom layer, building and distributing images just above to discussions on how to schedule a fleet of containers around the world at the very top. While the top layers get all the attention and buzz, this session will base-line the audiences' understanding of how to execute containers.

The talk will briefly recap the history of containers and containers in HPC and describe the challenges we faced and how the community overcame them; eventually converging towards a sustainable model to run HPC application at scale using containers. By attending this sessions viewers will come to understand that HPC runtime engines are not far apart anymore and what to look out for when adopting HPC containers in 2021. We'll also poke at the next layer up - Image build and distribution - to describe the challenge after we picked the runtime engine.

With Docker v1.9 a new networking system was introduced, which allows multi-host networking to work out-of-the-box in any Docker environment. This talk provides an introduction on what Docker networking provides, followed by a demo that spins up a full SLURM cluster across multiple machines. The demo is based on QNIBTerminal, a Consul backed set of Docker Images to spin up a broad set of software stacks.