César Gómez-Martín

Traditional container technology (i.e. Docker) was not meant for the deployment of HPC applications among large computing clusters. New container technology like Singularity focuses on the portability of compute, allowing us to ease deployment, reducing administrative overhead and user support. However, some issues still need special attention, for example how to handle different MPI configurations (e.g. over Infiniband), and how to handle special resources like GPUs.

Today, there is no need to have applications installed on any cluster or HPC/scientific resource; we can use container technology to encapsulate all the software and libraries of any specific application, ensuring the operation thereof in any possible operating system, software, and hardware architecture, avoiding  incompatibilities and retaining native or near-native performance. Our design strategy makes the container agnostic to the queue system i.e. the container works with SLURM, TORQUE and stand alone.

This talk will cover creating such portable containers using multiple resources, and showing how simple is to deploy a complex MPI dependant application across multiple TOP500 supercomputer with only a regular user account, including the ability to exploit existing resources like Infiniband and GPGPUs.

Also we’ll show how to add open source profiling tools to the container. We are going to show execution profiling samples of critical HPC/ML applications like VASP, Amber, Seismic Imaging and Chainer running on portable containers across different HPC systems.

Other people: Michael Bauer