Georg Rath

This talk will give an overview over how we use Slurm to schedule the workloads of over 6000 scientists at NERSC, while providing high throughput, ease of use and ultimately user satisfaction. With the emergence of data-intensive applications it was necessary to update the classic scheduling infrastructure to handle things like user defined software stacks (read: containers), data movement and storage provisioning. We did all of this and more through facilities provided by Slurm. In addition to these features we will discuss priority management and quality of service and how that can greatly improve the user experience of computational infrastructures.

This talk will be a walkthrough of the features that make Slurm great, using a supercomputing site as an example. All of the introduced interfaces are not specific to the site in question and can be used by the broader community. After an a brief introduction of the workings of a workload manager/scheduler in general and Slurm in particular, we'll go into some of the features and how those open up possibilities for frictionless working with all kinds of use-cases, way beyond classical HPC workloads: - Container integration - Data staging - On-demand filesystem provisioning - On-the-fly job rewriting - Cluster federation - and a healthy plugin ecosystem

No previous knowledge of high performance computing or batch processing required.

Containers are gaining significant traction as a means of deploying and distributing software in the scientific computing space. With a plethora of solutions to choose from, this talk will go into why there is a tendency for custom container runtimes in this area of computing. We will discuss portability, performance, security and ease of use, as well as direct hardware access as driving factors. As new processor architectures and specialized hardware for machine learning emerge, some of those factors will affect larger parts of the container community.

This talk will give an overview of (some of) the currently available container solutions and how they enable scientists to move their software stacks between machines and work in a reproducible way. The technical challenges and implementations of interfacing with the host system for access to hardware and highly optimized system libraries will be presented. Experiences of running containers on thousands of nodes at NERSC will be shared, which includes the good and the bad parts. The drawbacks of running portable but unoptimized code inside of containers and how that can have a profound impact on the performance of production workloads and what a solution for that could look like, will be addressed. Wrapping up I'd like to show how some of the work done in the scientific space can benefit the broader community and facilitate discussion about that.

With the advent of modern package managers for scientific applications (EasyBuild, Spack, etc.) automated building of large amounts of software is becoming easier, quickly giving rise to issues related to life cycle management of applications. This makes tracking the applications and libraries that actually get used considerably more important. Existing solutions (module load hooks, launch wrappers) do not account for user-built software, are hard to deploy or produce inconclusive results.

OGRT enables the tracking of jobs on a cluster with process-level granularity and without discernible performance penalty. It tracks used shared libraries, environment variables and loaded modules at the moment of process execution. It also supports watermarking executables and shared objects and reading those watermarks out of memory at runtime. Gathered information is collected and shipped to various backends.

OGRT aims to be a versatile tool, which can be used to:

• provide a census of used software (including user-built)
• troubleshoot problems with programs picking up unexpected shared libraries
• retroactively inform users about buggy libraries
• overlay process-level data onto existing job monitoring tools
• contribute to reproducibility of application runs

This presentation will give an overview of the design and implementation of OGRT, as well as demoing some of its capabilities when plugged into an Elasticsearch backend.