Todd Gamblin

The Spack package manager is widely used by HPC sites, users, and developers to install HPC software, and the Spack project began offering a public binary cache in June of 2022. The cache includes builds for x86_64, Power, and aarch64, as well as for AMD and NVIDIA GPUs and Intel's oneapi compiler. Currently, the system handles nearly 40,000 builds per week to maintain a core set of Spack packages.

Keeping this many different stacks working continuously has been a challenge, and this talk will dive into the build infrastructure we use to make it happen. Spack is hosted on GitHub, but the CI system is orchestrated by GitLab CI in the cloud. Builds are automated and triggered by pull requests, with runners both in the cloud and on bare metal. We will talk about the architecture of the CI system, from the user-facing stack descriptions in YAML to backend services like Kubernetes, Karpenter, S3, CloudFront, and the challenges of tuning runners to give good build performance. We'll also talk about how we've implemented security in a completely PR-driven CI system, and the difficulty of serving all the relevant HPC platforms when most commits are from untrusted contributors. Finally, we'll talk about some of the architectural decisions in Spack itself that had to change to better support CI.

Spack is a software distribution targeted at HPC systems, with over 6,800 packages. While Spack has long been a source-only distribution, in June 2022 we added public build caches that offer fixed and rolling binary releases. With 400-500 pull requests per month, most of them package updates, this was a non-trivial task. The build cache model for Spack is similar to Nix and Guix — it assumes no ABI compatibility. Any change to dependencies triggers rebuilds of dependents. Despite these challenges we have been able to build a CI system that builds and tests packages on pull request and on release branches for a subset of several thousand builds for x86_64, Power, and aarch64, as well as for AMD and NVIDIA GPUs and Intel's oneapi compiler. This talk will cover some of the main challenges we have faced: reliable build infrastructure, integration with pull request workflows, Kubernetes auto-scaling and AWS instance selection, and optimizing build performance in the cloud. We’ll talk about the infrastructure, as well as the algorithmic complexities of choosing CI commits carefully to minimize builds.

This talk is targeted at distribution maintainers, particularly people managing build farms for large distributions. The goal of the presentation is to highlight the challenges of managing a rolling release for continuously evolving distributions hosted on GitHub.

Spack manages 400-500 pull requests/month and is a very active repository. Keeping all the builds working all the time, particularly in a build-from-source system where the recipes are templated, is a big challenge. We may have the same package built many different ways in Spack (e.g. for different CPUs, GPUs, or MPI implementations), and all of those are managed by the same recipe and a very smart dependency/package configuration resolver.

The talk will cover how we’ve scaled build caching to manage PRs as well as release branches, and how we’ve set up infrastructure that allows us to request builders with specific processor microarchitectures. The CI system is orchestrated by a high availability GitLab CI instance in the cloud. Builds are automated and triggered by pull requests, with runners both in the cloud and on bare metal. We will talk about the architecture of the CI system, from the user-facing stack descriptions in YAML to backend services like Kubernetes, Karpenter, S3, CloudFront, and the challenges of tuning runners to give good build performance.

We'll also cover a bit about security in a completely PR-driven CI system. Specifically, we cover how we guarantee that arbitrary public CI users can’t inject malicious code into our binaries.

Choosing commits to build from the distribution tree has been a challenge as well, and in a build-cached system, choosing the right commit to merge a PR with can be a key factor in minimizing the number of builds that need to be done. Specifically, we are trying to maximize reuse of builds done on the rolling release branch while still allowing PRs to update frequently. Orchestrating this in CI, particularly when most systems want to merge with the head of the mainline, can be a challenge.

Finally, we'll talk about some of the architectural decisions in Spack itself that had to change to better support CI — specifically with the dependency resolver and how we choose exactly which stack configurations to build in CI for the rolling release.

All of this topics should help distribution managers better orchestrate build farms, and they should be particularly relevant to other distributions that are similar to Spack, like Nix and Guix.

Every software ecosystem seems to have a package manager these days, but reusing software across these ecosystems is still a challenge. Major Linux distributions package software from a wide range of languages, but they restrict the versions you can install, and they make deep assumptions about compilers and runtime libraries to keep everything compatible. If you need a newer libc or a newer Python than the OS offers, you're often on your own. Python packaging supports native libraries, but it imposes strict rules on package builders to ensure that the binaries work in many places. Some packagers break the rules, e.g., TensorFlow binary packages (wheels) will only work on Ubuntu, frustrating users of other distributions. Containers provide some hope, but only if you stay inside their sandbox. If you your container to talk to a fast network or a GPU, you'll need to ensure binary compatibility between the container and the host OS. Is every ecosystem a walled garden or can we make it easier to bridge the gap?

This talk takes a deeper look at the Application Binary Interface (ABI) and its place at the heart of nearly every software ecosystem. We'll look at the assumptions made by modern dependency management systems about toolchains and ABI. Most tools assume that toolchains and ABI are fixed, making it hard to take a package from one ecosystem and transplant it in another. We'll talk about how we can build dependency managers that can reason better about ABI compatibility, making decisions not just about which software versions to install but also about how those software versions should be built. We'll look at the types of metadata that need to be managed to enable a more flexible software environment, and how it affects the complexity of dependency resolution. Finally, we'll look at how some of these issues are being addressed in Spack, an open source package manager for high performance computing applications, which can span multiple languages, compilers, platforms, and software stacks.

In HPC, software is typically distributed as source code, so that users can build optimized software that takes advantage of specific microarchitectures and other hardware. While this approach provides a lot of flexibility, building software from source remains a huge barrier for users accustomed to simple, fast binary package mangers. Most package managers and container registries label binaries with a high-level architecture family name, e.g., x86_64 or ppc64le, but there is no standard way to label binaries for specific microarchitectures (haswell, skylake, power9, zen2, etc.).

We’ll present a new project called “archspec” that aims to bridge this gap. Archspec provides a standard set of human-understandable labels for many popular microarchitectures. It models compatibility relationships between microarchitectures, and it aggregates information on ISA extensions, compiler support, and compiler flags needed to optimize these machines. Finally, it provides a standard set of names for both microarchitectures and ISA features. These features allow container tools and package managers to detect, build, and use optimized binaries.

Archspec grew out of the Spack package manager, but it is intended for widespread use by other build, packaging, and containerization tools. We will describe how it has been used in practice so far, how it has simplified writing generic packages, and our plans to get contributions from vendors and the broader community.

Expected prior knowledge / intended audience: Audience should have basic knowledge of build systems, as well as some knowledge about processor architectures. There will be some brief background on this in the talk. This will be interesting to HPC users, developers, packagers, and admins, as well as to anyone writing tools that deal with microarchitecture metadata (like container systems).

Speaker bio: Todd Gamblin is a Senior Principal Member of Technical Staff in the Advanced Technology Office in Livermore Computing at Lawrence Livermore National Laboratory. His research focuses on scalable tools for measuring, analyzing, and visualizing parallel performance data. In addition to his research, Todd leads LLNL's DevRAMP (Reproducibility, Analysis, Monitoring, and Performance) team and the Software Packaging Technologies project in the U.S. Exascale Computing Project. He created Spack, a popular open source HPC package management tool with a community of over 450 contributors. Todd has been at LLNL since 2008.

Links to code / slides / material for the talk (optional): To be provided closer to FOSDEM.

Links to previous talks by the speaker: https://www.youtube.com/watch?v=DRuyPDdNr0M https://www.youtube.com/watch?v=edpgwyOD79E&t=2891s https://www.youtube.com/watch?v=BxNOxHu6FAI https://insidehpc.com/2019/03/spack-a-package-manager-for-hpc/ https://www.youtube.com/watch?v=iTLBkpHskzA

See https://tgamblin.github.io/cv/todd-cv.pdf for more (including tutorials and other presentations at major conferences)

Dependency resolution is deceptively complex; simply selecting a set of compatible versions for an arbitrary network of dependencies is NP-hard. Much effort has been spent on this problem for modern single-language ecosystems, but many of these ecosystems rely on natively compiled libraries, and dependency mangers often fail at managing the additional complexities that native libraries entail. Further, dependency resolution has traditionally been modeled as a SAT problem, where the package manager should find any workable solution to satisfy package constraints. However, any solution may not be good enough. Users want the most tested, most optimized, or most secure configuration, and this is a SAT problem coupled with complex optimization.

Spack is a package/dependency manager rapidly gaining popularity in High Performance Computing (HPC) that aims to address many of the complexities of native, multi-language, cross-platform dependency management. Spack has recently been reworked to use Answer Set Programming (ASP), a declarative logic programming paradigm that also provides sophisticated facilities for optimization. This talk will cover how we’ve been able to model the compiler toolchain, ISA, build options, ABI, and other constraints on native libraries. We’ll also talk about how ASP has been a useful tool for finding optimized dependency configurations. This work can be used to improve dependency resolvers in general — so that they can prefer more secure or tested configurations instead of simply selecting the most recent workable versions.

Expected prior knowledge / intended audience: Audience should have basic knowledge of build systems and compiled languages, but we'll explain this up front with some brief background. The talk is aimed broadly -- for users, developers, packagers, researchers, package manager implementors, and HPC administrators.

Spack is a package manager for cluster users, developers, and administrators, rapidly gaining populartiy in the HPC community. Like other HPC package managers, Spack was designed to build packages from source. However, we've recently added binary packaging capabilities, which pose unique challenges for HPC environments. Most binary distributions assume a lowest-common-denominator architecture, e.g. x86_64, and do not take advantage of vector instructions or architecture-specific features. Spack supports relocatable binaries for specific OS releases, target architectures, MPI implementations, and other very fine-grained build options.

This talk will introduce binary packaging in Spack and some of the open infrastructure we have planned for distributing packages. We'll talk about challenges to providing binaries for a combinatorially large package ecosystem, and what we're doing in Spack to address these problems. We'll also talk about challenges for implementing relocatable binaries with a multi-compiler system like Spack. Finally, We'll talk about how Spack integrates with the US exsascale project's open source software release plan, and how this will help glue together the HPC OSS ecosystem as a whole.

Panel discussion with some of the speakers from the Package Management Devroom throughout the day, discussion the future of package management.

Spack is a general-purpose package manager, used on machines ranging from laptops to high performance computing (HPC) clusters. It supports packages in C, C++, Fortran, as well as many other languages, and it allows users to build many different versions of packages with different compilers, build options, dependencies, patches, and target architectures.

Swapping a new compiler into a build may seem trivial, but this and other features in Spack add subtle constraints and complexities to the dependency model. We'll talk about modeling compiler runtime libraries in the DAG, compilers depending on other compilers, and how dependency resolution has to change for cross-compiling. We'll also touch on why all this is necessary for HPC environments, and how packaging has helped to enable new collaborations and software reuse across the scientific computing community.