Ricardo Wurmus

There are dozens of domain specific languages that allow scientists to describe complex workflows. From the humble generic GNU Make to large scale platforms like Apache Airflow you would think that there is something there to satisfy everyone. All of these systems have one thing in common: they have a strong focus on partitioning large computations and scheduling work units, but when it comes to managing the software environments that are the context of each of the planned computations, they are often remarkably shy to offer opinionated solutions.

Software management and deployment often seems like an afterthought. Workflow language designers increasingly seem to be following the devops trend of resorting to opaque application bundles to satisfy application and library needs. While this strategy has some advantages it also comes with downsides that rarely seem to be weighed carefully.

We present the Guix Workflow Language --- not as a solution to the question of software deployment in HPC workflows, but as an instance of convergent evolution: growing a workflow language out of a generic reproducible software management and deployment system (GNU Guix) instead of sprucing up a workflow language with software deployment features. We hope to encourage a discussion about the current state of workflow languages in HPC: when it comes to software and distributed computations, are we approaching the peak or do we circle a local maximum?

A discussion on the future of GNU Guix and GuixSD.

This talk introduces the functional package manager GNU Guix and the underlying concept of functional package management to a general audience.

Support teams of high-performance computing (HPC) systems often find themselves between a rock and a hard place: on one hand, they understandably administrate these large systems in a conservative way, but on the other hand, they try to satisfy their users by deploying up-to-date tool chains as well as libraries and scientific software. HPC system users often have no guarantee that they will be able to reproduce results at a later point in time, even on the same system—software may have been upgraded, removed, or recompiled under their feet, and they have little hope of being able to reproduce the same software environment elsewhere. We present GNU Guix and the functional package management paradigm and show how it can improve reproducibility and sharing among researchers with representative use cases.

Functional package management is a discipline that transcribes the functional programming paradigm to software deployment: build and installation processes are viewed as pure functions in the mathematical sense—whose result depends exclusively on the inputs—, and their result is a value—that is, an immutable directory. Since build and installation processes are pure functions, their results can effectively be “cached” on disk. Likewise, two independent runs of a given build process for a given set of inputs should return the same value—i.e., bit-identical files. This approach was first described and implemented in the Nix package manager. Guix reuses low-level mechanisms from Nix to implement the same paradigm, but offers a unified interface for package definitions and their implementations, all embedded in a single programming language.

This talk introduces the functional package manager GNU Guix and the underlying concept of functional package management to a general audience.

This talk introduces the functional package manager GNU Guix to a general audience.