What happens when you have access to large clusters, but have little control over the software installed on the machines? Unfortunately, this is the current scenario that researchers often find in HPC clusters, which include very old software stack, a brittle environment and non-cooperative sysadmins. We have been experimenting with Nix to provide an up-to-date system running on top of the already existing software, without root permissions with the help of user namespaces.
In this talk we give a tour on the problems we found and how we solved them:
- Starting from the installation and configuration of nix to be used by multiple users when we lack a shared
/nix store.
- Avoiding library contamination from
/usr/lib with an isolated root mount
- Interactive development while compiling the code inside the isolated environment with a patched nix-portable
- Adding custom compilers to the stdenv
- Building packages tuned to an specific CPU with vectorization optimizations in mind.
- Running the benchmarks with SLURM inside the isolated environment with multiple compute nodes.
- Improving MPI fast zero-copy transfer inside user namespaces.