Every software ecosystem seems to have a package manager these days, but reusing software across these ecosystems is still a challenge. Major Linux distributions package software from a wide range of languages, but they restrict the versions you can install, and they make deep assumptions about compilers and runtime libraries to keep everything compatible. If you need a newer libc or a newer Python than the OS offers, you're often on your own. Python packaging supports native libraries, but it imposes strict rules on package builders to ensure that the binaries work in many places. Some packagers break the rules, e.g., TensorFlow binary packages (wheels) will only work on Ubuntu, frustrating users of other distributions. Containers provide some hope, but only if you stay inside their sandbox. If you your container to talk to a fast network or a GPU, you'll need to ensure binary compatibility between the container and the host OS. Is every ecosystem a walled garden or can we make it easier to bridge the gap?
This talk takes a deeper look at the Application Binary Interface (ABI) and its place at the heart of nearly every software ecosystem. We'll look at the assumptions made by modern dependency management systems about toolchains and ABI. Most tools assume that toolchains and ABI are fixed, making it hard to take a package from one ecosystem and transplant it in another. We'll talk about how we can build dependency managers that can reason better about ABI compatibility, making decisions not just about which software versions to install but also about how those software versions should be built. We'll look at the types of metadata that need to be managed to enable a more flexible software environment, and how it affects the complexity of dependency resolution. Finally, we'll look at how some of these issues are being addressed in Spack, an open source package manager for high performance computing applications, which can span multiple languages, compilers, platforms, and software stacks.
Speakers: Todd Gamblin