Eduardo Arango

Singularity is increasingly recognized as the ideal container technology for AI, Machine/Deep Learning, compute-driven analytics, and Data Science. Recently released Version 3.0 of this open source software incorporates a number of significant enhancements that span from the core of the software itself to the enabling ecosystem that surrounds it. Thus the purpose of this presentation is to provide a technical overview of the following enhancements: reimplementation of the Singularity core in a combination Go and C; the introduction of the Singularity Image Format (SIF) as a file-based paradigm for encapsulating cryptographically signable and verifiable container images; expansion of the Singularity ecosystem through cloud-hosted services for signing and verifying cryptographic keys for SIF images, remotely building images as well as a repository for storing and sharing images; plus miscellaneous enhancements regarding instance support and networking management. Platform enhancements, together with an expanded and better-enabled container ecosystem, combine to set Singularity apart as the optimal choice for compute-driven workloads wherever they exist. Because the Go-based core and SIF enhancements are essential to the roadmap for Singularity, allusions are made here with respect to standards compliance as well as integration with Kubernetes for container orchestration.

Singularity is the most widely used container solution in High Performance Computing (HPC). Enterprise users interested in AI, Deep Learning, compute-driven analytics, and IoT, are increasingly demanding HPC-like compute infrastructures. Singularity has many features that make it the preferred container solution for this new type of “Enterprise Performance Computing” (EPC) workload.

2018 has proven to be a remarkable year for Singularity. In this talk we will take a look back at what has been accomplished this year, including what’s new in the Version 3.0 release. For this release, the core Singularity codebase was rewritten from a Bash/Python/C combination into a Go-C architecture. Motivated at the outset by the need to both modernize and unify the Singularity codebase, Go emerged as the obvious choice to replace scripting languages Bash and Python. While advantageous in and of itself, from the perspective of software lifecycle management for example, the choice of Go offered the potential for a significant strategic upside. Briefly, it is increasingly the case that Go is the go-to language for many of the major projects in the broader container ecosystem. The second big change is the format used for container images, as Singularity 3.0 introduced the Singularity Image Format (SIF) - a new file format specifically tailored for container images. SIF allows for storage of the different components that make up a container. For example, SIF-based containers may include OS partition images (read-only), user writable sections, recipes used to create the container, cryptographic signatures for data integrity and authenticity, and whatever else the community can think of - as SIF is an extensible format. A SIF file resembles a generic Linux file system in structure and content. A global header identifies the file as being of type SIF, and the necessary metadata to detail the remaining content of the image file. Following these descriptors is the corresponding data - namely the OS partition image, recipe, environment variables and signature blocks. As Singularity is drawn into full compliance with standards emerging from the Open Container Initiative (OCI), data encryption is a requirement slated for immediate action. Also related to OCI compliance, and involving SIF specifically, will be the need to harmonize the existing signing and verification capability inherent in SIF with those requirements currently emerging from within the OCI. Because extensibility was designed into SIF from the outset, OCI compliance can be addressed through technically efficient and effective means. Finally, and related to compliance with respect to the OCI Image Specification, will be efforts to ensure that SIF images can be appropriately ‘ingested’ – e.g., for subsequent use by an OCI-compliant runtime. In other words, SIF will become a packaging format for OCI-based containers. Still related to SIF, compliance will include the ability to mount an arbitrary number of SIF images as an OCI bundle. Needed for compliance with the OCI Runtime Specification, at a high level, this translates to mounting an arbitrary number of files – as SIF encapsulates each Singularity container into a single file. Keywords Go, HPC, Linux Containers, Singularity, Docker, OCI

One of the biggest problems in scientific HPC is ensuring that results are reproducible. That is, the code a scientist runs locally must be able to run identically on any computational resource. Until recently, the job of ensuring that fell to system administrators who needed to manage a complex web of tools and dependencies on those resources. However, with the introduction of HPC containers via Singularity, the ability to mobilize the compute environment has never been easier. Singularity allows anybody to run their own containers on HPC, ushering in a new era of computational mobility, validity, and reproducibility.

Singularity is the most widely used container solution in high-performance computing (HPC). Enterprise users interested in AI, Deep Learning, compute drive analytics, and IOT are increasingly demanding HPC-like resources. Singularity has many features that make it the preferred container solution for this new type of “Enterprise Performance Computing” (EPC) workload. Instead of a layered filesystem, a Singularity container is stored in a single file. This simplifies the container management lifecycle and facilitates features such as image signing and encryption to produce trusted containers. At runtime, Singularity blurs the lines between the container and the host system allowing users to read and write persistent data and leverage hardware like GPUs and Infiniband with ease. The Singularity security model is also unique among container solutions. Users build containers on resources they control or using a service like the Sylabs Remote Build Service. Then they move their containers to a production environment where they may or may not have administrative access and the Linux kernel enforces privileges as it does with any other application. These features make Singularity a simple, secure container solution perfect for HPC and EPC workloads.

One of the biggest problems in scientific HPC is ensuring that results are reproducible. That is, the code a scientist runs locally must be able to run identically on any computational resource. Until recently, the job of ensuring that fell to system administrators who needed to manage a complex web of tools and dependencies on those resources. However, with the introduction of HPC containers via Singularity, the ability to mobilize the compute environment has never been easier. Singularity allows anybody to run their own containers on HPC, ushering in a new era of computational mobility, validity, and reproducibility.

Singularity is the most widely used container solution in high-performance computing (HPC). Enterprise users interested in AI, Deep Learning, compute drive analytics, and IOT are increasingly demanding HPC-like resources. Singularity has many features that make it the preferred container solution for this new type of “Enterprise Performance Computing” (EPC) workload. Instead of a layered filesystem, a Singularity container is stored in a single file. This simplifies the container management lifecycle and facilitates features such as image signing and encryption to produce trusted containers. At runtime, Singularity blurs the lines between the container and the host system allowing users to read and write persistent data and leverage hardware like GPUs and Infiniband with ease. The Singularity security model is also unique among container solutions. Users build containers on resources they control or using a service like Singularity Hub. Then they move their containers to a production environment where they may or may not have administrative access and the Linux kernel enforces privileges as it does with any other application. These features make Singularity a simple, secure container solution perfect for HPC and EPC workloads.

Singularity blocks privilege escalation within the container so if a user wants to be root inside the container, it must be root outside the container. This usage paradigm mitigates many of the security concerns that exists with containers on multi-tenant shared resources. You can directly call programs inside the container from outside the container fully incorporating pipes, standard IO, file system access, X11, and MPI. Keywords HPC, Linux Containers, Singularity, Docker, LXC, portability

References Kurtzer, G. M. (2016). Singularity 2.1.2 - Linux application and environment containers for science. Zenodo. http://doi.org/10.5281/zenodo.60736 LinuxContainers.org Infrastructure for container projects. Project sponsored by Canonical Ltd. https://linuxcontainers.org/