Singularity is increasingly recognized as the ideal container technology for AI, Machine/Deep Learning, compute-driven analytics, and Data Science. Recently released Version 3.0 of this open source software incorporates a number of significant enhancements that span from the core of the software itself to the enabling ecosystem that surrounds it. Thus the purpose of this presentation is to provide a technical overview of the following enhancements: reimplementation of the Singularity core in a combination Go and C; the introduction of the Singularity Image Format (SIF) as a file-based paradigm for encapsulating cryptographically signable and verifiable container images; expansion of the Singularity ecosystem through cloud-hosted services for signing and verifying cryptographic keys for SIF images, remotely building images as well as a repository for storing and sharing images; plus miscellaneous enhancements regarding instance support and networking management. Platform enhancements, together with an expanded and better-enabled container ecosystem, combine to set Singularity apart as the optimal choice for compute-driven workloads wherever they exist. Because the Go-based core and SIF enhancements are essential to the roadmap for Singularity, allusions are made here with respect to standards compliance as well as integration with Kubernetes for container orchestration.
Singularity is the most widely used container solution in High Performance Computing (HPC). Enterprise users interested in AI, Deep Learning, compute-driven analytics, and IoT, are increasingly demanding HPC-like compute infrastructures. Singularity has many features that make it the preferred container solution for this new type of “Enterprise Performance Computing” (EPC) workload.
2018 has proven to be a remarkable year for Singularity. In this talk we will take a look back at what has been accomplished this year, including what’s new in the Version 3.0 release. For this release, the core Singularity codebase was rewritten from a Bash/Python/C combination into a Go-C architecture. Motivated at the outset by the need to both modernize and unify the Singularity codebase, Go emerged as the obvious choice to replace scripting languages Bash and Python. While advantageous in and of itself, from the perspective of software lifecycle management for example, the choice of Go offered the potential for a significant strategic upside. Briefly, it is increasingly the case that Go is the go-to language for many of the major projects in the broader container ecosystem. The second big change is the format used for container images, as Singularity 3.0 introduced the Singularity Image Format (SIF) - a new file format specifically tailored for container images. SIF allows for storage of the different components that make up a container. For example, SIF-based containers may include OS partition images (read-only), user writable sections, recipes used to create the container, cryptographic signatures for data integrity and authenticity, and whatever else the community can think of - as SIF is an extensible format. A SIF file resembles a generic Linux file system in structure and content. A global header identifies the file as being of type SIF, and the necessary metadata to detail the remaining content of the image file. Following these descriptors is the corresponding data - namely the OS partition image, recipe, environment variables and signature blocks. As Singularity is drawn into full compliance with standards emerging from the Open Container Initiative (OCI), data encryption is a requirement slated for immediate action. Also related to OCI compliance, and involving SIF specifically, will be the need to harmonize the existing signing and verification capability inherent in SIF with those requirements currently emerging from within the OCI. Because extensibility was designed into SIF from the outset, OCI compliance can be addressed through technically efficient and effective means. Finally, and related to compliance with respect to the OCI Image Specification, will be efforts to ensure that SIF images can be appropriately ‘ingested’ – e.g., for subsequent use by an OCI-compliant runtime. In other words, SIF will become a packaging format for OCI-based containers. Still related to SIF, compliance will include the ability to mount an arbitrary number of SIF images as an OCI bundle. Needed for compliance with the OCI Runtime Specification, at a high level, this translates to mounting an arbitrary number of files – as SIF encapsulates each Singularity container into a single file. Keywords Go, HPC, Linux Containers, Singularity, Docker, OCI
Speakers: Eduardo Arango