John Garbutt

Azimuth helps users self-service create Science Platforms, such as JuyterHub and Slurm. Sometimes this requires self-service creation of RDMA enabled Kubernetes clusters. OpenStack can use SR-IOV using VF-LAG to provide RoCE RDMA within VMs. We make use of K8s Cluster API to provision K8s using OpenStack servers. We then use multus and macvlan CNIs to give k8s pods RDMA networking. Testing the performance is automated using a Volcano based K8s operator. We are working on also bringing this power to OpenStack Magnum.

Why build #4 on the Green500 using OpenStack? It makes it easier to manage. Cambridge University started using OpenStack in 2015. Since mid 2020, all new hardware is controlled using OpenStack. Compute nodes, GPU nodes, Lustre nodes, Ceph nodes, almost everything. OpenStack allows large baremetal slurm clusters and dedicated TRE (trusted research environments) to share the same images. Is this a cloud native supercomputer?

We will explore how OpenStack is used to manage a supercomputer as a shared pool of hardware resources, that can be partitioned between a multitude of different platforms required by a diverse group of scientists. Ranging from Trusted Research Environments (TREs), on demand dedicated AI platforms, dedicated big data platforms, and to traditional shared Slurm clusters.

We will focus on providing a range of services from a single shared hardware pool, allowing for the delivery of both on demand interactive compute platforms for STFC's IRIS e-Infrastrcture and Slurm clusters such as the #4 in the Green500, called Wilkes-3: https://www.top500.org/system/179930/

This makes use of both OpenStack Ironic, for the baremetal deployment, and on-demand OpenStack KVM powered VMs running Cluster API provisioned Kubernetes, with KubeApps to deploy JuypterHub.

The next generation of research infrastructure and large scale scientific instruments will face new magnitudes of data. StackHPC have been working with Cambridge University on the Square Kilometre Array (SKA) radio telescope project (in collaboration with both CERN and Cambridge University) to build a performance prototype using OpenSource tools such as OpenStack and Kubernetes.

One of the biggest challenges is building storage fast enough to deal with the expected 4Tb/s Read and 0.5 Tb/s Write, yet flexible enough to deal with the expected variety of workloads being run on the system. This has involved much collaboration with CERN given the shared challenges posed by the High-Luminosity LHC upgrade.

The second aspect is making a system that keep up with the ever changing nature of the workloads running on the system. Key ideas from this project are being deployed in production on Cambridge University's Cumulus supercomputer (#87 in the Nov18 top500 https://www.top500.org/system/179577). We explore how this move away from a single Slurm cluster is helping meet a broader range of research computing needs.

You have your code working, it solves your problem, but now its time to get it upstream. What to do next? What should I have done?

Maybe we can use lessons from Harper Lee's To Kill a Mockingbird to improve how new and established OpenStack contributors work together?

John will share some of the lessons learnt from his OpenStack Journey, including his time as a private cloud packager, vendor driver maintainer, public cloud operator, nova-core, nova-specs-core, Blueprint Czar, Release CPL and now Nova Project Team Lead (PTL).