The next generation of research infrastructure and large scale scientific instruments will face new magnitudes of data. StackHPC have been working with Cambridge University on the Square Kilometre Array (SKA) radio telescope project (in collaboration with both CERN and Cambridge University) to build a performance prototype using OpenSource tools such as OpenStack and Kubernetes.
One of the biggest challenges is building storage fast enough to deal with the expected 4Tb/s Read and 0.5 Tb/s Write, yet flexible enough to deal with the expected variety of workloads being run on the system. This has involved much collaboration with CERN given the shared challenges posed by the High-Luminosity LHC upgrade.
The second aspect is making a system that keep up with the ever changing nature of the workloads running on the system. Key ideas from this project are being deployed in production on Cambridge University's Cumulus supercomputer (#87 in the Nov18 top500 https://www.top500.org/system/179577). We explore how this move away from a single Slurm cluster is helping meet a broader range of research computing needs.
Speakers: John Garbutt