Corne Lukken

Computational storage devices (or CSDs) are the new emerging technology that allows offloading computation to storage devices. In this technology, computation is pushed to the storage device (close to the data), and only the final result is returned to main system memory. The efficiency and performance gains come from the reduction in data movement over the I/O interconnects, thus relieving pressure on the memory bandwidth in the traditional Von Nuemann architecture where all data is first moved to the main memory before processing. Despite lots of enthusiasm, proposals, and research publications, there are no immediately available open-source ready to use CSDs available. Due to the lack of such prototypes, it is very hard and challenging to explore hardware, physical interfaces, application APIs (block-level, file system, key-value stores) for CSD devices.

In this talk, I will present OpenCSD, a completely open-source CSD exploration platform designed with QEMU. OpenCSD uses eBPF as the means to offload computation to the CSD, and includes an accompanying file system. FluffleFS, the file system, uses POSIX extended attributes to interact with the CSD device. The full, open-source implementation including a block-device, programming toolchain and a file system interface; allow anyone to explore the paradigm of computational storage.

Throughout Europe a large collection of 54 stations each with hundreds of antennae collect a large amount of data for radio astronomy. These stations separated by 2000 kilometers of distance combine at a core of 24 stations in Exloo, the Netherlands and two HPC clusters in Groningen, the Netherlands. Together this entire system is known as the LOw Frequency ARray (LOFAR).

This talk focuses on the 400gbps datastream and aggregation across the 2000 kilometers distance between stations and how both a GPU and CPU cluster translate these data into images. We will cover the open-source tools that enable this processing, discover some of the history about our clusters, gain insights into some of the pipelines and steps to do data transformations and finally show you some of LOFARs current achievements and future directions.