Anastassios Nanos

Unikernels promise fast boot times, small memory footprint and stronger security but lack in terms of manageability. Moreover, unikernels provide a non-generic environment for applications, with limited or no support for widely used libraries and OS features. This issue is even more apparent in the case of hardware acceleration. Acceleration libraries are often dynamically linked and have numerous dependencies, which directly contradict the statically linked notion of unikernels. Hardware acceleration functionality is almost non-existent in unikernel frameworks, mainly due to the absence of suitable virtualization solutions for such devices. ​ In this talk, we present an update on the vAccel framework we have built that can expose hardware acceleration semantics to workloads running on isolated sandboxes. We go through the components that comprise the framework and elaborate on the challenges in building such a software stack: we first present an overview of vAccel and how it works; then we focus on the porting effort of vAccel in various unikernel frameworks. Finally, we present a hardware acceleration abstraction that expose semantic acceleration functionality to workloads running as unikernels. ​ We will present a short demo of some popular algorithms running on top of Unikraft and vAccel show-casing the merits and trade-offs of this approach.

Serverless computing facilitates the use of resources without the burden of administering and maintaining infrastructure. The simplification of IaaS appears ideal (in theory) but providers and users are presented with several challenges: providers aim to reduce infrastructure maintenance overheads; users require isolation, flexibility and programming freedom.

Serverless deployments are mostly backed by sandboxed containers. To enable programming freedom for users, providers allow the use of containers for function deployment, however, to ensure strict isolation, these containers are sandboxed in VMs. As a result, this bloated stack brings complicated maintenance costs: (a) several layers of abstraction between the user function to be executed and the actual execution environment; (b) increased attack surface; (c) increased request-to-exec time; (d) reduced set of feature availability for functions (hardware acceleration).

Unikernels promise fast boot times, small memory footprint and stronger security but lack in terms of manageability. Additionally, Serverless frameworks only support containers. Moreover, unikernels provide a different environment for applications, with limited or no support for widely used libraries and OS features. This issue is even more apparent in the case of ML/AI workloads. ML/AI libraries are often dynamically linked and have numerous dependencies, which directly contradict the statically linked notion of unikernels. Finally, hardware acceleration is almost non-existent in unikernel frameworks, mainly due to the absence of suitable virtualization solutions for such devices.

In this talk, we present the design of a flexible serverless framework designed for the cloud and the edge, backed by unikernels that can access hardware accelerators. We go through the components that comprise the framework and elaborate on the challenges in building such a software stack: we first present an overview of the necessary components of a serverless framework; then we focus on the function execution framework based on two popular unikernel frameworks; finally, we present a hardware acceleration abstraction to expose semantic acceleration functionality to workloads running on top of this framework.

A short demo of the working components will be presented, discussing the challenges and trade-offs of this approach.

The debate on how to deploy applications, monoliths or micro services, is in full swing. Part of this discussion relates to how the new paradigm incorporates support for accessing accelerators, e.g. GPUs, FPGAs. That kind of support has been made available to traditional programming models the last couple of decades and its tooling has evolved to be stable and standardized (eg. CUDA, OpenCL/OpenACC, Tensorflow etc.).

On the other hand, what does it mean for a highly distributed application instance (i.e. a Serverless deployment) to access an accelerator? Should the function invoked to classify an image, for instance, link against the whole acceleration runtime and program the hardware device itself? It seems quite counter-intuitive to create such bloated functions.

Things get more complicated when we consider the low-level layers of the service architecture. To ensure user and data isolation, infrastructure providers employ virtualization techniques. However, generic hardware accelerators are not designed to be shared by multiple untrusted tenants. Current solutions (device passthrough, API-remoting) impose inflexible setups, present security trade-offs and add significant performance overheads.

To this end, we introduce vAccel, a lightweight framework to expose hardware acceleration functionality to VM tenants. Our framework is based on a thin runtime system, vAccelRT, which is, essentially, an acceleration API: it offers support for a set of operators that use generic hardware acceleration frameworks to increase performance, such as machine learning and linear algebra operators. vAccelRT abstracts away any hardware/vendor-specific code by employing a modular design where backends implement bindings for popular acceleration frameworks and the frontend exposes a function prototype for each available acceleration function. On top of that, using an optimized paravirtual interface, vAccelRT is exposed to a VM’s user-space, where applications can benefit from hardware acceleration via a simple function call.

In this talk we present the design and implementation of vAccel on two KVM VMMs: QEMU and AWS Firecracker. We go through a brief design description and focus on the key aspects of enabling hardware acceleration for machine learning inference for ligthweight VMs both on x86_64 and aarch64 architectures. Our current implementation supports jetson-inference & TensorRT, as well as Google Coral TPU, while facilitating integration with NVIDIA GPUs (CUDA) and Intel Iris GPUs (OpenCL).

Finally, we present a demo of vAccel in action, using a containerized environment to simplify configuration & deployment

• [1] https://blog.cloudkernels.net/posts/vaccel
• [2] https://blog.cloudkernels.net/posts/vaccel_v2
• [3] https://vaccel.org
• [4] https://github.com/nubificus/docker-jetson-inference

Running applications in the Cloud has changed the way users develop and ship their code. Quite recently, the community has given rise to microservices-based approaches, towards solutions that follow the paradigm of Platform-, Software-, and Function-as-a-Service (PaaS, SaaS, and FaaS respectively).

To accommodate user demands, while maintaining security and isolation, Cloud vendors have adopted a hybrid approach where user workloads are being executed in lightweight sandboxed environments, where micro-hypervisors provide the isolation and container-based images facilitate application deployment. As a result, lighter virtualization stacks remains a key aspect to maximize performance in a multi-tenant but isolated environment.

To this end, we started experimenting with various Virtual Machine Monitors (VMMs) that could provide the ideal trade-off between performance, flexibility and application portability. In this talk, we present the design of a minimal VMM, based on KVM, residing entirely in the Linux Kernel and showcase the merits and shortcomings (minimal footprint, security concerns), for each use-case (Cloud FaaS, edge multi-tenancy). Additionally, we present our experience from porting Firecracker to a low-power device (RPi4) demonstrating the merits of lightweight hypervisor stacks for flexible application execution at the edge.