Drivers are usually written in C for historical reasons, this can be bad if you want your driver to be safe and secure. We show that it is possible to write low-level drivers for PCIe devices in modern high-level languages. We are working on super-fast user space network drivers for the Intel 82599ES (ixgbe) 10 Gbit/s NIC in Rust, C#, go, OCaml, Haskell, Python, Swift, and a few more languages (WIP). All of our drivers are written from scratch and require no additional kernel code.

Check out our GitHub page with links to all implementations, performance measurements, and publications for further reading.

Supposedly modern user space drivers (e.g., DPDK or SPDK) are still being written in C in 2018 :(

This comes with all the well-known drawbacks of writing things in C that might be prevented by using safer programming languages. Also, did you ever see a kernel panic because a in-kernel driver did something stupid? It doesn't have to be that way, drivers should not be able to take down the whole system.

There are three steps to building better drivers:

• Write them in a safer programming language eliminating whole classes of bugs and security problems like bad memory accesses
• Isolating them from the rest of the operating system: user space drivers that drop privileges
• Isolating the hardware using the IOMMU

We are showing that it is possible to achieve all of these goals for PCIe drivers on Linux by implementing user space network drivers in all of the aforementioned programming languages. Our techniques are transferable to other drivers that would benefit from more modern implementations.

Our drivers in Rust, C#, go, OCaml, and Swift are completely finished, tuned for performance, evaluated, and benchmarked (they are about 80-90% as fast as our user space C driver which is as fast as older versions of DPDK).

The main thing to take away from this talk is: writing drivers is neither scary nor hard. You can write one in your favorite programming language, so go ahead and try that :)

Other people: Simon Ellmann

Drivers are usually written in C for historical reasons, this can be bad if you want your driver to be safe and secure. We show that it is possible to write low-level drivers for PCIe devices in modern high-level languages. We are working on super-fast user space network drivers for the Intel 82599ES (ixgbe) 10 Gbit/s NICs in different high-level languages. We've got fully working implementations in Rust, C#, go, OCaml, Haskell, and Swift. All of them are written from scratch and require no kernel code. Check out our GitHub page with links to all implementations, performance measurements, and publications for further reading.

Supposedly modern user space drivers (e.g., DPDK or SPDK) are still being written in C in 2018 :( This comes with all the well-known drawbacks of writing things in C that might be prevented by using safer programming languages. Also, did you ever see a kernel panic because a driver did something stupid? It doesn't have to be that way, drivers should not be able to take down the whole system. There are three steps to building better drivers: 1. Write them in a safer programming language eliminating whole classes of bugs and security problems like bad memory accesses 2. Isolating them from the rest of the operating system: user space drivers that drop privileges 3. Isolating the hardware using the IOMMU We show that it is possible to achieve all of these goals for PCIe drivers on Linux by implementing user space network drivers in all of the aforementioned programming languages. Our techniques are transferable to other drivers that would benefit from more modern implementations. Our drivers in Rust, C#, go, and Swift are completely finished, tuned for performance, evaluated, and benchmarked. And all of them except for Swift are about 80-90% as fast as our user space C driver and 6-10 times faster than the kernel C driver. We also investigate how garbage collectod languages affects the latency of a packet forwarder built on top of our drivers. We also got something for fans of functional languages: our implementations in OCaml and Haskell are working but not yet tuned for performance. We are also working on Python, Java, and Javascript. We take a brief look at Haskell, Swift, OCaml, and C# in the talk and a deeper dive into Rust and Go. The talk also features a quick summary from last year's talk about user space driver basics, so no previous knowledge is required. Another thing to take away from this talk is: writing drivers is neither scary nor hard. You can write one in your favorite programming language, so go ahead and try that :)

Other people: Simon Ellmann Sebastian Voit

Network cards are often seen as black boxes: you put data in a socket on one side and packets come out at the other end - or the other way around. Let's have a deeper look at how a network card actually works at the lower levels by writing a simple user space driver from scratch for a 10 Gbit/s NIC.

Packet processing in software is currently undergoing a huge paradigm shift. Connection speeds of 10 Gbit/s and beyond created new problems and operating systems couldn't keep keep up. Hence, there has been a rise of frameworks and libraries working around the kernel, sometimes referred to as kernel bypass or zero copy (the latter is a misnomer). Examples are DPDK, Snabb, netmap, XDP, pf_ring, and pfq. The first part of the talk looks at the background and performance of the kernel network stack and what changes with these new frameworks. They break with all traditional APIs and present new paradigms. For example, they usually provide an application exclusive access to a network interface and exchange raw packets with the app. There are no sockets, they don't even offer a protocol stack. Hence, they are mostly used for low-level packet processing apps: routers, (virtual) switches, firewalls, and annoying middleboxes "optimizing" your connection. It's now feasible to write quick prototypes of packet processing and forwarding apps that were restricted to dedicated hardware in the past, enabling everyone to build and test high-speed networking equipment with a low budget. These concepts are slowly creeping into operating systems and software routers/switches: FreeBSD ships with netmap today, XDP is coming to Linux, Open vSwitch can be compiled with a DPDK backend, pfSense is adopting DPDK as well, ... We need to look at the architecture of these frameworks to better understand what is coming for us. Most of these frameworks build on the original drivers that have been growing in complexity: a typical driver for a 10 or 40 Gbit/s NIC is in the order of 50,000 lines of code nowadays. Hundreds of thousands of lines of code are involved when handling a packet in a typical operating system, and tens of thousands when using one of these new frameworks. Reading and understanding so much code is quite tedious, so the obvious question is: How hard can it be to implement a driver for a modern 10 Gbit/s NIC from scratch while ignoring all of the existing software layers? Turns out that it's not very hard: I've written ixy, a user space driver for 10 Gbit/s NICs from the Intel 82599 family (X520, X540, X550) from scratch in about 1000 lines of C code. The second part of the talk focuses on user space drivers and the Intel 82599 architecture as it is easy to understand, has a great datasheet, and the core functionality is in the driver as opposed to a magic black-box firmware. ixy is a full user space driver: you get your raw packets delivered directly into your application and the operating system doesn't even know the NIC exists. User space drivers are also very hackable, you get direct access to the full hardware in your application in user space making it really easy to test out new features, no pesky kernel code needed. This is why it's important to have a simple driver like ixy: for hacking and educational purposes. Core functionality of the driver such as handling DMA buffers is never far away when writing an ixy app: you typically only need to look beneath one layer to see the guts of the driver. For example, when you send out a packet you call a transmit function that directly modifies a ring buffer of DMA descriptors. Check out the code of ixy on GitHub!

MoonGen is a scriptable high-speed packet generator suitable to test network devices or NFV deployments with millions of packets per second at rates of more than 10 Gbit/s. Each packet is crafted in real time by a user-defined Lua script to ensure the maximum possible flexibility to test complex scenarios. MoonGen is available as free and open source software on GitHub, a scientific paper describing it was published at the Internet Measurement Conference in October 2015.

What is MoonGen?

MoonGen is a flexible high-speed packet generator. It can saturate 10 GbE links with minimum-sized packets while using only a single CPU core by running on top of the packet processing framework DPDK. Linear multi-core scaling allows for even higher rates: We have tested MoonGen with up to 178.5 Mpps at 120 Gbit/s. Moving the whole packet generation logic into user-controlled Lua scripts allows us to achieve the highest possible flexibility. In addition, we utilize hardware features of commodity NICs that have not been used for packet generators previously. A key feature is the measurement of latency with sub-microsecond precision and accuracy by using hardware timestamping capabilities of modern commodity NICs. We address timing issues with software-based packet generators and apply methods to mitigate them with both hardware support and with a novel method to control the inter-packet gap in software. Features that were previously only possible with hardware-based solutions are now provided by MoonGen on commodity hardware. MoonGen is available as free software under the MIT license in our git repository at https://github.com/emmericp/MoonGen.

Using MoonGen for Complex Network Tests

MoonGen works with a user-defined Lua script that specifies the whole test by executing script code for each packet in real time. Support for packet reception allows rich tests that distinguish the throughput and latency of multiple different flows to evaluate QoS features.

One key feature is the precise and accurate measurement of latencies. MoonGen can timestamp thousands of packets per second, providing insights into the latency behavior of a system. Latency measurents are especially important in virtualized environments that are often plagued by bad worst-case behaviors that manifest in long-tail latencies. The 99th percentile (the long tail) of the latency is an important performance characteristic of a NFV setup.

This talk shows how to use MoonGen to evaluate networking devices and NFV setups with examples from network experiments conducted by the authors of MoonGen. It also discusses how SDN switches can be used to amplify and modify traffic to benchmark devices with speeds of multiple Terabit/s.

More Information on MoonGen

A scientific paper describing MoonGen was published at the Internet Measurement Conference in October 2015.

MoonGen is a scriptable high-speed packet generator suitable to test network devices with millions of packets per second at rates of above 10 Gbit/s. Each packet is crafted in real time by a user-defined Lua script to ensure the maximum possible flexibility. MoonGen is available as free and open source software on GitHub, a scientific paper describing it was published at the Internet Measurement Conference in October 2015.

What is MoonGen?

MoonGen is a flexible high-speed packet generator. It can saturate 10 GbE links with minimum-sized packets while using only a single CPU core by running on top of the packet processing framework DPDK. Linear multi-core scaling allows for even higher rates: We have tested MoonGen with up to 178.5 Mpps at 120 Gbit/s. Moving the whole packet generation logic into user-controlled Lua scripts allows us to achieve the highest possible flexibility. In addition, we utilize hardware features of commodity NICs that have not been used for packet generators previously. A key feature is the measurement of latency with sub-microsecond precision and accuracy by using hardware timestamping capabilities of modern commodity NICs. We address timing issues with software-based packet generators and apply methods to mitigate them with both hardware support and with a novel method to control the inter-packet gap in software. Features that were previously only possible with hardware-based solutions are now provided by MoonGen on commodity hardware. MoonGen is available as free software under the MIT license in our git repository at https://github.com/emmericp/MoonGen.

Using Lua for Networking Tasks

MoonGen is 90% Lua. Each single packet that is sent out is crafted in real time by a user-defined Lua script which needs to process millions of packets per second. There are several challenges that are addressed by MoonGen in order to achieve this. The most important one is multi-threading: all modern NICs are natively multi-core aware and linear multi-core scaling is a must for any modern packet processing app. This talk will address how MoonGen uses a shared-nothing architecture to implement multi-threading via multiple LuaJIT VMs, how the real-time requirements of a packet generator can be fulfilled despite interruptions from the JIT and the garbage collector.

More Information on MoonGen

A scientific paper describing MoonGen was published at the Internet Measurement Conference in October 2015.