Michael Tretter

Yosys and nextpnr made FPGA development very attractive for developers that prefer to use open-source tools over proprietary vendor tools. Affordable and well documented boards with ECP5 FPGAs lowered the entry threshold for hobbyists even further.

Michael will show how to use LiteX to use these tools to build soft-core RISC-V SoCs that are capable of running Linux and combine them with use-case specific cores to custom FPGA bitstreams.

LiteX provides a framework to build FPGA SoCs including a few examples of SoCs that are capable of running Linux. However, for fully leveraging the capabilities of FPGAs, one would not only reproduce existing designs, but also implement use-case specific cores and add them to the FPGA bitstream.

Michael will give an update on Steffen's and his experiences with using the open-source FPGA tools to build FPGA bitstreams. In recent years, they presented how to build the LiteX example SoCs, run Linux on them, and how to make the systems reproducible by using Yocto. This talk will focus on how to add custom cores that are written in Verilog and Migen to the SoC to implement use-case specific features.

The Xilinx ZynqMP SoC includes a powerful H.264/H.265 Video Codec Unit (VCU) which Xilinx supports with an out-of-tree driver and user space library. The allegro driver from mainline Linux allows to use the VCU using the standard V4L2 encoder API without requiring any custom user space.

I will explain why I wrote the driver, show how the VCU is used from a driver perspective, and present the current state of the driver and which features are still missing.

The Zynq UltraScale+ MPSoC integrates several ARM cores and an FPGA into a single chip. One variant of this SoC includes an H.264/H.265 Allegro DVT Video Codec Unit (VCU).

Although Xilinx did a good job supporting the ZynqMP in mainline Linux, the VCU was originally only supported via an out-of-tree driver. It consists of a user space library and a firmware blob, which interact via a custom IOCTL interface. Unfortunately, this has security implications as the user space handles physical buffer addresses.

Fortunately, the V4L2 mem2mem device API for video codecs avoids this problem by using abstract DMA buffers. Many applications already use these APIs, and various drivers for hardware codecs already implement these APIs. Thus, a V4L2 driver for the VCU allows reusing existing applications such as FFmpeg and GStreamer on the ZynqMP.

Video playback for embedded devices such as infotainment systems and media centers demands hardware accelerators to achieve reasonable performance. Unfortunately, vendors provide the drivers for the accelerators only as binary blobs. We demonstrate how we built a video playback system that uses hardware acceleration on i.MX6 by using solely open source software including Gstreamer, Qt QML, the etnaviv GPU driver, and the coda video decoder driver.

While hardware accelerators are necessary to provide reasonable performance for video playback on embedded devices, the drivers that are provided as binary blobs by hardware vendors cause a lot of problems. They are linked to specific versions of the Linux kernel, may contain security and performance issues, and are pretty much impossible to debug by developers trying to build a system based on these drivers.

We built an i.MX6 based embedded system that simultaneously decodes and displays four videos. Our system solely uses open source drivers to control the available hardware accelerators.

The GUI consists of a Qt application based on QML. Using Qt and QML allows us to use OpenGL for compositing the user interface. OpenGL is backed by the open source etnativ GPU driver and the Vivante GPU.

The Qt application receives the video streams from a Gstreamer pipeline (using playbin). The Gstreamer pipeline contains a v4l2 decoder element, which uses the coda v4l2 driver for the CODA 960 video encoder and decoder IP core (VPU in the Freescale/NXP Reference Manual), and a sink element to make the frames available to the Qt application.

The entire pipeline including the Gstreamer to Qt handover uses dma_bufs to avoid copies in software.

This example shows how to use open source drivers to ease the development of video and graphics applications on embedded systems.