Dario Faggioli

Immutable OSes are great, but not being able to --for instance-- install new distribution packages (or having to reboot every time you do so) may be really tedious, especially for developers. Except, you can install whatever you want without having to reboot, and without breaking the consistency and "the immutability" of the OS itself either. How? Well, in one word, with toolbox. In some more words: come to the talk.

So, let's say that your driver is an immutable OS, like Fedora Silverblue, openSUSE MicroOS, EndlessOS, etc. And let's say you are a power user and/or a developer. So, how do you perform your daily tasks, and how do you do your development on such a system, where you cannot install the compiler for the language you're writing the code in?

Why, you ask? Well, because immutable OSes are often cleaner, more consistent, more reliable, and a lot harder to break than traditional ones. But that comes at the price of (for instance, on Silverblue and MicroOS) having to reboot several times a day, which is often a showstopper. Unless we leverage containers in order to provide users with a read-write environment that can be easily spawned, and inside which they can "live" and do most of their work (especially if they're doing any development!). This is usually done with something called "toolbox". In fact, Fedora Silverblue has a toolbox. openSUSE MicroOS has a toolbox, which is similar but also different from Silverblue's one. But that's not all of it, e.g., someone came up with podbox and someone else with coretoolbox.

This talk will show the reasons why having some kind of toolbox is absolutely necessary inside an immutable OS. We will also try to explain the differences between some of the existing approaches and their implementations. And, last but not least, we will showcase --through real-life examples-- how, by taking advantage of a toolbox, you can elect an immutable OS to be the daily driver of your main development workstation, and not have to reboot it (unless you actually want to!)

If you can't modify the root filesystem of your distro (no, not even with sudo!!), you can't break it, right? Well... soft of. But what's the price to pay in terms of usability? How are you actually able to use an OS like that for common everyday user (web browsing, document editing) and developer (writing and building code) activities? Come to this talk and learn how it's being to use one of them, i.e., openSUSE MicroOS, as a daily driver since the last few months and how that compares with using other similar solutions, like Fedora Silverblue and Endless OS.

An Immutable OS is a GNU/Linux distribution where the user has limited (if at all!) changes to modify the root filesystem. And by "modify the root filesystem" we mean doing things like installing software with the package manager. Some examples of Immutable OSes that falls in this definition are Fedora Silverblue, openSUSE MicroOS, openSUSE Kubic and EndlessOS. In fact, on Fedora Silverblue, the user can't add software with dnf, not even after becoming "root". Similarly, on openSUSE MicroOS, zypper install just does not work. This brings the advantage of keeping the OS cleaner, consistent, reliable and a lot harder to break. For instance because what developers produce and tests is a lot more similar to what actual users use, with respect to what happens with "traditional OSes". And they usually come with a nice way of rolling back changes too, which is also really valuable.

But what are the technologies behind all that? Well, there are a few, and although the outcome is rather similar, their inner working and characteristics are quite different. In this talk, we will focus on two: ostree and transactional-update. The forme is used in Silverblue and Endless OS, the latter in openSUSE MicroOS and Kubic. We will try to provide a description of how both work and what are the main differences between them.

Even more important: what's the user experience with such distros? And what if one is not only a user, but a developer? Well, this talk will offer a perspective of how it is being to use one as one's own daily driver, for both user-like and developer-like tasks. That would be openSUSE MicroOS. We will cover what it is the current status of the "MicroOS as a Desktop" project, what are the current challenge and the upcoming developments. But we will also try to hint at how the user/developer experience may vary between MicroOS and others (more mature) immutable desktop solutions like Fedora Silverblue and Endless OS.

Tracing is awesome. Full stop. But what tracing? In fact, even just on Linux, there are quite a few tracing solutions, aren't there they? In this session we'll show off and compare ftrace and LTTng and, for visualizing the collected data, KernelShark and Trace Compass when tracing both baremetal and virtualized systems.

With tracing, we can get to understand what is really happening while our software is running at a great level of detail. Depending on the adopted tracing solution, that applies to user-space components, to OS kernels or even to both. The collected data can be very helpful when investigating bugs and misbehaviors, but can also be used for understanding the internals of the system and for chasing performance degradations or improvements.

On Linux, tracing can be done in a variety of ways. This talk will focus on two: ftrace and LTTng. The aim is describing their characteristics and show how each one works. After that we will show what are the main differences between them in terms of what they are capable of, how the are used and last but not least the overhead they introduce, measured with several benchmarks.

We will also cover the tools that one can use to visualize the often huge amount of data that is usually collected during a tracing session. Both textual ones, such as trace-cmd and babeltrace2, and graphical ones, such as KernelShark and Trace-Compass.

Last but not least, we will offer some insights on how to use these two mechanisms for tracing virtualized environments. I.e., how can you use them for tracing what is going on both on the host and inside one or more guests, at the same time, and how good are they for such purpose?

Virtualization brings many advantages, but what about the overhead it introduces? What about performance? This talk will show how great virtualization performance can be achieved, if proper tuning is applied to all the components of the system: hypervisor, host and guests, for both Xen and KVM. As a case study, we will describe how we tuned our OS in order to be able to reach, inside VMs, close to baremetal performance, on a server powered by a CPU from the AMD EPYC 7002 (codename "Rome") series. We will, of course, show the benchmarks proving that (run on KVM), even when memory encryption is used.

Virtualization is great because it decouples the software from the hardware on top of which it runs, and this brings benefits in terms of flexibility, security, reliability and cost savings. But what about the overhead that this, unavoidably, introduces?

Well, often enough, a virtualized system is really able to fulfill its goals with an acceptable quality of service, efficient exploitation of HW resources, satisfactory user experience, etc., only if all the components are configured properly. This is not entirely new, as baremetal systems need tuning too, but in a virtualized environment one has to take care of tuning both the the host and the guests. And beware that the interactions between all the different components may not always be straightforward, especially on a large server with complex CPU architecture, such anything based on the AMD EPYC 7002 (codename "Rome") series of processors.

This talk will go over some of the typical virtualization “tuning tricks” (for both Xen and KVM). Then, as a case study, we will illustrate how we managed to reach, inside Virtual Machines, a performance level that almost matches the one of the host, on a server powered by a CPU from the AMD EPYC 7002 series. In fact, we will show the results of running CPU and memory intensive benchmarks (on KVM) with and without the suggested tuning. Last (but not least :-D), we will show the impact that the Secure Encrypted Virtualization (SEV) technology has on performance.

What benchmark? How many VMs? How big each VM is? Are they all equal or are they different? What's the host OS? What are the guest OSes? I.e., when wanting to do virtualization performance testing, the matrix of test cases tends to explode pretty quickly. This talk will show how we enhanced an existing benchmarking suite, MMTests, in order to be able to deal a little bit better with such complexity. And what our further activities and plans are, for even more and better automation.

Functional testing is already hard enough, in virtualization. For instance, because we need to make sure that things work with different combinations of versions of the OSes in hosts and guests. Doing performance testing, even more so. In fact, there are much more things to consider, such as how many VMs we use, how big they are, whether or not they are equally big or different, what to run in them, how to partition the host resources for them... And this is true either in case you have a specific (virtualized) workload and some KPI to meet, in which case you need testing and benchmarking to figure out whether or not the tuning you have done has brought you there, or in case you wonder how good (or how bad) a certain configuration of both your host and your guests works, for a number of workloads,

This talk will introduce the problem, showing how the size and the complexity of a typical 'virtualization performance testing matrix' really tend to explode. We will, as an example, show how some specific characteristics of a virtualized system were, despite tuning, causing us to not be able to achieve the desired performance levels. Then we illustrate how, at SUSE, we do automated performance benchmarking, how we enhanced the tool that was in use the most for baremetal benchmarks (the MMTests suite) in order for it to be much more useful in virtualized systems and how we are integrating it with other tools to bring the level of automation even further and achieve something that really resembles a Virtualization Performance CI system.

Once upon a time, an operating system scheduler was TheOne_ entity in charge of deciding what "activity" would run on a physical processor. Furthermore, such "activity" would pretty much all the time be an application (or system) program. Well, guess what: these days, things are more complicated! Applications and physical processors have moved away from each other, as one or more layers have been put in between them. In fact, in a typical virtualization/cloud scenario, we have a hypervisor, providing the abstraction of virtual processors, and it is onto them that applications run. And here we are: running an application inside a virtual machine requires a virtual processor to be scheduled on a physical processor --at the host level-- and the application itself to be scheduler on the virtual processor --at the guest level. So, two schedulers (at least!).

But how do these two schedulers interact right now? How should they interact ideally? Should they be closely related, completely independent, or something in the middle? Are they better be based on the same algorithm? Or do different layers have different enough needs to require highly specific solutions?

This talk will go through the above (and more) questions, trying to outline pros and cons of a few approaches. That will happen both in a "theoretical" and practical sense. The latter meaning that we will be looking at some real scheduling traces (at both host and guest level), and at numbers from different benchmarks. The main focus, especially of the 'case study' part, will be on the Xen's scheduler(s) (i.e., on the interactions between Xen's and Linux's schedulers) but we will have a look at a few KVM data points as well.

One Scheduler to rule them all. One Scheduler to find them, One Scheduler to bring them all and...

Recently, while investigating, and trying to improve, performance of the Xen hypervisor, a number of non previously obvious interactions between host and guest level scheduling have been identified. This means, in case of Xen, interactions between the Xen's scheduler (at the host/hypervisor layer) and the Linux scheduler (at the guest layer).

Traces have been collected and benchmarks have been run, in order to figure out the exact nature of those interactions, and even more in order to amend or, at least, mitigate, the ones that were causing badly performing behaviours. Similar situations when other hypervisors (namely, KVM) were in use, have been inspected too, out of interest and for reference.

The results of such investigation, both in terms of generic thoughts and reasoning, and in terms of traces and numbers is what the talk will cover.

Embedded systems are becoming powerful enough that virtualization is now both possible and interesting. Xen, as a very tiny microkernel based hypervisor looks like a very good fit for the embedded environment, not to mention that it has been ported to ARM with the number of supported boards in constant increase.

This talk will outline the major strengths of the Xen architecture, when it comes to use Xen on embedded systems. It will also identify and discuss the areas where there is still room for improvement. It will go through preliminary experimental results on assessing some of the typical real-time requirements (such as responsiveness and predictability), for the benefit of everyone out there that would like to build its embedded product on top of Xen.

On the concrete side, we will show how to setup an 'Andorid on Xen' environment, which results from a fruitful collaboration between the Xen community and other interested parties from the Android community. We think this could be very useful as an example for anyone interested in working with us, with the aim of being successful in the embedded virtualization product space.

Full description: The goal of this talk is twofold: - provide a general enough view of the Xen architecture, focusing on

why it could be particularly well suited for the embedded
virtualization usecases

• show how the Xen community is tackling this new challenge, trying to involve all the (potentially) interesting parties and reaching the highest possible levels of cross-project collaboration

Outline: * Embedded virtualization: some typical usecases * The xen architecture, and why it fits so well there

 * it's tiny: small memory footprint, easier to certify
 * it's extensible: pluggable scheduling architecture
 * it's flexible: different Dom0s, driver domains

• Areas of further improvements:
  • new hard real-time schedulers
  • more determinism within the driver model
• Some benchmarks on how Xen behaves in low latency workloads
• Already ongoing projects/effort:
  • Xen on tablets, cars, industrial CNCs, ...
• Bootstrapping a (sub)community
  • getting in touch with all the interested parties
  • keeping in touch with them
  • handling feedback, coordinate collaboration
  • concrete example: setup a Android on Xen environment