Vlad Bogolin

Continuous Integration is a very important part in order to ensure a high quality software product. In order to be able to test the MariaDB server across multiple platforms, operating systems and configurations, the MariaDB Foundation has developed a continuous integration framework based on Buildbot. This talk will focus on describing the design decisions in order to be able to scale and test the MariaDB server on over 100 different configurations. We use a multi-master setup and combine Docker latent workers with LibVirt workers and classical Buildbot workers in order to achieve this.

In order to ensure that MariaDB runs smoothly, it needs to be tested on multiple platforms and configurations. In order to ensure this, the MariaDB Foundation has developed a new continuous integration framework, based on Buildbot, that thoroughly tests the server across multiple platforms, operating systems and configurations.

In order to achieve this, we use a mixture of builders, varying from Docker latent workers, to LibVirt workers but also the classical “always on” buildbot workers. For easier management, we have a separate master process that deals with each type of worker. In this talk, I will talk about our design decisions and some limitations that stand behind these. For example, the testing phase of the MariaDB server generates a very high amount of logs. Since Buildbot can deal with around 10k lines of logs per second, having multiple builds that run at the same time may saturate the master process. In order to deal with this, we use a master locking mechanism together with a multi-master setup.

Containers are a central point for the MariaDB buildbot (buildbot.mariadb.org). In fact, almost all our builds run in Docker containers. In this short presentation, I will talk about the container environment used in order to build MariaDB from source both on Linux and Windows. Then, I will present some of the challenges associated with running Windows in a Docker container and finally I will focus on some of the advantages of having a container based continuous integration infrastructure.

Due to their obvious advantages in terms of speed and flexibility, when developing our new continuous integration framework, we aimed to have an almost container only environment. Because of this, buildbot.mariadb.org uses almost exclusively Docker containers for all the workers.

In this way, using a single dockerfile, we can define in an easy and concise manner all the required environments. The process begins with choosing the operating system. Here, we have quite a large pool of 15+ operating systems varying from different Linux distributions to Windows, each running in a Docker container. The process continues by installing the MariaDB build dependencies. In the end, the buildbot components are installed. Finally, we need to make sure that the buildbot worker process starts on container startup. In this way, we obtain a clean environment that can easily be deployed to different platforms.

One of the most challenging things was to run Windows in a Docker container. While the main steps remain the same as described above, I will talk about the main differences between the Windows dockerfile in comparison to a Linux one to emphasise the similarities and differences between the two.

Recently, the MariaDB Foundation has been developing a new continuous integration framework for the MariaDB Server. The goal of buildbot.mariadb.org is to ensure that each change is properly tested on all supported platforms and operating systems. Our new CI uses almost exclusively latent workers, more exactly Docker latent workers. In this talk, I will present a main overview of the CI infrastructure, the advantages of using latent workers and talk about the challenges that we encountered along the way. This includes a broad range of aspects, ranging from misconfigurations to Buildbot code changes to ensure that everything runs smoothly.

In order to ensure that MariaDB runs smoothly, it needs to be tested on multiple platforms and configurations. In order to obtain a gain both in terms of speed and flexibility, we have decided to use Docker latent workers. In this way, each different environment is defined in a separate dockerfile. Besides having a clean and concise environment definition, using latent workers has the advantage of requiring minimal configurations on the worker machines. This makes the process of adding new hardware very easy, mainly involving installing Docker and configuring Docker remote access.

Now that we have the build environments defined, we can start testing. The process starts by cloning the MariaDB server repo and creating a source tarball. Then, all other configurations are triggered, all using the same source tarball.

While it seemed quite straightforward, the whole process turned out to be more challenging than we expected. This includes a quite long list of issues ranging from files mixups, weird sporadic failures where the main testing process was killed, grid view customizations to multi-master configuration and master-worker file transfer problems. In this talk, I will talk in more detail about these issues and tell you more about our experience and how we managed to overcome them.