Stephen Finucane

How does one manage and document change in Python projects, be that new features or deprecation or removal of a feature? Let's explore some of the tools a Python developer can keep in their toolbox for just this purpose.

Software rarely stands still (unless it's TeX). Things are added, things are removed, things break and are then hopefully fixed. Managing this, from both the developer and user perspective, can be tough. In this talk we examine and compare some of the tools that one can use to make this process easier, such as 'debtcollector', 'reno' and 'towncrier', and contrast these with alternatives used in other projects. This talk would mainly be of interest to developers of open source libraries, though the same tooling can be used for any Python library or application that wishes to maintain stable interfaces and/or document changes in their product.

Using Sphinx doesn't necessarily mean using reStructuredText for input and HTML for output. We explore Sphinx's newfound support for Markdown as well as it's broad range output formats available, before moving onto an overview of how you can develop your parser and builder extensions.

Sphinx is the documentation tool of choice for an increasing amount of projects, both inside and outside Python. Thanks to the success of platforms like Read the Docs, building documentation with this toolkit has never been easier. Most people associate Sphinx with the reStructuredText syntax provided by docutils, and typically output documents to HTML and, occasionally, PDF. However, Sphinx is capable of so much more. Since Sphinx 1.8, it is possible to use source documents written in the CommonMark syntax, while the amount of builders provided both in-tree and out-of-tree continue to grow.

Through this talk, we explore how one can build their documentation using Sphinx and Markdown source documents. We detail the variety of builders available as part of the standard Sphinx installation and as third-party extensions, including some basic configuration tips for the more commonly useful ones like man pages and LaTeX/PDF. Finally, we provide a high-level overview on how you can go about writing your own extensions to parse other plain-text documentation formats and output in additional documentation format. The latter of these will touch on docutils, a foundational component of Sphinx, so one can understand the intermediate documentation model it provides.

The Sphinx documentation tool provides tremendous extensibility and theming options; options which are massively underutilized. We demonstrate how you can go about writing your own extensions and themes for Sphinx and ultimately how you treat your docs like code.

If you work in open source, then it's increasingly likely that you've come across the Sphinx documentation tool. Originally written by and for the Python community, the past few years have seen an increasingly rapid adoption by non-Python projects. Recent adopters of Sphinx include the Linux kernel, Open vSwitch and the Dataplane Development Kit (DPDK), while other projects such as QEMU are preparing plans to migrate. Meanwhile, the Python ecosystem itself has only grown in strength and communities such as OpenStack have adopted the tool and have made extensive use of it.

Unfortunately, despite the widespread adoption and existence of multiple extensions, Sphinx is not always a very well understood tool. Extensions provide a pathway to greater functionality, for example, automatically generating documentation or generating docs in different output formats. Themes, meanwhile, provide a way to give a consistent, project-focused feel to your docs. The OpenStack community makes significant use of these extensibility features, which allow us to solve a number of problems in an automated manner.

• How can I avoid documenting a command line application twice: once in code and once in my docs?
• How can I take a large dataset and automatically convert this into multiple human-readable documents?
• How can I move documents about without breaking links?
• How can I automatically spell-check my documents?
• How can I style my docs to match my project website?

Through this talk we will demonstrate how each of the above problems, and how many more, can be solved in an easy and automated manner through the power of Sphinx extensions and themes. We also demonstrate how you can share your extensions and themes with the world, if you're so inclined

Real-time in OpenStack is a thing now. Let's explore what this actually looks like to a user, and (in brief) how it works under the hood.

Since Juno, OpenStack projects like nova have been on a steady march towards adding the features and capabilities that users with NFV and HPC workloads really need from their cloud. The latest of these nova features builds upon earlier work to finally make real-time workloads in an OpenStack cloud a reality. In this talk, we take a peek under the nova hood to see how this feature works and how it ties into earlier work in this area (hint: KVM + libvirt do most of the heavy lifting). We then demonstrate why this matters and how you can use this feature and others like it in your own applications.

What does it take to implement continuous integration-style automated testing into a mailing list-driven software project? Not a lot, actually. In this talk, we demonstrate how a simple but easily scaled testing system can be implemented for a such a project through the combination of Patchwork, the web-based patch tracking system, and open source CI tools such as Jenkins.

At FOSDEM 2016, developers working on Patchwork, the web-based patch tracking system, demonstrated some of the ongoing work in Patchwork. This work ranged from UI improvements to new features and APIs but, collectively, it had the goal of enabling automated testing functionality for software projects developed via mailing lists. The Patchwork developers have been busy since then and the application, in widespread use since 2008, recently hit the 2.0 milestone, marking this functionality as complete.

Projects such as the Dataplane Development Kit (DPDK) have quickly adopted the features that 2.0 brings, using them to enable real time, automated testing of patches sent to the mailing list. This automated testing provides a mechanism for developers to not only sidestep the more perfunctory of tasks, such as coding standard checks, but also to test changes in environments that they may not have at their disposal, such as differing hardware or OS configurations. As seen in projects pairing open source code collaboration tools like Gerrit or Rietveld with CI systems such as Jenkins or BuildBot, this continuous testing can provide huge improvements in developer velocity.

In this talk, we demonstrate how to build a basic "checkstyle" testing system through a combination of Patchwork and an off-the-shelf, open source CI system. This system will retrieve patches and dependencies, apply and test them, and report the results back to both Patchwork and a separate mailing list. This configuration demonstrates the best of Patchwork's new features, and can be easily extended to cover far more complex testing scenarios.

Mailing lists are the bedrock of many open source software projects, and have been so since the early days of the Internet. However, mailing lists can struggle to compete with code collaboration tools like Gerrit and Rietveld, many of which offer features such as integration with automated testing tools and patch tracking. How can such features be integrated into existing, mailing list-based projects like the Linux kernel or DPDK? The presenter reports on the ongoing work around the widely-deployed 'patchwork' tool to do just this.

Tracking of patches between development mailing lists and automated testing tools can be accomplished using patchwork. patchwork provides a web interface for patches submitted to mailing lists and presents them alongside any comments. patchwork also supports the tracking of the state of patches, be they Accepted, Rejected or Under Review. By design, patchwork is not intended to replace mailing lists but rather supplement them: these features are entirely optional, but where used can help ease the burden on maintainers by removing the need to perform tedious, time-consuming and low-value tasks like manual sorting. patchwork is already widely deployed for many projects, and instances can be found on kernel.org, dpdk.org and openembedded.org, to name but a few. There are also many projects, including Open vSwitch and QEMU, that can be found on ozlabs.org, which is maintained by the original author of patchwork.

This talk outlines the ongoing work to supplement development mailing lists with web-based workflows using patchwork. We demonstrate how features such as continuous integration support, automated patch and series tracking and a vastly expanded API help position the combination of the Mailing List and patchwork as a viable, feature-competitive alternative to tools such as Gerrit or Rietveld. We also detail some of the complexities of structuring the inherently unstructured data found on mailing lists. How do I identify a series of patches, for example? How do identify a reply to a patch? How do I identify a patch itself?

This talk be of particular interest to testers of open source, mailing list-based projects such as the Linux kernel, QEMU, DPDK or Open vSwitch. These users will be able take advantage of features like support for continuous integration servers and series tracking to develop automated, distributed testing infrastructures. Communities like OpenStack have shown that such infrastructures are critical for delivering software faster while reducing risk and maintenance cost. Such an infrastructure is currently being developed for the DPDK community, and it is easy to envision a similar roll-out for other projects with similar requirements.

It will also be of interest to the developers of these projects. These developers can already use patchwork as a tool to track patch backlogs, maintain personal TODO lists, or just browse submissions via a web UI/XML-RPC API. However, the new features provide these users with functionality that can help automate even more of the tedious yet necessary overhead tasks.