Pascal Costanza

Go is an excellent tool for concurrent programming. Goroutines and channels make it easy to coordinate multiple interacting processes. With careful design, good concurrent Go programs run efficiently on multi-core CPUs because multiple goroutines can run in parallel on multiple CPUs.

What is less known is that Go also works very well for expressing parallel algorithms that do not communicate via channels, but access shared memory directly. This goes against Go’s famous slogan: “Do not communicate by sharing memory; instead, share memory by communicating.” However, sometimes performance matters so much that you have to make an exception.

Fortunately, Go is a very powerful language and makes it easy to make such exceptions. In this talk, I am going to walk you through Pargo, an open-source Go library explicitly designed for parallel programming rather than concurrent programming, which is available at https://github.com/exascience/pargo. This library is based on several years of experience with parallel programming at the ExaScience Lab at imec in Belgium, with many programming languages (C, C++, Java), many parallel programming libraries (Cilk, OpenMP, Intel Threading Building Blocks), and many applications (physics simulations, DNA sequencing, statistics).

Go already provides primitives that help with parallel programming in the sync and sync/atomic packages of its standard library. Pargo builds on top of these primitives and offers high-level functions and data structures for expressing parallel algorithms. This covers simple convenience functions on top of goroutines, efficient parallel maps, parallel sort functions, and a very powerful pipeline concept for efficiently processing large data sets.

This talk will cover the following topics:

• How Go schedules goroutines, and why they can therefore be used for task-based parallel programming.
• How Pargo provides a number of parallel programming primitives to make task-based parallel programming easier.
• How parallel Quicksort and Mergesort can be implemented with Pargo.
• How Pargo allows you to write parallel pipelines for efficiently streaming, filtering, and transforming large data sets in parallel.

Pascal Costanza has a Ph.D. degree from the University of Bonn, Germany, and works as a senior researcher at imec, Leuven, Belgium. He was previously employed by Intel Corporation, and Vrije Universiteit Brussel, Belgium. He is an expert on high-performance and data-intensive computing with applications in bioinformatics, sequencing pipelines, and artificial intelligence.

While Go is not primarily designed for parallel programming, it nevertheless has features that end up being beneficial for parallelism as well, especially the inclusion of a work-stealing scheduler for goroutines and a concurrent, parallel garbage collector. For this reason, we have recently included Go as one of several candidate programming languages in an evaluation of their suitability for expressing sequencing pipelines. Other programming languages we have evaluated were C++ and Java. Go hits a sweet spot of performing very close to the best results with little programming effort and few compromises in terms of safety and generality.

This talk will present highlights of this experiments and the most important insights.

A DNA sequencer takes a DNA sample, such as human tissue, and applies chemical processes to eventually read small fragments of the DNA sample and output them as large files. These files are then fed into software pipelines that reconstruct the original DNA sequence from those fragments, among other things. Such sequencing pipelines need large amounts of storage, on disk and/or in RAM, and can strongly benefit from parallel execution to improve runtime performance. Data sets for human DNA samples are usually in the order of several hundreds of GB of uncompressed data, and runtimes are typically in the order of several hours for single samples.

Go is usually presented as a language designed for concurrent applications, and it is usually stressed that support for parallel programming was not a major design goal. It may therefore be surprising to a larger audience that Go actually fares very well for parallel programming tasks, with relatively little effort. This talk may encourage Go developers to explore the language for application areas that can benefit from parallelism, beyond the domains where Go is typically used.

Please note that this talk replaces one entitled "Writing a structured syslog parser in Go" that was due to have been given by Brian Knox, who has sent his apologies but is now unable to attend FOSDEM this year.