I'll present recent work on an implementation of "timely dataflow", a neat new data-parallel programming model, in Rust, a neat new-ish programming language from Mozilla. I'll talk through how Rust's take on ownership is a great fit for distributed big data programming, and in particular how it lets us write rich high-level dataflow programs that still have the performance characteristics we expect from hand-written code.
Rust is a new programming language which provides high-level programming idioms that compile down to code that is intended to be as good as (or better than) one could do by hand. At the core of Rust is the concept of "ownership", where all values, resources, allocations, handles, etc have a unique "owner", with a statically known lifetime. This allows Rust to provide automatic memory management, providing safe code with relatively few (not zero) leaks, without requiring a tracing garbage collector. In exchange, there is some amount of additional cognitive overhead when resources have weird or unknown lifetimes.
Timely dataflow is a model for data-parallel dataflow computations, introduced as the foundation for the high-throughput, low-latency distributed Naiad system. Naiad achieved top-tier performance by providing programmers with a well-engineered data plane and simple but effective control mechanisms, but otherwise staying out of the way. At the same time, much of Naiad's engineering work went in to fighting against the language and runtime in order to extract the intended performance, similar to what several JVM based systems are currently struggling against.
Having recently been given the opportunity, I re-implemented timely dataflow in Rust and found it to be a very liberating experience. Rust's concept of ownership dovetails very elegantly with the data-driven computation of most distributed dataflow systems. At the same time, by providing Rust clear information about data ownership the compiled code avoids the overheads (and pain) associated with a garbage collected runtime. The look and feel of user code is mostly the same as in managed languages; ownership issues are rarely raised to the data-driven, and so ownership-friendly, user code.
What result is a system that presents a high-level dataflow programming interface, but whose performance resembles that of carefully implemented HPC codes. I'll try and explain a few key moments that make this possible, and how systems implemented over other platforms should at least understand and explain what they are giving up.
