conferences | speakers | series

Building an actor library for Quickwit's indexing pipeline.

home

Building an actor library for Quickwit's indexing pipeline.
FOSDEM 2023

Quickwit is an opensource distributed search engine. The most challenging component of a search engine is its indexing pipeline.

It takes a stream of JSON documents, processes them in batches, and emits index artefacts we call splits.

We needed our implementation to be robust, testable, observable, and efficient... But even more importantly, our code had to be easy to work with.

I will explain why we chose to develop our own actor framework and discuss the unique features of our implementation.

Speakers: Paul Masurel