Paul Masurel

Quickwit is an opensource distributed search engine. The most challenging component of a search engine is its indexing pipeline.

It takes a stream of JSON documents, processes them in batches, and emits index artefacts we call splits.

We needed our implementation to be robust, testable, observable, and efficient... But even more importantly, our code had to be easy to work with.

I will explain why we chose to develop our own actor framework and discuss the unique features of our implementation.

Tantivy is a search engine library, akin to Lucene but for Rust.

In this talk, we will walk through the building blocks that make a scalable full-text search engine. While this talk will be centered on tantivy, most of the concepts introduced should apply to other inverted-list search engines like Lucene.