Alan Woodward

Lucene 8 will have some remarkable speed-ups when it comes to querying across large datasets. In this talk I will describe how this has been implemented, from new data structures through to changes in the scoring API, and the trade-offs required to make them possible.

• Lightning overview of the structure of an inverted index, showing how current queries are executed and documents scored
• Extension of an inverted index to include Impacts (single level, multi-level)
• Given impacts, show how a scorer can skip large blocks of documents
• Restrictions on Similarity implementations to make this possible (scores must increase with docfreq, must be greater than 0, etc)

Traditional searches take the form of individual queries over large mostly-stable corpuses of documents. In this talk, we'll show how we invert this paradigm to allow for searching over streams of documents by combining Samza, a distributed stream-processing framework, with Luwak, a library for efficiently running large numbers of queries over individual documents.

Real-time searching over streams is useful in a number of contexts. For example, companies may want to detect whenever they are mentioned in a news feed; or a Twitter user might want to see a continuous stream of tweets for a particular hashtag.

Luwak provides a mechanism for running many thousands of queries over a single document in a highly efficient manner, by filtering out queries that it can detect will not match. Luwak is designed to run on a single node, holding all registered queries in RAM. Scaling to higher document throughput, or to more queries, requires parallelization across multiple machines.

Samza provides a framework for such parallelization, by partitioning and recombining both the document streams and the query set (which can be treated as just another stream), and also provides fault-tolerance mechanisms that allows swift recovery from machine failure, without losing documents or queries.