Follow me as I walk through my journey of building lnx, a distributed search engine written in Rust akin to Elasticsearch or Algolia that aims to be faster and more efficient using tantivy.
I cover the challenges and solutions I encountered while developing lnx over the last year, the tradeoffs made and how you can build your own search engine using the tools the Rust ecosystem provides.
In the first part of the talk we'll look at what a search engine like Algolia or Elasticsearch does, how tantivy works, what things we need to do in order to make it suitable for user-facing search and how we implement it into our code to build our own application.
In the second part of the talk I talk about the biggest issue you run into when building a search engine or any other system designed to store data... Fault tolerance and replication, together we'll go through the challenges of implementing this for a search engine using tantivy, keeping the performance competitive, reliable and the tradeoffs that are made to make it work. We'll look at some existing implementations like Raft and take a dive into the world of eventual consistency!
With any luck, you'll have a better understanding of how we can build services like search engine using libraries like tantivy along with solving technical challenges like consensus and replication for distributed systems in Rust.
