Uwe Schindler

Spring 2019 will be the time for a new major release of Apache Lucene and the well-known search server Apache Solr. Since Lucene 7 a lot new features have been developed, mainly a new way to short circuit query execution if the total number of hits is not needed. This may improve query execution for common full text search use cases enormously. The talk will also present other new features and changes like change to a more standard BM25 implementation or optimizations for Java 9 and later Java versions using MR-JAR files. The second part of this talk will quickly cover news in Apache Solr: Solr 8 now also uses HTTP/2 for its cloud communication.

With the introduction of the Jigsaw module system many projects are confronted with issues running their code, now throwing new Exceptions like InaccessibleObjectException because they used to work around issues in the Java API. Although many features like the famous sun.misc.Unsafe are now provided by other public APIs, there is still the problem of making your code work with previous Java versions, but also dynamically using the new Java 9 replacements. Apache Lucene, the open source fulltext search library behind Apache Solr and Elasticsearch, is also using some of those "hacks" to work around issues in the Java API, but the developers also noticed that some code in the project hit some limits introduced by the module system. This talk will show on some examples how Apache Lucene managed to allow its code to run with different java versions (without using multi-release JAR files, as this brings additional burden with the build system) and still offer backwards compatibility to earlier versions. It will show also the common pitfalls that can be easily found with running your tests in a security manager to uncover bugs.

When Oracle released Java 7 GA, it contained a serious loop optimization bug, affecting Apache projects like Lucene, Solr, or Elasticsearch. Since that time, the Apache Lucene team works together with the Oracle and OpenJDK developers to test early-access releases with Apache Lucene's codebase. Due to Apache Lucene's heavy, low level code with many tight loops and unusual code patterns, it is an ideal testing environment for the Hotspot optimizer. Loops, iterators, and bitset operations are executed millions of times per full text query to execute finite state automata, collect and filter results, and sort them. Bugs caused by mis-compilation of code that are hard to reproduce in isolation can be easily reproduced with Lucene. In addition, Lucene uses a pseudo-randomized testing infrastructure triggering different Hotspot optimizations on each test execution. In this talk, Uwe will show the current state of Java 9 EA testing. He will also present the problems caused by hard to reproduce issues, e.g. occurred after the changes to arraycopy or AVX optimizations in Hotspot.

Around FOSDEM 2015, the first alpha/beta releases of Apache Lucene 5 will likely be downloadable. This talk will present the improvements and new features, but also some incompatible changes in the Lucene 5 release. Lucene 5 will focus on data safety: The move to Java 7 will be completed. Lucene now uses all the brand new features (NIO.2) of Java 7 to make the indexing process more stable and resulting indexes durable. Checksums are used during merging to prevent bugs in the underlying JVM or data corruption due to networking errors (e.g., while distributing indexes during recovery in Elasticsearch) to persist in newly created index segments.

The previous major version, Lucene 4 was a major release that introduced index codecs. In Lucene 5, the API around codecs will be cleaned up and will likely get more stable. There are also new features, like a common FilterCache that can be reused by Solr and Elasticsearch.

In parallel Apache Solr 5 will be released, the first version that will now work as a server out of the box, so Solr is no longer exposed as a webapp. Init.d scripts are included and configuration is managed easier through Zookeeper.