Maximilian Michels

Apache Beam is a programming model for composing parallel and distributed data processing jobs.

As many other Apache projects, Beam first used Java as its API language. Unsatisfied with the status quo, Beam developers launched the portability project to enable other languages to run with Beam. Currently, Beam has a Java, Python, and a Go API.

Ultimately, these languages won't just coexist in Apache Beam, but they will complement each other in cross-language data processing jobs.

In this talk we will learn how it is possible to support multiple languages and why it might be a good idea to combine these languages in data processing jobs.

Apache Beam is a programming model for composing parallel and distributed data processing jobs. Once composed, these jobs run on various execution engines like Apache Flink, Apache Spark, or Google Cloud Dataflow. But Apache Beam's vision goes beyond just running on multiple execution engines.

As many other Apache projects, Beam first used Java as its API language. Unsatisfied with the status quo, Beam developers launched the portability project to enable other languages to run with Beam. Currently, Beam has a Java, Python, and a Go API. That means users are not restricted to the Java ecosystem but can use their favorite Python libraries like Numpy or Tensorflow with Apache Beam.

Ultimately, these languages won't just coexist in Apache Beam, but they will complement each other in cross-language data processing jobs. For example, reading from Kafka can be done with the Java connector but the data can afterwards be processed in Python.

In this talk we will learn how it is possible to support multiple languages and why it might be a good idea to combine these languages in data processing jobs.

In this talk, I will give an introduction to CrateDB, its architecture, and demo a few things that people have built with it.

Search engines are databases that specialize in retrieving information from a data corpus. Compared to traditional databases like PostgreSQL, search engines allow you to work with text and other unstructured data very efficiently.

Projects like Xapian and Lucene can perform efficient indexing and querying of large amounts of documents. Projects like Solr and Elasticsearch have added clustering and distributed query execution to scale out the search features.

The most obvious gap between traditional databases and search engines is the query language. Whereas relational databases can typically be queried with SQL, search engines usually implement a custom search API.

At CrateDB, we don’t think you should have to give up SQL just because you’re using search engine features. That’s why we created a fully-functional SQL interface on top of Elasticsearch and Lucene. You get all the benefits of traditional databases, as well as the features of a distributed search engine.

Do you want to store huge amounts of data and search it in real time? Do you have unstructured and structured data? Do you want to run distributed joins? Do you want to add nodes and scale your cluster horizontally? Do you want to leverage the power of SQL? If so, CrateDB is a great match.