Lionel Montrieux

At Zalando, a small team of 9 engineers develops, maintains, and operates an internal deployment of Nakadi, the open source event streaming platform. Nakadi handles dozens of terabytes of data every day, produced and consumed by hundreds of teams, with use cases from big data to service-to-service communication. Nakadi also feeds data into Zalando’s data lake for batch analysis. In this talk, we will show how a small team can serve so many customers while still finding the time to work on new features and fixing bugs. Serving users at scale requires a completely self-service offering. Nakadi provides self-service stream creation, administration, schema validation and evolution, monitoring, authorization, SQL over streams, and a great UI so that even less technical users can use Nakadi.

Nakadi is an open source event broker, built on top of Kafka-like queues, that provides a REST API for easy integration with microservices. In this talk, we will introduce Nakadi, and demonstrate one of its most important features: timelines.

Timelines is a feature of Nakadi that allows administrators to transparently move queues between different clusters, without interrupting producers or consumers. This is useful to move between clusters, possibly using different technologies, but also as a fail-over for publishers in case of downtime on a cluster. Timelines can also be used to move event types between brokers on the same cluster, which is an alternative to rebalance operations. Contrary to rebalance operations, using timelines does not require additional data copy between brokers, which can slow them down. Future plans include using timelines for re-partitioning, i.e. changing the number of partitions in an event type.

Nakadi is used at Zalando, to produce around 15 terabytes and consume around 40 terabytes of data daily.