Nakadi is an open source event broker, built on top of Kafka-like queues, that provides a REST API for easy integration with microservices. In this talk, we will introduce Nakadi, and demonstrate one of its most important features: timelines.
Timelines is a feature of Nakadi that allows administrators to transparently move queues between different clusters, without interrupting producers or consumers. This is useful to move between clusters, possibly using different technologies, but also as a fail-over for publishers in case of downtime on a cluster. Timelines can also be used to move event types between brokers on the same cluster, which is an alternative to rebalance operations. Contrary to rebalance operations, using timelines does not require additional data copy between brokers, which can slow them down. Future plans include using timelines for re-partitioning, i.e. changing the number of partitions in an event type.
Nakadi is used at Zalando, to produce around 15 terabytes and consume around 40 terabytes of data daily.
Speakers: Lionel Montrieux