Loading from Kafka

Getting started

There are two supported mechanisms for ingesting data from Kafka:

  1. Tranquility Kafka, a process which reads messages from Kafka and then pushes these messages into Druid using the Tranquility framework. Setup instructions are available in our Tranquility Kafka tutorial and the Tranquility documentation.

  2. Kafka indexing service, an experimental Druid extension available starting in Imply 1.3.0 / Druid This offers exactly-once ingestion guarantees as well as the ability to ingest historical data. Additionally, this runs as part of the core Druid services and does not require any additional processes. Setup instructions are available in our Kafka indexing service tutorial and the Druid Kafka indexing service documentation.

Both options support partitioning and replication, and can be easily scaled out to achieve the desired ingestion throughput. While we provide support for both ingestion methods, please note that we are still considering the Kafka indexing service to be experimental until it has been thoroughly battle-tested in larger production systems.

If you are looking for the most stable option, you should proceed with Tranquillity Kafka. If you require exactly-once ingestion or have historical data, and are comfortable using experimental features, consider the Kafka indexing service.

How can we help?