In this session I would like to talk about the huge amount of data we ingested into Druid (raw data is 9 Terra per day) by using EMR, all orchestrated by Airflow. While the data grew we started experiencing many problems. After trying many scaling options, we decided to change the approach and came up with KUDRAS, the Kubernetes Druid Autoscaler.
This project is written in Python and is being used in our Apache Druid production environment. KUDRAS is a service developed using fastAPI which scales middlemanager nodes up and down in the most effective way, minimizing ingestion task costs to the bare minimum while maximizing ingestion speed.
Strategies for Managing Your Splunk Spend at Scale in 2026
Learn how a decoupled architecture for Splunk—powered by Imply Lumi and Federated Search—helps you keep more data searchable, reduce costs, and scale efficiently without changing existing Splunk workflows.
Watch nowObservability at a Breaking Point: How Decoupling Unlocks Speed, Scale, & Savings
Learn how decoupled observability helps you do more with your Splunk data, reduce costs, and scale efficiently with Federated Search.
Watch nowKeynote: Powering Event-Driven Data with Apache Druid
The distinction between OLTP and OLAP is becoming less relevant as data architectures shift toward entities and events. In this session, we’ll delve into how Apache Druid’s event-first approach synthesizes...
Watch now