In this session I would like to talk about the huge amount of data we ingested into Druid (raw data is 9 Terra per day) by using EMR, all orchestrated by Airflow. While the data grew we started experiencing many problems. After trying many scaling options, we decided to change the approach and came up with KUDRAS, the Kubernetes Druid Autoscaler.
This project is written in Python and is being used in our Apache Druid production environment. KUDRAS is a service developed using fastAPI which scales middlemanager nodes up and down in the most effective way, minimizing ingestion task costs to the bare minimum while maximizing ingestion speed.
Imply Lumi Observability Warehouse Demo
In this 30-minute session, you'll see a demo of Imply Lumi — the observability data layer built to help you store more, search faster, and reduce cost without changing your existing tools.
Watch nowLunch & Learn: Imply Lumi Observability Warehouse Demo
In this 30-minute session, you'll see a demo of Imply Lumi — the observability data layer built to help you store more, search faster, and reduce cost without changing your existing tools.
Watch nowImply Lumi: What’s New, What’s Next — and How to Unlock More Observability Value Today
Observability teams must retain more data, investigate faster, and control costs without disrupting existing tools. This live Imply Lumi update shows new ingestion, retention, search, Splunk interoperability,...
Watch now