This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community. We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our graphana dashboards. Finally we will list all the latest awesomeness in Druid 25.0 that helped us in this endeavour, how we discovered those midnight and our learnings.
Physical Hardware, Digital Analytics: IoT Challenges, Best Practices, and Solutions
Electric vehicle maker Rivian and German startup Thing-it were kind enough to talk us through how real-time data and analytics play a key part in the evolving landscape of IoT (Internet of Things). The wealth...
Watch nowAnalyzing streaming data with Apache Druid
Streaming data is not only data in motion—it’s a potential source of valuable insights, ready to be harvested and utilized. The challenge is to analyze streaming data at scale and extract these insights—before...
Watch nowReal-Time Analytics in the Real World
Engineering teams increasingly have to deliver insights in real-time. But as they aim to reduce latency from event-to-insight, they also face the challenge of dealing with larger and more complex data and concurrent...
Watch now