Currently Druid has two result formats: a JSON format and a protobuf-based format using Apache Calcite Avatica. Both of these formats are row-oriented while all of Druid’s internal data representations are column-oriented, as expected for high-performance analytics. This means that performance is being left on the table due to these transpositions.
This talk will describe a host of possible benefits for Apache Druid that would come by supporting Apache Arrow as an output format. Apache Arrow is an in-memory, columnar data format. It is used as the backing memory representation for utilities like pandas and polars, and allows for zero-copy data communication across many libraries. This session will cover various ways that we could improve the performance, interoperability and flexibility of Apache Druid, along with making it easier to integrate Druid with new data sources and analytics pipelines by leveraging Arrow, FlightSQL and ADBC (Arrow Database Connectivity).
Given the competitive landscape of data computation engines right now (Snowflake, BigQuery, Druid, Dremio, DuckDB), embracing support for Arrow will help Druid “keep up with the competition” and open doors for enhanced connectivity and performance!
Keynote: Powering Event-Driven Data with Apache Druid
The distinction between OLTP and OLAP is becoming less relevant as data architectures shift toward entities and events. In this session, we’ll delve into how Apache Druid’s event-first approach synthesizes...
Watch nowClosing Keynote: Charting the Future of Druid
What lies ahead for Apache Druid? Join us as we explore the evolving landscape of Druid’s query and storage engines, and how they are positioned to address the biggest challenges in event data for the future. Speaker: Gian...
Watch nowSalesforce: Tracing Service Dependencies at Scale with Druid and Flink
At Salesforce, we manage approximately 300 million distributed spans to infer service dependencies. We have successfully utilized a combination of Druid and Flink to handle this scale with high availability....
Watch now