Currently Druid has two result formats: a JSON format and a protobuf-based format using Apache Calcite Avatica. Both of these formats are row-oriented while all of Druid’s internal data representations are column-oriented, as expected for high-performance analytics. This means that performance is being left on the table due to these transpositions.
This talk will describe a host of possible benefits for Apache Druid that would come by supporting Apache Arrow as an output format. Apache Arrow is an in-memory, columnar data format. It is used as the backing memory representation for utilities like pandas and polars, and allows for zero-copy data communication across many libraries. This session will cover various ways that we could improve the performance, interoperability and flexibility of Apache Druid, along with making it easier to integrate Druid with new data sources and analytics pipelines by leveraging Arrow, FlightSQL and ADBC (Arrow Database Connectivity).
Given the competitive landscape of data computation engines right now (Snowflake, BigQuery, Druid, Dremio, DuckDB), embracing support for Arrow will help Druid “keep up with the competition” and open doors for enhanced connectivity and performance!
Druid Summit 2024 – Panel: Real-Time Data Experiences
Analytics applications powered by Apache Druid are fun places to be. Take a seat as user experience leaders from Google, the Data Visualisation Society, and Imply bring their minds together, and answer your...
Watch nowDruid Summit 2024 – Panel: Lakehouse Analytics
Apache Iceberg and Delta Lake are coming into the big top, and Apache Druid is ready! Get ready to hear from Apache Druid PMC members and industry experts on just where Apache Druid fits, how it can be used,...
Watch nowDruid Summit 2024 – Panel: Operations and Optimization
Come along and meet people who have implemented and tuned Druid in situations small and large – very very large! Panelists will share some of their key tips and tricks, and be open to your questions, whether...
Watch now