Delta Lake is an open-source storage layer that brings reliability and performance to data lakes. In this session, we will explore the fundamentals of Delta Lake and demonstrate how to ingest Delta Lake tables into Apache Druid for sub-second query latency. Additionally, we’ll introduce a new upcoming feature in Apache Druid: scheduled batch supervisors, which can facilitate continuous data ingestion natively in Druid, ensuring that your Druid tables stay up-to-date with the latest data from Delta Lake or any input source.
Speakers:
– Abhishek Balaji Radhakrishnan, Staff Software Engineer, Imply
– Venki Korukanti, Software Engineer, Databricks
[Timestamp] Table of Contents:
Introduction
[0:00] Agenda
[1:17] Motivation
[3:21] Overview of Delta Lake
Integrating Delta Lake
[4:10] Delta Lake: How does it work?
[9:36] Advanced Delta Feature: Deletion Vector
[11:52] Delta Kernel
[14:35] Integration Architecture
[15:46] DML Query Example
[17:29] Scheduled Batch Ingestion
[20:05] Demo
[24:34] Roadmap
Keynote: Powering Event-Driven Data with Apache Druid
The distinction between OLTP and OLAP is becoming less relevant as data architectures shift toward entities and events. In this session, we’ll delve into how Apache Druid’s event-first approach synthesizes...
Watch nowClosing Keynote: Charting the Future of Druid
What lies ahead for Apache Druid? Join us as we explore the evolving landscape of Druid’s query and storage engines, and how they are positioned to address the biggest challenges in event data for the future. Speaker: Gian...
Watch nowSalesforce: Tracing Service Dependencies at Scale with Druid and Flink
At Salesforce, we manage approximately 300 million distributed spans to infer service dependencies. We have successfully utilized a combination of Druid and Flink to handle this scale with high availability....
Watch now