In Apache Druid 0.18/Imply 3.3, we added support for SQL Joins in Druid. This capability, which has long been demanded from Druid by the community, opens the door to a large number of possibilities in the future. In this blog I want to highlight some of the motivations behind us undertaking the effort and give you, the reader, an understanding of how it can be useful and where we’re going with it.
Read MoreIn Apache Druid, Compaction basically helps with managing the segments for a given datasource. Using compaction, we can either merge smaller segments or split large segments to optimize segment size. One of the first options to consider would be to determine, if the segments could be generated optimally. If that isn’t possible, compaction would be required.
Read MoreThe Apache Druid community released Druid 0.18 on April 20th, 2020. This release contains over 200 new features, performance enhancements, bug fixes, and major documentation improvements from 42 contributors.
Read MoreWhen an engaged technical audience asks great questions, it’s easy to run out of time during Q&A. And that’s exactly what happened at Virtual Druid Summit! Because our speakers weren’t able to address all of your questions during the live sessions, we’re following up with the answers you deserve in a series of blog posts.
Read MoreThis short post describes how Druid compares against enterprise data warehouses. Druid is not a data warehouse. It's a real-time database for user-facing analytics application needing sub-second query response at high concurrency.
Read MoreThanks again to everyone who attended Virtual Druid Summit, and for being so engaged – as we previously mentioned, our speakers received more than 150 questions across their collective sessions! Unfortunately, there wasn’t enough time to answer all of your very good questions during the live sessions. In an effort to bring you some closure, we’ve invited our esteemed speakers to address the remaining questions in a series of blog posts.
Read MoreI want first to thank everyone involved with making our first, albeit a tad non-standard, Druid Summit a smashing success. We, and the hundreds of folks who had purchased tickets, were extremely disappointed to postpone our first physical Druid Summit to November, but we were fortunate to have 10 great speakers agree to do their presentations virtually.
Read MoreWe are excited to announce the general availability of Imply 3.3, which greatly expands the expressivity of SQL by adding the ability to do JOINs and interact with lookups in novel ways. This release includes over 200 new features, performance enhancements, bug fixes, and major documentation improvements.
Read MoreIn this post, I am going to talk a bit about Apache Druid and a recently documented configuration option that enables true NULL values to be stored and queried for better SQL compatibility: druid.generic.useDefaultValueForNull=false, and in the process do a deep dive into how it relates to a small sliver of the query processing system as we explore the performance of this feature.
Read MoreI am happy to tell you that 10 days after we officially postponed Druid Summit, we have now launched Virtual Druid Summit, which will take place as a series of online talks on April 15. Each talk will be a spicy 30 minutes of real-world information, followed by Q/A. There are 5 talks from Druid practitioners from a variety of industries and spanning 3 continents. The summit will open with a Apache Druid Roadmap and Vision talk from Apache Druid PMC Chair Gian Merlino, and will close with a 2-way voice-interactive “ask us anything” session featuring Druid authors and contributors.
Read MoreTijo Thomas, a Solutions Architect at Imply, recently wrote a reference architecture for Apache Druid on Microsoft Azure that includes some best practices for running on services such as Azure VM, Azure Blob Storage, Azure Database Service and HDInsight.
Read MoreAnalyzing the potential petabytes or more of data from all these devices goes way beyond existing data warehouses or data lakes. Fortunately companies have already implemented IoT analytics using Imply, the real-time intelligence platform built on Apache Druid, the leading open source real-time time analytics database.
Read MoreI’ve been leading engineering and product at Imply for 5 months now, and every day is more exciting than the one before. Just before Imply, I was a Partner at Andreessen Horowitz, directly involved in the investments in tens of companies and on the board of many. Many wonder: "having seen hundreds of companies at Andreessen Horowitz, why did I choose Imply?” The truth is it would have been a mistake not to.
Read MoreWe are delighted to announce that Imply 3.2 is now available! Imply 3.2 is based on Apache Druid 0.17 (Druid’s first Apache top level project release) and adds new cloud management, alerting, reporting, and data loading features.
Read MoreEarlier this week, the Apache Druid community released Druid 0.17.0. This is the project’s first release since graduating from the Apache Incubator, and it therefore represents an important milestone.
Read MoreMuthu Lalapet, a Solutions Architect at Imply, recently wrote a reference architecture for Apache Druid on Google Cloud Platform (GCP) that includes some best practices for leveraging GCP services such as Compute Engine, Cloud Storage and Cloud SQL. The document describes example cluster architectures and their accompanying machine types and configurations. As such, it’s a helpful resource for planning and implementing Druid on GCP.
Read MoreIf you are a Vertica customer, you probably already know this. Vertica is not built for real-time operational analytics at scale. If you do not know Vertica very well, you might be surprised. This statement may seem controversial. It’s not. Nearly ¼ of Imply customers were existing Vertica customers who purchased Imply, a commercially supported version of Apache Druid, because they were trying to implement operational analytics and hit limitations with Vertica. Other Vertica customers also use open source Druid and self-support.
Read MoreIf you are using Apache Druid to analyze customer-oriented data you are probably familiar with the General Data Privacy Regulation (GDPR), which went into effect May 25, 2018. However, you may be less familiar with a new law, the California Consumer Privacy Act (CCPA), which went into effect January 1, 2020 and is likely become a *de facto* standard in the US.
Read MoreToday, Imply announced a new round of growth funding raised from Andreessen Horowitz and Geodesic Capital. First I want to thank these firms for their support and for sharing our vision as we continue to grow. Thank you, too, to all of our customers for joining us on this journey. A milestone like this is a good time to reflect on the past and the future.
Read MoreI’m excited to announce that Imply has raised a $30M Series B funding round led by Andreessen Horowitz, with participation from Geodesic Capital and Khosla Ventures. We are excited to partner again with A16Z as the lead investor to continue to build out the company, and we look forward to working closely with Geodesic Capital towards international expansion.
Read MoreNielsen Marketing Cloud uses Druid to profile the various audiences that marketers and publishers would like to target on digital media, activate via various ad networks, and then gain insights on that activation after the fact.
Read MoreIn the 0.16 console, we have added a new layer of SQL awareness to the query view to help move the view away from its roots as a text-only interface, to a point and click one.
Read MoreWe are delighted to announce that Imply 3.1 is now available! Imply 3.1 is based on Apache Druid 0.16 and contains many improvements.
Read MoreLike every Druid release, this one has a huge amount and variety of fixes and new functionality. This particular one include over 350 new features, performance enhancements, bug fixes, and major documentation improvements from 50 contributors.
Read MoreApache Druid is commonly used for clickstream funnel analysis, and in this blog post we’ll deep dive into how you can collect and analyze funnel data. While there are applications designed for clickstream analysis, such as Google Analytics and Adobe SiteCatalyst (previously Omniture), Druid is ideal when you have significant scale.
Read MoreBelow is a transcript of a short interview we conducted with Chaitanya Bendre, Lead Data Engineer at Zeotap, where we discussed there use of Druid to help address the difficult problem of identity resolution and multi-channel attribution.
Read MoreBlueshift is an AI-powered customer data activation platform enabling CRM and product marketers to intelligently manage their audiences and orchestrate large-scale personalized messaging campaigns at scale. Blueshift offers real-time campaign analytics as a core capability in the platform. Campaign analytics break down engagement metrics like impressions, clicks, conversions etc by channel, trigger, experiment etc. Currently two billion+ user interactions are tracked on a monthly basis.
Read MoreA recent paper by independent researchers at the University of Minho in Portugal compared the performance of Apache Druid to well-known SQL-on-Hadoop technologies Hive and Presto. In the tests, Druid outperformed Presto from 10X to 59X (90% to 98% speed improvement) and Hive by over 100X.
Read MoreWe are delighted to announce that Imply 3.0 is now available! It contains many usability features such as a visual data loader, on-premises cluster and alerts functionality.
Read MoreWhen Hadoop is pushing data into Druid, Hadoop indexer performance is key and becomes challenging at scale. There are a quite a few things to consider when running large scale Hadoop indexing.
Read MoreThis document outlines the journey Zscaler made in building Zscaler Private Access (ZPA) and focuses on the analytics component of our solution. We will discuss some of our early requirements, why we picked certain technologies, such as Druid/Imply, and how we run things today.
Read MoreMoPub, a Twitter company, has just launched a new solution called MoPub Analytics based on Apache Druid and using Imply Pivot as the drag-and-drop UI. The solution allows users to determine the root cause of new data trends by interactively analyzing the data across many different time slices, dimensions, and metrics.
Read MoreToday, the Apache Druid community released Druid 0.15.0-incubating. Druid is known as an extremely high-performance database and much of the early design work has been focused on providing speed at scale. Lately we have made a pivot towards those “ease of” factors that help users get productive with Druid quickly.
Read MoreWe recently conducted our first Druid community survey. Every so often we’ll be asking our community a short set of questions to understand how they use Druid, and how they would like to see it improved.
Read MoreA triad of open source projects - Divolte, Apache Kafka and Apache Druid - can power real-time collection, streaming and interactive visualisation of clickstreams, so you can investigate and explore what’s happening on your digital channels as easily as looking out of your office window.
Read MoreAlthough Druid draws ideas from a number of TSDB concepts, it is designed for a wider range of analytic use cases than those for which a TSDB is usually employed.
Read MoreTo help you get to know GCP and Druid, the tutorial below will walk you through how to install and configure Druid to work with Dataproc (GCP’s managed Hadoop offering) for Hadoop Indexing. Then it will show you how to ingest and query data as well.
Read MoreIn this tutorial, we will step through how to set up Imply, Kafka, and Open-NTI to build an end-to-end streaming analytics stack that can handle Juniper Native streaming telemetry data.
Read MoreIn this tutorial, we will step through how to set up Imply, Kafka, and syslog-ng kafka to build an end-to-end streaming analytics stack that can handle many different forms of log data.
Read MoreBusinesses need to understand how their metrics change across many facets of their operations, and this is the core idea behind data analytics.
Read MoreImply 2.9 is based off of the just announced Druid 0.14. Druid 0.14 contains many new features, improvements, and bug fixes. This blog post will focus on the new Imply components and features not available in Druid 0.14.
Read MoreToday the Apache Druid community released Druid 0.14.0, our second release under the Apache umbrella and the first major release of 2019. I thought I'd take this opportunity to talk about what's new in this release and what's coming in the future.
Read MoreWalkMe uses Imply Cloud to monitor behavioral analytics for its leading Digital Adoption Platform.
Read MoreIn this tutorial, we will step through how to set up Imply, Kafka, and pmacct to build an end-to-end streaming analytics stack that can handle many different forms of networking data.
Read MoreHave you ever wanted more visibility in your AWS network traffic? This how-to blog covers how to analyze VPC flow logs with Imply.
Read MoreIf you are reading this because you are considering whether to use Apache Cassandra/DSE/ScyllaDB or Apache Druid/Imply, then you can just stop right now.
Read MoreTrafficGuard helps some of the world’s biggest digital advertisers and agencies protect their ad spend from fraud. Our clients need access to reliable reporting in real-time to allow them to optimise their ad campaigns with current insights.
Read MoreAt GameAnalytics, our user base has grown several times over in the past 12 months, and this growth has promoted us to rethink our user experience analytics system.
Read MoreOne of the key activities at the heart of any internet backbone is flow analytics, which enables visibility into global traffic for many technical, economical, and security use cases. By providing real-time traffic visibility and rapid explanation capabilities for this data, we unlock tremendous business value for the whole organization.
Read MoreImply 2.8 comes with the first Apache release of Druid and a host of features aimed at performance improvements and ease of use.
Read MoreWithin Druid there are multiple ways to enhance visibility for existing network flow records. This how-to blog covers one way to do this using Druid lookup tables.
Read MoreRubicon Project, one of the world’s largest digital advertising exchanges, has modernized their analytics stack with Druid and Imply.
Read MoreSecurity is a critical requirement in every deployment of a system that holds and processes data. In this blog post, we will discuss how we secured Apache Druid, and validated our implementation.
Read MoreIn Imply 2.7, we have added a selection of new visualizations and a new Explain feature that allows you to discover the contributing factors to any slice of data. We are also introducing advanced access control features, and have made several improvements to loading and managing data.
Read MoreWe ingested our internal AWS VPC netflows into Imply and found something surprising.
Read MoreImply 2.6 introduces time compares, data export, advanced aggregation measures, and more!
Read MoreImply 2.5 comes with an improved streaming data loader, a sunburst visualization, many dashboard improvements, and more.
Read MoreMarch 2018 Druid Bay Area Meetup - eBay Monitoring Platform, links to slides
Read MoreI'm excited to announce that Imply has raised a $13.3M Series A, led by Andreessen Horowitz, and joined by our seed investor, Khosla Ventures.
Read MoreToday, we are excited to announce that Imply Cloud, a fully managed service for AWS, is now generally available.
Read MoreImply 2.4 comes with a preview of a dataset manager, new features, improvements, bug fixes, and more!
Read MoreIt is now possible to deploy a druid cluster in a secure setting.
Read MoreNovember 2017 Druid Bay Area Meetup video, with talks from MZ and Slack, and a Druid roadmap update from Imply
Read MoreImply 2.3 comes with brand new apps, new features, improvements, bug fixes, and more!
Read MoreWe're excited to announce Imply 2.2, with tons of new features for Pivot.
Read MoreWe are extremely excited to announce Imply 2.1, a feature-packed release, available now for download. Native SQL comes to Druid and dashboarding is now possible in Pivot.
Read MoreOur last Druid meetup had great talks about how Druid is used at Branch, the ongoing work around better integrating Druid with the Hadoop ecosystem, and our roadmap plans.
Read MoreOur last Druid meetup at Sift Science had 3 great talks about use cases with Druid & Imply, and the upcoming Druid roadmap.
Read MoreWith the Druid 0.9.2 release, Druid has added additional column compression methods for longs to significantly improve query performance in certain use cases. In this blog post, we’ll highlight how these various compression methods impact data storage size and query performance.
Read MoreIt’s been some time since our last release of Pivot. Today, after much hard work, we are excited to announce version 0.10.27, which represents a significant step towards a more holistic data exploration experience.
Read MoreThe Druid community is pleased to announce our next major release, 0.9.2. We’ve added hundreds of performance improvements, stability improvements, and bug fixes.
Read MoreWe are extremely excited to announce Imply 2.0, our largest release ever, available now for download. This release contains significant updates to both Druid and Pivot.
Read MoreToday, many companies are turning to streaming solutions which are enabling them to understand and make business decisions from their data immediately, resulting in an operational agility that was unthinkable only a few years ago. The new Kafka indexing service is an exciting milestone in the maturity of Druid's ingestion technology, giving users a way to stream data into Druid with exactly-once correctness.
Read MoreWe are extremely excited to announce the next version of Imply Analytics Platform, IAP 1.3.0, available immediately on our download page. This is one of our biggest releases to date and includes major updates for both Druid and Pivot.
Read MoreWe are pleased to announce the latest version of the Imply Analytics Platform. This release focuses on improvements to Pivot and Tranquility as well as adding programmatic querying options to PlyQL.
Read MoreSome time ago at Imply, we launched PlyQL, a command line utility that provides an SQL-like interface to Druid via Plywood. We heard a lot of positive feedback as many people prefer to use SQL over Druid’s native JSON-over-HTTP interface. The most common question we hear about PlyQL is how one can interface to it programmatically either from user created apps or from existing SQL based BI tools.
Read MoreEverything is going to fail. If this is your first time working with or building out a distributed system, the fact that everything is going to fail may seem like an extremely scary concept, but it is one you will always have to keep in mind.
Read MoreI recently read a great article by Jeffrey Heer, Michael Bostock, and Vadim Ogievetsky that showcases the various techniques for visualizing and interacting with diverse data sets. I thought it may be useful to write something similar to showcase the various open source systems that exist in the “big data” space, including Druid, which is an open source data store I work on.
Read MoreA large part of what we do at Imply is help organizations build custom applications and visualizations on top of their data. While Druid is a powerful backend for powering applications, there are aspects of the development process that could definitely be easier. To enable people to better understand the power of Druid, we have released Pivot, an exploration UI that makes the most of the power of the Druid database.
Read MoreToday, Gian Merlino, Vadim Ogievetsky, and I are extremely excited to announce Imply, a company for interactive analytics at scale, centered around the Druid open source data store. We first began working on Druid at a startup called Metamarkets, and over the last few years, we’ve been proud to watch the project grow and take on a life of its own.
Read More