Living the Stream

Imply and Confluent are Stream Locomotives!

We are in the early stages of a stream revolution, as developers build modern transactional and analytic applications that use real-time data continuously delivered. The batch processes of past were forced upon us by the limitations of mainframe, minicomputers, and other technologies of past. As we move forward with the Internet, cloud computing, and the always-on world, streams are rapidly emerging as a key enabling technology. This was on display in London, at the Kafka Summit organized by Confluent and sponsored by Imply.

Apache Kafka

About a decade ago, a team at LinkedIn developed scalable, reliable tools to manage streaming data, naming it in honor of a favorite writer, Franz Kafka. Released as open source software, Kafka quickly became popular and widely used.

The creators of Kafka realized that there was a need for a company dedicated to helping others use, deploy, and run Kafka, so they created Confluent.

Today, Confluent provides a cloud-native experience, completing Kafka with a holistic set of enterprise-grade features to unleash developer productivity, operate efficiently at scale, and meet each customer’s architectural requirements before moving to production. 

Apache Druid

At the same time, another team at another technology company (now part of Snap) needed to quickly aggregate and query real-time data coming from website users across the Internet to analyze digital advertising auctions. This created large data sets, with millions or billions of rows.

They first implemented their product using relational databases, starting with Greenplum,  a fork of PostgreSQL. It worked, but needed many more machines to scale, and that was  too expensive.

They then used the NoSQL database HBase populated from Hadoop Mapreduce jobs. These jobs took hours to build the aggregations necessary for the product.  At one point, adding only 3 dimensions on a data set that numbered in the low millions took the processing time from 9 hours to 24 hours.. 

So, the team did something “crazy”: they created a new database and named it Druid. The first incarnation of Druid scanned, filtered, and aggregated 1 billion rows in 950 milliseconds.

Released as open source software, Druid quickly became popular and widely used. The creators of Druid realized that there was a need for a company dedicated to helping others use, deploy, and run Druid, so they created Imply.

Today, Imply provides the complete developer experience for Apache Druid – delivered as a fully-managed, hybrid-managed, and self-managed product. It builds on the speed and scalability of Apache Druid with committer-driven expertise, effortless operations, and flexible deployment to meet developers’ application requirements with ease.

Many customers use Confluent and Imply together. Confluent makes it easy to deploy and use Kafka streams to move data between services and systems, while Imply makes it easy to deploy and use Druid to analyze streams and enable people and machines to make better decisions.

IronSource

At the Kafka Summit, IronSource shared how they use the combination of Confluent and Imply.

As the leading business platform for the app economy, IronSource provides an array of services to monetize and scale applications, all using streams powered by Confluent and real-time dashboards powered by Imply.

Conference delegates heard from Elad Eldor and Or Anon how IronSource operates at scale.

Confluent + Druid

Confluent and Imply also use each other’s technologies: Confluent Cloud uses Druid to power real-time dashboards both for its own operations and for customers to understand and optimize their own stream operations. Imply Polaris, the Druid database-as-a-server, uses Confluent Cloud to manage stream delivery for data ingestion.

We’re on the cusp of data streams shifting from an emerging architecture to the new normal, with the change driven by developers worldwide. Confluent and Imply are two of the locomotives accelerating the change.

Are you a developer ready to jump onto the streaming train? Why not try a free trial of Imply Polaris – Druid Database-as-a-Service? There’s no commitment needed, not even a credit card – just your commitment to helping yourself and your team succeed in the stream-driven world!

© 2022 Imply. All rights reserved. Imply and the Imply logo, are trademarks of Imply Data, Inc. in the U.S. and/or other countries. Apache, Apache Druid, Druid and the Druid logo are either registered trademarks or trademarks of the Apache Software Foundation in the USA and/or other countries. All other marks are the property of their respective owners.