Smart Devices, Intelligent Insights: How Rivian and Thing-it use Apache Druid for IoT Analytics

The opportunity (and challenge) of the Internet of Things (IoT) has always been its ability to gather data from physical hardware, analyze it with digital software, and provide insights for teams and organizations to act on. To complicate matters, these conclusions can be time sensitive—expiring in seconds or minutes.

However, as more industries, from manufacturing to energy to healthcare, adopt IoT devices, the amount of data grows exponentially. If each IoT device can generate multiple events per second, when that is compounded across a network consisting of hundreds or thousands of devices, data could easily grow to terabytes or petabytes in volume.

To better understand how IoT users are dealing with this challenge, Eric Tschetter, the Chief Officer of Emerging Solutions at Imply, sat down for a conversation with Principal Architect Pramod Immaneni and Staff Software Engineer Connie Wang from Rivian, as well as Software Architect Robert Sauer from Thing-it. These companies are two leaders in the IoT space—Rivian’s electric smart vehicles are critically acclaimed, and Thing-it’s smart asset management platform has won a number of industry awards.

Here are some insights from their discussion.

The road to Apache Druid

The panel began with a very simple question: how did Rivian and Thing-it decide on Apache Druid? What was the impetus behind their switch from their initial databases to Druid?

In both cases, it had to do with the growth of their application and evolution of their business model—specifically outgrowing a database that had worked well in the past. 

“We started the first PoC of Druid to replace our previous system,” Connie Wang explained, with query performance being their biggest concern. At the time, Rivian was collecting data from “mostly prototype, pre-production models—[a] very limited set of engineering test vehicles. Even with that small quantity, [the prior database] was already hitting lots of limits with certain queries for even just one day of data.” 

This delay, which Wang estimated to be as high as 15 to 20 seconds, drastically impacted downstream applications, which were primarily “dashboarding applications” or those that relied on “quick, low-latency queries.” However, Wang and her team found that by switching to Druid, these exact same queries retrieved data in 500-700 milliseconds—a substantial improvement from the 20 second average.

Rivian’s Pramod Immaneni agreed, and mentioned an additional use case. “Anywhere a user dashboard is involved and they need to access telemetry data,” Immaneni added, “I think telemetry setup becomes very useful because users cannot wait…even for a few seconds. The user experience becomes very bad if they have to wait.”

For Thing-it, their primary concern was to simplify developer duties, though they did also appreciate Druid’s rapid query responses. Because their application executed a wide range of query types—both batch queries as well as “real-time or near real-time queries”—developers encountered significant complexity and overhead, fine-tuning indexes and even different schemas for individual customers.

As a result, Sauer and his team sought “a data system that…offers us enough headroom for the growth we are expecting and we’re aiming for that simplifies management and is much more cost effective.” Their database also needed to provide data isolation for customers, thorough technical support, and schema flexibility—all requirements that Druid fulfilled. 

Planning for the future was also particularly important for Sauer. “As of now, we have a relatively small schema with just a central set of extra properties,” Sauer explained. “But we already have use cases…to add more metadata here and there…and build out the schema.” Importantly, Druid enables the Thing-it team to grow without having to undertake big, painful initiatives like reindexing data or migrating years of data.

Unique challenges

While IoT is a catchall category for any applications that blend digital analytics with physical devices, in truth it spans a huge number of industries and sectors. Therefore, both Rivian and Thing-it had to contend with very different obstacles and difficulties.

For Rivian, that was unpredictability due to late arriving data. As Wang explained, “These are [adventure vehicles] that drive off into the wilderness with no connectivity for a week. And so that also comes with less predictability of what we can expect of data arriving in real time. Sometimes our data arrives late…[and] as we scale, potentially could introduce new production challenges.”

Another unique use case for Rivian was over-the-air (OTA) software updates—specifically determining which features have issues, the root causes of these problems, and which fixes should be released in each patch. “When users report an issue, there is a chance that it is an issue that could be resolved with an OTA,” Wang explained. “Being able to do the remote diagnostics without needing the customer to drive to a service center is a huge benefit…data-driven decisions are really useful to the company to prioritize [updates].”

Location was also a consideration. Rivian’s customers were primarily based in North America. As a result, times of peak demand matched up to the North American time zones—during these periods, vehicles produced the most data and queries were heaviest. Nighttime across North America, Wang noted, was when there was low traffic.

One unexpected use case was fleet management. “Rivian also sells delivery vans to Amazon,” Wang said. “Amazon has a fleet of…10,000 delivery vans that are electric and powered by Rivian [and used for] last mile deliveries. On the fleet side, there’s more of an aggregate view, a summary of a fleet instead of querying a single vehicle at a time.”

Because Thing-it is a smart management platform, they provide data access to customers through analytics tools and visualizations. Building managers retrieve weekly, monthly, or quarterly reports, aggregating data to determine different performance indicators such as occupant density, frequency of use, floors with the most occupants, and so forth. Users can even compare metrics such as occupancy or energy usage across multiple sites or floors.

Thing-it’s platform also includes some unexpected—and extremely useful—features. “One part of it is real-time information for users,” Sauer explained. “Imagine an employee getting to the office and looking for a desk to sit at. They may simply get an overview of what is already taken or the density in a certain area of the building. [Or] they may want to know where somebody with a certain expertise is located.”

Running Apache Druid

Both companies also had some interesting reasons for their choice of Druid deployments. 

“We’re running Druid on EKS [Amazon Elastic Kubernetes Service],” Wang explained, a straightforward solution for the Rivian team, given the ease of use of Amazon’s managed services as well as Rivian’s institutional familiarity with Kubernetes, which is the foundation for many of their other microservices. 

Because Thing-it is based in Germany, they comply with the European Union’s General Data Protection Regulation (GDPR), which provides exacting specifications around data privacy. “The hybrid model Imply provides is really, really good for us,” Sauer said, “because it means we still own the data in full—but we can benefit from the operational expertise Imply provides.” 

What’s next?

Given the success of Druid at these organizations, Tschetter was curious where they would go next—which new features they would adopt, or what new directions they would take for their Druid deployment. His guests provided some surprising answers.

“One thing we’re excited to play with is the time series analytics,” Sauer said, which will provide richer data analyses. “Specifically, a lot of the things we are tracking are effectively stateful…like a curve that is being sampled. We don’t track all the points in between, but we have an idea how the curve is behaving.”

In these circumstances, Sauer and his team find that it is helpful to have flexibility in modeling these curves based on sample data. In turn, this facilitates operations like smart averages and real averages—which go beyond simply adding and dividing but also incorporate an element of time into the analysis.

Because Thing-it is working on providing more powerful analytics to more users, Sauer and his team also find enriched data to be important. “Getting all the metadata joined,” Sauer says, will provide a “really enriched stream of data, partially pre-aggregated so [users] can start building on top of this, [get] additional insights…and [generate] even more interesting use cases.”

For his part, Immaneni wants to increase Druid usage at Rivian. “I think a lot of the data processing and analytics happen on the data lake, [where] people are more comfortable or familiar…So just showing sort of the awesome power of Druid (especially when it comes to real time), growing, putting more data sets on Druid.”

Immaneni pointed out that Rivian’s usage of Druid has grown significantly, especially when it comes to the numbers. “Now we have trillions of rows,” he said, “we’re able to query that at subsecond latencies and then…running everything stably in Kubernetes, and getting ingestion all going at high throughputs. The next goal is to expand the base.”

Wang agreed, and believed that the scale of Rivian’s data will only grow. “[In] 2022, we produced 25,000 vehicles, and [in] 2023 we produced slightly more than 50,000 vehicles. As the production rate continues to ramp, that’s still not the full capacity of our factory.”

For the full panel discussion, watch the webinar of their discussion.

For the easiest way to get started with Apache Druid, sign up for a free trial of Imply Polaris, the fully managed, Druid database-as-a-service.