Real-time analytics architecture with Imply Polaris on Microsoft Azure
This article provides an architectural overview of how Imply Polaris integrates with Microsoft Azure services to power real-time analytics applications.
Learn MoreWe are excited to share the latest enhancements in Imply Polaris, introducing time series analysis to revolutionize your analytics capabilities across vast amounts of data in real time.
Before we dive into these updates, a quick overview for those new to Polaris: Imply Polaris is a hassle-free, fully managed cloud environment that serves as the “easy button” for Apache Druid®. This database-as-a-service brings you all the performance advantages of Druid, plus add-in auto-scaling capabilities for seamless data ingestion, and an easy interface to build analytics applications that can be customized to visualize your time series data in real time. Within minutes, you can derive valuable insights from all your data without the complexities of procuring and managing infrastructure.
Starting with the basics, a time series is a sequence of events collected at successive points in time, typically expressed at uniform intervals. Analyzing time series data helps businesses track changes over time to identify trends and make predictions to maintain stable operations, drive business growth, reduce expenses, and optimize experiences.
Time series analysis is relevant for a broad range of use cases across IoT, gaming, finance and other industries. For example, time series analysis is used in finance for stock price prediction and risk management, helping investors make informed decisions. In manufacturing, it optimizes supply chain operations by forecasting demand and managing inventory levels. In IoT applications, it analyzes data from sensors like thermostats and manufacturing monitors to predict equipment failures and optimize energy consumption.
While compiling data for time series analysis might seem straightforward, it’s actually quite complicated in practice. There are two main factors to consider:
Time series data processing flow without vs. with Imply
That’s why we’ve introduced time series aggregation for Polaris. This feature works at persist-time to significantly reduce the data footprint of time series data and also reduces operational complexity by introducing mission-critical solutions for time series functions and visualization:
There are numerous applications for how these functions can be leveraged, combined and visualized, whether for real-time alerts or deeper analysis. To understand how they work in practice, check out the following mini demo or read on to dive deeper into an IoT scenario.
Demo Video: IoT Time Series Analysis with Imply Polaris
Imagine you work for a Smart Grid provider that is focused on optimizing energy distribution across ever-changing supply and load demands. Your goal is to enhance transmission and distribution efficiency and prevent outages while integrating renewable (but sometimes unreliable) energy sources seamlessly into the grid. With thousands or millions of IoT sensors deployed across your infrastructure, you face challenges in managing vast amounts of time series data to monitor energy consumption, predict maintenance needs, and ensure grid stability.
To contextualize these challenges, let’s say you’re working with three separate data sources:
With other systems, the first reaction is to try to model each of these data sets in their own independent table. Because they are independent schemas, this adds to operational overhead and complicates queries, as each new source of data or integration source is likely to add yet another table that needs to be managed. This is where the power of auto-detected schemas shines: We can very simply add all of this data into the same table and just let each source have its own independent schema.
Challenge: As your business scales, understanding data from thousands or millions of sensors across separate systems becomes increasingly burdensome. This results in significant performance issues, especially when attempting to view aggregated information across all assets or combining assets across locations. Without a real-time, scalable solution, it becomes nearly impossible to derive timely insights from sensor data in order to monitor and manage energy consumption patterns.
Solution: Time Series Aggregation enables data aggregation at various intervals, such as minutes or hours, significantly enhancing query performance to manage energy flows in real time. Polaris makes it easy to run fast queries with aggregates and roll-ups while preserving access to every granular event.
Example: Let’s start by creating a time series for power demand with a DOWNSAMPLED_SUM_TIMESERIES() function. If we want to show demand every hour in the first half of 2024, we can use:
SELECT
MAP_TIMESERIES(
DOWNSAMPLED_SUM_TIMESERIES("__time","consumed_wattage", '2024-01-01T00:00:00.000Z/2024-07-01T00:00:00.000Z', 'PT1H'),
'value / 1000.0'
) as demand_kwh
FROM table
WHERE source = 'consumer'
Here, we are using downsampling, which totals the data collected during each time period (in this case, 1 hour, using the ISO 8601 designation of ‘PT1H’). Note how in this one query we will get the combined values across all residential and commercial consumers, regardless of if there are 10 or 10 million consumers.
But it may be inaccurate, as, in real life, not every message from every sensor successfully arrives where it should. This means that some of the values we’re summing will be zero, which will give us a demand total that is lower than it should be. That’s where interpolation comes in.
Challenge: Data from IoT devices doesn’t always arrive reliably, so there are occasional gaps or other irregularities in the data stream, producing challenges when analyzing data. If the missing entries are treated as “zeros”, then it will produce incorrect results in analysis.
Solution: Interpolation estimates the values of missing data points in the time series. Interpolation can be used to resample or upsample the time series to a higher frequency, making it easier to analyze or compare with other time series sampled at different rates. Whether filling gaps in sensor readings, smoothing irregularly sampled data, or capturing complex patterns, interpolation enhances the reliability of insights derived from IoT data, contributing to more informed decision-making processes.
Example: While Imply supports multiple options for interpolation, the most common choice is linear interpolation, which fills in data gaps with a common mathematics function.
SELECT
Meter_id,
LINEAR_BOUNDARIES(
TIMESERIES("__time", "ambient_temperature", '2024-01-01T00:00:00.000Z/2024-07-01T00:00:00.000Z'),
‘PT1H’
) as demand_kwh
FROM table
WHERE source = 'consumer'
GROUP BY meter_id
Here, we are using linear interpolation to get an hourly indication of the ambient temperature across all of the meters at the “minute boundaries” (the 00 second of every minute).
Challenge: Sometimes, you may have multiple IoT sensors or devices measuring different aspects of the same phenomenon or process. For example, you may have different sensors to measure voltage, current, temperature, load, power quality, supply and demand.
Solution: By adding or subtracting time series data, you can combine these signals to get a more comprehensive representation of the overall system or process.
Example: Comparing two IoT data streams can aid in anomaly detection or fault analysis. In this case, we want to compare supply with demand to be sure we aren’t running low, and risking blackouts or brownouts.
WITH solar as
( SELECT LINEAR_INTERPOLATION(DOWNSAMPLED_SUM_TIMESERIES("__time","AC_POWER",'2024-01-01T00:00:00.000Z/2024-06-30T23:59:59.999Z', 'PT1M'),'PT1H') as supply_kwh FROM SOLAR_GENERATION )
SELECT ADD_TIMESERIES(solar.supply_kwh,
LINEAR_INTERPOLATION(DOWNSAMPLED_SUM_TIMESERIES("__time","AC_POWER",'2024-01-01T00:00:00.000Z/2024-06-30T23:59:59.999Z', 'PT1M'),'PT1H') as supply_kwh FROM GRID_SUPPLY )
From here, we can use SUBTRACT_TIMESERIES() to subtract demand from supply to get a time series that shows how much buffer power (or deficit) we have for each hour.
SELECT
SUBTRACT_TIMESERIES(
ADD_TIMESERIES(
DOWNSAMPLED_SUM_TIMESERIES("__time","output_AC", '2024-01-01T00:00:00.000Z/2024-07-01T00:00:00.000Z', 'PT1H') FILTER (WHERE source = 'solar'),
DOWNSAMPLED_SUM_TIMESERIES("__time","output_AC", '2024-01-01T00:00:00.000Z/2024-07-01T00:00:00.000Z', 'PT1H') FILTER (WHERE source = 'grid')
),
DOWNSAMPLED_SUM_TIMESERIES("__time","consumed_wattage", '2024-01-01T00:00:00.000Z/2024-07-01T00:00:00.000Z', 'PT1H')
) AS excess_capacity
FROM table
WHERE source = IN('consumer', grid', solar')
While the use cases above were focused on a hypothetical IoT scenario, you can check out this blog to explore a real example of how Inowatts was able to simplify utility management.
Time series analysis is also applicable to numerous other industries across IoT and beyond. Here are a few more concrete examples that may be relevant for your business:
This example visualization applies the TIMESERIES function by filtering the data to a specific period. The time column on the x-axis is Time, with Number shown on the y-axis. The data is grouped by Browser. The visualization displays the top three browsers with the largest number of events. There’s a point on the chart for every 5 minutes as set in the time series bucket.
Whether you’re new to time series analysis or trying to optimize current processes, we’re here to help.
Start your journey with a free 30-day trial of Imply Polaris – no credit card required! Or, take Polaris for a test drive and experience firsthand how easy it is to build your next analytics application.
If you have questions or want to learn more, set up a demo with an Imply expert. We’d love to help you make the most of Imply Polaris for your real-time analytics needs.
Real-time analytics architecture with Imply Polaris on Microsoft Azure
This article provides an architectural overview of how Imply Polaris integrates with Microsoft Azure services to power real-time analytics applications.
Learn MoreUsing Upserts in Imply Polaris
Transform your data management with upserts in Imply Polaris! Ensure data consistency and supercharge efficiency by seamlessly combining insert and update operations into one powerful action. Discover how Polaris’s...
Learn MoreMake Imply Polaris the New Home for your Rockset Data
Rockset is deprecating its service—so try Imply Polaris, built on Apache Druid, a popular database designed for real-time Kafka and Kinesis streaming, and highly concurrent, subsecond queries on petabytes...
Learn More