How to Monitor Your IoT Environment in Real Time

Today, interconnected smart devices are everywhere, from consumer appliances like thermostats and cars to industrial-scale applications such as wind farms or factories. 

As Internet of Things (IoT) adoption increases, it also generates increasing amounts of data. By one estimate, 2025 will see 55.7 billion IoT devices generating 80 zettabytes of data—approximately 80 billion terabytes. Needless to say, managing and monitoring all of this IoT data is a vast, expansive task.

But it’s also necessary. Beyond simply troubleshooting issues such as latencies or outages, teams can also rapidly access insights and even automate the decision making process, leading to improved safety, more precise predictive maintenance, reduced resource consumption, lower expenses, and more.

What is IoT monitoring?

At its most basic level, IoT monitoring includes the collection, organization, analysis, and management of IoT devices and networks, all to ensure continued performance, security, and efficiency. In general, this includes several steps:

Device discovery

The first step is to identify, catalog, and connect all IoT devices within a single ecosystem, which can be a complex initiative. After all, an IoT environment can consist of multiple models of devices, each generating different types of data and running various firmware. 

There are several types of IoT devices: sensors, which collect data; controllers, which analyze and act on data; and actuators, which are instructed by controllers to execute actions, such as closing a valve or releasing steam in a controlled manner. Controllers vary in capacity and size, ranging from a simple microcontroller that’s the size of a computer chip, to a more sophisticated system that could be the size of a modern internet modem. 

Device monitoring and intervention

Many IoT use cases, such as energy or wastewater treatment, run nonstop, with downtime only for maintenance. As a result, IoT sensors must constantly collect data, generating a continuous stream of timestamped event data for metrics such as power levels, operational states, connectivity, and more. 

Alerting and automation

Because failures can create safety consequences (power outages or oil well blowouts), teams must monitor their IoT environments in real time, so that they can alert on anomalies—and promptly intervene or troubleshoot any issues that arise. Therefore, IoT monitoring does require a real-time IoT database capable of rapid analytics, aggregations, and time series support.

An organization may also automate actions, especially if the safety margins are too tight for human intervention. One example is communications-based train control (CBTC), a digital train control system that uses computers to route train traffic and prevent collisions or other hazards. A use case like CBTC will likely automate actions such as braking or train spacing, perhaps providing computerized assistance (in the form of navigation or alerts) to conductors and dispatchers, or even directly intervening in case of human error.

A real-world example

For instance, a utility-scale solar installation, which includes panels, inverters, and battery storage, could have a large range of different devices, even on a single asset. One solar panel may have sensors to gather data on panel angles, a controller to assess production, and an actuator to tilt the panel, following the sun as it travels across the sky. Inverters, which convert the solar panel outputs into electricity, may also need sensors to monitor their energy efficiency. Lastly, any on-site battery storage also requires sensors to collect metrics like temperature and charge, and perhaps a controller and an actuator to discharge excess electricity into the grid to avoid overload.

Each of these devices generate different data types, such as temperatures, angles, voltage, sunlight intensity, state of charge, cycle count, and more. To process this data efficiently, a team will need a database model that can accommodate multiple data models, query and analyze this data instantaneously, and provide insights to act on immediately.

What are the benefits of IoT monitoring?

IoT monitoring opens up a new world of possibilities for everyone involved—teams, organizations, and customers alike. Without this capability, stakeholders cannot visualize their operations, compliance, or security. In fact, many of the key capabilities of IoT are impossible without strong, scalable monitoring.

The first benefit is system health and performance. Monitoring provides continuous, real-time visibility into the state of IoT devices, networks, and systems, helping teams ensure that they function as needed. This also facilitates the troubleshooting process—teams can quickly detect and preempt any issues that arise, preventing them from escalating into a larger crisis like a full-blown failure. For instance, a team can monitor their network routers, and if any fail or show signs of latency or other trouble, dispatch field technicians to fix them.

Another important benefit is optimizing performance. By gathering and analyzing IoT data, teams can come to conclusions and act on them, ideally in real time and with a degree of automation. For example, a fleet management solution could automatically reroute delivery or pickup vehicles without human intervention by relying on rapid, real-time insights and triggers linked to common issues such as traffic jams, delays, emergencies, and more.

Security can also be strengthened through IoT monitoring. Teams can detect security threats, vulnerabilities, and unauthorized access attempts in IoT networks, better identifying anomalies and protecting against cyberattacks and data breaches. This could take the form of monitoring software patches, firmware updates, and unusual transmissions or connections.

IoT monitoring also enables predictive maintenance, so that infrastructure can last longer and failures can be minimized. By analyzing past performance and trends, teams can determine wear and tear and fine tune maintenance for important infrastructure—like oil well drill bits or trains—for repairs and overhauls, extending service lives and maintaining safety margins.

Similarly, IoT monitoring enables cost optimization. Practices like predictive maintenance can make components last longer, but teams can also identify other inefficiencies, better utilizing resources and improving profitability. For instance, a smart HVAC system could alert operators on failing boilers or other equipment anomalies, or even turn off lighting or climate control in rooms or floors with no occupants.

End users will also benefit from IoT monitoring, which can analyze their usage, build patterns, and alter their behaviors to better suit human preferences. A smart home security system can automatically adjust settings based on resident schedules (perhaps someone is working the night shift and gets home early in the morning), or even distinguish between harmless events (the family cat returning through a pet door) and more suspicious ones.

Lastly, IoT monitoring enables data-driven decision making, helping teams come to conclusions that will improve their business model or organization. One example is data in farming—through data analytics, a farmer can refine their irrigation schedule, fertilizer process, and pest control, providing maximum crop yield for minimum resource usage.

What are the challenges of IoT monitoring?

While no IoT use cases are exactly the same, all IoT teams do face similar challenges and obstacles.

High volume of data

By nature, connected devices for IoT are always on and transmitting metrics, events, logs, and other types of data. Take the example of sensors on assembly lines or thermostats for both residential and commercial use: for safety and efficiency reasons, both devices must run continuously, without any downtime. If either device generates one event per second, over the course of a day, that would equate to approximately 86,400 events.

When compounded across an entire network of devices, this could result in a significant amount of data—perhaps too much for most databases to organize, process, and analyze, at least in a timely manner. Some legacy databases, such as those utilizing relational data models, have been known to struggle with large data volumes, forcing teams to find workarounds or manually scale their data infrastructure up and down.

High velocity

In IoT applications, data is created, ingested, and analyzed at high speeds, because insights are time sensitive and can expire rapidly. High volumes of data can hinder the high velocity at which IoT environments must perform.

Delays in ingestion, analysis, and retrieval can also have knock-on effects, especially on applications. Regardless of whether these applications are for customers or internal users, significant latency (up to several seconds or minutes) will make them unusable. Any lag will also bring serious consequences to downstream applications or components, such as a train control system or an oil drill controller.

High variety

IoT devices are not at all the same, simply because they carry out separate functions—a controller is not a sensor, and a sensor is not an actuator. Therefore, IoT devices likely generate (and work with) a wide range of data, which can be difficult to accommodate in a single database.

In fact, there is great variety even within IoT sensors working in the same environment, and  integrating these different data types will likely require changes to a database schema, perhaps with regular updates. A sensor on a network router may generate logs, while a sensor on a network switch might collect and transmit metrics concerning resource utilization, and port status. Further, due to inconsistent updates, even the same sensor types might run different firmware versions which collect different data fields.

Complexity only grows depending on the length and stages involved in a team’s data pipeline. For instance, if a data pipeline requires data to be exported, transformed, and loaded into their database before analysis or aggregations can be executed, then a team will need to add extra steps into their workflow—which could increase latency and further slow downstream applications. 

How does IoT monitoring work?

The actual IoT monitoring process is fairly straightforward: find devices, gather and send data, store and manage data, analyze data for insights, and finally, display data in a graphical format, such as charts or maps.

Device discovery

There are several ways to identify and register IoT devices, including network scanning, service discovery protocols, or simply just adding them to a centralized IoT platform. For instance, devices configured with protocols such as Universal Plug and Play (UPnP) and Multicast DNS (mDNS) will broadcast their presence locally for other nearby devices to connect.

After they are discovered, teams need to add unique identifiers, such as device IDs or MAC addresses, to devices and define metadata tags for each device, including device type, location, firmware version, and more. For security concerns, devices will need to be authenticated before they are allowed onto the network.

Data collection and transmission

IoT sensors will capture important metrics and then stream them to a database. These parameters will vary widely depending on industry: for instance, monitoring an assembly line might require temperature, error rates, or speed, while a team running a solar panel installation would need to look at angle, voltage and output, inverter performance, and sunlight intensity.

Streaming data (usually via Amazon Kinesis or Apache Kafka) is the preferred method of data ingestion, because it is very suitable for real-time speeds. Data will be transferred via protocols such as Messaging Queuing Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), or HTTPS—all of which balance security with efficiency and resource usage. Depending on an organization’s needs, they may also add an additional layer of encryption or even role-based access control (RBAC).

Data storage and management

After data is ingested, it has to be stored, either in the cloud, on premises, or a mix of the two. The exact mode of storage will depend on the company’s needs—for compliance purposes, some companies may be required to keep some (or all) of their data in on-premises servers, especially in highly regulated fields such as finance or healthcare. 

Otherwise, an organization can choose (or switch between) storage types as needed, taking advantage of the scalability, flexibility, and accessibility of cloud databases or settling for the data security and relatively higher degree of control of on-premises hardware. Alternatively, they can choose both—putting sensitive data on in-house servers while keeping less important data on the cloud. 

There are also different types of data solutions available. Some companies may prefer data lakes for efficient retrieval or specialized databases (like time series) to meet their needs. In fact, some organizations prefer time series databases, given that most IoT data come in the form of timestamped events, which time series databases are optimized for.

Data also has to be organized for efficient storage and querying, usually in either structured, semi-structured, or unstructured formats. Each have their disadvantages and advantages: while structured databases are consistent and organized for fast retrieval, they aren’t as flexible or scalable. Conversely, unstructured databases, which have no predefined schema, can easily capture all types of data, but are less suitable for use cases where data consistency, fast queries, and processing speeds are necessary. 

Semi-structured tries to blend both strengths, accommodating more data types through JSON or XML and better representing a wider array of hierarchical and other relationships—but struggling with data consistency, overhead, and query speeds. 

At this stage, teams also need to set up tiering, automating tasks such as data archival. Given that fast memory (usually NVMe) is the most expensive, it is usually reserved for hot, recent data that needs to be immediately accessed. In contrast, cold storage is used for older data, which could be anywhere from six weeks to six months (and longer). 

Data analysis and insights

Before data can be squeezed of insights, it has to be prepared—cleaned of duplicates, filtered of noise, and finally transformed into a suitable format for analysis. This pre-processing usually takes place through a stream processor such as Apache Flink, which will execute these steps in real time.

Afterwards, data is analyzed, run through advanced operations such as statistical analysis, machine learning, or predictive modeling. At this stage, teams may also elect to apply anomaly detection, highlighting outliers and flagging them for human investigation or alternatively, for an automatic action. For instance, a healthcare device might detect fluctuations in vital signs, which could trigger automatic alerts to healthcare providers for a response. Alternatively, a solar panel moving too far out of its optimal angle could trigger motors to bring it back into alignment, without notifying any humans.

After data is analyzed, insights are generated to assist stakeholders in making decisions, optimizing performance, or troubleshooting issues. This could be an improved predictive maintenance schedule for airplane engines, shortening the periods between different checks and spreading the work out amongst teams to avoid burnout. Another option could be traffic analysis that provides insights into patterns and congestion, leading to better traffic signal performance through altered timings.

Visualization and alerts

After data is analyzed and insights created, the results have to be communicated, ideally through an intuitive, appealing visual format. Solutions like Imply Pivot enable the creation of interactive, shareable graphics, such as charts, maps, stack areas, and more, so that customers, employees, and executives alike can access and dissect data at their leisure. 

Alerts are another important option. As mentioned previously, teams can be alerted in situations where human intervention is required, through methods such as email, SMS, or push notifications. THe important thing to remember is that alerts, done properly, can help a team get ahead of an evolving crisis and take corrective actions as necessary.

What are some common metrics monitored in IoT systems?

Even across different IoT use cases, such as manufacturing or energy or supply chain management, organizations will gather and analyze similar metrics. Here are some examples:

Device health

Device performance is essential for IoT operations, as IoT analytics and insights cannot be generated without data. These metrics include:

Battery life, which is essential for device usage. A device shutting down due to a depleted battery will result in downtime, data loss, and service interruptions. Therefore, teams have to track this metric and manage their batteries accordingly, swapping out inoperable batteries or recharging batteries regularly.

Signal strength—either WiFi, Bluetooth, Long Range Wide Area Network (LoRaWan), or more—which is a key component of wireless communication, and the primary method by which IoT devices transmit data, connect with other nodes in the network, and receive instructions. Weak signals result in errors or failures, and teams can guard against this by deploying signal boosters, adjusting antennas, or even removing barriers to connectivity.

CPU usage is a key indicator of how IoT devices are utilizing resources, processing workloads, and balancing their priorities. Too-high CPU usage could be a sign of inefficiencies in the above processes, so teams have to monitor this metric in order to optimize performance.

Storage capacity, which refers to the amount of data that can be stored on a single device. Insufficient storage could lead to data loss, system failure, application crashes, or even data corruption. Organizations have to track storage capacity metrics and implement data retention policies, tiering, or data compression to better manage resources.

Sensor data metrics

Because there are so many different sectors and industries within IoT, sensors probably track the most diverse types of metrics, including:

Temperature, which can be used across environmental monitoring, HVAC systems, logistics (generally around transporting food and other perishable goods), and manufacturing. Any anomalies that fall outside optimal temperature ranges could be indicators of malfunctions, safety hazards, or other problems—and should trip alerts or automated triggers for actions.

Pressure, which is essential to different fields, such as manufacturing, oil and gas, saturation diving, and meteorology. Pressure fluctuations could be indications of pipeline leaks, equipment failures, or operating conditions—all vital safety issues that must be addressed rapidly.

Humidity, which is vital for agriculture, food storage, and climate control, among other use cases. Deviating from normal humidity can have impacts on product quality, equipment performance, and comfort. Automated systems also need to track humidity data in order to execute actions (lowering temperatures for instance) or alert human team members to faults.

Location, which is important for asset tracking, fleet management, or any application that requires geographic data. Teams may use this for optimizing routes (avoiding traffic and diverting deliveries in real time), identifying inefficiencies, and improving asset utilization (such as removing or relocating underperforming devices based on their locations).

Performance metrics

Rather than being limited to single devices, teams also have to assess performance metrics across the entire network as a whole. 

Transmission latency measures how long it takes for data to travel from IoT devices to the cloud or server, which helps assess network performance and reliability. High latency has downstream effects such as data processing delays, slow responses to crises, and poor user experiences.

Throughput measures whether data is traveling efficiently through the network, and is vital for ensuring good use of network bandwidth and timely data delivery. Poor throughput can lead to congestion, packet loss, and ultimately, degraded IoT application performance.

Response times measures how long it takes for requests or commands to be sent to IoT devices, and serve as yet another indicator of user experience or system responsiveness. Slow response times lead to delays, congestion, and inefficient responses from IoT applications and devices.

Security metrics

Even if they’re not solely on the cloud, IoT devices are still connected to servers and each other, making them vulnerable to hacking. Therefore, teams have to monitor the following security metrics:

Login attempts and authentication events are used to detect unauthorized access—and potentially security breaches. Multiple failed logins, as well as logins from unfamiliar IP addresses or locations, can be signs of credential stuffing or other brute force attacks such as Distributed Denial of Service.

Suspicious activity, such as unusual device interaction or data access and transfer patterns, can also indicate security incidents. This activity is usually found in security logs and events, and are best detected through security information and event management (SIEM) solutions and anomaly detection algorithms.

Potential vulnerabilities are more miscellaneous, but include things like patch deployment or OS firmware version, as older variants can be more susceptible to hacking; compliance with security standards, which could reveal vulnerabilities; and risk level.

What are some use cases for IoT monitoring?

Predictive maintenance 

By analyzing sensor data, teams can anticipate equipment failures before they occur—and perhaps even prevent them entirely by tailoring maintenance schedules accordingly. Done properly, this will lead to less downtime and longer equipment life.

Anomaly detection 

By establishing thresholds for metrics and detecting anomalies in sensor readings, teams can more easily identify underlying issues in an IoT device. This could be temperature readings for assembly lines or revolutions per minute (RPM) for an engine. Any events that fall beyond these thresholds can be flagged for intervention, or trigger automatic actions.

Fleet tracking and asset management

Any business model which involves fleets of vehicles will benefit from IoT monitoring. This could be GPS data for altering routes in response to real-time traffic conditions such as congestions or accidents; improved routes for more efficient deliveries, pickups, and dropoffs; and usage- or hotspot-based data to better dispatch vehicles.

Environmental monitoring

Whether it’s air, water, or soil quality, sensors can provide a real-time, big-picture view of environmental quality at any given point in time. This requires a database that can handle high-cardinality, geographic, and timestamped data.

Industrial IoT monitoring

For factories and assembly lines, real-time monitoring is essential for both safety and operational efficiency. Teams can use this data to optimize processes, identify bottlenecks, assess performance, and assure both quality and productivity. 

What are some key features IoT monitoring solutions must have?

Scalability and real-time data ingestion

With the exponential growth of IoT devices and data, scalability is key to accommodating data loads without sacrificing performance or reliability. Any IoT monitoring solution has to scale dynamically to handle the high volume and velocity of data generated by IoT devices. That means they have to ingest data in real time, ideally via streaming, and offer features that can quickly process and access such data.

Apache Druid, a real-time analytics database that serves as the foundation for Imply products, is natively compatible with top streaming platforms such as Apache Kafka and Amazon Kinesis, enabling users to set up streaming data ingestion with several clicks and no additional workarounds. Imply also includes features such as query on arrival to make streaming data accessible upon ingestion, and deduplication to ensure data consistency.

Data storage and management

Storing and organizing data properly facilitates data integrity, accessibility, and compliance with regulatory requirements. Therefore, any IoT monitoring platform has to be capable of flexibly accommodating diverse data formats from different sensors, including structured, semi-structured, and unstructured data types. In addition, the underlying database should also provide either cloud-based, on-premise, or hybrid options with strong data retention policies, versioning, and encryption.

Imply can automatically detect schema and alter its model accordingly, providing the flexibility of a schemaless database while retaining the performance advantages of a strongly-typed database. This also offloads some of the work expected of developers or database administrators, freeing them up for other tasks.

To learn more, read this article on schema autodiscovery.

Ability to handle late-arriving data

Some IoT devices may be located in areas with poor connectivity or struggle with packet loss, which can result in data transmission delays. However, late-arriving data is still important, and should be seamlessly incorporated into analytics.

Imply can help automatically backfill late-arriving data, slotting them into the proper place without human intervention. To learn more about how this helped one solar inverter manufacturer with their operations, read this ebook.

Real-time queries

In order to detect and address issues promptly, as well as to make informed decisions, teams have to query data in real time. Regardless of factors such as the size of their dataset, the rate of queries, or the number of concurrent users, queries should ideally be returned in seconds or less.

Built on Apache Druid’s unique architecture, Imply products can ensure subsecond query responses even amid high rates of simultaneous operations and users. This is accomplished through the scatter-gather method—queries are broken down into discrete parts, routed to the columns where the relevant data is stored, and then reassembled by a broker node before being returned to the user. Importantly, these queries can proceed in parallel, as columns and segments are not locked—thus further improving query retrieval times.

To learn more about how this process works, read the Apache Druid architecture whitepaper.

Advanced analytics and insights

For an organization to make better (and more timely) decisions, they need to identify trends, patterns, and anomalies. They can build predictive models, algorithms to detect anomalies, and more.

Imply supports integration with external analytics libraries and frameworks, such as Apache Spark and TensorFlow. This enables organizations to leverage advanced analytics techniques, including machine learning, predictive modeling, and anomaly detection, to derive insights from IoT data.

Imply is also optimized for time-series data analysis, making it well-suited for IoT use cases where data is collected over time. This includes time-based partitioning, interpolation and backfill, windowing functions, and time-based aggregations.

Data visualization and alerting

Intuitive dashboards and visualizations are essential for presenting complex IoT data in an easy-to-understand format. Paired with automated alerts, dashboards can help reduce confusion and improve communication, especially during crisis situations.

Imply Pivot is a GUI for building and sharing interactive visualizations—such as bar charts, line graphs, maps, and more—either by embedding them into applications or as browser links. Each user action, such as a zoom in or drag and drop, will create multiple SQL operations on the backend, which Pivot will execute in milliseconds—providing a truly responsive, fast experience.

Conclusion

As interconnected smart devices spread to more sectors of our society and economy, monitoring and managing them will become more important. Teams have to detect and respond to latencies and outages, fine tune predictive maintenance to extend equipment lives and safety parameters, and ensure more efficient, profitable operations.

To learn more about Imply and how it can help facilitate key IoT functions, read this ebook.
For the easiest way to get started with Apache Druid, sign up for a free trial of Imply Polaris, the fully managed, Druid database-as-a-service.