Are you consuming a lot of IoT data? Are long query times slowing you down? Are users complaining about sluggish graphs?

Downsampling was the solution for us at Hydrologiq when we faced similar challenges — it might be the one you need too. Here’s why ⬇️

The problem

IoT devices (1/2 — see glossary) often generate data at very high frequencies (3) — sometimes every few seconds. Since IoT devices usually run 24/7, this can lead to massive datasets over time. Storing and querying (4) this data at its original frequency over long periods (months or years) can cause significant challenges, including:

1. 🤕 Slow Querying Times: Querying large datasets generated by IoT devices often results in longer response times, impacting system efficiency and user experience.
2. 🤑 High Costs: Storing and processing vast amounts of high-frequency data drives up operational and infrastructure expenses.
3. 😣 Complex Data Management: Handling and maintaining large datasets is complicated, requiring significant expertise and increasing the risk of errors or system inefficiencies.

You may be wondering, why capture so much data in the first place, good question. High-frequency data not only allows for real-time decision-making but also leads to more accurate insights, as it captures finer details and trends. The trade-off, however, is that a large amount of data is stored, which also needs to be available for historical analysis and reporting purposes.

What is downsampling?

Downsampling is the process of reducing the number of data points (5) collected by summarizing them over longer time intervals. It retains the critical information needed for historical analysis and reporting, while significantly reducing data storage and processing requirements.

Example
An IoT device monitors power usage on a construction site, where real-time data is essential for monitoring immediate power consumption. The device sends a power reading every 15 seconds, resulting in 5,760 data points per day 🤯. However, when generating hourly historical insights — such as the average and max power — we don’t need that level of detail. This is where downsampling comes in. By summarizing the data into 1-hour intervals, we only need to store the average and max power for each hour, reducing the data to just 48 points — 120x smaller, without losing key insights.

Why should you downsample your IoT data?

1. ⚡ Faster Query Times: High-frequency data is slow to query, especially over long periods. Downsampling reduces the number of points in a query, speeding up data retrieval.
2. 🤑 Reduced Costs: Storing every second of data gets expensive. Downsampling compresses data into meaningful summaries (e.g. hourly or daily), reducing storage costs.
3. 🚀 Scaleable: Ensures that your system can handle the increasing volume of data generated by the growing number of IoT devices in your system without compromising performance or inflating costs.
4. 👌 Simplified Logic: Pre-calculating common aggregations (6) reduces the need for additional logic and calculations in the queries serving your system.

Additional considerations

While downsampling is beneficial, there are a few things to watch out for:

• Retaining Critical Information: Downsampling reduces data, but keeping essential details is important. Choose appropriate time intervals and metrics — sometimes peak or outlier values are more relevant than averages.
• Hybrid Approaches: You should keep raw, high-frequency data for a short time (e.g., 30 days) for detailed analysis and downsample the rest for long-term storage and reporting.
• Evolving Needs: Downsampling isn’t a one-time process. Be prepared to revisit or adjust your downsampling logic as business needs, data patterns, or system requirements evolve.

Methods to downsample your IoT data

On-Device processing

On-device downsampling is performed directly on the IoT device before publishing data points (see Figure 1), meaning only the downsampled data points are stored in a time series database.

Benefits

• Lower Transmission Costs: Sending smaller, downsampled data reduces bandwidth usage and communication costs.
• Distributed Processing: Each device processes its data, lightening the load on your infrastructure.

Challenges

• Limited Computational Power: Many IoT devices have limited resources (e.g., CPU, memory), which may constrain the complexity of the downsampling.
• Maintaining Consistency: Ensuring that all devices apply the same downsampling logic and frequency consistently can be challenging, especially when updating configurations across a distributed network.

Stream processing

Stream downsampling is usually performed after data points have been published but before they are stored in a time series database (see Figure 2), usually the downsampled data points are stored alongside the raw data points.

Benefits

· Scalability: Designed to handle large, continuous data flows without overwhelming your infrastructure and built to scale scale as the volume of IoT data grows.

· Faster Insights: Monitoring systems or predictive analytics can immediately use aggregated data to trigger alerts or actions without waiting for additional processing.

Challenges

· Complexity: Setting up and maintaining real-time stream processing systems can be complex and any delays in data transmission or processing can impact real-time insights.

· Error Handling: Data loss or inaccuracies need to be addressed in real-time, which requires robust error management and monitoring systems.

Batch processing

Batch downsampling is usually performed at fixed intervals, once the raw data points have been published and stored in a time series database (see Figure 3), the downsampled data points are stored in addition to the raw data points.

Benefits

· Improved Data Accuracy: Allows you to back-fill any data captured offline, correct any inconsistencies, and validate the accuracy of the data before running downsampling.

· Efficient for Larger Data Sets: Enables you to downsample at higher intervals over longer periods massively reducing the data stored for long-term storage whilst making it easier to analyse historical data.

Challenges

· Latency: Since batch processing isn’t real-time, there is a delay in getting insights which will have a big impact on real-time use cases.

· High Resource Usage: Processing large batches of data can require significant computational resources during the job, especially if the downsampling logic is complex.

What’s next?

If you’re struggling with large-scale IoT data and slow performance, downsampling might be a good fit for your system. Here’s a quick roadmap to guide your implementation:

1. Identify Key Metrics: Identify the key business metrics and underlying data points that need to be retained. This ensures you downsample without losing crucial insights.
2. Define Frequency & Aggregation: Determine the minimum frequency and the appropriate aggregation methods (e.g., max, min, mean, sum) required for each metric.
3. Select Method: Select the best downsampling method for your data. You may need multiple methods depending on the data and use cases.
4. Assess Impact: Test the impact on costs and performance to ensure the benefits are worth the investment.

In an upcoming blog, I’ll dive deeper into an example downsampling scenario we handled at Hydrologiq using InfluxDB and batch-processing. Stay tuned for more details!

Have questions about downsampling or IoT monitoring systems? Feel free to connect with me on LinkedIn

Glossary

1. IoT (Internet of Things): A network of interconnected devices that communicate and share data over the internet. It’s like a group of overly chatty friends texting each other every little detail: “I’m turning on the lights!” “I just spilled coffee!” 📱💡
2. IoT device: A machine equipped with sensors and software that connects to the internet to send and receive data. Imagine a smart toaster that not only burns your toast perfectly but also texts your fridge to chill the orange juice. 🍞📱🥤
3. High-Frequency Data: Data collected at very short intervals, often seconds apart. It’s like your smartwatch pinging you every 5 seconds to remind you that, yes, you’re still sitting. ⏱️
4. Query/Querying: The process of asking a database for specific information. It’s like asking your fridge, “Where’s the leftover pizza?” and it tells you the exact shelf. 🍕
5. Data point: A single piece of information collected from a device, part of a larger dataset. It’s like one sprinkle on a donut — small on its own, but when combined with thousands of others, it creates a delicious treat. 🍩
6. Aggregation: Summarizing multiple data points into a single value, like averaging or totaling them. Think of it as making a giant burrito by combining all your fridge leftovers into one tasty wrap. 🌯

Originally published on Medium by Adam Ayre-Storie, Digital Tech Lead