We can think about the data generated by an IoT device as going through three separate phases: the data creation itself, collection, and making data valuable.
We can think about the data generated by an IoT device as going through three separate phases. The initial phase is that of the data creation itself. This phase takes place on the level of the IoT device from where the data is transmitted over a network. The second phase of the journey of IoT data is that of collection and organization. The third phase involves the actual use of that data. This is the process of making that data valuable for a wealth of contexts.
In what follows, we trace the process of IoT data collection and data storing across these stages.
In the world of IoT, each event creates data. Sending the data over involves standard protocols such as MQTT, WAMP, HTTP, CoAP, or Sigfox. Each of them comes with its strong points and adjacent use cases. These protocols support the fetching of updates or other information from the IoT device to send it to a given centralized location for actual processing.
At this stage, one has to decide how the data is aggregated and stored for future use. How one is to proceed from now on depends on how the IoT data is to be consumed. Here you select an approach and determine whether the data should be sent over in real time or in batches. Also, you determine in what order the data points should be created for maximally accurate IoT analytics.
Using real-time data safeguards maximum accuracy. This approach guarantees access to all data generated by each IoT device. However, this usually means massive amounts of incoming data. Putting the right timestamp for sorting data coming from multiple IoT devices becomes a challenge. One should consider systems that are able to accommodate the speed and volume of that incoming IoT data.
Opting for the approach of collecting all available IoT data in real-time and analyzing it all should be backed by a clear rationale. Otherwise, the sheer volume of big data may have an unnecessarily heavy impact on the cloud systems—that is, the network resources and computational resources required to keep the inflow of IoT data.
One should also consider concrete IoT applications. These have unique requirements in terms of latency, energy consumption, and accuracy. Some applications may allow for delays. Others, such as security applications, are rather seen as time-critical and cannot have room for delays.
Many use cases may not require high accuracy and would allow for sending the data over in batches. If the data is sent in batches or micro-batches, you still get a record of all data. This takes place not in real time but only at given pre-established intervals. The choice of a particular use case depends on the requirements. In some scenarios, you will need accurate real-time data for your analytics. In other scenarios, historic data will do just as well.
If planning to analyze the incoming sensor data directly, at this stage, you need an IoT platform capable of ingesting IoT device data coming from large numbers of devices streaming raw data in real time. One such platform has to be able to respond to temporary connectivity issues. These may include a lost connection, outage or a server failure.
The platform helps you to avoid loss of data. You, therefore, do not risk impairing the accuracy of the results to be generated on the basis of that data. On the platform, data is stored in a ready-made form for data modeling and IoT data analytics. From there, you use the collected data for building machine learning models, developing your IoT application, and supplementing with data visualization.
In broad brushstrokes, again, the approach selected in collecting and storing IoT data heavily depends on the target requirements of a given use case. These involve, but are not limited to, data collection procedures motivated by needs as diverse as the accuracy of the data, energy consumption, response time, and privacy protection. Proven approaches for reducing the volumes of collected IoT data are data aggregation, filtering, interpretation, and compression at the level of the IoT sensors, as close to the data source as possible.
In what follows, we summarize some of the key data collection approaches based on the intended application as laid out in this study.
This IoT data collection strategy takes into account the tradeoff between the frequency of the measurement requests and the accuracy of the data. Here the frequency is adapted with an eye towards the target data accuracy. It may be the case that a given use case requires only a certain level of data accuracy. A higher level of accuracy does not bring any additional value to the effort.
As part of this strategy, one needs to reduce the frequency of the data measurements. This reduces the resources necessary for the collection of the IoT data while retaining the accuracy target that has been calculated as optimal for that use case.
In this approach to IoT data collection, a maximum delay value is established in advance. The time that has elapsed since the timestamp of the last measurement has to remain below that maximum delay value. In IoT data collection scenarios motivated by time-related requirements, each new measurement that is received minus the timestamp of the last measurement has to be below the maximum delay. Hereby the elapsed time corresponds to the “freshness” of the data measurements.
When opting for this approach, energy consumption is a key factor. The effort is to achieve the targeted accuracy while optimizing power consumption. An energy-driven strategy for IoT data collection aims at maximum efficiency. The benefit here is measured as the difference between the utility achieved for some targeted data accuracy minus the power consumption for the measurements necessary to achieve that data accuracy.
Similar to the accuracy-driven scenario, here it is assumed that, depending on the application, one will aim for a specific target accuracy knowing that a higher accuracy will offer no additional benefit.
If the privacy of the end users is of prevalent importance, this approach can be adopted. According to the study, here there will be an effort to minimize the number of times a sensor is asked for data and to add noise to the results using privacy techniques.
The goal is to safeguard the privacy of end users by modifying the accuracy of individual measurements while at the same time maintaining a certain level of “adequate accuracy” for the results as a whole.
Adding “noise” to the result is achieved by using “differential privacy”. According to the study, this permits the extraction of data pertaining to a population of users without revealing any information about individuals. This is thanks to the addition of “noise” during the very data extraction process without significantly altering the statistical result.
This approach recommends adding noise at the IoT data collection stage and not at the data processing stage. The related experiments show that an intervention of this kind has a non-significant effect on the final statistics.
This has been just a taste of the various data collection approaches and trends in IoT data harvesting right now. Within the platform, the studio for data analysis allows you to connect to your IoT data source and stream IoT data directly into a compact lightweight data warehouse. The platform makes it possible to adjust the exact way you want to have the data collected as well as monitor and control the data streams at all times.
But your IoT solution does not end with ingesting real-time data from a connected device. Taking it from here, you build your data management strategy, make use of the extensive data storage capabilities of the cloud, and use the data in various cloud computing scenarios including IoT app development. This way, you utilise the platform as the springboard for building and maintaining a whole IoT system – starting from the smart device all the way up to sophisticated IoT solutions using artificial intelligence. Get in touch for an in-depth discussion.
