How IoT data collection works

The IoT data collection process

At its heart, an IoT platform is a network of devices, similar to any conventional enterprise network in service today. IoT's complexities and challenges are rooted in the extremes of its paradigm:

• Every IoT device produces enormous amounts of data.
• Even modest IoT infrastructures involve hundreds, thousands or even tens of thousands of IoT devices – exponentially multiplying the data burden.
• IoT data is time sensitive, rapidly losing value when stored, elevating demand for efficient handling and transmission of IoT data.
• These volumes of real-time IoT data also need quick processing for timely application, requiring a ready and capable computing infrastructure.

Consequently, business and technology leaders must have a clear understanding of IoT data collection and its unique implications for enterprise IT. There are typically four broad aspects of the complete IoT data collection process: creation, collection, preparation and analysis.

IoT data creation

IoT data collection starts with IoT devices, typically segregated into two broad categories: sensors and actuators.

Sensors measure a specific physical condition, translate that physical condition into meaningful real-time data, and then make that data available to a network for collection, preparation and analysis. Sensors are input devices that produce three broad types of IoT data:

• Raw physical data includes motion, pressure, temperature, lighting level and location, such as GPS data.
• Operational data, sometimes called automation data, includes mechanical metrics, device health or operating condition, usage details and log information.
• User-specific data includes usage patterns, preferences and other user interactions, such as those of home devices such as smart thermostats or medical wearables.

Actuators, on the other hand, are output devices designed to perform specific tasks or take certain actions in the real world. For example, a smart home security system uses an actuator to remotely lock or unlock a door or control lighting. As another example, an industrial plant uses an actuator to open or close a valve.

IoT data collection

Individual IoT devices – generally small, extremely low-power components with a bare minimum of onboard processing capabilities – cannot perform or assist in data processing tasks. IoT data must be moved from IoT devices and collected at a centralized location, where it is then prepared and processed.

IoT data is placed onto a common network, such as a traditional Ethernet network. Though this network is sometimes shared with other devices, such as servers and storage subsystems, the preferred approach is a dedicated secondary network for IoT devices, ensuring exclusive access for data transmission and collection.

An IoT gateway, a common addition to an IoT infrastructure, acts as a multifunction bridge. Among their tasks, IoT gateways reconcile and interface varied device types and communication protocols, ensure IoT device security with encryption and authentication and perform some initial IoT data preparation and processing, such as data aggregation and filtering, before passing data to the cloud or main data center for analysis.

For example, basic IoT deployments use an array of sensors and other IoT devices in an edge computing environment, passing data to the local IoT gateway using either wired or wireless networking technologies. The IoT gateway collects the data and stores it locally, often performing simple data preparation processes from formatting to deduplication. These actions reduce overall data volume and streamline centralized processing. Then the IoT gateway passes the collected data along a common Ethernet network to the cloud or a primary data center, where servers and enterprise applications conduct comprehensive data analysis.

IoT platforms employ a wide variety of network communication protocols, including the following:

• Advanced Message Queuing Protocol, or AMQP, for connecting applications and devices.
• Bluetooth and Bluetooth Low Energy, or BLE, for short-range device communication.
• Constrained Application Protocol messaging, or CoAP messaging, for limited bandwidth.
• Data Distribution Service, or DDS, for real-time data distribution.
• Conventional Hypertext Transfer Protocol, or HTTP.
• Long-Range WAN, or LoRaWAN, for wide area networks.
• Message Queuing Telemetry Transport messaging, or MQTT messaging, for limited bandwidth.
• Narrowband IoT, or NB-IoT, for long-range, low-power cellular communication.
• OPC Unified Architecture, or OPC UA, for industrial automation and control.
• WiFi for wireless high-speed communication.
• Extensible Messaging and Presence Protocol, or XMPP, for IoT command and control.
• Zigbee for long range mesh networks.
• Z-Wave for home automation.

It's important to initiate a network with appropriate bandwidth and latency to maintain real-time data transmission from all IoT devices. Usually, lost or dropped data is not sent again because of the emphasis on timeliness in IoT systems.

IoT data preparation

Real-world, real-time data collection from IoT devices is rarely perfect. Data elements are inaccurate or dropped due to device malfunction, lack of maintenance, a network bottleneck or network disruption. Different device types, manufacturers and configurations yield different data formats or units, such as temperatures in Fahrenheit from some devices and Celsius from others. Accommodations are also needed for myriad data types, such as temperature data from one group of sensors, pressure data from another and motion data from a third set.

All this extensive and diverse data needs preparation, or cleaning, before it's useful in any practical analytics. After all, analyzing faulty data yields faulty conclusions. Proper data cleaning ensures accurate, consistent data is ready for use. This scrubbing of IoT data includes numerous data quality processes, such as:

• Managing missing values.
• Finding and correcting erroneous data.
• Deduplication, or removing duplicate data.
• Using consistent or standard data formats.
• Treating outliers in any unusual or unexpected data.

Rules-based processes typically guide IoT data preparation, but rapid advances in ML and artificial intelligence (AI) improve current outlier recognition and address certain data quality issues.

Although it's possible to prepare IoT data in a cloud or data center, this task is often addressed remotely in IoT gateways, located at the edge where data is originally collected. By preparing data at the edge, it's already validated and suitable for centralized processing, which saves time, storage and computing resources.

IoT data analysis

Ultimately, the goals of IoT data analysis are as varied as the organizations that use it. Some businesses are simply data mining – finding useful data to sell for profit. Others use IoT data for ML and the development of AI platforms. Still other businesses employ IoT data to identify opportunities, gain insights, improve supply chain efficiencies, enhance customer experience and forestall downtime with predictive maintenance on manufacturing equipment.

Analytics are often expressed as easily understood data visualization charts for the benefit of human business and technology leaders. Comprehensive reporting adds extensive context and background to enrich IoT data processing practices.

Challenges of IoT data collection

Given the complexity and diversity of IoT systems, adopters must carefully consider several important challenges with IoT data collection. They include:

• Network and IoT device performance. IoT devices are subject to wear, damage, defects and loss. Select IoT devices suitable for the task and ensure devices meet the durability and power requirements for their uses. Do not overlook reliable and uninterrupted network connectivity with ample bandwidth to support current and future IoT fleets.
• IoT data quality. IoT success depends on the quality of its data. Bad data means bad decisions, rendering an entire IoT deployment worthless to the business. IoT data must be accurate, as complete as possible and timely, using reliable sensors to deliver scrubbed data across a network without disruption or delay. Adopters must develop a comprehensive plan to address data quality problems, such as lost or missing data.
• IoT data security. IoT data is just as much a business asset as a key customer database and demands appropriate protection. IoT infrastructures must use encryption and authentication to govern device access and prevent vulnerabilities that precede data eavesdropping or cyber attacks. To prevent data breaches, IoT data must be stored and safeguarded with encryption, role-based access control, data protection and retention and destruction policies. Further, especially sensitive IoT data – related to individuals, for example – must meet current compliance standards for data privacy.
• IoT data management. Because different IoT device types use different communication protocols and produce various data types in various forms, consider how device data is formatted, integrated as a common data resource, then scaled as the IoT fleet grows and evolves. Develop a uniform plan to organize and manage IoT data. Similarly, software tools must be available to review, visualize, edit and manage vast datasets. IoT data management often overlaps with IoT security and compliance issues.

Stephen J. Bigelow, senior technology editor at TechTarget, has more than 30 years of technical writing experience in the PC and technology industry.