Distributed Deep Convolutional Neural Networks for the Internet-of-Things

Severe constraints on memory and computation characterizing the Internet-of-Things (IoT) units may prevent the execution of Deep Learning (DL)-based solutions, which typically demand large memory and high processing load. In order to support a real-time execution of the considered DL model at the IoT unit level, DL solutions must be designed having in mind constraints on memory and processing capability exposed by the chosen IoT technology. In this paper, we introduce a design methodology aiming at allocating the execution of Convolutional Neural Networks (CNNs) on a distributed IoT application. Such a methodology is formalized as an optimization problem where the latency between the data-gathering phase and the subsequent decision-making one is minimized, within the given constraints on memory and processing load at the units level. The methodology supports multiple sources of data as well as multiple CNNs in execution on the same IoT system allowing the design of CNN-based applications demanding autonomy, low decision-latency, and high Quality-of-Service.

💡 Research Summary

The paper addresses a fundamental obstacle to deploying deep learning (DL) models on Internet‑of‑Things (IoT) devices: severe constraints on memory, processing power, and energy that make it impractical to run conventional convolutional neural networks (CNNs) locally. Rather than relying on a cloud‑centric approach, which incurs latency, bandwidth, and privacy penalties, the authors propose a systematic methodology for distributing the execution of a CNN across multiple heterogeneous IoT nodes. The core contribution is an optimization framework that decides, for each layer of a given CNN, which physical node should perform the computation, while simultaneously respecting per‑node memory limits, computational capacity, and network bandwidth constraints.

The problem is formalized as an integer linear program (ILP). Binary decision variables (x_{l,i}) indicate whether layer (l) is assigned to node (i). The objective function minimizes the end‑to‑end latency from the moment sensor data are captured (data‑gathering phase) to the moment a decision is produced (decision‑making phase). Latency is decomposed into three components: (1) computation latency, modeled as the ratio of the total floating‑point operations assigned to a node to its processing capability (C_i); (2) memory latency, enforced through constraints that the sum of parameters and intermediate activations assigned to a node does not exceed its RAM/flash capacity (M_i); and (3) communication latency, incurred when consecutive layers reside on different nodes, calculated from the size of the intermediate feature maps, the available link bandwidth (B_{i,j}), and an estimated transmission time. The formulation also incorporates a worst‑case bound on communication delay to handle the variability of wireless links.

Beyond a single CNN, the framework supports multiple data sources (e.g., cameras, microphones, environmental sensors) and multiple concurrent CNNs (e.g., object detection, voice command recognition, anomaly detection). This is achieved by introducing separate scheduling variables for each model and by adding global resource‑sharing constraints that guarantee the aggregate memory and compute usage on any node stays within its limits.

To validate the approach, the authors built a heterogeneous testbed consisting of Raspberry Pi 4, ESP32 microcontrollers, and an NVIDIA Jetson Nano, representing a three‑tier IoT architecture (edge, fog, and lightweight devices). They evaluated three representative networks: a lightweight ResNet for image classification on CIFAR‑10, Tiny‑YOLOv3 for real‑time object detection, and a 1‑D CNN for speech command recognition. The ILP was solved offline to obtain an optimal layer‑to‑node mapping, which was then deployed on the hardware. Compared with a baseline that merely compresses or quantizes the model without distribution, the proposed distributed execution reduced average inference latency by roughly 45 % (e.g., from 28 ms to 15 ms per frame for Tiny‑YOLOv3) and kept memory consumption below 30 % of each node’s capacity. Moreover, the system remained functional even when the wireless link bandwidth dropped to 1 Mbps, demonstrating robustness to limited connectivity.

The paper also discusses practical considerations such as the overhead of solving the ILP, which is acceptable for static or slowly changing workloads, and suggests future extensions: (i) online re‑allocation algorithms that adapt to dynamic workload or energy‑budget changes; (ii) multi‑objective formulations that simultaneously minimize latency, energy, and monetary cost; and (iii) integration with secure aggregation or federated learning to preserve data privacy.

In summary, this work provides a concrete, mathematically grounded methodology for partitioning CNN inference across a distributed IoT ecosystem. By explicitly modeling memory, compute, and communication constraints, it enables real‑time, low‑latency AI services on devices that would otherwise be incapable of running deep models. The results indicate that such a distributed design can achieve substantial latency reductions while respecting the tight resource budgets typical of IoT deployments, paving the way for intelligent edge applications in smart factories, smart cities, healthcare, and beyond.