To Offload or Not To Offload: Model-driven Comparison of Edge-native and On-device Processing In the Era of Accelerators

Computational offloading is a promising approach for overcoming resource constraints on client devices by moving some or all of an application’s computations to remote servers. With the advent of specialized hardware accelerators, client devices can now perform fast local processing of specific tasks, such as machine learning inference, reducing the need for offloading computations. However, edge servers with accelerators also offer faster processing for offloaded tasks than was previously possible. In this paper, we present an analytic and experimental comparison of on-device processing and edge offloading for a range of accelerator, network, multi-tenant, and application workload scenarios, with the goal of understanding when to use local on-device processing and when to offload computations. We present models that leverage analytical queuing results to derive explainable closed-form equations for the expected end-to-end latencies of both strategies, which yield precise, quantitative performance crossover predictions that guide adaptive offloading. We experimentally validate our models across a range of scenarios and show that they achieve a mean absolute percentage error of 2.2% compared to observed latencies. We further use our models to develop a resource manager for adaptive offloading and show its effectiveness under variable network conditions and dynamic multi-tenant edge settings.

💡 Research Summary

The paper investigates the trade‑off between performing compute‑intensive tasks on modern, accelerator‑equipped client devices and offloading them to edge servers that also host powerful accelerators. The authors argue that the traditional belief—edge offloading is always faster despite network latency—no longer holds in the era of on‑device AI accelerators and edge GPUs. To answer when each approach is preferable, they develop a rigorous analytical framework based on queuing theory.

Two separate queuing systems are modeled: (1) the on‑device path, where incoming requests join a local queue and are dispatched to k_dev parallel accelerator cores, and (2) the edge‑offload path, which includes network transmission delays (request and response), network queuing on both the device side and the edge side, and processing on an edge accelerator with parallelism k_edge. The expected end‑to‑end latency formulas are derived as:
T_edge = w_net_dev + n_req + w_proc_edge + s_edge + w_net_edge + n_res
T_dev = w_proc_dev + s_dev
where w denotes queuing delays, s denotes service times, and n denotes transmission times. Service times for each workload on each accelerator are obtained either by empirical profiling or by a lightweight neural‑network predictor.

The model explicitly incorporates key parameters: request arrival rate λ, degree of parallelism k, network round‑trip time, bandwidth, and, crucially, multi‑tenant interference at the edge (modeled by aggregating arrival rates of all tenants). By solving the closed‑form expressions, the authors can predict the “crossover point” at which T_edge equals T_dev, thus indicating which strategy yields lower latency under given conditions.

Experimental validation spans four accelerators (Google Edge TPU, NVIDIA Jetson TX2, Jetson Orin Nano, NVIDIA A2 GPU) and three workload families (deep neural networks, recurrent networks, large language models). Network conditions are varied from 5 ms to 100 ms RTT and 10 Mbps to 1 Gbps bandwidth; edge servers are configured with 1–8 concurrent processing slots to emulate multi‑tenant load. Across all scenarios the analytical predictions achieve a mean absolute percentage error of only 2.2 %, with 91.5 % of predictions within ±5 % of measured latency and every prediction within ±10 %. This demonstrates that the relatively simple queuing models capture the dominant dynamics of real systems.

Building on the analytical insight, the authors design an adaptive offloading manager. The manager continuously monitors network RTT and request arrival rates, queries the pre‑computed latency formulas, and dynamically switches between local execution and edge offloading whenever the predicted optimal strategy changes. Two case studies illustrate its effectiveness: (a) an AR/VR streaming application experiencing fluctuating wireless latency, and (b) a smart‑city video analytics service where the edge server’s tenant count spikes. In both cases the manager reduces average latency by 15–30 % compared to static policies.

Key contributions are: (1) a queuing‑theory‑based, closed‑form model for on‑device versus edge offloading that is hardware‑aware and does not require workload‑specific training; (2) extensive empirical validation showing high prediction accuracy across diverse accelerators, networks, and workloads; (3) extension of the model to multi‑tenant edge environments and split (device‑edge collaborative) processing; (4) a practical resource‑management algorithm that leverages the model for real‑time adaptation.

The study concludes that the decision to offload is a nuanced function of accelerator performance, network conditions, and edge server contention. The presented model provides system designers and operating‑system developers with an interpretable, low‑overhead tool to make optimal placement decisions in today’s heterogeneous, accelerator‑rich edge computing landscape. Future work may explore additional accelerator types (ASICs, FPGAs) and broader hierarchical architectures involving cloud, edge, and device layers.