LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detection

The proliferation of edge devices has created an urgent need for security solutions capable of detecting malware in real time while operating under strict computational and memory constraints. Recently, Large Language Models (LLMs) have demonstrated remarkable capabilities in recognizing complex patterns, yet their deployment on edge devices remains impractical due to their resource demands. However, in edge malware detection, static or centrally retrained models degrade under evolving threats and heterogeneous traffic; locally trained models become siloed and fail to transfer across domains. To overcome these limitations, in this paper, we present a continuous learning architecture for edge-based malware detection that combines local adaptation on each device with global knowledge sharing through parameter-efficient LoRA adapters. Lightweight transformer models (DistilBERT, DistilGPT-2, TinyT5) run on edge nodes and are incrementally fine-tuned on device-specific traffic; only the resulting LoRA modules are aggregated by a lightweight coordinator and redistributed, enabling cross-device generalization without exchanging raw data. We evaluate on two public IoT security datasets, Edge-IIoTset and TON-IoT, under multi-round learning to simulate evolving threats. Compared to isolated fine-tuning, the LoRA-based exchange yields up to 20-25% accuracy gains when models encounter previously unseen attacks from another domain, while maintaining stable loss and F1 across rounds. LoRA adds less than 1% to model size (~0.6-1.8 MB), making updates practical for constrained edge hardware.

💡 Research Summary

The paper addresses the pressing need for real‑time malware detection on resource‑constrained edge devices. While large language models (LLMs) have shown remarkable ability to capture complex patterns, their size and computational demands make direct deployment on edge nodes impractical. Moreover, static or centrally retrained models quickly become obsolete as malware evolves and as traffic characteristics differ across heterogeneous environments. To overcome these challenges, the authors propose a continuous‑learning framework that couples local adaptation on each edge device with global knowledge sharing via Low‑Rank Adaptation (LoRA) adapters, a parameter‑efficient fine‑tuning technique.

The architecture uses lightweight transformer models—DistilBERT, DistilGPT‑2, and TinyT5—chosen for their reduced parameter count (≈40 % of the original BERT/GPT‑2) while retaining strong representational power for text‑based network‑traffic analysis. Each device fine‑tunes only the LoRA modules (low‑rank matrices inserted into the attention layers) on its own traffic; the backbone of the model remains frozen. After a local training round, the device transmits only the learned LoRA parameters (typically 0.5‑1 % of the total model size, amounting to 0.6–1.8 MB) to a central coordinator. The coordinator aggregates the adapters—using a simple average or weighted average—and redistributes the merged LoRA back to all devices. This process repeats over multiple rounds, simulating the arrival of new malware families.

Two public IoT security datasets, Edge‑IIoTset and TON‑IoT, are employed to evaluate the approach. The experiments simulate incremental learning by introducing previously unseen attack types in successive rounds. Three baselines are compared: (a) isolated fine‑tuning on each device, (b) full‑model federated learning, and (c) the proposed LoRA‑based sharing. Results show that the LoRA approach yields a 20‑25 % increase in detection accuracy and a comparable boost in F1‑score relative to isolated fine‑tuning, while maintaining stable loss curves and exhibiting minimal catastrophic forgetting. Communication overhead is dramatically reduced because only the adapters—not the entire model—are exchanged, and the model size increase stays below 1 % of the original, making over‑the‑air updates feasible on devices such as Raspberry Pi 4, ESP‑32, or Jetson Nano.

The authors discuss why their method differs from conventional federated learning. Traditional FL requires transmitting full model weights or employing complex consensus protocols (e.g., PBFT) to handle non‑IID data, leading to high latency and bandwidth consumption. In contrast, LoRA’s low‑rank updates are lightweight, and the central aggregation simplifies synchronization, which is crucial for real‑time security applications.

Limitations are acknowledged. Repeated aggregation of LoRA adapters may cause parameter conflicts or over‑fitting if not regularized; the current simple averaging does not account for the varying importance of different attack types across devices. Moreover, the evaluation is confined to IoT traffic, leaving open the question of generalization to mobile or cloud environments. Future work is outlined, including weighted aggregation strategies, hybrid approaches that combine LoRA with memory‑based replay to further mitigate forgetting, and broader cross‑domain testing.

In summary, the paper demonstrates that combining lightweight LLMs with LoRA‑based parameter‑efficient updates enables a practical, scalable, and privacy‑preserving continuous learning system for edge‑based malware detection. It provides a concrete roadmap for deploying advanced language‑model reasoning at the edge, balancing accuracy, adaptability, and stringent resource constraints.