Memory Constrained Dynamic Subnetwork Update for Transfer Learning

📝 Original Info

• Title: Memory Constrained Dynamic Subnetwork Update for Transfer Learning
• ArXiv ID: 2510.20979
• Date: 2025-10-23
• Authors: 정보 제공되지 않음 (논문에 명시된 저자 정보가 없습니다).

📝 Abstract

On-device neural network training faces critical memory constraints that limit the adaptation of pre-trained models to downstream tasks. We present MeDyate, a theoretically-grounded framework for memory-constrained dynamic subnetwork adaptation. Our approach introduces two key innovations: LaRa (Layer Ranking), an improved layer importance metric that enables principled layer pre-selection, and a dynamic channel sampling strategy that exploits the temporal stability of channel importance distributions during fine-tuning. MeDyate dynamically resamples channels between epochs according to importance-weighted probabilities, ensuring comprehensive parameter space exploration while respecting strict memory budgets. Extensive evaluation across a large panel of tasks and architectures demonstrates that MeDyate achieves state-of-the-art performance under extreme memory constraints, consistently outperforming existing static and dynamic approaches while maintaining high computational efficiency. Our method represents a significant step towards enabling efficient on-device learning by demonstrating effective fine-tuning with memory budgets as low as a few hundred kB of RAM.

💡 Deep Analysis

📄 Full Content

Reference