Multi-Task Neural Architecture Search Using Architecture Embedding and Transfer Rank

Multi-task neural architecture search (NAS) enables transferring architectural knowledge among different tasks. However, ranking disorder between the source task and the target task degrades the architecture performance on the downstream task. We propose KTNAS, an evolutionary cross-task NAS algorithm, to enhance transfer efficiency. Our data-agnostic method converts neural architectures into graphs and uses architecture embedding vectors for the subsequent architecture performance prediction. The concept of transfer rank, an instance-based classifier, is introduced into KTNAS to address the performance degradation issue. We verify the search efficiency on NASBench-201 and transferability to various vision tasks on Micro TransNAS-Bench-101. The scalability of our method is demonstrated on DARTs search space including CIFAR-10/100, MNIST/Fashion-MNIST, MedMNIST. Experimental results show that KTNAS outperforms peer multi-task NAS algorithms in search efficiency and downstream task performance. Ablation studies demonstrate the vital importance of transfer rank for transfer performance.

💡 Research Summary

The paper introduces KTNAS, an evolutionary multi‑task neural architecture search (NAS) framework that explicitly tackles the “ranking disorder” problem—where architectures that rank highly on a source task perform poorly on a target task. KTNAS first converts each neural architecture into a directed acyclic graph and applies node2vec to obtain a 256‑dimensional embedding vector. This low‑dimensional representation enables fast similarity computation via cosine distance, dramatically reducing the cost of architecture‑level comparisons.

The core novelty is the “transfer rank” mechanism, an instance‑based classifier that leverages a historical transferred set (HTS). Each previously transferred architecture is labeled +1 if its children rank within the top r % on the target task (positive transfer) or –1 otherwise (negative transfer). For a current candidate, KTNAS computes distances to all HTS members, finds the two nearest neighbors, and sums their labels to produce a transfer‑rank score. Candidates with higher scores are deemed more likely to yield positive transfer and are selected as the transfer population (size M). This data‑driven selection mitigates negative transfer without expensive re‑training of transferred models.

KTNAS maintains an independent population for each task, performs self‑evolution (tournament selection, crossover, mutation) on the union of the task’s own population and its transfer population, and then selects the top K individuals as the next generation. Hyper‑parameters include the positive‑transfer threshold r %, the number of saved generations m for HTS, and the transfer population size M. Sensitivity analyses show that reasonable settings balance transfer effectiveness and computational overhead.

Empirical evaluation spans three search spaces: NASBench‑201, Micro‑TransNAS‑Bench‑101 (seven vision tasks), and the DARTS space (CIFAR‑10/100, MNIST, Fashion‑MNIST, MedMNIST). Compared with state‑of‑the‑art multi‑task NAS methods such as EMT‑NAS and MT‑NAS, KTNAS achieves higher validation accuracy (≈2–3 % gain) while reducing total search time by up to 30 %. Ablation studies confirm that removing transfer rank or replacing node2vec embeddings with naïve encodings degrades performance, underscoring the importance of both components.

In summary, KTNAS offers a practical solution for cross‑task knowledge transfer in NAS by (1) embedding architectures into a compact graph‑based space, (2) employing a lightweight, instance‑based transfer‑rank classifier to prioritize beneficial transfers, and (3) integrating these mechanisms into a standard evolutionary search loop. The approach delivers superior search efficiency and downstream task performance across diverse vision benchmarks, establishing a new direction for scalable, multi‑task neural architecture optimization.