6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks

This paper introduces 6G-Bench, an open benchmark for evaluating semantic communication and network-level reasoning in AI-native 6G networks. 6G-Bench defines a taxonomy of 30 decision-making tasks (T1–T30) extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories. Starting from 113,475 scenarios, we generate a balanced pool of 10,000 very-hard multiple-choice questions using task-conditioned prompts that enforce multi-step quantitative reasoning under uncertainty and worst-case regret minimization over multi-turn horizons. After automated filtering and expert human validation, 3,722 questions are retained as a high-confidence evaluation set, while the full pool is released to support training and fine-tuning of 6G-specialized models. Using 6G-Bench, we evaluate 22 foundation models spanning dense and mixture-of-experts architectures, short- and long-context designs (up to 1M tokens), and both open-weight and proprietary systems. Across models, deterministic single-shot accuracy (pass@1) spans a wide range from 0.22 to 0.82, highlighting substantial variation in semantic reasoning capability. Leading models achieve intent and policy reasoning accuracy in the range 0.87–0.89, while selective robustness analysis on reasoning-intensive tasks shows pass@5 values ranging from 0.20 to 0.91. To support open science and reproducibility, we release the 6G-Bench dataset on GitHub: https://github.com/maferrag/6G-Bench

💡 Research Summary

The paper presents 6G‑Bench, an open, standards‑driven benchmark designed to evaluate the semantic communication and network‑level reasoning capabilities of foundation models (e.g., large language models) in AI‑native sixth‑generation (6G) networks. Recognizing that existing LLM assessments in telecommunications focus on isolated knowledge recall or simple numeric constraints, the authors construct a comprehensive evaluation suite that reflects the multi‑dimensional decision‑making required in future 6G systems.

First, the authors extract 30 concrete decision‑making tasks (T1–T30) from ongoing standardization work in 3GPP, IETF, ETSI, ITU‑T, and the O‑RAN Alliance. These tasks are organized into five capability categories that align with the emerging 6G vision: (1) intent and policy reasoning, (2) network slicing and resource management, (3) trust and security awareness, (4) AI‑native networking and agentic control, and (5) distributed intelligence and emerging use cases. Each task encapsulates heterogeneous inputs such as radio measurements, service‑level requirements, policy constraints, and security contexts, thereby demanding multi‑step quantitative reasoning, uncertainty handling, and worst‑case regret minimization over multi‑turn horizons.

To generate the dataset, the authors start from 113 475 scenarios derived from the α3‑Bench repository. Task‑conditioned prompts are crafted for each of the 30 tasks, enforcing very‑hard difficulty, multi‑step calculations, and adversarial reasoning patterns. A heterogeneous set of state‑of‑the‑art reasoning models is used to produce candidate multiple‑choice questions (MCQs). Automatic deduplication, heuristic anti‑pattern filters, and difficulty balancing yield a pool of 10 000 MCQs.

A two‑stage validation pipeline then refines this pool. The first stage performs automated structural, logical, and numerical checks, ensuring that each question is well‑formed, internally consistent, and that the worst‑case scenario is correctly modeled. The second stage involves expert reviewers from the telecommunications field who manually verify semantic correctness, adherence to standards, and the realism of the network context. After this rigorous filtering, 3 722 high‑confidence MCQs remain as the official evaluation set; the remaining questions are released for model pre‑training and fine‑tuning.

The benchmark is used to assess 22 contemporary foundation models spanning dense and mixture‑of‑experts architectures, code‑specialized and general‑purpose models, multimodal and long‑context designs (up to 1 M tokens), and both open‑weight and proprietary systems. Performance is reported using deterministic pass@1 (single‑shot accuracy) and selective pass@5 (top‑5 accuracy) metrics, aggregated both at the task level and across the five capability categories. Results show a wide spread in deterministic accuracy, ranging from 0.22 to 0.82. Models excel in the “intent and policy” category, achieving 0.87–0.89 accuracy, but struggle on “trust & security” and “distributed intelligence” tasks, where pass@1 can be as low as 0.20. Pass@5 values vary from 0.20 to 0.91, indicating that many models can produce a correct answer within a small candidate set if allowed limited retries. Notably, mid‑scale models sometimes outperform larger ones, suggesting that sheer parameter count does not guarantee superior reasoning in this domain.

The authors discuss the implications of these findings for the deployment of AI‑native 6G networks. They argue that domain‑specific fine‑tuning on 6G‑Bench data, careful prompt engineering, and robust verification pipelines are essential before foundation models can be trusted as high‑level reasoning layers above standardized network functions. Moreover, the benchmark’s alignment with ongoing standardization work provides a common ground for researchers, vendors, and standards bodies to evaluate and iterate on model capabilities in a reproducible manner.

Finally, the paper contributes the full 6G‑Bench dataset (both the curated 3 722‑question evaluation set and the larger 10 000‑question pool) on GitHub, encouraging open science, reproducibility, and community‑driven extensions. By bridging the gap between large‑scale language modeling and concrete 6G network decision‑making, 6G‑Bench sets a foundation for systematic, comparable, and standards‑compliant evaluation of future AI‑native communication systems.