Mind the Gap: Missing Cyber Threat Coverage in NIDS Datasets for the Energy Sector

Network Intrusion Detection Systems (NIDS) developed using publicly available datasets predominantly focus on enterprise environments, raising concerns about their effectiveness for converged Information Technology (IT) and Operational Technology (OT) in energy infrastructures. This study evaluates the representativeness of five widely used datasets: CIC-IDS2017, SWaT, WADI, Sherlock, and CIC-Modbus2023 against network-detectable MITRE ATT&CK techniques extracted from documented energy sector incidents. Using a structured five-step analytical approach, this article successfully developed and performed a gap analysis that identified 94 network observable techniques from an initial pool of 274 ATT&CK techniques. Sherlock dataset exhibited the highest mean coverage (0.56), followed closely by CIC-IDS2017 (0.55), while SWaT and WADI recorded the lowest scores (0.38). Combining CIC-IDS2017, Sherlock, and CIC-Modbus2023 achieved an aggregate coverage of 92%, highlighting their complementary strengths. The analysis identifies critical gaps, particularly in lateral movement and industrial protocol manipulation, providing a clear pathway for dataset enhancement and more robust NIDS evaluation in hybrid IT/OT energy environments.

💡 Research Summary

The paper addresses a critical gap in the evaluation of network intrusion detection systems (NIDS) for the energy sector, where converged IT and OT environments present unique cyber‑risk profiles. While most NIDS research relies on publicly available datasets that were originally created for enterprise networks, the authors question whether these datasets adequately represent the threat landscape of modern power‑generation, transmission, and distribution infrastructures.

To answer this, the study first compiles a comprehensive list of MITRE ATT&CK techniques that have been observed in documented energy‑sector incidents. Starting from the full ATT&CK for Enterprise and ATT&CK for Industrial Control Systems matrices, the authors extract 274 techniques. They then filter this pool to 94 techniques that are observable at the network level—i.e., those that can be detected through packet headers, flow records, or session‑level metadata without deep payload inspection.

Next, five widely cited public datasets—CIC‑IDS2017, SWaT, WADI, Sherlock, and CIC‑Modbus2023—are subjected to a structured five‑step analytical workflow: (1) data acquisition and cleaning, (2) verification and augmentation of existing labels, (3) mapping of each traffic trace to the selected ATT&CK techniques, (4) calculation of per‑technique coverage within each dataset, and (5) aggregation of results to produce mean coverage scores. The mapping process is meticulous, ensuring that each technique is only counted when the underlying traffic exhibits the required observable behavior (e.g., lateral movement via SMB, protocol‑specific command injection, or credential dumping visible in network flows).

The quantitative results reveal a pronounced disparity among the datasets. Sherlock achieves the highest mean coverage (0.56), closely followed by CIC‑IDS2017 (0.55). Both datasets contain a balanced mix of enterprise‑style attacks (phishing, web‑shells) and OT‑relevant scenarios (protocol manipulation, PLC command injection). In contrast, the SWaT and WADI testbeds, which are primarily designed for control‑system safety studies, score only 0.38 on average, reflecting their limited focus on a narrow set of attacks (mostly replay and simple command injection). CIC‑Modbus2023, while strong in Modbus‑specific manipulations, remains middle‑of‑the‑road overall because it does not cover many lateral‑movement or credential‑theft techniques.

When the three best‑performing datasets—CIC‑IDS2017, Sherlock, and CIC‑Modbus2023—are combined, the aggregate coverage reaches 92 % of the 94 network‑observable techniques. This demonstrates that the datasets are complementary: each fills gaps left by the others, especially in areas such as multi‑protocol attacks and complex lateral‑movement chains. However, the analysis also uncovers persistent blind spots. Techniques related to lateral movement across segmented OT zones, sophisticated industrial protocol manipulation (e.g., IEC 61850 GOOSE spoofing), and advanced credential‑stealing that manifest only in subtle timing anomalies are under‑represented, accounting for more than 30 % of the technique pool.

The authors discuss the implications of these findings for both researchers and practitioners. For researchers, the lack of coverage means that NIDS models trained on existing datasets may overfit to enterprise‑centric patterns and under‑perform when deployed in hybrid energy environments. For operators, reliance on such models could lead to missed detections of attacks that specifically target process control logic or exploit OT‑specific communication standards.

To bridge the gap, the paper proposes concrete actions: (1) augment public datasets with real‑world traffic captures from substations, SCADA networks, and DER (distributed energy resource) installations; (2) adopt a standardized labeling schema aligned with ATT&CK to ensure consistent mapping across datasets; (3) enrich datasets with multi‑protocol scenarios (Modbus, DNP3, IEC 61850, OPC UA) and realistic lateral‑movement pathways that cross IT‑OT boundaries; and (4) provide extensive metadata (timestamp, device role, network segment) to facilitate reproducibility and fine‑grained analysis. By implementing these recommendations, the community can develop more robust NIDS benchmarks that truly reflect the complex threat landscape of modern energy infrastructures, ultimately strengthening cyber resilience in this critical sector.