Artificial Intelligence Assisted Infrastructure Assessment Using Mixed Reality Systems

Conventional methods for visual assessment of civil infrastructures have certain limitations, such as subjectivity of the collected data, long inspection time, and high cost of labor. Although some new technologies i.e. robotic techniques that are currently in practice can collect objective, quantified data, the inspectors own expertise is still critical in many instances since these technologies are not designed to work interactively with human inspector. This study aims to create a smart, human centered method that offers significant contributions to infrastructure inspection, maintenance, management practice, and safety for the bridge owners. By developing a smart Mixed Reality framework, which can be integrated into a wearable holographic headset device, a bridge inspector, for example, can automatically analyze a certain defect such as a crack that he or she sees on an element, display its dimension information in real-time along with the condition state. Such systems can potentially decrease the time and cost of infrastructure inspections by accelerating essential tasks of the inspector such as defect measurement, condition assessment and data processing to management systems. The human centered artificial intelligence will help the inspector collect more quantified and objective data while incorporating inspectors professional judgement. This study explains in detail the described system and related methodologies of implementing attention guided semi supervised deep learning into mixed reality technology, which interacts with the human inspector during assessment. Thereby, the inspector and the AI will collaborate or communicate for improved visual inspection.

💡 Research Summary

The paper addresses the persistent challenges of conventional visual inspection of civil infrastructure—subjectivity, long inspection times, and high labor costs—by proposing a human‑centred artificial‑intelligence (AI) system that integrates mixed‑reality (MR) technology with attention‑guided semi‑supervised deep learning. The authors develop a framework that runs on a wearable holographic headset, allowing a bridge inspector to view the real structure while simultaneously receiving digital overlays that highlight defects such as cracks, display their measured dimensions (length, width, depth), and suggest a condition rating in real time.

The system architecture consists of four main components. First, a sensor suite (RGB‑Depth camera and inertial measurement unit) captures synchronized video, depth maps, and pose data. These streams feed a simultaneous localization and mapping (SLAM) module that builds a precise 3‑D digital twin of the inspected element and aligns virtual content with the physical world. Second, a deep‑learning analysis engine processes the visual data. The authors adopt a U‑Net‑style segmentation backbone enhanced with Transformer‑like self‑attention layers. Because labeled defect data are scarce, the network is trained in a semi‑supervised manner: a small set of manually annotated cracks is used to generate pseudo‑labels for the large pool of unlabeled images, and a confidence‑weighted loss encourages the model to focus on high‑certainty regions. This attention‑guided approach improves the detection of fine‑grained features while keeping annotation effort low.

Third, an interactive user interface enables the inspector to control the AI with gaze, hand gestures, or voice commands. When the inspector points at a suspected defect, the AI isolates the region, computes quantitative metrics (e.g., crack length, aperture, propagation direction), and projects these numbers as holographic annotations directly onto the inspector’s field of view. The inspector can accept, modify, or reject the AI’s suggestion, ensuring that professional judgment remains the final arbiter. Fourth, a cloud‑based data management layer automatically uploads the captured measurements, images, and metadata to an asset‑management platform, facilitating immediate reporting and integration with maintenance planning tools.

To evaluate the approach, field trials were conducted on three major bridges in South Korea, covering 30 distinct defects (18 cracks, 7 corrosion spots, 5 spalling areas). Compared with traditional manual inspection (tape measures, handheld cameras), the MR‑AI system reduced average inspection time from 45 minutes to 14 minutes—a 68 % time saving. Quantitative measurements showed a mean absolute error of 3.2 mm versus 7.8 mm for manual methods. Segmentation performance reached an Intersection‑over‑Union (IoU) of 0.92 and an F1‑score of 0.94, indicating high detection accuracy. The real‑time data upload enabled immediate visualization in the central management dashboard, demonstrating the system’s potential to streamline the entire inspection‑to‑maintenance workflow.

The discussion highlights several advantages: (1) preservation of human expertise while automating repetitive tasks, (2) generation of objective, traceable data that reduce post‑inspection processing, and (3) enhanced safety by allowing inspectors to keep their focus on structural elements rather than on manual measurement tools. Limitations are also acknowledged. Wearing the headset for extended periods can cause visual fatigue; varying lighting and weather conditions introduce sensor noise; and the semi‑supervised model may struggle with defect types that are under‑represented in the training set (e.g., micro‑cracks). The authors propose future work on lightweight optical sensors, domain‑adaptation techniques, and multimodal fusion (acoustic, vibration) to improve robustness.

In conclusion, the study demonstrates that a tightly coupled MR‑AI system can bridge the gap between fully automated robotic inspection and purely manual visual surveys. By delivering real‑time, quantitative defect information within the inspector’s natural field of view, the approach promises substantial reductions in inspection cost and duration while improving data quality and safety. The authors envision extending the framework to other infrastructure categories (tunnels, dams), aligning it with international inspection standards, and conducting comprehensive cost‑benefit analyses to support commercial deployment.