Beyond the Desktop: XR-Driven Segmentation with Meta Quest 3 and MX Ink

Medical imaging segmentation is essential in clinical settings for diagnosing diseases, planning surgeries, and other procedures. However, manual annotation is a cumbersome and effortful task. To mitigate these aspects, this study implements and evaluates the usability and clinical applicability of an extended reality (XR)-based segmentation tool for anatomical CT scans, using the Meta Quest 3 headset and Logitech MX Ink stylus. We develop an immersive interface enabling real-time interaction with 2D and 3D medical imaging data in a customizable workspace designed to mitigate workflow fragmentation and cognitive demands inherent to conventional manual segmentation tools. The platform combines stylus-driven annotation, mirroring traditional pen-on-paper workflows, with instant 3D volumetric rendering. A user study with a public craniofacial CT dataset demonstrated the tool’s foundational viability, achieving a System Usability Scale (SUS) score of 66, within the expected range for medical applications. Participants highlighted the system’s intuitive controls (scoring 4.1/5 for self-descriptiveness on ISONORM metrics) and spatial interaction design, with qualitative feedback highlighting strengths in hybrid 2D/3D navigation and realistic stylus ergonomics. While users identified opportunities to enhance task-specific precision and error management, the platform’s core workflow enabled dynamic slice adjustment, reducing cognitive load compared to desktop tools. Results position the XR-stylus paradigm as a promising foundation for immersive segmentation tools, with iterative refinements targeting haptic feedback calibration and workflow personalization to advance adoption in preoperative planning.

💡 Research Summary

This paper presents the design, implementation, and preliminary evaluation of an extended‑reality (XR) segmentation platform that combines the Meta Quest 3 standalone headset with the Logitech MX Ink stylus. The authors motivate the work by highlighting the limitations of conventional desktop‑based medical image segmentation tools, which typically require users to switch between 2‑D slice viewers and 3‑D renderers, rely on mouse‑and‑keyboard input that is ill‑suited for fine‑grained anatomical delineation, and impose a high cognitive load due to fragmented workflows. To address these issues, the system offers a unified spatial workspace where 2‑D cross‑sectional images and 3‑D volumetric reconstructions can be manipulated simultaneously, while the stylus provides a pen‑on‑paper feel that mirrors traditional annotation practices.

Technical implementation is built on Unity 2022.3 with the Meta XR SDK and OpenXR for cross‑platform compatibility. Medical image data are ingested as TIFF stacks (converted from DICOM) using a custom TiffLib parser that preserves original voxel spacing. The stacks are loaded as 8‑bit textures (512 × 512) and pre‑buffered on the GPU to ensure smooth slice navigation. The MX Ink stylus is integrated via ray‑casting: the tip’s 3‑D position is projected onto a virtual canvas, UV‑mapped to image coordinates, and linearly interpolated to produce continuous strokes. Brush size, eraser size, and additive/subtractive modes are dynamically adjustable, and haptic pulses are emitted when the stylus contacts the canvas, giving users tactile confirmation.

For 3‑D visualization, the platform employs a Marching Cubes algorithm to generate iso‑surfaces from either raw CT data or user‑created masks. To keep the UI responsive, the algorithm yields results every 1,000 cubes, preventing UI thread blocking. Generated meshes undergo quadric edge‑collapse decimation, reducing triangle counts by 70‑80 % while preserving critical anatomical detail, followed by vertex deduplication and view‑dependent down‑sampling. The optimized pipeline achieves interactive frame rates of ~72 FPS with latencies below 200 ms on the Quest 3 hardware.

A user study with ten participants (medical students and practicing clinicians) evaluated usability. Participants received a 10‑minute training session, then spent 30 minutes segmenting structures in a publicly available craniofacial CT dataset (Open‑Full‑Jaw). Afterward they completed the System Usability Scale (SUS) and a 10‑item questionnaire derived from ISO 9241‑110, covering suitability, self‑descriptiveness, controllability, error tolerance, and overall satisfaction. SUS scores averaged 66 / 100, slightly below the conventional benchmark of 68 but within the accepted standard deviation for digital‑health applications (±12.5). The ISO questionnaire yielded mean item scores between 2.5 and 4.1, with the highest ratings for self‑descriptiveness (4.1/5) and ease of learning (≥3.7/5). Internal consistency was high (Cronbach’s α = 0.89).

Qualitative feedback highlighted several strengths: the stylus’s ergonomic design and realistic pen feel reduced cognitive effort; the hybrid 2‑D/3‑D workspace facilitated spatial reasoning; and the wireless, standalone nature of the Quest 3 eliminated desk clutter and allowed flexible positioning of the virtual canvas on real‑world surfaces. Participants suggested improvements such as more robust edge‑tracking for strokes near the canvas border, an undo/redo mechanism, and finer error‑correction tools.

In discussion, the authors note that a SUS of 66 is acceptable for an early‑stage prototype and indicates that the system is on a viable trajectory toward clinical adoption. They argue that the XR‑stylus paradigm bridges the gap between traditional pen‑based annotation and immersive 3‑D interaction, thereby lowering the learning curve for clinicians. Potential applications extend beyond segmentation to medical education, where the mixed‑reality environment could serve as a scaffold for teaching complex anatomical relationships, and to collaborative scenarios enabled by the Quest’s native multi‑user capabilities.

Future work will focus on refining stylus tracking accuracy, integrating an undo/redo stack, incorporating AI‑assisted pre‑segmentation to enable semi‑automatic workflows, and expanding multi‑user collaboration for training and team‑based planning. The authors also plan to conduct larger clinical studies to assess segmentation accuracy and workflow efficiency compared with standard desktop tools.

Overall, the paper demonstrates that an XR‑based segmentation platform leveraging the Meta Quest 3 and MX Ink stylus can provide a more intuitive, spatially integrated, and ergonomically favorable alternative to conventional desktop solutions, laying groundwork for broader adoption in pre‑operative planning, education, and collaborative clinical practice.