Enriching physical-virtual interaction in AR gaming by tracking identical objects via an egocentric partial observation frame

Augmented reality (AR) games, particularly those designed for head-mounted displays, have grown increasingly prevalent. However, most existing systems depend on pre-scanned, static environments and rely heavily on continuous tracking or marker-based solutions, which limit adaptability in dynamic physical spaces. This is particularly problematic for AR headsets and glasses, which typically follow the user’s head movement and cannot maintain a fixed, stationary view of the scene. Moreover, continuous scene observation is neither power-efficient nor practical for wearable devices, given their limited battery and processing capabilities. A persistent challenge arises when multiple identical objects are present in the environment-standard object tracking pipelines often fail to maintain consistent identities without uninterrupted observation or external sensors. These limitations hinder fluid physical-virtual interactions, especially in dynamic or occluded scenes where continuous tracking is infeasible. To address this, we introduce a novel optimization-based framework for re-identifying identical objects in AR scenes using only one partial egocentric observation frame captured by a headset. We formulate the problem as a label assignment task solved via integer programming, augmented with a Voronoi diagram-based pruning strategy to improve computational efficiency. This method reduces computation time by 50% while preserving 91% accuracy in simulated experiments. Moreover, we evaluated our approach in quantitative synthetic and quantitative real-world experiments. We also conducted three qualitative real-world experiments to demonstrate the practical utility and generalizability for enabling dynamic, markerless object interaction in AR environments. Our video demo is available at https://youtu.be/RwptEfLtW1U.

💡 Research Summary

The paper addresses a fundamental challenge in head‑mounted display (HMD) based augmented reality (AR) games: maintaining consistent identities for multiple visually identical physical objects when continuous scene observation is infeasible. Traditional AR pipelines rely on pre‑scanned static environments, continuous visual tracking, or fiducial markers, all of which are unsuitable for wearable devices with limited power and processing capacity, especially when the user’s head motion yields only partial, ego‑centric views. Moreover, standard multiple‑object tracking (MOT) methods struggle to discriminate between identical instances because they depend on uninterrupted video streams and often assume a stationary camera.

To overcome these limitations, the authors propose a lightweight, optimization‑driven framework that re‑identifies identical objects using a single partial observation frame captured by the headset. The key insight is that, in many AR games, the spatial relationships among objects (relative positions and orientations) remain relatively stable over short time intervals, even if the objects themselves are moved or rearranged. By exploiting this spatial consistency, the system can infer object identities without continuous tracking.

Methodology

1. 6‑DOF Pose Estimation – Each detected object’s position, orientation, and size are estimated from RGB or RGB‑D input. The authors employ existing solutions such as Objectron or a custom computer‑vision pipeline when off‑the‑shelf methods are unavailable.
2. Cost Formulation – For every possible pairing between a newly observed object i and an object j from the previously stored layout, three costs are computed:
   • Translation cost: Euclidean distance between the two centroids.
   • Rotation cost: Angular difference between orientations (e.g., quaternion distance).
   • Dimension cost: Discrepancy in bounding‑box or model dimensions.
     These costs are combined linearly with user‑defined weights to produce a scalar compatibility score.
3. Integer Programming (IP) Model – Binary variables (x_{ij}) indicate whether observation i is assigned the identity of layout object j. The IP enforces: (a) each observation is matched to exactly one layout object, and (b) each layout object receives at most one match. The objective is to minimize the total cost across all assignments.
4. Voronoi‑Based Pruning – To keep the IP tractable, the authors construct a Voronoi diagram from the layout object positions. An observation is only considered for objects whose Voronoi cell contains the observation’s projected location, dramatically reducing the number of candidate pairs before the IP is solved.

Experimental Evaluation

• Synthetic Tests: The authors generate virtual scenes with varying numbers of identical objects (5–12), random translations, rotations, and occlusion levels. The framework consistently achieves ~91 % correct re‑identification while the Voronoi pruning cuts average solve time from 1.02 s to 0.48 s (≈50 % reduction).
• Real‑World Tests: Using a Microsoft HoloLens 2 and an RGB‑D sensor, physical objects (e.g., toy huts, coops) are rearranged between game rounds. With up to eight identical items, the system maintains 89 %+ accuracy and introduces less than 30 ms latency per frame, satisfying real‑time gaming requirements.
• Qualitative Demonstrations: Three AR scenarios showcase practical utility: (1) a “farm‑to‑table” game where moving identical coops does not break animal‑to‑building logic, (2) a storytelling setup where props are shuffled but narrative cues stay aligned, and (3) a robot‑delivery simulation where the robot’s path adapts to relocated identical containers.

Contributions and Significance

1. Introduces a marker‑less, observation‑efficient method for identical‑object re‑identification in dynamic AR scenes.
2. Formulates the problem as an integer‑programming label‑assignment task grounded in spatial consistency.
3. Employs Voronoi‑diagram pruning to achieve near‑real‑time performance on commodity AR hardware.
4. Validates the approach across synthetic benchmarks, controlled real‑world experiments, and diverse application scenarios.

Limitations and Future Work

• The method assumes relatively stable inter‑object relationships; abrupt large‑scale rearrangements can degrade cost reliability.
• Accuracy depends on the quality of the initial layout scan; errors propagate if the baseline geometry is inaccurate.
• Pose estimation is currently delegated to external libraries; inaccuracies there directly affect assignment quality.

Future directions include integrating graph‑neural‑network models to capture more complex relational dynamics, developing an end‑to‑end learnable pose‑and‑assignment pipeline, and exploring distributed or incremental IP solvers to scale to scenes with dozens or hundreds of objects.

Conclusion
By leveraging a single ego‑centric observation frame, spatial consistency, and an efficiently pruned integer‑programming formulation, the paper delivers a practical solution for maintaining object identity in headset‑based AR games. The approach balances computational efficiency with high re‑identification accuracy, paving the way for more fluid physical‑virtual interactions in power‑constrained wearable AR platforms.