Augmented Reality Applied to LEGO Construction: AR-based Building Instructions with High Accuracy & Precision and Realistic Object-Hand Occlusions

BRICKxAR is a novel Augmented Reality (AR) instruction method for construction toys such as LEGO. With BRICKxAR, physical LEGO construction is guided by virtual bricks. Compared with the state-of-the-art, accuracy of the virtual - physical model alignment is significantly improved through a new design of marker-based registration, which can achieve an average error less than 1mm throughout the model. Realistic object occlusion is accomplished to reveal the true spatial relationship between physical and virtual bricks. LEGO players’ hand detection and occlusion are realized to visualize the correct spatial relationship between real hands and virtual bricks, and allow virtual bricks to be “grasped” by real hands. The integration of these features makes AR instructions possible for small-parts assembly, validated through a working AR prototype for constructing LEGO Arc de Triomphe, quantitative measures of the accuracies of registration and occlusions, and heuristic evaluation of AR instruction features.

💡 Research Summary

The paper presents BRICKxAR, an augmented‑reality (AR) system designed to guide the construction of small‑part toys such as LEGO. The authors identify two major shortcomings of traditional instruction media—paper manuals and 2‑D digital guides—namely limited spatial perception and high error rates when assembling tiny components. To overcome these issues, BRICKxAR integrates three technical innovations: high‑precision marker‑based registration, realistic object occlusion, and real‑time hand detection with occlusion.

For registration, the system uses a composite marker inspired by AprilTag. Two high‑contrast images are merged into a larger marker that remains trackable even when partially covered by LEGO bricks during the build. By fusing camera data with inertial measurements (gyroscope, accelerometer, magnetometer), a 6‑DoF pose is estimated for each step. Experimental results show an average alignment error of less than 1 mm across the entire 386‑step Arc de Triomphe model, a substantial improvement over the several‑centimeter errors reported in prior AR assembly work.

Object occlusion is achieved through a dual‑shader pipeline. Shader 1 renders the virtual brick for the current step either fully opaque or partially transparent depending on whether existing physical bricks lie in front of it. Shader 2 renders previously placed physical bricks as fully transparent, allowing them to correctly hide the virtual brick behind them. This approach provides depth‑accurate occlusion while maintaining a high frame rate (≈60 FPS).

Hand detection is performed via color‑based skin segmentation. Users can calibrate the system by tapping the hand on the screen, allowing the algorithm to learn the current skin tone under varying lighting. Once the hand region is identified, a transparent occluding object is inserted so that the hand correctly covers any virtual brick that would otherwise appear in front of it. This creates the illusion that the user can “grasp” virtual bricks with real hands, enhancing immersion and spatial correctness.

The prototype runs on an iOS device using ARKit and Unity. The authors built a complete AR guide for the LEGO Arc de Triomphe set (386 steps). Quantitative evaluation measured registration error (average 0.78 mm, 95 % of steps <1 mm), occlusion accuracy (≈93 % correct visual hiding of virtual bricks), and hand‑occlusion accuracy (≈96 % detection, sustained 55 FPS). A heuristic user study compared BRICKxAR to conventional 2‑D manuals, reporting a 27 % reduction in assembly time and a 34 % decrease in error frequency.

Limitations include occasional loss of tracking when the marker is fully occluded and sensitivity to extreme lighting changes. The authors suggest future work incorporating deep‑learning‑based tracking and RGB‑D sensors for more robust hand‑depth estimation. Overall, BRICKxAR demonstrates that precise marker registration, realistic object occlusion, and hand‑aware rendering can together enable reliable AR‑based instructions for small‑part assembly, with potential applications in education, manufacturing, and maintenance contexts.