Augmented Reality Applied to LEGO Construction: AR-based Building Instructions with High Accuracy & Precision and Realistic Object-Hand Occlusions

Enabling spatial play, construction toys serve an important role as metaphors for scientific principles, spatial skills and math aptitude [14,24,29,42]. The improvement of the performance of construction toy players in terms of building time and number of building errors contributes to the players' overall learning [11]. The performance heavily depends on the instructions that guide the construction process. There are two primary tasks in designing the instructions: (1) Planning a sequence of assembly operations for users to understand and follow easily, and (2) Presenting the assembly operations clearly in a series of diagrams [3]. This project has developed an Augmented Reality (AR)-based instruction method: BRICKxAR. With BRICKxAR, physical toy construction (LEGO as an example) is guided by virtual bricks in the right place at the right time, step by step. Information augmentation is created for specific bricks about architecture and construction knowledge. Physical and virtual object occlusion is implemented to enable a natural appearance of virtual bricks on the physical model. Players' hand detection and occlusion are accomplished to allow a realistic immersive AR experience, in which virtual bricks can be "grasped" by the real hand, revealing correct spatial relationship of objects.

In BRICKxAR, the physical model can be moved and rotated freely on a desk surface, and the AR device camera can move in 6-Degrees of Freedom (DoF); in the meantime, high accuracy of AR registration -the virtual model’s alignment with the physical model -is achieved through Computer Vision-powered marker-based registration using camera and motion sensors (including gyroscope, accelerometer, and magnetometer). The average error of the registration is less than 1mm throughout the entire model when a major part or full of the marker is within the AR camera’s field of view. Compared with the state-of-the-art of AR instructions, BRICKxAR significantly improved accuracy of model registration. To validate the project contributions, LEGO Architecture 21036 Arc de Triomphe in Paris is built completely for its 386 steps with a working prototype of BRICKxAR in experiments for quantitative measures of the accuracies of registration and occlusions, and for heuristic evaluation of AR instruction features by comparing BRICKxAR with design principles of assembly instructions and AR design guidelines suggested by literature.

Major assembly and construction projects are increasingly complex [45]. In recent years, LEGO sets also become more complex and labor intensive [27]. Some sets have thousands of pieces, e.g. LEGO 75192 Star Wars Millennium Falcon has 7,541 pieces and an instruction booklet of 496 pages. In the meantime, Augmented Reality (AR) can superimpose digital images on the real world view of users, having the potentials to benefit manufacturing, building construction, and part assembly significantly. AR has been studied as education and research tools, e.g., in the interactive AR assembly guidance and learning of Tou-Kung (a sophisticated cantilevered bracket system in traditional East Asian architecture) [10], in tutoring machine tasks with AR-assisted spatial interactions [9], and in the research of spatial design and urban planning problems by projecting data visualization onto LEGO models [4]. The LEGO Group uses AR to mix touch-screen game components, such as audiovisual effects, with the physical models, in order to make LEGO play more engaging and fun [26]. However, one of the core LEGO building experiences -the construction process -may also be enriched from the transformational AR technology, which is the goal of BRICKxAR.

Instructions A recent study about a 2D projection-based AR-assistive system shows only small benefits in the training scenario, in which AR training does not reach the personal training in terms of speed and recall precision after 24 hours [8]. However, many studies have verified that 3D-model-based AR instructions can significantly save task completion time [20] and reduce the error rate [35,41]. Funk et al. proposed using Duplo (a larger version of LEGO with each dimension doubled) and an artificial industry task for evaluating AR instructions in terms of assembly time and errors, with paper-based instructions as baselines [16]. Westerfield et al. evaluated participants’ performance for assembly of computer motherboards, and found significant improvement using AR with a feedback system [44]. They also pointed out the drawback of limited accuracy in AR tracking. Schwald and de Laval presented an AR prototype for assisting in equipment maintenance and concluded that the results were positive, but improvements were needed for the accuracy of 3D augmentation [36]. Tang et al. compared an AR-based instruction system with a printed manual, computer-assisted instructions on a monitor, and a Head-Mounted Display for Duplo assembly [40]. Their user evaluations support the proposition that AR systems impro

…(Full text truncated)…

This content is AI-processed based on ArXiv data.