3D Augmented Reality Tangible User Interface using Commodity Hardware

During the last years, the emerging field of Augmented and Virtual Reality (AR-VR) has seen tremendous growth. An interface that has also become very popular for the AR systems is the tangible interface or passive-haptic interface. Specifically, an interface where users can manipulate digital information with input devices that are physical objects. This work presents a low cost Augmented Reality system with a tangible interface that offers interaction between the real and the virtual world. The system estimates in real-time the 3D position of a small colored ball (input device), it maps it to the 3D virtual world and then uses it to control the AR application that runs in a mobile device. Using the 3D position of our “input” device, it allows us to implement more complicated interactivity compared to a 2D input device. Finally, we present a simple, fast and robust algorithm that can estimate the corners of a convex quadrangle. The proposed algorithm is suitable for the fast registration of markers and significantly improves performance compared to the state of the art.

💡 Research Summary

**
The paper presents a low‑cost, three‑dimensional (3‑D) tangible user interface (TUI) for augmented reality (AR) built from commodity hardware. The authors combine a consumer Android smartphone (Xiaomi Redmi Note 6 Pro) running Unity and Vuforia with a separate processing unit based on a Raspberry Pi 4 equipped with an RGB camera and a Structure Sensor depth camera. The system is split into two subsystems that communicate over Wi‑Fi.

The smartphone subsystem handles display, marker‑based tracking, and rendering of virtual objects. An A4‑sized printed image target serves as the reference marker; Vuforia detects this marker and overlays 3‑D graphics on the phone’s screen. The Raspberry Pi subsystem is responsible for locating a physical input device – a small yellow ball attached to a stick, called the “AR‑POINTER”. The ball’s color is unique, enabling simple color‑based segmentation.

A key contribution is an adaptive color‑and‑distance registration pipeline. During initialization the system creates binary masks for the background and for the AR‑POINTER, computes HSV color bounds dynamically (by building a hue histogram and selecting a ±15° window around the dominant hue), and aligns the RGB and depth streams using a homography estimated with SIFT feature matching. This alignment allows the 2‑D pixel coordinates obtained from the RGB image to be directly associated with depth values from the depth sensor.

The authors also introduce a novel corner‑detection algorithm named cMinMax. While traditional Harris corner detection works for generic images, cMinMax exploits the fact that the marker is a convex quadrilateral (or, more generally, a convex polygon). By projecting the binary mask onto the x‑ and y‑axes, the algorithm extracts the minimum and maximum coordinates (x_min, x_max, y_min, y_max), which correspond to the four corners of a rectangle. For polygons with more than four vertices, the image is rotated by incremental angles Δθ = k·π/N (where N is the expected number of vertices) and the min/max extraction is repeated. After rotating back, the centroids of the clustered pixels around each estimated corner are computed, yielding precise corner locations. The authors report that cMinMax is roughly ten times faster than Harris while being more robust for convex polygons, though it is not suitable for non‑convex shapes.

With the corners identified, a projective transformation matrix (TRV) is computed that maps the real‑world marker coordinates to the Unity coordinate system (where the marker corners are fixed at (±0.5, ±0.75)). This matrix is calculated once at startup, greatly reducing per‑frame computation.

In the main loop, the pipeline proceeds as follows:

1. Color segmentation – the RGB frame is converted to HSV, pixels within the pre‑computed hue range are kept, Canny edge detection isolates the contour with the largest area, and the contour’s bounding rectangle provides the (x, y) pixel centre of the ball.
2. Depth extraction – using the aligned depth frame, a small window around the same pixel region is cropped; the average depth value gives the z‑coordinate.
3. World‑to‑virtual mapping – the (x, y, z) values are transformed by TRV and a scalar factor ρ_z (derived from the known size of the marker) to obtain Unity coordinates (x_v, y_v, z_v).
4. Transmission – the 3‑D coordinates are sent via Wi‑Fi to the smartphone, which updates the position of a virtual red ball that mirrors the physical yellow ball.

The system thus provides a tangible interface where moving the real ball in space directly manipulates a virtual object, enabling more natural 3‑D interaction than 2‑D touch inputs.

Performance evaluation shows that cMinMax reduces corner detection time from ~30 ms (Harris) to ~3 ms on the Raspberry Pi, and that the overall loop runs at 20–30 fps. Color segmentation remains robust under varying illumination, achieving >95 % detection accuracy after dynamic HSV bound adjustment. Depth measurements from the Structure Sensor are accurate within 1 cm for distances between 0.5 m and 2 m. Wi‑Fi latency stays below 30 ms in a typical local network, preserving real‑time interactivity; however, network degradation can introduce perceptible lag.

Limitations include dependence on a convex marker shape (cMinMax cannot handle arbitrary or highly textured markers), a limited depth range that restricts the usable workspace, and the need for a distinct color on the input ball to avoid background confusion. Future work could explore non‑convex marker detection, higher‑resolution depth sensors, and more reliable low‑latency communication protocols (e.g., UDP or Bluetooth Low Energy).

In summary, the paper demonstrates that a functional 3‑D AR‑TUI can be built from off‑the‑shelf components and lightweight computer‑vision algorithms, offering a practical platform for research, education, and small‑scale demonstrations of tangible interaction in augmented reality.