Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success

3D reconstruction serves as the foundational layer for numerous robotic perception tasks, including 6D object pose estimation and grasp pose generation. Modern 3D reconstruction methods for objects can produce visually and geometrically impressive meshes from multi-view images, yet standard geometric evaluations do not reflect how reconstruction quality influences downstream tasks such as robotic manipulation performance. This paper addresses this gap by introducing a large-scale, physics-based benchmark that evaluates 6D pose estimators and 3D mesh models based on their functional efficacy in grasping. We analyze the impact of model fidelity by generating grasps on various reconstructed 3D meshes and executing them on the ground-truth model, simulating how grasp poses generated with an imperfect model affect interaction with the real object. This assesses the combined impact of pose error, grasp robustness, and geometric inaccuracies from 3D reconstruction. Our results show that reconstruction artifacts significantly decrease the number of grasp pose candidates but have a negligible effect on grasping performance given an accurately estimated pose. Our results also reveal that the relationship between grasp success and pose error is dominated by spatial error, and even a simple translation error provides insight into the success of the grasping pose of symmetric objects. This work provides insight into how perception systems relate to object manipulation using robots.

💡 Research Summary

The paper addresses a critical gap in robotics research: while 3D reconstruction and 6‑D object pose estimation have each seen impressive advances, they are traditionally evaluated with geometric metrics that do not reflect their impact on downstream manipulation tasks. To bridge this gap, the authors introduce a large‑scale, physics‑based benchmark that directly measures how reconstruction fidelity and pose estimation errors affect robotic grasping success.

Using the YCB‑Video (YCB‑V) dataset, nine widely used grippers, and the PyBullet simulator, the authors generate millions of grasp attempts. They define a transformation chain linking world, camera, object, and gripper frames, allowing them to compute the gripper’s target pose from an estimated object pose (T_est^c→o) and a pre‑computed canonical grasp (T_o→g). In simulation, the robot executes the grasp based on the estimated pose but interacts with a ground‑truth object, faithfully reproducing the real‑world scenario where perception is imperfect.

Three experimental conditions are explored: (1) an ideal baseline where both grasp generation and pose estimation use the perfect CAD model; (2) isolation of pose error by using the perfect CAD model for grasp generation but a reconstructed mesh for pose estimation; and (3) a fully realistic end‑to‑end case where the same reconstructed mesh is used for both grasp generation and pose estimation. Two functional metrics are introduced: Grasp Generation Success Rate (S_gen), which measures the proportion of sampled grasps that are physically successful on a given mesh, and Estimated Success Rate (S_est), which measures how many grasps that succeed with the ground‑truth pose also succeed when the estimated pose is used. Failure modes are further broken down into slip, no‑contact, and collision.

The benchmark evaluates a diverse set of reconstruction techniques—including NeRF‑based Instant‑NGP, NeRFacto, Neuralangelo, implicit surface methods such as UniSurf, MonoSDF, VolSDF, BakedSDF, and a commercial photogrammetry pipeline (RealityCapture)—and two state‑of‑the‑art pose estimators, MegaPose and FoundationPose.

Key findings are: (i) reconstruction artifacts dramatically reduce the number of viable grasp candidates (lower S_gen), but when the pose estimate is accurate, the final grasp success (S_est) is largely unaffected; thus, pose accuracy dominates grasp performance. (ii) Spatial (translation) errors have a stronger negative impact than rotational errors, especially for symmetric objects where a simple translation error already predicts success or failure. (iii) Position errors exceeding ~5 mm cause a sharp rise in “no‑contact” failures, while rotation errors become critical only beyond larger thresholds. (iv) Among grippers, the Robotiq 2F‑85 and WSG 50 achieve the highest average S_gen, confirming that gripper geometry interacts strongly with object shape. (v) Both MegaPose and FoundationPose maintain high S_est (>90 %) when their ADD errors stay below 2 mm, but performance collapses when errors exceed ~10 mm.

The authors conclude that, for manipulation‑oriented perception pipelines, investing in more accurate pose estimation yields greater returns than pursuing ever‑higher mesh fidelity, provided that the mesh is sufficient to generate a reasonable set of grasps. Nevertheless, poor reconstruction can limit the grasp search space, suggesting that future work should jointly optimize mesh quality and grasp sampling strategies, and that error‑aware grasp planners could compensate for predictable translation errors, especially for symmetric objects. This benchmark offers a reproducible framework for evaluating perception‑manipulation pipelines in a function‑centric manner, paving the way for more robust, real‑world robotic systems.