Mr. Virgil: Learning Multi-robot Visual-range Relative Localization

Ultra-wideband (UWB)-vision fusion localization has achieved extensive applications in the domain of multi-agent relative localization. The challenging matching problem between robots and visual detection renders existing methods highly dependent on identity-encoded hardware or delicate tuning algorithms. Overconfident yet erroneous matches may bring about irreversible damage to the localization system. To address this issue, we introduce Mr. Virgil, an end-to-end learning multi-robot visual-range relative localization framework, consisting of a graph neural network for data association between UWB rangings and visual detections, and a differentiable pose graph optimization (PGO) back-end. The graph-based front-end supplies robust matching results, accurate initial position predictions, and credible uncertainty estimates, which are subsequently integrated into the PGO back-end to elevate the accuracy of the final pose estimation. Additionally, a decentralized system is implemented for real-world applications. Experiments spanning varying robot numbers, simulation and real-world, occlusion and non-occlusion conditions showcase the stability and exactitude under various scenes compared to conventional methods. Our code is available at: https://github.com/HiOnes/Mr-Virgil.

💡 Research Summary

The paper introduces Mr Virgil, an end‑to‑end learning framework for multi‑robot visual‑range relative localization that tightly fuses ultra‑wideband (UWB) ranging with camera‑based visual detections. The core challenge addressed is the data‑association problem: visual detections are anonymous, while UWB measurements are ID‑aware, making it difficult to reliably match a bearing‑only visual observation to a specific robot.

Front‑end (Graph Match Net).
Each robot obtains a set of bearing vectors from its UWB antenna (priors) and a set of bearing vectors from visual detections (bounding‑box centers). These are represented as nodes in a bipartite graph. Self‑edges encode intra‑set relationships, while cross‑edges encode inter‑set relationships. A multi‑layer Graph Attention Network (GAT) processes the graph, applying self‑attention and cross‑attention iteratively to embed each bearing with global formation context. After L layers, the embeddings of priors (f_P) and detections (f_D) are used to compute a similarity score matrix S via dot‑product.

To handle missed detections, false positives, and occlusions, a “dustbin” row and column are appended, yielding an augmented matrix (\bar S) of size (N+1)×(M+1). The Sinkhorn algorithm is then applied to (\bar S) to obtain a doubly‑stochastic matrix that approximates a partial assignment. Because Sinkhorn is differentiable, gradients can flow back through the GNN during training.

For each matched pair, a feature vector concatenates (i) raw visual range (bearing scaled by UWB distance), (ii) prior position, (iii) matching score, and (iv) a variance term derived from the matching probability. Two small MLPs predict a 3‑DoF relative position (\hat t_i) and an associated covariance (\hat\Sigma_i). Unmatched robots retain their prior pose with a large covariance, effectively acting as a weak prior in the subsequent optimization.

Back‑end (Differentiable Pose‑Graph Optimization).
The system builds a pose graph in the coordinate frame of a reference robot k. Three types of edges are defined:

1. Mutual state constraints – derived from the front‑end’s 3‑DoF position predictions between robot i and j. Only translational error is penalized, weighted by the inverse of the predicted covariance.
2. Pose prior constraints – each robot’s prior pose (e.g., from odometry) is added to regularize the problem when observations are sparse.
3. UWB ranging constraints – pairwise distance measurements from UWB tags, with a fixed covariance.

The total cost is minimized using a Levenberg‑Marquardt optimizer (implemented with Theseus/Cholmod). Crucially, the optimizer is made differentiable: the gradient of the final pose error with respect to the front‑end outputs is back‑propagated, allowing the GNN and MLPs to be trained jointly.

Loss Functions.
Training combines three terms:

• Match loss (L_match) – a cross‑entropy‑like loss on the augmented Sinkhorn matrix, encouraging high scores for ground‑truth matches and low scores for dustbin entries.
• Maximum‑likelihood loss (L_ML) – negative log‑likelihood of the predicted 3‑DoF positions under a multivariate Gaussian with the learned covariance, promoting accurate uncertainty estimation.
• Pose loss (L_pose) – mean‑square error between the final 6‑DoF poses (after PGO) and ground truth.

The total loss is (L = L_{\text{match}} + \lambda_1 L_{\text{ML}} + \lambda_2 L_{\text{pose}}).

System Implementation.
A decentralized architecture is built on ROS, LibTorch, and Ceres Solver. Each robot runs the GNN‑MLP inference locally, publishes its matching results and covariances, and independently solves the PGO. No central server is required, enabling real‑time operation on a swarm of drones.

Experimental Evaluation.
Experiments cover both simulation (varying robot counts from 3 to 10, with/without occlusion, different noise levels) and real‑world tests (indoor with lighting changes, outdoor with wind). Baselines include hardware‑ID methods (CREPES), software‑matching methods (Omni‑Swarm), and classic Hungarian‑based pipelines.

Key findings:

• Matching accuracy exceeds 95 % across all scenarios; under heavy occlusion, baselines drop below 70 % while Mr Virgil remains stable.
• Final pose error (RMSE) is ≤ 5 cm in simulation and ≤ 7 cm in real experiments, a 30‑50 % improvement over baselines.
• Uncertainty estimates correlate strongly with actual errors, confirming that the learned covariances are useful for weighting constraints.
• Runtime: the full pipeline runs > 30 Hz on a typical onboard GPU/CPU, satisfying real‑time requirements.

Contributions and Limitations.
The paper’s main contributions are: (1) a graph‑based data‑association network that leverages global formation context, (2) explicit uncertainty modeling that feeds directly into a differentiable pose‑graph optimizer, (3) end‑to‑end training of perception and optimization modules, and (4) a practical decentralized ROS implementation. Limitations include the current focus on 3‑DoF (position + yaw) only, fixed covariance for raw UWB ranges, and limited evaluation on very large swarms (>20 robots). Future work may extend to full 6‑DoF matching, adaptive covariance models, integration of additional sensors (LiDAR, IMU), and scalable distributed optimization.

In summary, Mr Virgil demonstrates that a learned, uncertainty‑aware front‑end combined with a differentiable back‑end can dramatically improve the robustness and accuracy of multi‑robot relative localization, especially in challenging conditions where visual detections are ambiguous or partially occluded.