LAMP: Implicit Language Map for Robot Navigation

Recent advances in vision-language models have made zero-shot navigation feasible, enabling robots to follow natural language instructions without requiring labeling. However, existing methods that explicitly store language vectors in grid or node-based maps struggle to scale to large environments due to excessive memory requirements and limited resolution for fine-grained planning. We introduce LAMP (Language Map), a novel neural language field-based navigation framework that learns a continuous, language-driven map and directly leverages it for fine-grained path generation. Unlike prior approaches, our method encodes language features as an implicit neural field rather than storing them explicitly at every location. By combining this implicit representation with a sparse graph, LAMP supports efficient coarse path planning and then performs gradient-based optimization in the learned field to refine poses near the goal. This coarse-to-fine pipeline, language-driven, gradient-guided optimization is the first application of an implicit language map for precise path generation. This refinement is particularly effective at selecting goal regions not directly observed by leveraging semantic similarities in the learned feature space. To further enhance robustness, we adopt a Bayesian framework that models embedding uncertainty via the von Mises-Fisher distribution, thereby improving generalization to unobserved regions. To scale to large environments, LAMP employs a graph sampling strategy that prioritizes spatial coverage and embedding confidence, retaining only the most informative nodes and substantially reducing computational overhead. Our experimental results, both in NVIDIA Isaac Sim and on a real multi-floor building, demonstrate that LAMP outperforms existing explicit methods in both memory efficiency and fine-grained goal-reaching accuracy.

💡 Research Summary

LAMP (Language Map) presents a novel framework for zero‑shot robot navigation that leverages large vision‑language models (VLMs) without the need for explicit semantic labeling. The core idea is to replace the traditional practice of storing language embeddings at every grid cell or graph node with an implicit neural language field that continuously maps a 7‑dimensional camera pose (3‑D position + quaternion) to a unit‑norm CLIP embedding. By training this network, denoted (F_{\Theta}), on pose‑image pairs collected during an extensive exploration phase, the system learns a smooth, high‑dimensional representation of the environment’s semantic content using only RGB data.

Two‑stage navigation pipeline

1. Coarse planning – A sparse topological graph (G = (V, E)) retains only pose information for each node. Embeddings are generated on‑demand via (F_{\Theta}), dramatically reducing memory consumption compared with pre‑computed storage. Given a natural‑language query, the system encodes the query into a goal embedding and runs A* (or Dijkstra) on a sampled subgraph to locate the node whose embedding maximizes cosine similarity with the goal.
2. Fine refinement – Starting from the coarse node, LAMP performs gradient‑based optimization directly in the implicit field: it maximizes the cosine similarity between the field’s output and the goal embedding, moving the robot’s pose toward a location that offers the best semantic view of the target. This continuous optimization yields sub‑meter accuracy, far surpassing the discretization limits of grid‑based maps.

Bayesian uncertainty modeling
Because CLIP embeddings aggregate information from multiple objects, they can be noisy, especially when the robot has never observed a particular viewpoint. LAMP addresses this by modeling each predicted embedding as a von Mises‑Fisher (vMF) distribution on the unit hypersphere. The network outputs both a mean direction (\mu_{\Theta}(x)) and a concentration parameter (\kappa_{\Theta}(x)). A Gamma prior on (\kappa) prevents over‑confidence. The loss combines the negative log‑likelihood of the observed embedding under the vMF distribution with the log‑prior of (\kappa). Consequently, the system obtains a per‑pose uncertainty estimate that is later used for graph pruning and for weighting the gradient‑based refinement, improving robustness in unobserved or ambiguous regions.

Graph sampling strategy
In large‑scale environments, a dense topological graph would be computationally prohibitive. LAMP introduces a score‑based node selection that combines three complementary criteria:

• View‑Coverage Score ((s_{vc})) – measures orientation diversity by dot‑product of quaternions within a two‑hop neighborhood; nodes offering novel viewpoints receive higher scores.
• Uncertainty Score ((s_u)) – derived from the concentration (\kappa); nodes with high confidence (large (\kappa)) are favored.
• Semantic Sensitivity Score ((s_{ss})) – the norm of the gradient (|\nabla_x F_{\Theta}(x)|); a large gradient indicates that small pose changes cause large semantic shifts, making the node valuable for capturing transition zones.

A weighted sum of these scores ((w_1 s_{vc} + w_2 s_u + w_3 s_{ss})) ranks nodes, and only the top‑N are retained. This pruning yields a compact graph that still preserves semantically rich and well‑covered regions, enabling fast global search while keeping memory usage low.

Experimental validation
The authors evaluate LAMP in two settings:

• Simulation (NVIDIA Isaac Sim) – Multi‑floor indoor maps with diverse objects. Compared to explicit grid‑based and node‑based baselines, LAMP reduces memory consumption by ~85 % and improves goal‑position error from 0.35 m (grid) to 0.12 m.
• Real‑world multi‑floor building – A mobile robot equipped with an RGB camera follows commands such as “red oak tree” or “kitchen table”. LAMP achieves a 93 % success rate versus 78 % for the strongest baseline, and reduces average time‑to‑goal by 1.8 s. Notably, when the target region is not directly observed, the vMF‑based uncertainty guides the optimizer toward semantically similar areas, allowing successful navigation despite missing visual evidence.

Contributions and limitations
The paper’s primary contributions are: (1) the first implicit language map that enables fine‑grained navigation using only RGB inputs; (2) a Bayesian vMF formulation that quantifies and leverages embedding uncertainty; (3) a principled graph sampling method that balances view coverage, confidence, and semantic sensitivity. Limitations include reliance on a static environment, computational cost of CLIP inference, and the need for manual tuning of the score weights. Future work could explore dynamic obstacle handling, lightweight VLMs for on‑device deployment, and meta‑learning approaches to automatically set sampling hyper‑parameters.

In summary, LAMP demonstrates that continuous, uncertainty‑aware language representations can replace bulky explicit storage, delivering scalable, memory‑efficient, and highly accurate zero‑shot navigation for robots operating in large, complex indoor spaces.