User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments

Open-set object detection (OSOD) localizes objects while identifying and rejecting unknown classes at inference. While recent OSOD models perform well on benchmarks, their behavior under realistic user prompting remains underexplored. In interactive XR settings, user-generated prompts are often ambiguous, underspecified, or overly detailed. To study prompt-conditioned robustness, we evaluate two OSOD models, GroundingDINO and YOLO-E, on real-world XR images and simulate diverse user prompting behaviors using vision-language models. We consider four prompt types: standard, underdetailed, overdetailed, and pragmatically ambiguous, and examine the impact of two enhancement strategies on these prompts. Results show that both models exhibit stable performance under underdetailed and standard prompts, while they suffer degradation under ambiguous prompts. Overdetailed prompts primarily affect GroundingDINO. Prompt enhancement substantially improves robustness under ambiguity, yielding gains exceeding 55% mIoU and 41% average confidence. Based on the findings, we propose several prompting strategies and prompt enhancement methods for OSOD models in XR environments.

💡 Research Summary

The paper investigates how realistic user‑generated natural‑language prompts affect the performance of open‑set object detection (OSOD) models in extended‑reality (XR) environments. While recent OSOD systems such as GroundingDINO and YOLO‑E have achieved strong results on standard benchmarks (COCO, LVIS), those evaluations assume clean, well‑formed textual inputs. In practice, XR users often issue prompts that are ambiguous, under‑specified, or overly detailed, creating a source of uncertainty that has not been systematically studied.

To fill this gap, the authors curate a dataset of 264 real‑world AR images drawn from the DiverseAR and DiverseAR+ collections, covering indoor scenes with clutter, occlusions, and virtual overlays. For each image a target object is manually selected and precisely annotated with bounding boxes, ensuring a consistent ground‑truth reference across all prompt variants.

Prompt generation is automated using a vision‑language model (VLM). Four prompt categories are defined:

1. Standard prompts – concise, containing only essential attributes (e.g., “red water bottle”).
2. Underdetailed prompts – missing key descriptors (e.g., “bottle”).
3. Overdetailed prompts – contain many attributes, some redundant or conflicting (e.g., “transparent plastic, glossy, with a rounded top and metallic lid”).
4. Pragmatically ambiguous prompts – the user’s intent is clear from context but the language is indirect (e.g., “I’m thirsty, I need water”).

Both GroundingDINO (cross‑modal transformer) and YOLO‑E (lightweight real‑time detector with language head) are evaluated on each prompt type. Metrics include mean Intersection‑over‑Union (mIoU) and average confidence score. Results show that both models are robust to standard and underdetailed prompts, but suffer substantial drops under pragmatic ambiguity. Overdetailed prompts particularly degrade GroundingDINO, while YOLO‑E is less affected.

To mitigate these failure modes, the authors propose two VLM‑based prompt‑enhancement strategies:

• Key object extraction – the VLM parses the user’s sentence and extracts the core noun phrase (the object name), discarding extraneous modifiers.
• Semantic category grounding – the VLM maps descriptive attributes to a predefined set of semantic categories (color, shape, material) and reconstructs a concise, well‑structured prompt that preserves the most salient information.

Applying these enhancements before feeding the prompt to the OSOD models yields dramatic improvements: mIoU gains exceeding 55 % and average confidence increases over 41 % for ambiguous prompts. GroundingDINO recovers notably from overdetailed inputs, while YOLO‑E shows the largest boost under pragmatic ambiguity.

The paper formalizes prompt‑conditioned robustness as a new robustness dimension that isolates performance variability caused solely by linguistic changes while keeping the visual scene fixed. This complements traditional robustness analyses focused on visual distribution shifts or unseen categories.

Finally, the authors outline practical prompting guidelines for XR applications: (1) run the user’s raw input through a VLM to extract the key object, (2) optionally apply semantic grounding to prune or prioritize attributes, and (3) feed the refined prompt to the OSOD model. This pipeline can be executed in real time, preserving user experience while substantially increasing detection reliability.

In summary, the study demonstrates that OSOD models are highly sensitive to the quality and structure of natural‑language prompts in XR settings, identifies under‑specification, over‑specification, and pragmatic ambiguity as primary failure patterns, and shows that VLM‑driven prompt post‑processing is an effective, scalable remedy. Future work should explore interactive, conversational correction loops and multimodal feedback mechanisms to further close the gap between user intent and model perception in immersive XR systems.