Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception

Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE.

💡 Research Summary

The paper addresses a critical bottleneck in training large multimodal models (LMMs): the quality of descriptive image captions that bridge vision and language. Existing caption sources fall into two categories. Human‑annotated datasets such as COCO or LAION provide scalable data but are often sparse, describing only salient objects and lacking fine‑grained attributes, spatial relations, or interaction cues. Caption generation from state‑of‑the‑art LMMs (e.g., ShareGPT‑4V, InternVL2) improves coverage but still omits important objects and relational details, as illustrated in Figure 1. These shortcomings limit the ability of LMMs to develop a comprehensive visual understanding required for complex reasoning tasks.

To remedy this, the authors propose the Enhancing Descriptive Captions Engine (EDC), a pipeline that leverages off‑the‑shelf “visual specialists” – pretrained models dedicated to specific visual tasks – and a large language model (LLM) to synthesize rich, human‑like captions. Visual specialists include detectors for object size, depth estimation networks, emotion recognizers (applied to person regions), fine‑grained classifiers for animals, plants, aircraft, logos, OCR engines, and human‑object interaction (HOI) models. Table 1 enumerates each attribute, the corresponding specialist, and the extraction procedure, ensuring reproducibility.

EDC operates in three stages. First, all specialists run in parallel on an image, producing a set of instance‑level attributes (size, depth, emotion, fine‑grained category, OCR text) and relational attributes (2‑D/3‑D spatial relations, HOI). Second, the attribute set for each detected region is fed to an LLM via carefully crafted prompts, prompting the model to generate a “region caption” that narrates the region’s visual properties in natural language. Third, another round of prompting merges the region captions with the relational attributes, yielding a single, coherent image‑level caption that captures objects, their fine‑grained properties, and their interactions. This two‑step prompting mimics the human cognitive process of first perceiving individual elements and then integrating them into a holistic description.

The authors applied EDC to annotate 1.1 million images, forming two corpora: EDC‑1M (a diverse million‑image set) and EDC‑118K (real‑world scene images). These captions were used to pre‑train two prominent LMMs, LLaVA‑v1.5 and LLaVA‑NeXT. Evaluation across 14 benchmarks—including visual question answering, image‑text retrieval, and complex reasoning tasks—showed consistent gains over baselines trained with CC3M, COCO, or ShareGPT‑4V captions. Improvements ranged from 3 to 7 percentage points, with the most pronounced benefits on tasks that require spatial reasoning or HOI understanding. Figure 2 visualizes the attribute coverage, confirming that EDC captions describe the largest number of objects, attributes, OCR tokens, and relational facts.

Technical strengths of EDC are threefold. (1) Cost‑effectiveness: it relies solely on publicly available visual specialists and LLMs, eliminating expensive human annotation. (2) Modularity: new specialists or attributes can be added without redesigning the whole pipeline, facilitating future extensions to domains such as medical imaging or video. (3) Quality: LLM‑driven text synthesis produces fluent, human‑readable sentences while preserving the detailed factual content supplied by the specialists.

The paper also acknowledges limitations. Errors from any specialist (e.g., mis‑detected depth or incorrect emotion label) can propagate into the final caption. The multi‑stage pipeline incurs higher inference latency compared to end‑to‑end caption generators. Moreover, the current work focuses on static images; extending the approach to video streams, where temporal consistency of attributes matters, remains an open challenge.

Future directions suggested include (a) confidence‑aware fusion of specialist outputs to mitigate error propagation, (b) tighter integration of visual and language modules via multimodal attention mechanisms, and (c) adaptation of the framework to video captioning and interactive embodied agents.

In summary, EDC demonstrates that enriching image captions with fine‑grained visual attributes and explicit relational information—extracted from specialized vision models and assembled by a large language model—substantially boosts the perceptual and reasoning capabilities of large multimodal models. The approach offers a scalable, extensible pathway to generate high‑quality training data, thereby advancing the state of the art in vision‑language alignment.