Object Counts! Bringing Explicit Detections Back into Image Captioning

The use of explicit object detectors as an intermediate step to image captioning - which used to constitute an essential stage in early work - is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding. We argue that explicit detections provide rich semantic information, and can thus be used as an interpretable representation to better understand why end-to-end image captioning systems work well. We provide an in-depth analysis of end-to-end image captioning by exploring a variety of cues that can be derived from such object detections. Our study reveals that end-to-end image captioning systems rely on matching image representations to generate captions, and that encoding the frequency, size and position of objects are complementary and all play a role in forming a good image representation. It also reveals that different object categories contribute in different ways towards image captioning.

💡 Research Summary

The paper revisits the role of explicit object detection in image captioning, a step that early approaches treated as essential but has largely been omitted by modern end‑to‑end neural models that condition language generation directly on a global image embedding. The authors argue that object detections provide a rich, interpretable semantic signal that can both improve caption quality and shed light on why end‑to‑end systems work.

To investigate this, they construct a simple “bag‑of‑objects” representation using the 80 COCO object categories. Each image is encoded as an 80‑dimensional vector where each dimension can be (i) the raw frequency of that category, (ii) a normalized proportion, or (iii) a binary presence flag. Despite its low dimensionality and extreme sparsity, this representation alone yields CIDEr scores that match or exceed those obtained with a ResNet‑152 POOL5 feature vector when fed into a standard Karpathy‑Fei‑Fei style LSTM caption generator. This finding demonstrates that merely knowing what objects are present is a surprisingly strong cue for caption generation.

The authors then enrich the representation with spatial cues derived from the bounding boxes of detected objects: (a) the normalized area of the largest instance per category (object size) and (b) the normalized Euclidean distance from the centre of the closest instance to the image centre (object distance). They experiment with each cue separately, as well as concatenated. Adding size information raises CIDEr from 0.807 (frequency only) to 0.836, while distance alone yields 0.759. Combining both reaches 0.849, indicating that size and centrality are complementary; size proves more informative than distance, likely because larger objects dominate visual attention and are more frequently mentioned in captions.

Pooling strategies are examined (max, min, mean) for aggregating multiple instances. Max‑pooling consistently outperforms the others, supporting the intuition that the most salient (largest or most central) instance drives caption content.

A detailed ablation study explores how much each object category contributes. By retaining only a subset of categories—selected randomly, by frequency, by size, or by proximity to the centre—and zero‑ing out the rest, the authors show that performance drops sharply when core categories such as “person”, “bicycle”, or “bench” are removed, while eliminating rarely mentioned categories (e.g., “spoon”, “banana”) has minimal impact. This aligns with prior work suggesting that captions preferentially mention semantically salient objects rather than all detected entities.

The paper also compares results using ground‑truth COCO annotations versus detections from an off‑the‑shelf YOLO‑v3 model trained on the same 80 categories. Ground‑truth based experiments achieve higher CIDEr (up to 0.849), whereas detector‑based runs fall to around 0.743, highlighting that detection errors—missed objects or false positives—directly degrade the bag‑of‑objects signal. Consequently, improving detector accuracy is a clear path to further gains.

Methodologically, the captioning model follows the Karpathy‑Fei‑Fei pipeline: the image vector is linearly projected and passed through an ELU activation to produce a 256‑dimensional initialization vector for a two‑layer LSTM with 128‑dimensional word embeddings and 256‑dimensional hidden states. Training minimizes cross‑entropy loss with teacher forcing; inference uses greedy decoding to isolate the effect of the visual representation. Hyper‑parameter details and additional metric results (BLEU, METEOR, ROUGE‑L) are provided in the appendix.

In sum, the study demonstrates that a lightweight, interpretable representation consisting of object counts, sizes, and positions can rival deep CNN features for image captioning. This challenges the prevailing belief that high‑dimensional global embeddings are indispensable and opens the door to more transparent, computationally efficient captioning systems, especially in resource‑constrained or real‑time settings. Future work should focus on jointly optimizing detection and language components, and on extending the approach to richer relational representations beyond simple bags of objects.