An Empirical Evaluation of Current Convolutional Architectures Ability to Manage Nuisance Location and Scale Variability

We conduct an empirical study to test the ability of Convolutional Neural Networks (CNNs) to reduce the effects of nuisance transformations of the input data, such as location, scale and aspect ratio. We isolate factors by adopting a common convolutional architecture either deployed globally on the image to compute class posterior distributions, or restricted locally to compute class conditional distributions given location, scale and aspect ratios of bounding boxes determined by proposal heuristics. In theory, averaging the latter should yield inferior performance compared to proper marginalization. Yet empirical evidence suggests the converse, leading us to conclude that - at the current level of complexity of convolutional architectures and scale of the data sets used to train them - CNNs are not very effective at marginalizing nuisance variability. We also quantify the effects of context on the overall classification task and its impact on the performance of CNNs, and propose improved sampling techniques for heuristic proposal schemes that improve end-to-end performance to state-of-the-art levels. We test our hypothesis on a classification task using the ImageNet Challenge benchmark and on a wide-baseline matching task using the Oxford and Fischer’s datasets.

💡 Research Summary

This paper presents a thorough empirical investigation into how well modern convolutional neural networks (CNNs) handle nuisance transformations—specifically location, scale, and aspect‑ratio variations—in visual recognition tasks. The authors contrast two processing strategies: (1) feeding the entire image to a CNN, thereby relying on the network’s internal mechanisms to marginalize over nuisance variables, and (2) first generating object proposals (bounding boxes) that explicitly reduce the variability in position, size, and shape, then applying the CNN to each proposal and averaging the resulting class‑conditional posteriors.

Using two well‑known architectures, AlexNet and VGG‑16, the authors evaluate performance on the ImageNet 2014 validation set (50 K images) and on wide‑baseline matching benchmarks (Oxford and Fischer datasets). When the CNN is applied to the whole image, top‑5 error rates are 19.96 % (AlexNet) and 13.24 % (VGG‑16). Restricting the input to the ground‑truth bounding box slightly worsens performance (20.41 % and 12.44 %) because valuable contextual information outside the box is lost. Adding a modest 10‑pixel rim around the ground‑truth box restores context and reduces error dramatically to 17.66 % and 17.65 % respectively. A systematic sweep of rim sizes shows that a rim covering roughly 25 % of the image yields the lowest error (≈15 % for AlexNet, ≈8 % for VGG‑16), indicating that current CNNs exploit only a limited neighbourhood of the object.

The authors then explore “domain‑size pooling”: they sample multiple concentric regions around the object (four or eight scales) and average the softmax outputs across these samples. This anti‑aliasing style averaging improves top‑5 error to 15.96 % (AlexNet, 4 scales) and 14.43 % (AlexNet, 8 scales), and similarly for VGG‑16 (16.00 % → 14.22 %). The improvement demonstrates that explicit marginalization over sampled nuisance transformations can outperform the implicit marginalization performed by the network.

To test the approach without ground‑truth boxes, the authors employ EdgeBoxes to generate up to 80 proposals per image. They select the proposal with the highest Intersection‑over‑Union (IoU) with the ground truth for a subset of experiments, and they introduce an information‑theoretic pruning step based on inverse Rényi entropy to discard low‑information proposals. Combining proposal‑based classification with domain‑size pooling and standard data augmentation (horizontal flips, multi‑crop) yields a 5–15 % mean average precision gain on the matching benchmarks and a 9–10 % relative reduction in top‑5 error on ImageNet compared with the baseline of 150 regular crops.

Overall, the study reveals that contemporary CNNs are not fully invariant to location‑scale nuisances; they rely heavily on limited surrounding context and benefit from explicit sampling and averaging of transformed inputs. This finding aligns with the Data Processing Inequality, which predicts that conditioning on a subset of the data (proposals) should not improve performance unless the conditioning discards irrelevant information. The authors’ results show the opposite: careful proposal selection and anti‑aliased averaging can surpass the naïve whole‑image approach. The paper contributes a practical pipeline—proposal generation, entropy‑based pruning, and domain‑size pooling—that achieves state‑of‑the‑art single‑model performance on ImageNet while remaining computationally tractable. All code and scripts are publicly released for reproducibility.