Variance & Greediness: A comparative study of metric-learning losses

Metric learning is central to retrieval, yet its effects on embedding geometry and optimization dynamics are not well understood. We introduce a diagnostic framework, VARIANCE (intra-/inter-class variance) and GREEDINESS (active ratio and gradient norms), to compare seven representative losses, i.e., Contrastive, Triplet, N-pair, InfoNCE, ArcFace, SCL, and CCL, across five image-retrieval datasets. Our analysis reveals that Triplet and SCL preserve higher within-class variance and clearer inter-class margins, leading to stronger top-1 retrieval in fine-grained settings. In contrast, Contrastive and InfoNCE compact embeddings are achieved quickly through many small updates, accelerating convergence but potentially oversimplifying class structures. N-pair achieves a large mean separation but with uneven spacing. These insights reveal a form of efficiency-granularity trade-off and provide practical guidance: prefer Triplet/SCL when diversity preservation and hard-sample discrimination are critical, and Contrastive/InfoNCE when faster embedding compaction is desired.

💡 Research Summary

The paper introduces a diagnostic framework—VARIANCE (intra‑ and inter‑class variance) and GREEDINESS (active‑sample ratio and gradient norm)—to systematically analyze how different metric‑learning loss functions shape embedding geometry and training dynamics. Seven representative supervised losses are examined: Contrastive, Triplet, N‑pair, InfoNCE, ArcFace, Supervised Contrastive Learning (SCL), and Center Contrastive Loss (CCL). Experiments are conducted on five image‑retrieval benchmarks of varying granularity (CIFAR‑10, Cars196, CUB‑200, Tiny‑ImageNet, FashionMNIST) using a frozen Vision Transformer (ViT‑B/32) backbone and a 128‑dimensional L2‑normalized projection head. All models are trained under identical settings (Adam, 1e‑4 learning rate, 100 epochs, batch size 512).

VARIANCE quantifies the final embedding space: intra‑class variance (σ²_intra) reflects how dispersed samples of the same class are, while inter‑class variance (σ²_inter) captures the spread of class centroids. GREEDINESS captures optimization efficiency: the active‑sample ratio measures the proportion of batch elements that incur non‑zero loss, and the overall gradient L2 norm indicates the magnitude of parameter updates. High active ratio with low gradient norm denotes “greedy” learning (many small updates), whereas low active ratio with high gradient norm denotes “non‑greedy” learning (few but strong updates).

Key empirical findings:

1. Triplet and SCL consistently produce the largest σ²_intra, preserving substantial within‑class diversity, while also achieving high σ²_inter, yielding clear class margins. Their active ratios are modest (~38 %), but gradient norms are relatively high (~0.27), indicating focused learning on hard examples. This behavior translates into superior top‑1 recall on fine‑grained datasets (Cars196, CUB‑200).

2. Contrastive and InfoNCE achieve the smallest σ²_intra, rapidly compressing each class into tight clusters. They exhibit high active ratios (~60‑65 %) and low gradient norms (~0.12), reflecting many small updates across the batch. Consequently, they converge quickly and excel on coarse‑grained tasks (CIFAR‑10, FashionMNIST) but tend to plateau on harder, fine‑grained cases where subtle intra‑class structure matters.

3. N‑pair often yields large inter‑class mean distances but with high inter‑class variance, indicating uneven spacing of class centroids. This irregularity can cause nearest‑neighbor failures despite strong average separation, especially on datasets with many classes (Tiny‑ImageNet).

4. ArcFace and CCL report near‑zero intra‑class variance, a symptom of metric‑scale mismatch (angular margin vs. cosine distance) rather than genuine clustering quality. Their recall scores are consistently lower than other losses.

The authors synthesize these observations into an “efficiency‑granularity trade‑off”: greedy losses prioritize rapid embedding compaction and early performance gains, at the cost of eroding intra‑class diversity; non‑greedy, margin‑based losses sacrifice speed for richer class structure and better discrimination in fine‑grained settings.

Practical guidance derived from the study:

• For tasks requiring fine‑grained discrimination (e.g., vehicle model retrieval, bird species identification), prefer Triplet or SCL to maintain intra‑class variance and focus learning on hard samples.
• For tasks where fast convergence and compact embeddings are more valuable (e.g., generic object retrieval), Contrastive or InfoNCE are advantageous.
• N‑pair may be useful when large inter‑class separation is desired, but one must monitor inter‑class variance to avoid uneven spacing.
• ArcFace and CCL, in their current formulation with cosine distance, are less suitable for standard retrieval pipelines.

Overall, the paper contributes a methodological toolkit (VARIANCE‑GREEDINESS diagnostics) that complements traditional performance benchmarks, enabling researchers and practitioners to make informed loss‑function choices aligned with dataset granularity and application priorities.