UniHash: Unifying Pointwise and Pairwise Hashing Paradigms

Effective retrieval across both seen and unseen categories is crucial for modern image retrieval systems. Retrieval on seen categories ensures precise recognition of known classes, while retrieval on unseen categories promotes generalization to novel classes with limited supervision. However, most existing deep hashing methods are confined to a single training paradigm, either pointwise or pairwise, where the former excels on seen categories and the latter generalizes better to unseen ones. To overcome this limitation, we propose Unified Hashing (UniHash), a dual-branch framework that unifies the strengths of both paradigms to achieve balanced retrieval performance across seen and unseen categories. UniHash consists of two complementary branches: a center-based branch following the pointwise paradigm and a pairwise branch following the pairwise paradigm. A novel hash code learning method is introduced to enable bidirectional knowledge transfer between branches, improving hash code discriminability and generalization. It employs a mutual learning loss to align hash representations and introduces a Split-Merge Mixture of Hash Experts (SM-MoH) module to enhance cross-branch exchange of hash representations. Theoretical analysis substantiates the effectiveness of UniHash, and extensive experiments on CIFAR-10, MSCOCO, and ImageNet demonstrate that UniHash consistently achieves state-of-the-art performance in both seen and unseen image retrieval scenarios.

💡 Research Summary

The paper addresses a fundamental challenge in large‑scale image retrieval: achieving strong performance on both “seen” categories (those present during training) and “unseen” categories (new classes that appear after deployment). Existing deep hashing approaches are typically confined to a single training paradigm. Pointwise (center‑based) methods excel at seen categories by pulling samples toward learnable class prototypes, but they suffer from poor generalization to unseen classes because the prototypes are fixed. Pairwise (relationship‑based) methods, on the other hand, learn similarity/dissimilarity relations among samples, which yields better generalization to novel categories but often lags behind pointwise methods on the known classes.

UniHash proposes a dual‑branch architecture that simultaneously runs a pointwise branch and a pairwise branch. Both branches share a common backbone (e.g., ResNet‑50) that extracts visual features, but each branch has its own hashing head. The pointwise branch learns a binary hash center for each class and optimizes a cross‑entropy loss (L_C) over the cosine similarity between a sample’s continuous hash vector u_c and all class centers. The pairwise branch optimizes a log‑sigmoid pairwise loss (L_P) that encourages inner‑product similarity I_ij of continuous hash vectors u_p to match the semantic similarity matrix S_ij derived from label overlap.

The novelty lies in how the two branches interact. A mutual learning loss (L_M) measures the cosine distance between u_c and u_p and forces them to align. During training, one branch is detached each epoch so that the other branch receives a stable target, enabling bidirectional knowledge flow. This alignment allows the pointwise branch to inherit relational cues from the pairwise branch, while the pairwise branch benefits from the global semantic structure imposed by the class centers.

To further enhance cross‑branch communication, the authors introduce the Split‑Merge Mixture of Hash Experts (SM‑MoH) module, an adaptation of the Mixture‑of‑Experts (MoE) paradigm for hashing. Two independent gating networks (G_c for the center branch and G_p for the pairwise branch) compute scores over a pool of m lightweight expert networks {E_i}. For each input feature v, the top‑k experts are selected (split step) and their outputs are weighted by the normalized gate scores. The weighted sum yields the branch‑specific continuous hash vector u_s (merge step). Because the same set of experts is shared across both branches, the merge step enforces a common transformation, while the split step preserves branch‑specific specialization. This design promotes sparse, expert‑driven routing and ensures that the two branches learn mutually consistent yet complementary hash representations.

Theoretical analysis is provided under several assumptions: bounded feature extractor, Lipschitz experts, sparse routing (top‑k), well‑separated hash centers, and data lying on a union of semantic manifolds with small intra‑class diameter and large inter‑class separation. Under these conditions, the quantization error decays exponentially with code length q, and the mutual learning term reduces the structural discrepancy between the two paradigms from a fixed error floor to a vanishing term ε, effectively eliminating the gap. Complexity analysis shows that the statistical cost scales as O(q k log m · n).

Empirical evaluation on CIFAR‑10, MS‑COCO, and ImageNet follows both seen‑category and unseen‑category protocols. UniHash consistently outperforms state‑of‑the‑art pointwise and pairwise baselines in mean average precision (mAP) for both settings. Notably, even with short hash codes (≤48 bits), UniHash maintains robust performance, and the SM‑MoH component contributes an average 3–5 % mAP gain over a version without expert routing. Ablation studies confirm the importance of each component (mutual loss, SM‑MoH, top‑k routing) and demonstrate that the default hyper‑parameters (λ₁=4, λ₂=1, λ₃=1) are near‑optimal across datasets.

In summary, UniHash successfully unifies the strengths of pointwise and pairwise deep hashing within a single framework. By coupling mutual alignment with a split‑merge expert architecture, it mitigates the classic trade‑off between seen‑category precision and unseen‑category generalization, delivering high‑quality binary codes suitable for real‑time, large‑scale image retrieval systems.