Quantization based Fast Inner Product Search

We propose a quantization based approach for fast approximate Maximum Inner Product Search (MIPS). Each database vector is quantized in multiple subspaces via a set of codebooks, learned directly by minimizing the inner product quantization error. Then, the inner product of a query to a database vector is approximated as the sum of inner products with the subspace quantizers. Different from recently proposed LSH approaches to MIPS, the database vectors and queries do not need to be augmented in a higher dimensional feature space. We also provide a theoretical analysis of the proposed approach, consisting of the concentration results under mild assumptions. Furthermore, if a small sample of example queries is given at the training time, we propose a modified codebook learning procedure which further improves the accuracy. Experimental results on a variety of datasets including those arising from deep neural networks show that the proposed approach significantly outperforms the existing state-of-the-art.

💡 Research Summary

The paper tackles the Maximum Inner Product Search (MIPS) problem, which is central to large‑scale recommendation, classification, and retrieval systems. Traditional approaches either rely on brute‑force linear scans (O(nd) time) or transform the problem into a nearest‑neighbor search via dimensionality augmentation and asymmetric hashing (e.g., ALSH, SRP‑LSH). These transformations increase memory footprints and often require careful tuning of augmentation parameters.

The authors propose a fundamentally different strategy called QUIP (Quantization‑based Inner Product). The method proceeds in three stages: (1) a fixed random permutation (or rotation) of vector coordinates, followed by a deterministic chunking of each d‑dimensional vector into K equal‑sized sub‑vectors (subspaces) of dimension l = d/K; (2) independent learning of a codebook U^{(k)} ∈ ℝ^{l×C} for each subspace k using a modified k‑means objective that minimizes the expected squared error of the inner‑product approximation; (3) at query time, only the database vectors are quantized, while the query remains in its original form. The inner product qᵀx is approximated by Σ_{k=1}^{K} q^{(k)ᵀ} u^{(k)}{c(x)} where u^{(k)}{c(x)} is the selected codebook entry for subspace k. Because the query is not quantized, the approximation can be evaluated extremely quickly via K pre‑computed lookup tables containing q^{(k)ᵀ} U^{(k)}.

Two learning scenarios are considered. In the basic setting, only the database X is available. The authors compute the non‑centered query covariance Σ_Q^{(k)} (or, when queries are unavailable, approximate it with the data covariance Σ_X^{(k)}) and run a Mahalanobis‑distance k‑means in each subspace. This yields codebooks that are the empirical means of their clusters, guaranteeing that the estimator is unbiased (Lemma 3.1).

When a modest set of example queries Q is available during training, the authors augment the objective with a hinge‑loss term that penalizes any violation of the ordering “the quantized representation of the true top‑ranked database vector must have a larger inner product with the query than any other quantized vector.” The resulting constrained optimization (Eq. 5) is solved by alternating updates: (i) identify a limited set of violated triplets (query, true top, competitor), (ii) reassign each database sub‑vector to the codebook entry that minimizes a combined Mahalanobis distance plus hinge penalty, and (iii) update each codebook entry by a gradient step that balances reconstruction error and the hinge term. This procedure reduces to the basic Mahalanobis k‑means when no constraints are violated, but empirically yields a noticeable boost in recall when the query distribution differs from the data distribution.

Theoretical contributions include two concentration results. Theorem 4.1 shows that a random permutation of coordinates makes the chunked subspaces η‑balanced in expectation, i.e., each subspace captures roughly an equal fraction of the total ℓ₂ energy. Theorem 4.2 proves that, assuming the data lie inside a ball of radius r and the subspaces are η‑balanced, the probability that the quantized inner product deviates by more than a factor ε from the true inner product decays exponentially with K (the number of subspaces) and quadratically with C (the codebook size). This formalizes the intuition that increasing the granularity of the product quantization improves approximation quality dramatically.

Empirical evaluation spans four real‑world datasets: MovieLens and Netflix (collaborative‑filtering recommendation), ImageNet (deep visual classification), and VideoRec (video recommendation). The authors compare QUIP against state‑of‑the‑art MIPS methods, including ALSH, SRP‑LSH, PCA‑Tree, and standard product quantization. Experiments are conducted under two resource regimes: (a) fixed memory budget (varying K and C to meet a space constraint) and (b) fixed latency budget (adjusting lookup table sizes). QUIP consistently achieves higher recall@N and lower query latency than competitors. The variant that incorporates example queries (QUIP‑Q) further improves performance, especially when the query distribution is skewed relative to the database.

In conclusion, the paper demonstrates that MIPS can be approximated efficiently without any dimensionality augmentation, solely by learning high‑quality subspace codebooks that directly minimize inner‑product quantization error. The combination of unbiased estimation, strong concentration guarantees, and practical hinge‑loss constraints yields a method that outperforms existing hashing‑based and tree‑based solutions across diverse tasks. Future directions suggested include dynamic codebook updates for streaming data, extensions to non‑Euclidean embeddings (e.g., hyperbolic space), and hardware‑aware implementations leveraging SIMD or GPU parallelism.