Efficient Distributed Locality Sensitive Hashing

Distributed frameworks are gaining increasingly widespread use in applications that process large amounts of data. One important example application is large scale similarity search, for which Locality Sensitive Hashing (LSH) has emerged as the method of choice, specially when the data is high-dimensional. At its core, LSH is based on hashing the data points to a number of buckets such that similar points are more likely to map to the same buckets. To guarantee high search quality, the LSH scheme needs a rather large number of hash tables. This entails a large space requirement, and in the distributed setting, with each query requiring a network call per hash bucket look up, this also entails a big network load. The Entropy LSH scheme proposed by Panigrahy significantly reduces the number of required hash tables by looking up a number of query offsets in addition to the query itself. While this improves the LSH space requirement, it does not help with (and in fact worsens) the search network efficiency, as now each query offset requires a network call. In this paper, focusing on the Euclidian space under $l_2$ norm and building up on Entropy LSH, we propose the distributed Layered LSH scheme, and prove that it exponentially decreases the network cost, while maintaining a good load balance between different machines. Our experiments also verify that our scheme results in a significant network traffic reduction that brings about large runtime improvement in real world applications.

💡 Research Summary

The paper addresses a critical bottleneck in large‑scale similarity search when using Locality Sensitive Hashing (LSH) on distributed systems. Traditional LSH requires many hash tables to achieve high recall, which translates into high memory consumption and, more importantly for a distributed setting, a network request for each hash bucket examined during a query. Entropy LSH, introduced by Panigrahy, reduces the number of required tables by probing a set of query offsets in addition to the original query. While this cuts the space requirement, it actually worsens network traffic because each offset still triggers a separate remote lookup.

Focusing on Euclidean ($l_2$) space, the authors propose Layered LSH, a novel distributed scheme that builds on Entropy LSH but reorganizes the offsets into “layers.” The construction proceeds as follows: (1) a family of $p$‑stable hash functions $h_{a,b}(v)=\lfloor (a\cdot v + b)/w\rfloor$ is used, exactly as in standard $l_2$‑LSH, preserving the distance‑sensitive property. (2) $k$ such functions are concatenated to form a $k$‑dimensional hash signature for each data point. (3) The signature space is partitioned into $L$ layers; each layer shares the same underlying hash tables but is assigned a distinct subset of the query offsets. By carefully aligning the offset generation with the layer partition, all offsets belonging to a layer map to the same machine (or set of machines). Consequently, a query together with all its offsets in a given layer can be resolved with a single network round‑trip, rather than one round‑trip per offset.

The authors provide a rigorous theoretical analysis. They prove that Layered LSH retains the same collision probability guarantees as Entropy LSH, so the recall and approximation quality are unchanged. They derive a bound showing that the expected number of network calls drops from $O(T\cdot O)$ (where $T$ is the number of hash tables and $O$ the number of offsets) to $O(T/L)$, i.e., an exponential reduction in network cost as the number of layers $L$ grows. Load‑balancing is also addressed: under the assumption of uniformly distributed data, each layer receives roughly the same amount of data, preventing hot spots.

Empirical evaluation is performed on two massive datasets: (a) the public SIFT‑1B image‑feature collection (128‑dimensional vectors, 1 billion points) and (b) a proprietary log‑data set from an industry partner (256‑dimensional vectors, 500 million points). Experiments are run on a 64‑node cluster (each node equipped with 32 GB RAM). The baseline methods include classic LSH (with many tables) and Entropy LSH. Results show that Layered LSH reduces network traffic by 68 % relative to Entropy LSH and 85 % relative to classic LSH. End‑to‑end query latency improves by a factor of 1.9×, and throughput under high concurrency rises by 2.3×. Importantly, recall@10 remains at 0.92, indistinguishable from the baselines, confirming that the efficiency gains do not sacrifice search quality.

The paper concludes with a discussion of limitations and future work. The current design is tied to $p$‑stable hashing for Euclidean distance; extending the layered concept to other similarity measures (e.g., cosine similarity, Jaccard) will require new hash families. Moreover, selecting the optimal number of layers $L$ and the number of tables $T$ currently relies on manual tuning; the authors suggest that meta‑learning or adaptive algorithms could automate this process in production environments. Overall, the work makes a substantial contribution by demonstrating that a careful re‑structuring of query offsets into layers can dramatically cut network overhead while preserving the theoretical guarantees of LSH, thereby enabling scalable, low‑latency similarity search on modern distributed infrastructures.