How does hnsw work

Overview

HNSW (Hierarchical Navigable Small World) is a graph-based algorithm for approximate nearest neighbor (ANN) search that has become a cornerstone of modern vector search systems. Developed in 2016 by Yury Malkov and colleagues at the Russian Academy of Sciences, HNSW addressed limitations of earlier ANN methods like Locality-Sensitive Hashing (LSH) and Inverted File (IVF) approaches. The algorithm builds on the concept of navigable small world graphs, which were first studied in network theory in the 1990s. HNSW's development coincided with the rise of neural embeddings and the need for efficient similarity search in high-dimensional spaces (typically 100-1000 dimensions). The algorithm was first presented at the 2016 International Conference on Similarity Search and Applications (SISAP) and has since been implemented in major vector databases like Milvus, Weaviate, and Qdrant. Its popularity stems from providing state-of-the-art performance on benchmarks, with typical recall rates of 90-99% at speeds thousands of times faster than exact search.

How It Works

HNSW constructs a multi-layered graph where each layer is a subset of the previous layer, forming a hierarchy. The bottom layer (layer 0) contains all data points, while higher layers contain progressively fewer points, selected with probability p=1/ln(M) where M is a parameter controlling layer size (typically 16-64). During insertion, new points are assigned to layers probabilistically, then connected to their nearest neighbors in each layer using a greedy search. For querying, HNSW starts at the top layer with a random entry point and performs greedy search to find the nearest neighbor in that layer. It then uses this as the entry point for the next layer down, repeating until reaching the bottom layer where the final nearest neighbors are found. This hierarchical approach dramatically reduces search complexity from O(n) to O(log n) by limiting the search space at each layer. The algorithm maintains two key properties: small world characteristics (short paths between any two nodes) and navigability (ability to find these paths efficiently).

Why It Matters

HNSW has become fundamental to modern AI applications that rely on similarity search, powering recommendation systems at companies like Spotify and Netflix, image search at Google and Pinterest, and semantic search in chatbots like ChatGPT. The algorithm enables real-time retrieval from billion-scale vector databases with millisecond latency, making previously impractical applications feasible. In e-commerce, HNSW powers visual search that increased conversion rates by 15-30% at retailers like Amazon. In healthcare, it accelerates drug discovery by finding similar molecular structures 1000x faster than traditional methods. The algorithm's efficiency has been crucial for the deployment of large language models, where retrieving relevant context from knowledge bases requires searching millions of embeddings. HNSW's open-source implementations have democratized high-performance vector search, with libraries like FAISS and hnswlib seeing over 10 million downloads annually.