I’m Nino and happy to share that I’ve joined MLPills to write about AI! I am a Founding ML Engineer at AIRecruitPro, an open-source contributor, tech writer and a self-taught builder who learns by creating things from the ground up. My work is at the intersection of AI systems, product engineering, and startups, where I focus on designing multimodal models, GPU-accelerated training, and ML infrastructure. I love building companies, solving hard problems, and turning ideas into real products that people use. Say Hi on X or visit my LinkedIn!
💊 Pill of the week
The claim “Our vector database can handle 100 million embeddings on a single machine.” Sounds impressive, but the math doesn’t work using just naive storage. Let’s take a standard embedding: 768 dims, float32. That’s 3,072 bytes per vector. Multiply by 100 million: 307 GB. Just for the vectors. No index. No metadata. No IDs. No breathing room for the OS. Just raw floats sitting in memory.
Most machines have 64-128 GB of RAM. We’re 3-5× over budget before we’ve even started. So how do production systems actually pull this off? The answer is a systems design pattern built on one key insight: you don’t need full-precision vectors in RAM. You need just enough precision to find candidates, then you refine.
The way is to:
Compress the vectors (307 GB → ~10 GB using Product Quantization)
Partition the search space (don’t scan everything)
Score cheaply on compressed codes
Refine only the top candidates with full precision
This post walks through the exact math, the compression techniques that make it possible, and a runnable demo you can use to verify the claims yourself. By the end, you’ll understand precisely where every byte goes and how to tune the tradeoffs for your own system.
The Math
Say you’re building a RAG system. You’ve chunked your document corpus, embedded each chunk with a model like OpenAI’s text-embedding-3-small, and now you need to store and search those vectors. Here are the realistic numbers: 100 million vectors, 768 dims each, stored as float32 (4 bytes per value).
The storage calculation:
Memory = N × d × bytes_per_float
= 100,000,000 × 768 × 4
= 307,200,000,000 bytes
= 307.2 GBThat’s 307 GB for the vectors alone.
But a working system also needs an index structure, HNSW graph edges or IVF posting lists, which add 10-100+ bytes per vector depending on the method. You need vector IDs to map results back to your documents, typically 8 bytes each. Metadata like timestamps, permissions, and filter fields pile on more. Then there’s allocator overhead, fragmentation, alignment, and padding, which eat another 5-15%. And if you care about reliability, you’re replicating the data, which doubles everything.
Now consider what you’re working with. A standard cloud VM comes with 64 GB of RAM. Memory-optimized instances give you 128-256 GB. High-memory machines can reach 512 GB or more, but you’ll pay dearly for them. The gap is brutal. So, a typical production machine has 64-128 GB of RAM, but our vectors alone need 307 GB.
Something has to give. Either we throw money at bigger machines (which doesn’t scale), distribute across many nodes (which adds latency and complexity), or we find a way to radically compress what we keep in RAM.
Where Memory Actually Goes
Before we can fix the problem, we need to understand where the bytes actually go. Not all memory is created equal; some components are compressible, others aren’t. Some scale linearly with N, others don’t.
Let’s break down the four main buckets:
Vectors
Indexes
IDs and Metadata
Overhead
Vectors
In an uncompressed system, this is where most of your RAM goes. The good news is that this is also the most compressible part of the system. The entire point of techniques like Product Quantization is to shrink this from hundreds of gigabytes to single digits. Everything else in this breakdown is noise compared to solving the vector storage problem.
Indexes
You can’t just store vectors in a flat array and scan all 100 million on every query. You need an index to narrow the search space. But indexes have their own memory footprint, and it varies dramatically by method.
HNSW (Hierarchical Navigable Small World) builds a graph where each vector connects to its approximate neighbors. With a typical configuration like 32 edges per vector across multiple layers, you’re adding 128-256 bytes per vector just for graph connectivity. At 100M vectors, that’s another 12-25GB for the graph alone. HNSW gives excellent latency and recall, but it comes with a memory overhead.
IVF (Inverted File Index) clusters vectors into partitions and stores posting lists of which vectors belong to each cluster. The overhead per vector is much smaller but you pay for the cluster centroids and list management. For 100M vectors, IVF structures typically add 1-3GB depending on configuration.
The choice between HNSW and IVF often comes down to this tradeoff: HNSW is faster and more accurate but memory-heavy; IVF is leaner but requires more tuning to match HNSW’s recall.
You can check our previous issue on HNSW:
IDs and metadata
Every vector needs an identity. At minimum, you need a way to map search results back to your source documents. A 64-bit ID costs 8 bytes per vector, that’s 800MB at 100M scale. Sounds small compared to 307GB of vectors, but once you compress the vectors down to ~10GB, suddenly 800MB of IDs represents 8% of your memory budget!
Metadata compounds this. Timestamps for freshness filtering. Permission flags for access control. Chunk offsets for retrieval. Category tags for faceted search. Each field you add multiplies across 100M rows. A system with 32 bytes of metadata per vector adds another 3.2 GB.
Overhead
Memory allocators don’t pack data perfectly. You lose bytes to alignment requirements (8 or 16 byte boundaries), internal fragmentation (allocated blocks are often larger than requested), and bookkeeping (the allocator itself needs to track what’s allocated where). In practice, expect 5-15% overhead on top of your calculations.
The key insight
In an uncompressed system, vectors account for 80-90% of total memory. Index structures, IDs, metadata, and overhead split the remaining 10-20%. This means that you should compress the vectors first. If you can take vectors from 307GB to 10GB, you’ve solved 90% of the problem. The rest is optimization at the margins!
🎓Full Stack AI / LLM Engineering*
*Sponsored: by purchasing any of their courses you would also be supporting MLPills.
The Core Idea: Multi-Stage Retrieval + Compression
Think about what search actually requires. You have a query vector. You want the 10 or 100 most similar items. You don’t care about the precise distance to vector #47,382,019 in the middle of the ranking. You only care whether it’s in your top results or not.
Production vector search systems exploit this with a multi-stage pipeline. Each stage trades precision for speed, progressively narrowing the candidate set until only the final results need careful scoring.
Stage 1: Candidate generation
The first stage doesn’t try to find the best results. It tries to find a reasonable superset that contains the best results. This is where approximate nearest neighbor (ANN) algorithms like IVF and HNSW earn their keep. IVF partitions the vector space into clusters and only searches the few clusters closest to your query — maybe 1-5% of the data. HNSW navigates a graph structure, hopping from node to node toward the query region without ever examining most of the corpus.
The goal is to go from 100 million candidates to a few thousand — a 10,000× reduction — while maintaining high probability that the true top results are somewhere in that shortlist. You’re not ranking here. You’re filtering.
Stage 2: Cheap scoring
Now you have a few thousand candidates. You need to rank them, but you still don’t need full precision. This is where compressed distance calculations shine. With Product Quantization, each vector is represented as a short code — maybe 96 bytes instead of 3,072. Computing approximate distances from these codes is fast: just a series of table lookups and additions, no floating-point multiplications, no loading full vectors from memory.
You score all candidates with these compressed distances and keep the top 100 or so. The ranking won’t be perfect, some vectors will be slightly misordered due to quantization error but the rough ordering is preserved. The true top-10 results are almost certainly somewhere in your top-100 candidates.
Stage 3: Refine
If you need more precision, you can re-score your top candidates using the original float32 vectors. This means fetching 100 full vectors (307 KB) instead of 100 million (307 GB) — a perfectly tractable amount to load from SSD or a secondary store.
You compute exact distances, re-rank, and return the final top-10. This stage recovers most of the accuracy lost to compression, but it only runs on a tiny fraction of the data.
Stage 4: Rerank
Cross-encoder rerankers take your query and each candidate document as raw text, feeding them through a model that directly predicts relevance. This is far more accurate than any vector similarity — it can catch semantic nuances that embedding distance misses — but it’s expensive. Running a cross-encoder on 100 million documents is unthinkable. Running it on your top 20 candidates takes milliseconds.
This is the pattern: use cheap, approximate methods to shrink the haystack, then apply expensive, precise methods to find the needle.
Check our previous issue about reranking:
The pipeline in practice
100M vectors → ANN filter → 5,000 candidates → PQ scoring → 100 candidates → exact refine → 20 candidates → cross-encoder → 10 results
This is how every production vector search system works: Pinecone, Weaviate, Qdrant, Milvus, pgvector at scale, you name it. The specific algorithms differ, the boundaries between stages blur, but the fundamental pattern is universal.
Compression Options: From Simple to PQ
We’ve established that vectors are the problem: 307 GB of float32 data that needs to fit in 64 GB of RAM. Now let’s look at the solutions, starting with the obvious approaches and building to the technique that actually works at scale.
Simple approaches (and why they’re not enough)
Float16 — half precision
The simplest compression: cut your floats in half. Float16 uses 2 bytes instead of 4, giving you an immediate 2× reduction.
100M × 768 × 2 = 153.6 GBBetter, but still 153 GB. You’ve gone from “impossible” to “still impossible.” Float16 is worth using as a baseline optimization — there’s rarely a good reason to keep float32 if your embeddings don’t need the precision, but it won’t solve the fundamental problem.
Scalar quantization (int8)
Take each float and map it to an 8-bit integer. You lose precision in the value range, but you cut storage to 1 byte per dimension.
100M × 768 × 1 = 76.8 GBNow we’re at 77GB, a 4× reduction from float32. This is actually usable on high-memory machines. Some production systems stop here, especially if they can afford 128-256 GB instances. But we’re targeting 64 GB or less, and we haven’t accounted for index overhead. Scalar quantization gets us closer, but it won’t deliver “100M vectors on a commodity machine.” We need something more aggressive.
Product Quantization — the real enabler
Product Quantization (PQ) is the technique that makes large-scale vector search feasible. It’s been the backbone of billion-scale systems since Jégou, Douze, and Schmid introduced it in 2011, and it remains the dominant approach today.
The core idea is deceptively simple: don’t store vectors, store references to a codebook.
Step 1: Split the vector into subvectors
Take your 768-dim vector and divide it into m equal chunks. If m=96, each chunk contains 8 dims.
Original: [v₁, v₂, v₃, ..., v₇₆₈]
↓
Subvectors: [v₁...v₈], [v₉...v₁₆], [v₁₇...v₂₄], ..., [v₇₆₁...v₇₆₈]
└───┬───┘ └───┬────┘ └────┬────┘ └─────┬─────┘
s₁ s₂ s₃ s₉₆Step 2: Learn a codebook for each subspace
For each of the 96 subspaces, run k-means clustering on the training data to find k representative centroids. Typically k=256, which means each centroid can be identified by a single byte (2⁸ = 256). After training, you have 96 codebooks, each containing 256 centroids of 8 dimensions.
Step 3: Encode each vector as codebook indices
For each vector, find the nearest centroid in each subspace and store only the index.
Original subvector s₁ = [0.23, -0.41, 0.87, ...] (8 floats = 32 bytes)
Nearest centroid in codebook₁ = index 147
Stored: just the byte "147"Repeat for all 96 subspaces. Your 768-dimensional vector is now 96 bytes.
The compression math
Component Calculation Size Original float32 100M × 768 × 4 bytes 307.2 GB PQ codes 100M × 96 bytes 9.6 GB Codebooks 96 codebooks × 256 centroids × 8 dims × 4 bytes 0.75 MB. The codebook overhead is negligible — less than a megabyte regardless of database size. It’s a fixed cost shared across all vectors. The PQ codes scale linearly, but at 96 bytes per vector instead of 3,072. That’s a 32× compression ratio.
307 GB → 9.6 GB
Now we’re talking! Single-digit gigabytes for the vector payload, with room left for index structures, IDs, and metadata.
What you’re actually storing
It’s worth being concrete about what the compressed representation looks like. Each vector becomes a sequence of 96 bytes:
Vector #0: [147, 23, 201, 88, 45, ..., 156] ← 96 code indices
Vector #1: [92, 178, 34, 88, 212, ..., 41]
Vector #2: [147, 55, 201, 12, 45, ..., 203]
...
Vector #99,999,999: [84, 23, 19, 241, 178, ..., 92]That’s it. No floats. Just bytes pointing into codebooks. The codebooks themselves sit in a small lookup table that fits in CPU cache.
How search works on compressed vectors (ADC)
Compression is useless if you can’t search efficiently. The magic of PQ is that you can compute approximate distances directly on the codes without ever reconstructing the original vectors. The technique is called Asymmetric Distance Computation (ADC).
When a query arrives, you don’t compress it. You keep all 768 float32 values. The asymmetry is intentional — you’re comparing one high-precision query against millions of low-precision database vectors.
Before scanning any codes, you split the query into the same 96 subvectors and compute the distance from each query subvector to all 256 centroids in the corresponding codebook.
Query subvector q₁ = [0.15, -0.33, 0.91, ...]
Distance to centroid 0: 0.234
Distance to centroid 1: 0.891
Distance to centroid 2: 0.156
...
Distance to centroid 255: 0.445
Store as: lookup_table₁[256]You build 96 such tables, one per subspace. Total work: 96 × 256 = 24,576 distance calculations. This happens once per query, not once per vector.
Distance = sum of table lookups
Now the scan is trivial. For each database vector, you look up its code in each table and sum the results.
Database vector codes: [147, 23, 201, ...]
Distance ≈ lookup_table₁[147] + lookup_table₂[23] + lookup_table₃[201] + ...That’s 96 table lookups and 95 additions per vector. No floating-point multiplications. No memory fetches beyond the codes themselves and the cached lookup tables.
Why is it fast?
Three reasons:
memory bandwidth. You’re reading 96 bytes per vector instead of 3,072. That’s 32× less data moving from RAM to CPU. At scale, memory bandwidth is often the bottleneck, not compute.
cache efficiency. The lookup tables (96 × 256 × 4 bytes ≈ 96 KB) fit comfortably in L2 cache. The codes stream through sequentially. There’s no random access pattern to blow out your cache lines.
simple operations. Table lookups and integer additions are about as cheap as it gets. No floating-point pipeline stalls, no branch mispredictions, no complex instruction sequences.
The result: PQ-based scanning can be 10-30× faster than brute-force float32 distance calculations, depending on hardware. Combined with an IVF index that limits which vectors you scan at all, you get practical query times even at 100M scale.
The tradeoff
None of this is free. PQ introduces quantization error — the distance you compute is an approximation of the true distance. Some vectors will be slightly misordered in your rankings.
The key point: PQ gives you a knob to turn. More bytes per vector (higher m) means less error but more memory. Fewer bytes means more compression but more error. You choose the tradeoff that fits your constraints.
Index Choice Determines the Rest of the RAM Story
Compressing vectors from 307 GB to 10 GB solves the dominant term. But you still can’t scan 100 million vectors on every query — even with PQ’s fast distance calculations, that’s too slow for production latencies.
HNSW — fast but memory heavy
HNSW builds a graph where each vector connects to its approximate nearest neighbors. It’s the performance king: sub-millisecond queries, 95%+ recall, minimal tuning required. The cost is memory. Each vector stores neighbor lists across multiple graph layers. With typical parameters (M=32), you’re adding 20-30 bytes per vector just for graph edges. At 100M vectors, that’s 25+ GB for the index alone on top of your vector storage. If you’ve compressed vectors to 10 GB with PQ, then add 25 GB for HNSW, you’re at 35 GB before IDs or metadata. Workable on a 64 GB machine, but tight.
IVF-PQ — the scale play
IVF takes a different approach: partition the space into clusters, then search only the relevant clusters. At index time, k-means creates nlist cluster centroids (typically 4,096-16,384). Each vector gets assigned to its nearest cluster. At query time, you find the nprobe closest clusters to your query and scan only those posting lists using PQ distance calculations. The memory overhead is minimal, just the centroids and some bookkeeping for posting lists. Maybe 200-300 MB total, regardless of database size.
Why IVF-PQ wins at scale
The math makes the choice obvious:
PQ + HNSW: 10 GB vectors + 25 GB graph = 35 GB
PQ + IVF: 10 GB vectors + 0.3 GB index = 10.3 GB
IVF-PQ uses 3× less memory than HNSW for the same compressed vectors. That’s the difference between “fits on a 32 GB machine” and “needs 64 GB minimum.”
The tradeoff is recall and tuning complexity. IVF-PQ typically achieves 80-90% recall@10 versus HNSW’s 95%+, and it requires more parameter tuning (nlist, nprobe) to get there. But for memory-constrained deployments at 100M+ scale, that tradeoff is almost always worth it.
The Two-Tier Storage Pattern (Hot vs Cold)
Here’s a secret about production vector databases: they don’t keep everything in RAM. They keep just enough in RAM to identify candidates, then fetch the rest on demand. This hot/cold split is what makes “100M on one machine” practical, not just theoretically possible.
Hot tier — what stays in RAM
The hot tier contains everything needed to answer the question: “which vectors are closest to this query?” That means PQ codes (~10 GB for 100M vectors), IVF cluster structures (~300 MB), and vector IDs to map results back to documents (~800 MB). Maybe a few bytes of metadata per vector for basic filtering. Call it 12-15 GB total. This is your working set. It needs to be in RAM because you’re scanning millions of PQ codes per query and you can’t afford disk latency in that loop.
Cold tier — what lives on SSD
Everything else goes to disk: Original float32 vectors, if you want an optional refinement stage. Full document text or chunk content for RAG retrieval. Extended metadata — timestamps, permissions, tags, whatever your application needs. Audit logs, versioning information, anything that doesn’t need sub-millisecond access. SSDs are cheap and fast enough. Reading 100 vectors × 3 KB each = 300 KB from NVMe takes under a millisecond. That’s negligible compared to network round-trips to your LLM.
The pipeline in practice
Hot path (RAM): IVF identifies relevant clusters, PQ scores candidates, returns top 1,000 vector IDs
Cold path (SSD): Fetch original vectors for top 100, recompute exact distances, rerank
Retrieval (SSD): Load actual document chunks for top 20 results
Downstream: Send chunks to cross-encoder or LLM
Why this works for RAG
RAG applications don’t actually need vectors at the end of the pipeline. They need text. Your user asks a question. You embed it, search vectors, and get back document IDs. Then you load those documents and feed them to an LLM. The vectors were just an intermediate step to find relevant content.
This means the cold tier isn’t optional overhead, it’s where your actual content lives. The hot tier is a compressed index into that content. Keeping them separate is natural, not a compromise.
The budget reality
With hot/cold separation, your RAM requirement drops dramatically:
Component Location Size PQ codes RAM 9.6 GB IVF structures RAM 0.3 GB Vector IDs RAM 0.8 GB Minimal metadata RAM 1-2 GB Hot tier total RAM ~12 GB Original vectors SSD 307 GB Document chunks SSD Variable Extended metadata SSD Variable
A 32 GB machine handles the hot tier comfortably. A 2 TB SSD handles everything else. Total hardware cost: a few hundred dollars, not tens of thousands. This is why “100M on one machine” is feasible — not because we solved an impossible compression problem, but because we only keep the index hot and let everything else stay cold.
Worked Memory Budget (The Proof)
The setup
100 million vectors
768 dimensions (standard embedding size)
PQ with m=96 subspaces, k=256 centroids
IVF with nlist=4,096 clusters
PQ codes — the compressed vectors themselves. Each vector becomes 96 bytes (one byte per subspace). That’s 100,000,000 × 96 = 9,600,000,000 bytes. 9.6 GB.
Codebooks — the lookup tables for reconstruction and distance computation. You have 96 codebooks, each with 256 centroids of 8 dimensions stored as float32. That’s 96 × 256 × 8 × 4 = 786,432 bytes. Under 1 MB. This is constant regardless of database size.
Vector IDs — mapping search results back to documents. Using 64-bit integers for safety, that’s 100,000,000 × 8 = 800,000,000 bytes. 0.8 GB.
IVF structures — cluster centroids plus posting list overhead. The centroids themselves are 4,096 × 768 × 4 = 12.6 MB. Posting list bookkeeping (offsets, lengths) adds another few hundred MB. Call it 0.5 GB total.
Minimal metadata — basic fields you might filter on. Say 16 bytes per vector for timestamps, flags, and a category ID. That’s 100,000,000 × 16 = 1,600,000,000 bytes. 1.6 GB.
Allocator overhead — fragmentation, alignment, bookkeeping. Estimate 10% on top of everything. ~1.2 GB.
The total
Component Size PQ codes 9.6 GB Codebooks < 1 MB Vector IDs 0.8 GB IVF structures 0.5 GB Minimal metadata 1.6 GB Allocator overhead ~1.2 GB Hot tier total ~13.7 GB. Round up for safety: 15 GB for a fully operational 100M vector index.
The comparison
Raw float32 vectors alone would cost 307 GB. Our compressed, indexed system fits in 15 GB. That’s a 20× reduction — from “needs a server with half a terabyte of RAM” to “fits on a laptop.”
A 32 GB machine runs this index comfortably with room for the OS, file caches, and your application code.
A 64 GB machine gives you headroom for growth, more metadata, or a hybrid HNSW layer for frequently-accessed vectors.
A 128 GB machine is overkill for vectors alone, but gives you space to keep original float32 vectors in RAM for refinement without hitting disk.
The cold tier, the original vectors, full documents, and extended metadata live on a 1-2 TB SSD. NVMe drives handle this for a few hundred dollars.
Reproducible Demo
This repository builds everything we’ve discussed: PQ compression, IVF indexing, recall measurement, and lets you confirm the numbers yourself. Run it at 1M vectors on your laptop, extrapolate to 100M, and see that the math holds.
What the demo measures
The script generates synthetic embeddings at feasible scale (1M vectors by default), builds both an exact index and a compressed IVF-PQ index, then compares them head-to-head:
Metric Exact (FlatL2) IVF-PQ Index size 2.86 GB 0.11 GB Search time (10K queries) 12.0s 1.4s Recall@10 100% 81.4%
The compression ratio is 26× on the actual index files. The speedup is 8.5×. The recall lands squarely in the 80-90% range we’ve been claiming.
Extrapolation to 100M
The script scales these measurements to show what 100M vectors would require: Component Size Raw float32 vectors 286 GB PQ codes only 8.9 GB Full IVF-PQ index ~10.9 GB. Fits in 64 GB RAM? YES
Three things to remember
“100M vectors on one machine” is a systems design outcome, not a single algorithm. It’s the combination of compression, indexing, and tiered storage — each solving a different part of the problem. Skip any piece and the math falls apart.
The winning recipe: partition → compress → shortlist → refine. IVF partitions the search space so you don’t scan everything. PQ compresses vectors so they fit in RAM. ANN search shortlists candidates cheaply. Exact refinement recovers precision where it matters. This pipeline is how every production system works at scale.
PQ is the key that unlocks everything else. Without Product Quantization, you’re stuck at 307 GB with no path forward. With it, vectors drop to 10 GB and suddenly the rest of the system — indexing, metadata, hot/cold storage — becomes tractable. Compress the dominant term first; everything else follows.