Vector Databases
Storing meaning as coordinates, finding neighbours fast, and the retrieval half of RAG.
Meaning As Coordinates
That is a genuinely different query from anything SQL offers. Keyword search finds documents containing your words; vector search finds documents that mean something similar, including ones sharing no vocabulary with the query at all.
| Measure | Notes |
|---|---|
| Cosine similarity | Angle only, ignores magnitude — the usual choice for text |
| Dot product | Equivalent to cosine when vectors are normalised |
| Euclidean distance | Straight-line distance; sensitive to magnitude |
Approximate Nearest Neighbour
Comparing a query against every vector is exact and linear — fine for ten thousand, hopeless for fifty million. ANN indexes trade a little recall for orders of magnitude of speed.
| Index | Idea | Trade-off |
|---|---|---|
| Flat | Compare against everything | Exact, slow, no build time |
| IVF | Cluster, then search the nearest clusters | Fast; misses neighbours near cluster edges |
| HNSW | Navigable small-world graph | Excellent recall and speed; memory hungry |
| Product quantisation | Compress vectors into codes | Big memory saving, some accuracy lost |
The knobs all trade the same three things: recall, latency and memory. Measure recall against an exact flat search on a sample before tuning for speed — a fast index that returns the wrong neighbours is worse than a slow correct one, and the difference is invisible without measurement.
Chunking, Filtering & Hybrid Search
Retrieval quality is decided long before the index. Chunking is the biggest lever: too large and a chunk's embedding averages several topics into mush; too small and it loses the context that made it meaningful.
| Decision | Guidance |
|---|---|
| Chunk size | Split on structure — headings, paragraphs — not a fixed character count |
| Overlap | A little, so a sentence spanning a boundary survives |
| Metadata | Store source, date, permissions — filtering needs them |
| Hybrid search | Combine vector and keyword: exact ids and names need lexical matching |
| Reranking | Retrieve 50 cheaply, rerank to 5 with a stronger model |
Retrieval In RAG
In retrieval-augmented generation the database supplies the context a language model answers from. When RAG answers badly, the fault is far more often retrieval than generation — the model cannot use a passage it was never given.
| Symptom | Usually |
|---|---|
| Confidently wrong answers | Nothing relevant retrieved; the model filled the gap |
| Right topic, wrong detail | Chunks too large, detail averaged away |
| Ignores recent information | Index stale, or no recency weighting |
| Leaks other tenants' data | Missing metadata filter |
| Slow responses | Retrieving far more context than the answer needs |
Interview Questions
What does a vector database do that SQL cannot?
Nearest-neighbour search over embeddings — finding items that mean something similar, including documents that share no words with the query.
Why is changing embedding model expensive?
Vectors are only comparable within the model that produced them, so switching requires re-embedding and re-indexing the entire corpus.
What do ANN indexes trade?
Recall for latency and memory. HNSW gives excellent recall and speed at a memory cost; IVF is cheaper but misses neighbours near cluster boundaries.
Why is chunking the biggest lever in RAG?
Chunks that are too large average several topics into one vector; too small and they lose the context that made them meaningful. Splitting on structure beats a fixed character count.
Why combine vector and keyword search?
Embeddings are poor at exact tokens — an order number, a product code, a person's name. Hybrid search covers both semantic similarity and literal matching.
A RAG system answers confidently but wrongly. Where do you look?
Retrieval first. Check whether the correct passage was in the top-k at all; if it was never retrieved, no amount of prompt work will fix the answer.