Everyone shipping retrieval has an intuition that embeddings capture meaning. That intuition is close enough to be useful and wrong enough to cause outages.
A vector is a compression, and compressions have priorities
An embedding model is trained to make some pairs close and others far. Which pairs depends entirely on the objective it was trained against. A model trained on question–answer pairs puts a question near its answer. A model trained on paraphrase puts a sentence near its restatement. These are different geometries, and a question is not near its answer in a paraphrase-trained space.
This is the most common silent failure in RAG: retrieving with a similarity model when the task is question-answering. The scores look fine. The neighbours are topical. The answer is not in them.
The geometry is anisotropic
Embedding spaces are not evenly populated. Vectors cluster in a narrow cone, which means raw cosine similarity between two arbitrary texts is high almost regardless of content — often 0.7 or above for unrelated documents. Absolute thresholds tuned on one corpus rarely transfer to another.
Work in ranks, not scores. “Top ten neighbours” is portable. “Everything above 0.82” is a number that will betray you the week your corpus changes.
Length, format and boilerplate dominate more than you expect
Chunk two documents that share a navigation header and a legal footer, and you have partly embedded the template rather than the content. Long chunks average away the specific sentence that mattered. Short chunks lose the referent — a paragraph saying “this must not exceed the limit” is useless when the limit was named two paragraphs earlier.
Strip boilerplate before embedding, and carry a little structural context — document title, section heading — into each chunk so the vector knows what it belongs to.
Test the space, not just the pipeline
Build a small labelled set from real questions and the passages that genuinely answer them — fifty is enough to be informative. Then measure recall@k for your actual k. If recall@20 is 0.6, no amount of prompt engineering downstream will fix it: two out of five answers were never in the room.
Do this before you tune anything else. It is the cheapest diagnostic in the stack and it routinely overturns assumptions about which model to use.
What to take away
Pick the embedding model by the relationship you need — question to answer, or text to similar text — not by benchmark position. Compare ranks rather than absolute scores. Clean boilerplate. Measure recall against real questions. The geometry is a tool with a shape, and most retrieval problems are the shape being wrong for the job.