Dense Embedding

Dense embedding maps text into a vector space where semantically similar text is close. Similarity is measured by the cosine of the angle (direction matters, length does not).

Evolution:

• Word2Vec — static word vectors, does not distinguish polysemy (“bank”);
• BERT and BGE-M3 — contextual representations: the same word in different contexts gets different vectors.

ANN indexes:

• ANNOY — tree, fast, no incremental updates;
• HNSW — graph, incremental, more accurate.

Related: Sparse BM25 Search, Hybrid Search, RAG