Introduction to Vector Embeddings

The retriever's effectiveness depends on its ability to understand the meaning behind both the user's query and the documents in the knowledge base. Computers, however, don't inherently understand language the way humans do. They operate on numbers. This fundamental gap necessitates a way to translate text into a numerical format that captures its semantic essence. This is where vector embeddings come into play.

Vector embeddings are dense numerical representations of text (which could be words, sentences, or even entire documents) in a multi-dimensional mathematical space. Think of each piece of text being mapped to a specific point, or vector, in this space. A vector is essentially a list of numbers, for example, [0.05, -0.21, 0.98, ..., 1.52]. The "multi-dimensional" aspect means these vectors can have many components, often hundreds or even thousands (e.g., 768 or 1024 dimensions are common).

The remarkable property of these embeddings is that they are designed to capture semantic relationships. Text snippets with similar meanings are expected to have vectors that are "close" to each other in this high-dimensional space. Conversely, texts with dissimilar meanings will have vectors that are farther apart. For instance, the embedding for "machine learning" would likely be closer to the embedding for "artificial intelligence" than it would be to the embedding for "stock market".

Consider a simplified 2D space:

A 2D visualization where related words like "Dog" and "Puppy" or "Cat" and "Kitten" are positioned closer together than unrelated words like "Apple". Real embeddings exist in much higher dimensions.

This proximity isn't based on simple keyword overlap but on learned contextual understanding derived from analyzing extensive amounts of text data during the training of specialized embedding models. These models, often based on neural network architectures like Transformers, learn how words are used in context and encode that understanding into the numerical vectors. We will look at specific types of models in the next section.

Why is this representation so significant for RAG? When a user submits a query, the RAG system first converts this query into its vector embedding. The retriever component then uses this query vector to search the vector database (which stores the pre-computed embeddings of all the document chunks). The goal is to find the document chunk embeddings that are closest to the query embedding in the vector space. This closeness is typically measured using mathematical similarity metrics, like cosine similarity, which we will discuss shortly.

By using embeddings, the retrieval process moves past simple keyword matching. It can identify documents that are relevant to the query, even if they don't use the exact same words. This ability to grasp semantic meaning is fundamental to retrieving genuinely useful context for the generator LLM, leading to more accurate and relevant final answers.

Build LLM apps faster with Kerb

Cleaner syntax. Built-in debugging. Production-ready from day one.

Built for the AI systems behind ApX Machine Learning

Was this section helpful?

References

Efficient Estimation of Word Representations in Vector Space, Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013 arXiv preprint arXiv:1301.3781 DOI: 10.48550/arXiv.1301.3781 - Introduces Word2Vec, a foundational method for learning dense word embeddings that capture semantic and syntactic relationships, illustrating the core idea of mapping words to vectors in a continuous space.
Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems 30, Vol. 30 (Neural Information Processing Systems Foundation, Inc. (NeurIPS)) DOI: 10.5555/3295222.3295349 - Presents the Transformer architecture, which has become a basis for many modern embedding models, allowing for the learning of contextual text representations.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2019 Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Association for Computational Linguistics) DOI: 10.18653/v1/N19-1423 - Introduces BERT, a significant advancement in pre-trained language models based on Transformers, important for generating context-aware embeddings.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers and Iryna Gurevych, 2019 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics) DOI: 10.18653/v1/D19-1410 - Describes Sentence-BERT, a method for creating semantically meaningful sentence and document embeddings from pre-trained BERT-like models, optimized for tasks like semantic similarity search.