Vector Databases and Semantic Search Implementation
Chapter 1: Embeddings and Vector Spaces
From Data to Vectors: A Refresher
Survey of Embedding Models
Understanding Vector Dimensionality
Introduction to Dimensionality Reduction
Measuring Similarity in Vector Space
Hands-on Practical: Generating and Comparing Embeddings
Quiz for Chapter 1
Chapter 2: Introducing Vector Databases
What Defines a Vector Database?
Core Architectural Components
Data Models and Schemas
Vector Operations: CRUD
Metadata Filtering
Scaling Considerations
Hands-on Practical: Basic Vector DB Interaction
Quiz for Chapter 2
Chapter 3: Approximate Nearest Neighbor (ANN) Search
The Need for Approximation
Core Concepts of ANN
Algorithm Overview: HNSW
Algorithm Overview: IVF
Algorithm Overview: LSH
Indexing Parameters and Tuning
Evaluating ANN Performance
Hands-on Practical: Experimenting with Index Parameters
Quiz for Chapter 3
Chapter 4: Building Semantic Search Systems
Semantic vs. Keyword Search Revisited
Architecture of a Semantic Search Pipeline
Data Preparation and Chunking Strategies
Query Processing and Embedding
Result Ranking and Re-ranking
Implementing Hybrid Search
Evaluating Semantic Search Relevance
Hands-on Practical: Designing a Search Query Flow
Quiz for Chapter 4
Chapter 5: Vector Databases in Practice
Choosing a Vector Database Platform
Working with Pinecone Client
Working with Weaviate Client
Working with Milvus Client
Working with ChromaDB Client
Indexing Large Datasets Efficiently
Monitoring and Maintenance
Hands-on Practical: Build a Small Semantic Search App
Quiz for Chapter 5
Evaluating Semantic Search Relevance
Was this section helpful?
- Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, 2008 (Cambridge University Press) - A foundational textbook covering information retrieval theory, including comprehensive details on ranking evaluation metrics.
- Cumulated gain-based evaluation of IR techniques, Kalervo Järvelin and Jaana Kekäläinen, 2002 ACM Transactions on Information Systems (ACM TOIS), Vol. 20 (ACM) DOI: 10.1145/582415.582418 - Introduces and details the Normalized Discounted Cumulative Gain (NDCG) metric, a standard for evaluating ranked retrieval systems with graded relevance.
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, Tong Wang, 2016 Proceedings of the Neural Information Processing Systems (NIPS) Workshop on Conversational AI DOI: 10.48550/arXiv.1611.09268 - Describes the MS MARCO dataset, a large-scale collection widely used for training and evaluating information retrieval and reading comprehension systems, including relevance judgments.
- Neural Information Retrieval: A Review, Sean MacAvaney, Daniel Cohen, Nazir Nayal, and Andrew Yates, 2020 Foundations and Trends® in Information Retrieval, Vol. 14 (now publishers) DOI: 10.1561/1500000072 - Provides a comprehensive overview of modern neural information retrieval, covering evaluation practices and challenges specific to these systems.