Hybrid Search Implementation

LangChain Implementation Notes

LangChain provides specific abstractions to facilitate hybrid search. For instance:

EnsembleRetriever: This class in langchain.retrievers accepts a list of retrievers (such as a BM25 retriever and a vector store retriever) and unifies their results. It typically uses the Reciprocal Rank Fusion (RRF) algorithm to rerank documents based on their positions in the individual result lists. You can assign weights to prioritize specific retrievers within the RRF calculation.

# Example using EnsembleRetriever (assumes retrievers are initialized)
from langchain.retrievers import EnsembleRetriever
# Assume bm25_retriever and faiss_retriever are already configured
# Example: pip install rank_bm25, faiss-cpu

# Initialize sparse retriever (e.g., BM25)
# ... setup bm25_retriever ...

# Initialize dense retriever (e.g., FAISS vector store)
# ... setup faiss_retriever ...

# Combine them using EnsembleRetriever (uses Weighted Reciprocal Rank Fusion)
ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, faiss_retriever],
    weights=[0.4, 0.6] # Optional: Weights applied to the RRF calculation
)

# Use the hybrid retriever
query = "How to fix connection timeout?"
hybrid_results = ensemble_retriever.invoke(query)

print(hybrid_results)

Vector Store Integrations: Some vector databases (e.g., Pinecone, Weaviate, Elasticsearch with dense vectors) offer built-in support for hybrid search, allowing you to query using both sparse keywords and dense vectors simultaneously via their APIs. LangChain integrations often expose these capabilities. Check the documentation for your specific vector store integration.

Build LLM apps faster with Kerb

Cleaner syntax. Built-in debugging. Production-ready from day one.

Built for the AI systems behind ApX Machine Learning

Was this section helpful?

References

Okapi at TREC-3, Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, Mike Gatford, 1994 Proceedings of The Third Text REtrieval Conference, TREC 1994, Vol. 500-225 (National Institute of Standards and Technology (NIST)) - Describes the Okapi BM25 ranking function, a cornerstone of sparse retrieval, illustrating its effectiveness in early information retrieval evaluations.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers, Iryna Gurevych, 2019 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics) DOI: 10.18653/v1/D19-1410 - Presents Sentence-BERT, a modification of BERT that generates semantically meaningful sentence embeddings, a fundamental technique for dense retrieval methods.
Hybrid Search: An Overview, Adam Benayoun, 2024 (Pinecone) - A practical overview of hybrid search, explaining its components, benefits, and common implementation strategies from a vector database vendor's perspective.
LangChain Documentation: EnsembleRetriever, LangChain Developers, 2024 (LangChain) - Official documentation for LangChain's EnsembleRetriever, detailing its usage and how it combines results from multiple retrievers, including sparse and dense methods.