This module implements a two-stage retrieval system that combines the strengths of BM25 and dense vector retrieval.

The hybrid search approach:

1. Stage 1 (Retrieval): Use BM25 to retrieve top-K candidates (e.g., top-1000)
2. Stage 2 (Reranking): Rerank the candidates using semantic similarity (dot product of embeddings)

This combines:

• BM25's strength: Fast retrieval and exact keyword matching
• Vector search's strength: Semantic similarity and understanding

# For dev queries
python -m python.reranking.hybrid_search \
  --bm25-results results/bm25_dev.txt \
  --queries data/queries_dev_eval_embeddings.h5 \
  --output results/hybrid_dev.txt

# For TREC 2019 eval
python -m python.reranking.hybrid_search \
  --bm25-results results/bm25_eval_one.txt \
  --queries data/queries_dev_eval_embeddings.h5 \
  --output results/hybrid_eval_one.txt

# For TREC 2020 eval
python -m python.reranking.hybrid_search \
  --bm25-results results/bm25_eval_two.txt \
  --queries data/queries_dev_eval_embeddings.h5 \
  --output results/hybrid_eval_two.txt

• --bm25-results: Path to BM25 results file
• --queries: Path to query embeddings h5 file
• --output: Output file path for reranked results
• --doc-embeddings: Path to document embeddings (default: ./data/collection_embeddings_subset.h5)
• --top-k: Number of results to return after reranking (default: 100)
• --run-name: Run identifier in output (default: HYBRID)

The script reads BM25 results (typically top-100 or top-1000 per query):

query_id\tpassage_id\trank\tscore\tBM25

• Query embeddings from h5 file
• Document embeddings from h5 file
• Both are normalized to unit length for dot product similarity

For each query:

1. Get the query embedding
2. For each BM25 candidate:
   • Get the document embedding
   • Compute similarity: score = query_emb · doc_emb
3. Sort by semantic similarity (descending)
4. Return top-K results

Output format matches BM25 and vector search:

query_id\tpassage_id\trank\tscore\tHYBRID

Typical performance on 1M document collection:

• Reranking speed: ~300-700 queries/sec
• Memory: ~4GB (for 1M documents with 384-dim embeddings)
• Time:
  • Dev (1000 queries): ~1.5s
  • Eval (43-54 queries): ~0.15s

Based on MS-MARCO passage ranking:

1. MRR@10: Hybrid performs best, showing excellent top-result quality

   • Hybrid improves MRR over BM25 by ~46% (dev), ~23% (TREC 2019), ~11% (TREC 2020)
   • Slight improvements over pure vector search

2. Recall@100: Vector search has the best recall

   • Vector: Best at finding all relevant documents
   • Hybrid: Good balance (better than BM25, slightly lower than Vector)
   • BM25: Lower recall due to vocabulary mismatch issues

3. NDCG: Hybrid shows strong ranking quality

   • Consistently outperforms BM25
   • Competitive with or slightly better than pure vector search

4. Trade-offs:

   • BM25: Fast, interpretable, but lower accuracy
   • Vector: Best recall, but expensive to search 1M docs
   • Hybrid: Best overall accuracy, efficient (only reranks BM25 candidates)

Uses normalized dot product (equivalent to cosine similarity):

similarity = query_emb · doc_emb  # Both normalized to unit length

• Loads document embeddings once at startup
• Builds passage ID → index mapping for fast lookup
• Processes queries in batches with progress tracking

The hybrid approach is more efficient than pure vector search because:

1. BM25 quickly narrows down candidates (e.g., 1000 out of 1M)
2. Only computes semantic similarity for candidates
3. Avoids expensive nearest neighbor search over all 1M documents

• h5py - Reading embeddings
• numpy - Numerical operations
• tyro - CLI argument parsing
• tqdm - Progress bars

• python/semantic_search/search_index.py - Pure vector search
• python/evaluate_batch.py - Batch evaluation of all systems
• python/filter_results.py - Filter results by query IDs