Semantic Search Systems

Traditional search (Ctrl+F, Google circa 2005) looks for exact keyword matches. Semantic Search understands the meaning behind words. If you search for "movie about a sinking ship," it should find "Titanic" even though those exact words don't appear in the title.

1. The Evolution of Search 📜

Generation 1: Keyword Search (BM25)

• Method: Count word frequencies, rank by TF-IDF
• Example: Search "car accident" → finds documents with those exact words
• Problem: Misses "automotive collision," "vehicle crash"

Generation 2: Semantic Search (Dense Vectors)

• Method: Convert queries and documents to vectors, find nearest neighbors
• Example: Search "car accident" → finds "traffic collision," "automobile crash"
• Magic: Understands synonyms, context, and related concepts

Generation 3: Hybrid Search (Best of Both)

• Method: Combine keyword matching + semantic similarity
• Why: Keywords catch exact matches (product IDs, names); vectors catch concepts
• Result: Most production systems use this approach

2. Dense vs. Sparse Representations 🔎

Sparse Search (BM25)

• Representation: High-dimensional sparse vectors (one dimension per word)
• "apple": [0,0,0,...,1,0,0] (only position 47839 is 1)
• Pros: Fast, explainable, works for exact matches
• Cons: Vocabulary mismatch problem

Dense Search (Neural Embeddings)

• Representation: Low-dimensional dense vectors (e.g., 768 dimensions)
• "apple": [0.23, -0.15, 0.87, ..., 0.42] (all dimensions have values)
• Pros: Captures semantics, handles synonyms
• Cons: Computationally expensive, harder to debug

3. The Embedding Model 🧠

You need a model to convert text → vectors. This is the heart of semantic search.

Popular Embedding Models

Fine-Tuning Embeddings for Your Domain

Pre-trained embeddings are great, but domain-specific fine-tuning boosts accuracy:

• Medical: "discharge" (hospital) vs "discharge" (electrical)
• Legal: "party" (lawsuit) vs "party" (celebration)

4. Similarity Metrics 📏

Once you have vectors, how do you measure "closeness"?

Cosine Similarity (Most Common)

Measures the angle between vectors, ignoring magnitude.

• Formula: cos(θ) = (A · B) / (||A|| × ||B||)
• Range: -1 (opposite) to 1 (identical)
• Why it works: Text length shouldn't matter ("I love AI" vs "I really truly deeply love AI")

Dot Product

Combines angle AND magnitude.

• Formula: A · B = Σ(ai × bi)
• Use case: When you want to factor in document length/importance

Euclidean Distance (L2)

Straight-line distance in vector space.

• Formula: √(Σ(ai - bi)²)
• Use case: Clustering, anomaly detection

5. Building a Production Search Engine 🏗️

Architecture Overview

Step 1: Indexing (Offline)

1. Load documents (PDFs, web pages, etc.)
2. Chunk them (see previous chapter)
3. Generate embeddings for each chunk
4. Store in Vector DB (Pinecone, Weaviate, Qdrant, Milvus)

Step 2: Querying (Real-time)

1. User enters query
2. Generate query embedding
3. Vector DB finds nearest neighbors (HNSW, IVF algorithms)
4. Optional: Rerank with a more powerful model
5. Return top results

Advanced: Reranking

The 2-stage approach:

1. Retrieval (fast, approximate): Vector search finds top 100 candidates
2. Reranking (slow, accurate): Cross-encoder model scores top 10

6. Hybrid Search Strategy 🎯

Combine BM25 (keyword) + Vector (semantic) for best results.

Reciprocal Rank Fusion (RRF)

Simple algorithm to merge two ranked lists:

• score = 1/(k + rank_bm25) + 1/(k + rank_vector)
• where k is a constant (usually 60)

Quiz

6. Training Embedding Models: Bi-Encoders & Cross-Encoders 🏋️

How are semantic search models trained? Understanding the architecture explains why 2-stage retrieval is necessary:

Bi-Encoders (Sentence Transformers)

• Architecture: Query and Document pass through the Transformer separately.
• Vector Caching: Document vectors are pre-computed and indexed in a vector DB.
• Speed: Searching 10M documents takes < 10ms via HNSW.
• Training: Trained with Multiple Negatives Ranking Loss (MNRL) or InfoNCE loss — pulling relevant (query, doc) pairs together and pushing irrelevant pairs apart in vector space.

Cross-Encoders (Rerankers)

• Architecture: Query and Document are concatenated ([CLS] Query [SEP] Document [SEP]) and passed through the Transformer together.
• Attention: Every word in the query directly attends to every word in the document.
• Accuracy: Extremely high accuracy — captures fine-grained nuances.
• Speed: Cannot pre-compute vectors! Must run inference for every candidate pair at search time (~50-100ms for 50 candidates).

Bi-Encoder:     Query ──> Transformer ──> Vector Q ┐  Dot Product (Fast)
                Doc   ──> Transformer ──> Vector D ┴────────────────────> Score

Cross-Encoder:  [Query + Doc] ──> Transformer (Full Attention) ─────────> Score (Slow)

7. SPLADE: Learned Sparse Representations ⚡

BM25 uses fixed word counts. SPLADE (Sparse Lexical and Expansion Model) uses a Transformer (like BERT) to predict learned sparse weights for terms:

• Term Expansion: Searching "automobile" automatically activates the dimension for "car" and "vehicle".
• Sparse Indexing: Uses traditional inverted indexes (like Elasticsearch / Lucene) — fast, explainable, and handles rare jargon.

8. End-to-End Semantic Search Implementation 💻

Official Resources

• 📖 Sentence Transformers Documentation — Complete guide to bi-encoder & cross-encoder models
• 📖 Cohere Rerank API — Production cross-encoder reranking service
• 📖 OpenAI Embeddings API — text-embedding-3 models
• 📖 MTEB Leaderboard — Benchmark for evaluating text embeddings
• 📖 BEIR Search Benchmark — Heterogeneous benchmark for zero-shot evaluation

Quiz

Key Takeaways

✅ Semantic Search understands intent, not just keywords.
✅ Bi-Encoders pre-compute document vectors for fast ANN vector search.
✅ Cross-Encoders perform full joint attention for high-precision 2nd-stage reranking.
✅ SPLADE provides learned sparse representations for term expansion in inverted indexes.
✅ Hybrid Search (BM25 + Vector) with Reranking is the gold standard production architecture.

What's Next?

English is powerful, but the world speaks 7,000 languages. Can AI truly understand them all? Next Chapter: Multilingual NLP.