Word Embeddings

How do you explain to a computer that "King" is similar to "Queen"? With Word Embeddings, we represent words as dense vectors (lists of numbers) in a high-dimensional space where similar words cluster together.

1. One-Hot Encoding vs. Embeddings 🆚

• One-Hot Encoding:

  • Cat = [1, 0, 0, 0]
  • Dog = [0, 1, 0, 0]
  • Problem: No relationship between vectors. They are orthogonal. Huge memory usage for large vocabularies.

• Dense Embeddings:

  • Cat = [0.2, 0.9, -0.1]
  • Dog = [0.2, 0.8, -0.2]
  • Result: Similar words have similar vectors. Compact representation.

2. Word2Vec 🧠

Created by Google in 2013. It learns word associations from a large corpus of text. Key idea: "You shall know a word by the company it keeps." (J.R. Firth, 1957)

• CBOW (Continuous Bag of Words): Predict target word from context words.
  • Input: ["The", "sat", "on"] → Predict: "cat"
• Skip-Gram: Predict context words from target word. Better for rare words.
  • Input: "cat" → Predict: ["The", "sat", "on"]

3. GloVe (Global Vectors) 🌍

Developed at Stanford, GloVe captures global co-occurrence statistics rather than local windows.

Key Insight: GloVe directly factorizes the word co-occurrence matrix.

• If words A and B frequently appear together → their vectors will be close.
• The ratio P(ice|solid) / P(steam|solid) captures the relationship "solid → ice".

Advantage over Word2Vec: Better captures global semantic relationships and analogy tasks.

📚 Official Resources

• GloVe Project Page — Pre-trained vectors, paper
• Word2Vec Paper (Original) — Mikolov et al., 2013
• Gensim Word2Vec — Python implementation

4. Vector Arithmetic ➕

The most famous example of Word2Vec/GloVe magic:

King - Man + Woman = Queen

This shows that the model captured the concept of "Gender" and "Royalty" as directions in the vector space:

5. FastText: Subword Embeddings 🔤

Created by Meta AI, FastText improves on Word2Vec by representing words as bags of character n-grams:

• "playing" → ["pla", "lay", "ayi", "yin", "ing"] (n-grams of size 3)
• The word vector = sum of its n-gram vectors

Why this matters:

• Handles out-of-vocabulary (OOV) words — "Techcoder" can be embedded via its n-grams
• Better for morphologically rich languages (Turkish, Finnish, German)
• Works well for misspelled words in noisy text

📚 Official Resource

• FastText Documentation — Meta's fast text classification + embeddings

6. Contextual Embeddings: The BERT Revolution 🌊

Word2Vec, GloVe, and FastText create static embeddings — each word has ONE vector regardless of context.

Problem: "Bank" has the same vector in:

• "I went to the bank to deposit money." (financial institution)
• "The river bank was muddy." (river edge)

BERT's solution: Contextual Embeddings

BERT generates a different vector for "bank" depending on the surrounding words:

7. Embedding Models Comparison 📊

📚 Official Resources

• MTEB Leaderboard — Compare ALL embedding models on standardized benchmarks
• Sentence Transformers — Best library for generating sentence embeddings locally

8. Cosine Similarity 📐

To measure how similar two words/sentences are, we calculate the cosine of the angle between their vectors.

• 1.0: Identical direction (Synonyms).
• 0.0: Unrelated (Orthogonal).
• -1.0: Opposite direction (Antonyms).

Quiz

Key Takeaways

✅ Static Embeddings (Word2Vec, GloVe, FastText) create one vector per word — fast but context-blind.
✅ Contextual Embeddings (BERT, GPT) create different vectors per word based on context — essential for disambiguation.
✅ FastText handles out-of-vocabulary words via character n-grams — great for noisy or multilingual text.
✅ Cosine Similarity measures semantic proximity between vectors.
✅ The MTEB Leaderboard is the best resource for choosing an embedding model for your task.

What's Next?

Word2Vec creates static embeddings. But what architecture makes BERT's contextual embeddings possible?

Next Chapter: Transformers & Attention.