NLP Essentials

Natural Language Processing (NLP) is the field of AI focused on the interaction between computers and human language. Before we can feed text into a model, we must turn it into numbers — and doing this well is where great NLP begins.

1. Tokenization ✂️

Tokenization is the process of breaking text into smaller chunks called tokens. Tokens can be words, characters, or sub-words depending on the algorithm.

Word-Level Tokenization (Simple but Limited)

• Sentence: "I love AI."
• Word Tokens: ["I", "love", "AI", "."]
• Problem: Vocabulary explosion — "run", "running", "ran" are 3 separate tokens. Unknown words become <UNK>.

Subword Tokenization — The Modern Standard

Modern LLMs use subword algorithms that balance vocabulary size with coverage of rare words.

BPE (Byte Pair Encoding) — Used by GPT, LLaMA

BPE starts with individual characters and iteratively merges the most frequent adjacent pairs:

WordPiece — Used by BERT

Similar to BPE but merges based on likelihood rather than frequency. Uses ## prefix for continuation tokens.

• "playing" → ["play", "##ing"]
• "unbelievably" → ["un", "##believ", "##ably"]

Tokenizer Comparison

📚 Official Resources

• HuggingFace Tokenizers Library — Production tokenizers for all algorithms
• OpenAI tiktoken — Fast BPE tokenizer used by GPT-4
• Google SentencePiece — Unigram/BPE implementation used by T5, LLaMA

2. Cleaning Text 🧹

Raw text is messy. We often perform these steps:

• Lowercasing: "AI" → "ai"
• Removing Punctuation: "Hello!" → "Hello"
• Stop Word Removal: Removing common words like "the", "is", "and" that don't carry much meaning.

[!NOTE] For modern LLMs, you often skip these steps — the model learns from raw text. Classical NLP tasks (TF-IDF, BM25, keyword search) benefit most from preprocessing.

3. Stemming vs. Lemmatization 🌱

We want to treat "running", "runs", and "run" as the same word.

• Stemming: Chops off the end of words. Fast but crude.
  • running → run
  • better → better (fails to map to good)
• Lemmatization: Uses a dictionary to find the root form (lemma).
  • better → good
  • was → be

4. Named Entity Recognition (NER) 🏷️

NER identifies and classifies named entities in text — people, places, organizations, dates, and more. It's one of the most practically useful NLP tasks.

📚 NER Tools

• spaCy Documentation — Industrial-strength NLP with fast NER
• NLTK Documentation — Classic NLP toolkit for research

5. Part-of-Speech (POS) Tagging 🏷️

POS tagging identifies the grammatical role of each word: Noun, Verb, Adjective, Adverb, etc.

• "The quick (ADJ) brown (ADJ) fox (NOUN) jumps (VERB) over the lazy (ADJ) dog (NOUN)."

POS tags are used in:

• Information extraction (find all noun phrases)
• Grammar checking
• Named entity disambiguation

6. Bag of Words (BoW) 👜

A simple way to represent text. We count how many times each word appears. It ignores grammar and word order but works surprisingly well for classification tasks.

7. TF-IDF 📊

Term Frequency - Inverse Document Frequency. It highlights words that are important to a specific document but rare across the entire dataset.

• TF: How often a word appears in this document.
• IDF: How rare the word is across all documents.
• Combined: TF-IDF(t, d) = TF(t, d) × log(N / df(t))

Where N = total documents, df(t) = documents containing term t.

Interactive Challenge: Build a Tokenizer

Let's write a function to clean and tokenize text — and compare word vs. subword tokenization.

Quiz

Key Takeaways

✅ BPE Tokenization is the standard for modern LLMs — it handles unknown words via subword splits.
✅ Lemmatization produces real dictionary words; stemming is faster but cruder.
✅ NER extracts structured information (people, places, organizations) from unstructured text.
✅ TF-IDF helps find important keywords — still used in search engines and BM25 retrieval.

What's Next?

Counting words is useful, but it doesn't capture meaning. "King" and "Queen" are just different strings to a computer. How can we teach it that they are related?

Next Chapter: Word Embeddings.