NLP Essentials

Natural Language Processing (NLP) is the field of AI focused on the interaction between computers and human language. Before we can feed text into a model, we must turn it into numbers.

1. Tokenization ✂️

Tokenization is the process of breaking text into smaller chunks called tokens. Tokens can be words, characters, or sub-words.

• Sentence: "I love AI."
• Word Tokens: ["I", "love", "AI", "."]

2. Cleaning Text 🧹

Raw text is messy. We often perform these steps:

• Lowercasing: "AI" -> "ai"
• Removing Punctuation: "Hello!" -> "Hello"
• Stop Word Removal: Removing common words like "the", "is", "and" that don't carry much meaning.

3. Stemming vs. Lemmatization 🌱

We want to treat "running", "runs", and "run" as the same word.

• Stemming: Chops off the end of words. Fast but crude.
  • running -> run
  • better -> better (fails to map to good)
• Lemmatization: Uses a dictionary to find the root form (lemma).
  • better -> good

4. Bag of Words (BoW) 👜

A simple way to represent text. We count how many times each word appears. It ignores grammar and word order.

5. TF-IDF 📊

Term Frequency - Inverse Document Frequency. It highlights words that are important to a specific document but rare across the entire dataset.

• TF: How often a word appears in this document.
• IDF: How rare the word is across all documents.

Interactive Challenge: Build a Tokenizer

Let's write a simple function to clean and tokenize text.

Quiz

Key Takeaways

✅ Tokenization is the first step in any NLP pipeline.
✅ Preprocessing (lowercasing, lemmatization) reduces noise.
✅ TF-IDF helps find important keywords.

What's Next?

Counting words is useful, but it doesn't capture meaning. "King" and "Queen" are just different strings to a computer. How can we teach it that they are related?

Next Chapter: Word Embeddings.