Large Language Models (LLMs) — Deep Dive

A Large Language Model is a Transformer trained at a scale that produces emergent capabilities — abilities that weren't explicitly programmed and only appear beyond certain size thresholds.

1. Scaling Laws: How Big is Big Enough? 📏

The Chinchilla Scaling Laws (Hoffmann et al., 2022) answered one of the most important questions in AI: "Given a compute budget, what's the optimal model size?"

The key finding: Previous models (GPT-3) were significantly undertrained. The optimal ratio is:

• ~20 tokens of training data per parameter

| Model | Parameters | Training Tokens | Optimal? | |---|---|---|---| | GPT-3 | 175B | 300B | ❌ Underfit by ~10x | | Chinchilla | 70B | 1.4T | ✅ Optimal | | LLaMA 3 8B | 8B | 15T | ✅ Extremely overtrained (better inference) |

[!IMPORTANT] The Chinchilla insight for practitioners: A smaller model trained on more data will outperform a larger model trained on less data for the same compute budget. This is why LLaMA 7B models can rival GPT-3.5 — they're massively overtrained on high-quality data.

📚 Official Resource

• Chinchilla Scaling Laws Paper — Training Compute-Optimal Large Language Models

2. Mixture of Experts (MoE) 🧩

Instead of activating the whole model for every token, MoE splits the model into "experts" (specialized sub-networks) and only activates a few per token:

3. Speculative Decoding ⚡

Large models (70B+) are slow because generation is sequential. Speculative Decoding uses a small "draft" model to generate multiple tokens quickly, then the large "oracle" model verifies them in parallel:

1. Draft: Small model (7B) generates 5 tokens very fast.
2. Verify: Large model (70B) checks all 5 tokens in one forward pass.
3. Accept/Reject: Accept tokens the large model agrees with; regenerate from the first rejection.
4. Result: 2-3x speedup with identical output quality.

Used in: Google Gemini production, Anthropic Claude, HuggingFace TGI.

4. Structured Output & JSON Mode 📋

One of the most important practical features: forcing the LLM to output valid JSON.

5. LLM Comparison: Choosing the Right Model 🏆

📚 Official Docs for Major LLM APIs

• OpenAI API Documentation — GPT-4o, embeddings, DALL-E
• Anthropic Claude Documentation — Claude 3.5, tool use, streaming
• Google Gemini API — Gemini Pro, 1M context, multimodal
• Meta LLaMA GitHub — Open-weights models
• Mistral AI Documentation — Mistral, Mixtral MoE models

6. In-Context Learning & Prompt Engineering 💬

LLMs learn from examples in the prompt itself — without gradient updates:

• Zero-Shot: "Classify this review as positive/negative."
• One-Shot: "Example: 'Great product!' → positive. Now classify: 'Terrible quality.'"
• Few-Shot: 5-10 examples. Dramatically improves consistency.

System Prompt Architecture

┌─────────────────────────────────────────────┐
│  SYSTEM PROMPT                              │
│  ├─ Role definition: "You are a..."         │
│  ├─ Capabilities: "You can..."              │
│  ├─ Constraints: "Never discuss..."         │
│  ├─ Output format: "Always respond as JSON" │
│  └─ Examples (few-shot)                     │
├─────────────────────────────────────────────┤
│  USER TURN (conversation history)           │
├─────────────────────────────────────────────┤
│  ASSISTANT TURN (previous responses)        │
└─────────────────────────────────────────────┘

Interactive Challenge: LLM Cost Calculator

Quiz

Key Takeaways

✅ Chinchilla Laws tell us smaller + more data > bigger + less data for the same compute budget.
✅ MoE gives large model capacity with efficient inference (only K of N experts activate per token).
✅ Speculative Decoding uses a fast small model + slow large model in tandem for 2-3x speedup.
✅ Structured Outputs + Pydantic enforce schema compliance from LLMs.
✅ Model selection depends on context length, cost, privacy, and task complexity.

What's Next?

We understand LLMs deeply. Now let's put them to work — building real NLP pipelines for document processing and semantic search.

Next Module: Applied NLP — Document Intelligence & Semantic Search.