Fine-Tuning Generative Models
From generalist to specialist. Master Parameter-Efficient Fine-Tuning (LoRA, QLoRA, DoRA), alignment algorithms (RLHF, DPO, GRPO), model merging, and fine-tuning frameworks.
From generalist to specialist. Master Parameter-Efficient Fine-Tuning (LoRA, QLoRA, DoRA), alignment algorithms (RLHF, DPO, GRPO), model merging, and fine-tuning frameworks. This hands-on tutorial focuses on practical implementation of fine-tuning generative models concepts.
Fine-Tuning Generative Models
Big models like GPT-4, LLaMA 3, or Stable Diffusion are generalists. They know a little bit about everything. Fine-tuning takes these base models and turns them into domain experts — specialized for legal analysis, medical diagnosis, enterprise codebases, or custom artistic styles.
1. Why Fine-Tune vs. Prompt Engineering? 🤔
| Factor | Prompt Engineering | Fine-Tuning | |---|---|---| | Best For | Prototyping, zero-shot tasks, external data | Custom style, task specialization, lower latency | | Token Cost | High (long system prompts & examples per call) | Low (short prompts, knowledge in weights) | | Latency | Slower (large prompt context to process) | Faster (smaller prompts) | | Data Privacy | Context sent to API every request | Base weights updated, runs on self-hosted infra |
2. Parameter-Efficient Fine-Tuning (PEFT) ⚡
In the past, fine-tuning meant updating all billions of parameters (Full Fine-Tuning). Today, PEFT allows us to update only 0.1–3% of parameters while achieving 99% of full fine-tuning performance.
LoRA (Low-Rank Adaptation)
LoRA freezes base weights $W_0$ and adds two low-rank matrices $A$ and $B$: $$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} (B \cdot A)$$
Where $r$ is the rank (e.g., $r=8$ or $16$) and $\alpha$ is the scaling factor.
QLoRA (Quantized LoRA)
QLoRA quantizes the base model to 4-bit NormalFloat (NF4) and adds 16-bit LoRA adapters.
- Result: Fine-tune a 70B parameter model on a single 48GB GPU!
DoRA (Weight-Decomposed LoRA)
DoRA (Liu et al., 2024) decomposes weights into magnitude and direction components:
- Standard LoRA updates magnitude and direction together, which can cause instability.
- DoRA updates them independently, matching or exceeding full fine-tuning performance.
| PEFT Method | Mechanism | Trainable Params | VRAM (7B Model) |
|---|---|---|---|
| Full Fine-Tuning | Update all parameters | 100% (7B) | ~80 GB VRAM |
| LoRA | Low-rank adapter matrices | ~0.1–1% (20M) | ~16 GB VRAM |
| QLoRA | 4-bit NF4 base + 16-bit LoRA | ~0.1–1% (20M) | ~6 GB VRAM (Consumer GPU) |
| DoRA | Magnitude/Direction decomposition | ~0.1–1% (22M) | ~16 GB VRAM |
3. The Alignment Pipeline: RLHF, DPO & GRPO ⚖️
Raw fine-tuning (SFT) makes a model complete text. Alignment teaches it to be helpful, honest, and harmless.
RLHF (Reinforcement Learning from Human Feedback)
- SFT: Supervised fine-tuning on high-quality Q&A pairs.
- Reward Model: Train a scoring model on human preference pairs (A vs B).
- PPO Optimization: Use Proximal Policy Optimization (PPO) to steer the LLM toward higher reward scores.
DPO (Direct Preference Optimization)
DPO mathematically proves that you don't need a separate Reward Model! It optimizes the LLM directly on preference pairs $(x, y_w, y_l)$ where $y_w$ is the winning response and $y_l$ is the losing response.
GRPO (Group Relative Policy Optimization)
Used in DeepSeek-R1 and DeepSeek-Math, GRPO removes the critic network from PPO. It samples a group of outputs ${o_1, o_2, \dots, o_G}$ for each prompt, scores them using rule-based reward functions (e.g., exact math correctness or code pass/fail), and normalizes rewards relative to the group: $$A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$$
[!IMPORTANT] Why GRPO is revolutionary: It allowed DeepSeek-R1 to train reasoning models without human preference data by evaluating outputs against automated verification tests (verifiable rewards).
4. Modern Fine-Tuning Frameworks 🛠️
Instead of writing PyTorch training loops from scratch, production teams use optimized libraries:
5. Model Merging: Combining Expert Models 🧩
What if you have one fine-tuned model for Coding and another for Legal Analysis? Model Merging combines their weight matrices into a single model without retraining:
- SLERP (Spherical Linear Interpolation): Smoothly interpolates weights on a high-dimensional sphere.
- TIES-Merging: Trims redundant small updates, resolves sign conflicts, and merges task vectors.
- DARE (Drop And REscale): Randomly drops 90% of fine-tuned delta weights and rescales the remaining 10%, allowing merging of multiple distinct LoRAs into one base model.
6. Dataset Curation: Quality > Quantity 💎
The LIMA paper ("Less Is More for Alignment") proved that 1,000 carefully curated, high-quality examples outperform 50,000 low-quality scraped pairs.
Data Pipeline: Raw Sources ──> Deduplication ──> Quality Filtering (LLM-as-Judge) ──> Formatted Pair (SFT)
Official Resources
- 📖 HuggingFace PEFT Documentation
- 📖 HuggingFace TRL (Transformer Reinforcement Learning)
- 📖 Unsloth Documentation & GitHub
- 📖 Axolotl GitHub
- 📖 LM Evaluation Harness
Quiz
Quiz
Question 1 of 3What is the main advantage of DoRA over standard LoRA?
Key Takeaways
✅ PEFT methods (LoRA, QLoRA, DoRA) enable fine-tuning 70B models on accessible GPUs.
✅ Alignment techniques evolved from RLHF $\to$ DPO $\to$ GRPO (verifiable rewards for reasoning models).
✅ Unsloth and Axolotl accelerate training workflows by up to 5x with memory-optimized Triton kernels.
✅ Model Merging combines specialized skills into a unified checkpoint without retraining.
✅ Data quality > quantity: 1,000 clean examples beat 50,000 noisy samples.
What's Next?
We can build and fine-tune models. Now, how do we make sure they stay safe, compliant, and on-topic in production?
Next: Guardrails & AI Safety Systems.