Model Evaluation

You trained a model. It has 99% accuracy. Is it good? Maybe not.

If you are detecting a rare disease (1% of population), a model that always guesses "Healthy" will be 99% accurate but 100% useless.

1. Confusion Matrix 😵

For classification problems, we use a Confusion Matrix to break down errors.

• TP: Sick person correctly identified as Sick.
• TN: Healthy person correctly identified as Healthy.
• FP (Type I Error): Healthy person told they are Sick (False Alarm).
• FN (Type II Error): Sick person told they are Healthy (Missed Detection).

2. Metrics Beyond Accuracy 📏

Precision

"Of all the people we said were sick, how many actually were?" Precision = \frac{TP}{TP + FP}

Recall (Sensitivity)

"Of all the people who actually were sick, how many did we find?" Recall = \frac{TP}{TP + FN}

F1-Score

The harmonic mean of Precision and Recall. Good for imbalanced datasets. F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}

3. Overfitting vs. Underfitting 📉

• Underfitting: The model is too simple (e.g., fitting a line to a curve). High bias.
• Overfitting: The model memorized the training data (including noise). High variance. It fails on new data.

Solution: Cross-Validation.

4. Cross-Validation 🔄

Instead of just one Train/Test split, we split the data into $K$ folds (e.g., 5). We train 5 times, each time using a different fold as the test set.

Interactive Challenge: Calculate Metrics

Quiz

Key Takeaways

✅ Accuracy can be misleading.
✅ Precision = Quality of positive predictions.
✅ Recall = Quantity of positive predictions found.
✅ Cross-Validation prevents overfitting.

What's Next?

You have mastered traditional Machine Learning. Now it's time to enter the world of Deep Learning.

Next Module: Module 4 — Deep Learning & Neural Networks.