Working with Data

Real-world data is messy. It has missing values, typos, and weird formats. Before we can train an AI model, we must preprocess the data.

This chapter covers the standard data pipeline:

1. Loading
2. Cleaning
3. Scaling
4. Visualization

1. Loading Data 📥

Data usually comes in CSV (Comma Separated Values) or JSON files. Pandas makes loading them easy.

import pandas as pd

# Load a CSV file
df = pd.read_csv('data.csv')

# Load a JSON file
df_json = pd.read_json('data.json')

2. Cleaning Data 🧹

"Garbage In, Garbage Out." If your data has errors, your model will fail.

Handling Missing Values

You can either drop missing rows or fill them (imputation).

# Check for missing values
print(df.isnull().sum())

# Option 1: Drop rows with missing values
df_clean = df.dropna()

# Option 2: Fill missing values with the average (mean)
df['Age'] = df['Age'].fillna(df['Age'].mean())

Removing Duplicates

df = df.drop_duplicates()

3. Scaling Data ⚖️

AI models like numbers to be on a similar scale.

• Age: 0 - 100
• Income: 20,000 - 200,000

If we don't scale, the model might think "Income" is more important just because the numbers are bigger.

Normalization (Min-Max Scaling): Scales data between 0 and 1.

# Simple normalization formula
df['Income_Scaled'] = (df['Income'] - df['Income'].min()) / (df['Income'].max() - df['Income'].min())

4. Data Visualization 📈

Always visualize your data to understand it better.

Interactive Challenge: Clean the Dataset

We have a dataset with a missing value and a duplicate. Fix it!

Quiz

Key Takeaways

✅ Cleaning: Handle missing values and duplicates.
✅ Scaling: Bring all features to a similar range (0-1).
✅ Visualization: Use charts to understand your data.

What's Next?

Data is ready. Tools are ready. Now, let's build our first Machine Learning Model.

Next Module: Module 3 — Core Machine Learning.