Handling Missing Values

In real-world datasets, it is common to find incomplete or missing information. Missing values may occur due to human errors, data collection issues, equipment failure, or incomplete survey responses. Before training an Artificial Intelligence (AI) or Machine Learning (ML) model, it is essential to identify and handle these missing values to improve the accuracy and reliability of predictions.

The Pandas library provides simple yet powerful functions to detect, remove, and replace missing values. Data cleaning is one of the most important steps in every AI project because the quality of an AI model depends largely on the quality of the data used for training.

---

Learning Objectives

• Understand missing values.
• Identify missing values in a dataset.
• Remove rows or columns containing missing values.
• Replace missing values with appropriate values.
• Understand the importance of data cleaning in AI.

---

What are Missing Values?

A missing value is a data item that has not been recorded or is unavailable in a dataset. In Pandas, missing values are generally represented by NaN (Not a Number).

---

Example Dataset with Missing Values

Name	Marks	City
Aman	92	Jaipur
Neha	NaN	Delhi
Rahul	88	NaN
Priya	95	Mumbai

The above dataset contains missing values in the Marks and City columns.

---

Why Do Missing Values Occur?

• Human error during data entry.
• Incomplete survey responses.
• Faulty sensors or equipment.
• Data corruption during transfer.
• Information not available at the time of collection.

---

Why is Handling Missing Values Important?

• Improves prediction accuracy.
• Prevents incorrect analysis.
• Reduces bias in Machine Learning models.
• Improves data quality.
• Produces more reliable AI systems.

---

Workflow for Handling Missing Values

Collect Dataset

      │

      ▼

Identify Missing Values

      │

      ▼

Remove or Replace Missing Values

      │

      ▼

Clean Dataset

      │

      ▼

Machine Learning Model

---

Checking Missing Values

The isnull() function checks whether each value in the DataFrame is missing.

import pandas as pd

df = pd.read_csv("student.csv")

print(df.isnull())

---

Counting Missing Values

To count the number of missing values in each column, use sum() with isnull().

print(df.isnull().sum())

Sample Output

Name      0

Marks     1

City      1

dtype: int64

---

Removing Missing Values using dropna()

The dropna() function removes rows or columns containing missing values.

Syntax

df.dropna()

---

Example 1: Remove Rows with Missing Values

import pandas as pd

df = pd.read_csv("student.csv")

new_df = df.dropna()

print(new_df)

Only rows without missing values will remain.

---

Removing Columns with Missing Values

new_df = df.dropna(axis=1)

Setting axis=1 removes columns containing missing values.

---

Replacing Missing Values using fillna()

Instead of deleting data, missing values can be replaced using the fillna() function.

Syntax

df.fillna(value)

---

Example 2: Replace Missing Values with Zero

df.fillna(0, inplace=True)

print(df)

---

Example 3: Replace Missing Values with Average

import pandas as pd

df = pd.read_csv("student.csv")

average = df["Marks"].mean()

df["Marks"].fillna(average, inplace=True)

print(df)

Replacing missing numerical values with the average helps preserve valuable records instead of deleting them.

---

Comparison: dropna() vs fillna()

dropna()	fillna()
Removes missing records.	Replaces missing values.
May reduce dataset size.	Keeps all records.
Suitable when missing data is very small.	Suitable when retaining data is important.

---

Case Study

A school maintains the marks of 2,000 students. During data entry, some marks were accidentally left blank. Instead of deleting those students' records, the school replaces the missing marks with the average marks of the class. This ensures that the dataset remains complete for analysis while minimizing the impact of missing information.

---

Flow of Data Cleaning

Raw Dataset

      │

      ▼

Detect Missing Values

      │

      ▼

dropna() / fillna()

      │

      ▼

Clean Dataset

      │

      ▼

Machine Learning

---

Real-Life Applications

• Cleaning hospital patient records.
• Preparing banking transaction data.
• Processing weather datasets.
• Cleaning e-commerce customer information.
• Preparing datasets for AI and Machine Learning.

---

Think Like an AI Engineer

A hospital dataset contains missing blood pressure values for several patients. Should the entire records be deleted or should the missing values be replaced? Explain your answer.

Click to View Answer

If only a few values are missing, replacing them with an appropriate value such as the average or median is usually better than deleting the entire record. This preserves valuable information and maintains the size of the dataset.

---

Competency-Based Question

A company collected customer information from an online survey. Some customers did not mention their age. Explain how Pandas can be used to identify and handle these missing values before training an AI model.

---

Common Beginner Mistakes

• Ignoring missing values before analysis.
• Deleting large portions of data unnecessarily.
• Using fillna() with inappropriate replacement values.
• Forgetting to use inplace=True when updating the original DataFrame.
• Confusing rows (axis=0) with columns (axis=1).

---

Quick Revision

---

Memory Trick

---

Exam Tips

• Remember the purpose of isnull(), dropna(), and fillna().
• Know the difference between deleting and replacing missing values.
• Understand when to use each approach.
• Practice writing Python programs for handling missing values.

---

Summary

• Missing values are common in real-world datasets.
• Pandas represents missing values using NaN.
• isnull() identifies missing values.
• dropna() removes incomplete rows or columns.
• fillna() replaces missing values while preserving the dataset.
• Handling missing values is an essential step before training Artificial Intelligence and Machine Learning models.