3.3 Data Inspection — Understand a Dataset with Pandas

3.3 Data Inspection with Pandas

After importing a dataset into a pandas DataFrame, the next step is data inspection.

Data inspection means examining the structure, size, columns, data types, missing values, and basic statistical properties of a dataset before performing cleaning, transformation, visualization, or analysis.

1. Sample Dataset

Throughout this lesson, we will use a small student-performance dataset.

import pandas as pd

data = {
    "Name": [
        "Alice",
        "Brian",
        "Clara",
        "David",
        "Eva"
    ],

    "Age": [
        20,
        21,
        19,
        22,
        20
    ],

    "Math": [
        85,
        91,
        76,
        88,
        95
    ],

    "Science": [
        82,
        89,
        79,
        92,
        94
    ]
}

df = pd.DataFrame(data)

print(df)

The DataFrame contains:

• 5 rows
• 4 columns
• One text column
• Three numerical columns

2. Using head()

The head() method displays the first rows of a DataFrame.

df.head()

By default, head() displays the first 5 rows.

print(df.head())

Example output:

    Name  Age  Math  Science
0  Alice   20    85       82
1  Brian   21    91       89
2  Clara   19    76       79
3  David   22    88       92
4    Eva   20    95       94

3. Displaying a Specific Number of Rows

You can provide a number to head() to control how many rows are displayed.

df.head(3)

This displays the first three rows.

df.head(10)

This displays the first ten rows, or all available rows if the DataFrame contains fewer than ten.

4. Using tail()

The tail() method displays the last rows of a DataFrame.

df.tail()

By default, it displays the last five rows.

You can specify the number of rows:

df.tail(3)

This displays the last three rows.

5. Understanding shape

The shape attribute returns the dimensions of a DataFrame.

df.shape

Example:

(5, 4)

The first value represents the number of rows and the second value represents the number of columns.

6. Finding Number of Rows and Columns

You can separately extract the dimensions.

rows, columns = df.shape

print("Rows:", rows)
print("Columns:", columns)

Output:

Rows: 5
Columns: 4

You can also access them individually:

print(df.shape[0])   # Rows
print(df.shape[1])   # Columns

7. Inspecting Column Names

The columns attribute returns the column labels of a DataFrame.

df.columns

Example:

Index(['Name', 'Age', 'Math', 'Science'], dtype='object')

To convert the column labels into a regular Python list:

print(df.columns.tolist())

Output:

['Name', 'Age', 'Math', 'Science']

8. Inspecting the DataFrame Index

The index attribute displays the row labels.

df.index

For the default DataFrame index, you may see:

RangeIndex(start=0, stop=5, step=1)

This means the DataFrame uses integer row labels beginning at 0.

9. Inspecting Data Types with dtypes

The dtypes attribute displays the data type of every column.

df.dtypes

Example:

Name       object
Age         int64
Math        int64
Science     int64
dtype: object

Data types help determine what operations can be performed safely on each column.

10. Using info()

The info() method provides a compact summary of a DataFrame.

df.info()

It typically provides information such as:

• Number of rows
• Index information
• Column names
• Number of non-null values
• Data types
• Approximate memory usage

11. Understanding info() Output

A typical output may look similar to:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   Name     5 non-null      object
 1   Age      5 non-null      int64
 2   Math     5 non-null      int64
 3   Science  5 non-null      int64
dtypes: int64(3), object(1)

Here:

• 5 entries means there are five rows.
• 4 columns means there are four variables.
• Non-Null Count tells you how many values are present.
• Dtype indicates the data type of each column.

12. Detecting Possible Missing Values with info()

Suppose a dataset contains:

data = {
    "Name": ["Alice", "Brian", "Clara", "David"],
    "Age": [20, 21, None, 22],
    "Score": [85, 91, 78, None]
}

df = pd.DataFrame(data)

df.info()

The non-null counts may show:

Name     4 non-null
Age      3 non-null
Score    3 non-null

This immediately indicates that Age and Score contain missing values.

13. Using describe()

The describe() method generates descriptive statistics for suitable columns.

df.describe()

For numerical columns, the output commonly includes:

• count
• mean
• std
• min
• 25%
• 50%
• 75%
• max

14. Understanding describe() Statistics

15. Example of describe()

df = pd.DataFrame({
    "Math": [85, 91, 76, 88, 95],
    "Science": [82, 89, 79, 92, 94]
})

print(df.describe())

A typical result will contain statistics for both numerical columns.

The 50% row represents the median, while 25% and 75% represent the first and third quartiles.

16. Describing Categorical Data

By default, describe() focuses on numerical columns when numerical data is available.

You can explicitly request information about object/string columns.

df.describe(
    include=["object"]
)

For categorical/text data, useful statistics can include:

• count
• unique
• top
• freq

17. Describing Multiple Data Types

You can request descriptive information across different column types using:

df.describe(
    include="all"
)

The exact statistics shown depend on the data types present in the DataFrame.

18. Counting Non-Missing Values

The count() method returns the number of non-missing values in each column.

df.count()

This is useful when you want a quick column-by-column view of data availability.

19. Checking Missing Values with isna()

The isna() method identifies missing values.

df.isna()

It returns Boolean values:

• True → missing value
• False → value is present

To count missing values in each column:

df.isna().sum()

Example:

Name       0
Age        1
Math       0
Science    1

20. Checking Non-Missing Values with notna()

The notna() method performs the opposite logical check.

df.notna()

Here:

• True → value is present
• False → value is missing

21. Checking Unique Values

The unique() method returns the distinct values in a Series.

df["Age"].unique()

To count the number of distinct values:

df["Age"].nunique()

22. Using value_counts()

value_counts() counts the frequency of distinct values in a Series.

df["Age"].value_counts()

It is particularly useful for categorical columns.

Example:

20    2
21    1
19    1
22    1

23. Checking Memory Usage

The memory_usage() method provides information about the memory consumed by DataFrame columns.

df.memory_usage()

You can also request deeper memory analysis through info().

df.info(
    memory_usage="deep"
)

This can be particularly useful when working with large datasets.

24. Using T for Quick Inspection

The T attribute transposes the DataFrame, exchanging rows and columns.

df.T

This can sometimes make a small DataFrame easier to inspect when it contains many columns.

25. Inspecting Random Rows with sample()

sample() selects random rows from a DataFrame.

df.sample()

Select multiple random rows:

df.sample(3)

This is useful when you want to inspect records that are not necessarily at the beginning or end of the dataset.

26. Inspecting Individual Columns

A specific column can be inspected using bracket notation.

df["Math"]

Multiple columns can also be selected:

df[
    ["Name", "Math", "Science"]
]

You can then inspect statistics for selected numerical columns:

df[
    ["Math", "Science"]
].describe()

27. Inspecting a Single Series

DataFrame inspection and Series inspection are related but not identical.

df["Math"].head()

df["Math"].describe()

df["Math"].dtype

df["Math"].shape

This allows focused inspection of an individual variable.

28. Complete Inspection Example

import pandas as pd

df = pd.DataFrame({
    "Name": [
        "Alice",
        "Brian",
        "Clara",
        "David",
        "Eva"
    ],

    "Age": [
        20,
        21,
        None,
        22,
        20
    ],

    "Math": [
        85,
        91,
        76,
        88,
        95
    ],

    "Science": [
        82,
        89,
        79,
        92,
        None
    ]
})

print("First rows:")
print(df.head())

print("\nShape:")
print(df.shape)

print("\nColumns:")
print(df.columns)

print("\nData types:")
print(df.dtypes)

print("\nInformation:")
df.info()

print("\nStatistics:")
print(df.describe())

print("\nMissing values:")
print(df.isna().sum())

This compact inspection routine gives a strong first understanding of the dataset before cleaning or analysis.

29. Recommended Data Inspection Workflow

A systematic inspection process is better than randomly executing methods.

1. Check the first records
   Use df.head().
2. Check the last records
   Use df.tail().
3. Check dimensions
   Use df.shape.
4. Check column names
   Use df.columns.
5. Check data types and non-null counts
   Use df.info().
6. Check detailed data types
   Use df.dtypes.
7. Check missing values
   Use df.isna().sum().
8. Check numerical statistics
   Use df.describe().
9. Check categorical frequencies
   Use value_counts().
10. Check representative records
    Use sample().

30. Pandas Data Inspection Cheatsheet

31. Pandas Data Inspection Interview Questions

df.shape

df.isna().sum()

df.dtypes

df["Column"].nunique()

32. Examination Questions

Multiple Choice Questions

Short Answer Questions

1. Define data inspection.
2. What is the purpose of head()?
3. What information does info() provide?
4. Explain the meaning of df.shape.
5. What is the purpose of describe()?
6. Differentiate between shape and size.
7. How can you find missing values in a DataFrame?
8. How can you determine the number of unique values in a column?

33. Practical Examination Questions

print(df.head())

print(df.shape)

print(df.columns)

print(df.dtypes)

df.info()

print(df.describe())

print(df.isna().sum())

print(
    df["Department"].value_counts()
)

34. How to Interpret an Unknown Dataset

Imagine receiving a dataset that you have never seen before. Do not immediately start building charts or machine-learning models.

First answer these questions:

35. Data Inspection Best Practices

1. Never assume the dataset is clean. Inspect it before analysis.
2. Check dimensions first. Know how much data you are dealing with.
3. Inspect both the beginning and the end. Use head() and tail().
4. Check data types. Incorrect types can cause analytical errors.
5. Check missing values. Missing data can affect calculations and models.
6. Use descriptive statistics. Understand the numerical distribution before visualization.
7. Inspect categorical frequencies. This can reveal unexpected categories and imbalances.
8. Do not rely on one inspection method. Each method reveals a different aspect of the dataset.

37. Key Takeaways

• head() displays the first rows of a DataFrame.
• tail() displays the last rows.
• shape gives the number of rows and columns.
• columns identifies the available variables.
• dtypes shows the data type of each column.
• info() provides a compact structural overview.
• describe() summarizes numerical data statistically.
• isna().sum() helps identify missing values.
• value_counts() helps understand categorical frequencies.
• Data inspection should happen before serious data cleaning and analysis.