4.4 Structured EDA Framework: Step-by-Step Workflow to Explore Any Dataset

4.4 Structured EDA Framework

Exploratory Data Analysis (EDA) is the systematic process of examining a dataset before performing statistical analysis, visualization, or machine learning.

When a dataset is unfamiliar, the first task is not to immediately build a model or create attractive charts. The first task is to understand the data.

A structured EDA process helps avoid assumptions and ensures that important data-quality problems are discovered before analysis.

1. Why Is EDA Important?

A dataset may contain thousands or millions of records, but its structure may not be immediately obvious.

EDA helps identify:

• Number of rows and columns
• Column names
• Data types
• Missing values
• Duplicate records
• Invalid values
• Outliers
• Distribution patterns
• Relationships between variables
• Potential target variables
• Potential data leakage
• Unexpected categories
• Possible business or analytical questions

2. Universal EDA Workflow

A practical EDA workflow can be organized into the following stages:

1. Define the Objective
        ↓
2. Load the Dataset
        ↓
3. Understand the Dataset Structure
        ↓
4. Inspect Data Types
        ↓
5. Check Data Quality
        ↓
6. Analyze Missing Values
        ↓
7. Check Duplicate Records
        ↓
8. Explore Individual Variables
        ↓
9. Analyze Distributions
        ↓
10. Detect Outliers
        ↓
11. Analyze Relationships
        ↓
12. Perform Group-Based Analysis
        ↓
13. Generate Visualizations
        ↓
14. Identify Important Patterns
        ↓
15. Formulate Questions / Hypotheses
        ↓
16. Document Findings
        ↓
17. Prepare Data for Next Stage

3. Step 1 — Define the Analytical Objective

Before opening the dataset, determine why the data is being analyzed.

Possible objectives include:

• Understanding student performance
• Analyzing sales trends
• Finding customer behavior patterns
• Identifying factors affecting revenue
• Preparing data for machine learning
• Detecting unusual transactions
• Comparing performance across categories

4. Step 2 — Load the Dataset

pandas is commonly used to load tabular datasets.

import pandas as pd

df = pd.read_csv("students.csv")

After loading the dataset, do not immediately start modifying it. First create an initial understanding of the raw data.

5. Step 3 — Take the First Look

Use head() to inspect the first records.

df.head()

Inspect the last records as well:

df.tail()

Display a random sample:

df.sample(5)

6. Step 4 — Understand Dataset Size

The shape attribute returns:

df.shape

The result is:

(number_of_rows, number_of_columns)

For example:

(5000, 12)

means the dataset contains:

• 5,000 rows
• 12 columns

7. Step 5 — Inspect Column Names

df.columns

Convert column names into a list:

df.columns.tolist()

Ask:

• What does each column represent?
• Are column names meaningful?
• Are there duplicate column names?
• Are there unnecessary columns?
• Could some columns contain identifiers rather than useful analytical variables?

8. Step 6 — Inspect Data Types and Structure

The info() method provides a compact structural overview.

df.info()

It can reveal:

• Column names
• Number of non-null values
• Data types
• Approximate memory usage

9. Step 7 — Inspect Data Types Directly

df.dtypes

Typical pandas data types include:

10. Step 8 — Generate a Statistical Summary

Use describe() to obtain descriptive statistics for numerical columns.

df.describe()

Common statistics include:

• Count
• Mean
• Standard deviation
• Minimum
• 25th percentile
• Median
• 75th percentile
• Maximum

11. Step 9 — Explore Categorical Variables

Numerical descriptive statistics are not enough for categorical columns.

Use value_counts().

df["Class"].value_counts()

To include missing values:

df["Class"].value_counts(
    dropna=False
)

To obtain proportions:

df["Class"].value_counts(
    normalize=True
)

12. Step 10 — Detect Missing Values

Missing data is one of the first data-quality issues to investigate.

df.isnull().sum()

Missing-value percentages can be calculated as:

missing_percentage = (
    df.isnull().mean() * 100
)

print(missing_percentage)

13. Create a Missing-Value Report

missing_report = pd.DataFrame({
    "Missing_Count": df.isnull().sum(),
    "Missing_Percentage":
        df.isnull().mean() * 100
})

missing_report = (
    missing_report
    .sort_values(
        "Missing_Percentage",
        ascending=False
    )
)

missing_report

This creates a compact report for prioritizing data-cleaning work.

14. Step 11 — Check Duplicate Records

Duplicate records can distort counts, averages, and other statistics.

df.duplicated().sum()

To view duplicate rows:

df[
    df.duplicated()
]

Before removing duplicates, determine whether they are truly accidental duplicates or legitimate repeated observations.

15. Step 12 — Inspect Unique Values

For categorical variables:

df["Class"].unique()

Count unique values:

df["Class"].nunique()

This can reveal unexpected categories such as:

"Male"
"male"
"MALE"
" M "

Such inconsistencies may represent the same category but appear different to the computer.

16. Step 13 — Check Data Validity

Data can be non-null but still invalid.

Domain knowledge is essential when deciding whether a value is actually invalid.

17. Step 14 — Identify Numerical Variables

numeric_columns = (
    df.select_dtypes(
        include="number"
    ).columns
)

print(numeric_columns)

Numerical variables can then be analyzed for:

• Central tendency
• Spread
• Distribution
• Skewness
• Outliers
• Relationships

18. Step 15 — Identify Categorical Variables

categorical_columns = (
    df.select_dtypes(
        include=["object", "category", "string"]
    ).columns
)

print(categorical_columns)

Categorical variables should generally be explored through frequencies, proportions, and group comparisons.

19. Step 16 — Analyze Distributions

A distribution describes how values are spread across a variable.

Important questions include:

• Is the distribution symmetric?
• Is it skewed?
• Are there multiple peaks?
• Are there extreme observations?
• Is the range reasonable?

A histogram is a common starting point.

import plotly.express as px

fig = px.histogram(
    df,
    x="Score",
    title="Score Distribution"
)

fig.show()

20. Step 17 — Examine Skewness

Skewness describes asymmetry in a distribution.

df["Score"].skew()

A rough interpretation is:

These are descriptive guidelines rather than absolute rules.

21. Step 18 — Detect Outliers

An outlier is an observation that is unusually distant from other observations according to a chosen criterion.

One common approach is the IQR method.

Q1 = df["Score"].quantile(0.25)
Q3 = df["Score"].quantile(0.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[
    (df["Score"] < lower) |
    (df["Score"] > upper)
]

outliers

22. Do Not Automatically Delete Outliers

An unusual value is not necessarily an error.

It may represent:

• A genuine extreme observation
• A rare but important event
• A measurement error
• A data-entry mistake
• A different population or subgroup

23. Visual Outlier Detection

Box plots provide a useful visual way to inspect distributions and potential outliers.

fig = px.box(
    df,
    y="Score",
    title="Score Distribution"
)

fig.show()

24. Step 19 — Analyze Relationships Between Numerical Variables

Correlation can help identify linear relationships between numerical variables.

correlation = df.corr(
    numeric_only=True
)

print(correlation)

A correlation matrix can be visualized using Plotly.

fig = px.imshow(
    correlation,
    text_auto=True,
    title="Correlation Matrix"
)

fig.show()

25. Step 20 — Use Scatter Plots for Relationships

A scatter plot allows individual observations to be examined.

fig = px.scatter(
    df,
    x="Attendance",
    y="Score",
    title="Attendance vs Examination Score"
)

fig.show()

Look for:

• Positive relationships
• Negative relationships
• Clusters
• Outliers
• Non-linear patterns
• Changes in spread

26. Step 21 — Compare Numerical Variables Across Categories

Grouping allows you to ask questions such as:

• Which class has the highest average score?
• Which department has the highest revenue?
• Does customer behavior differ by region?

df.groupby(
    "Class"
)["Score"].mean()

For multiple statistics:

df.groupby(
    "Class"
)["Score"].agg([
    "count",
    "mean",
    "median",
    "min",
    "max"
])

27. Visualize Group Differences

fig = px.box(
    df,
    x="Class",
    y="Score",
    color="Class",
    title="Score Distribution by Class"
)

fig.show()

A grouped visualization often reveals differences that are hidden by an overall average.

28. Step 22 — Explore Two-Variable and Multi-Variable Patterns

EDA should progress from individual variables to relationships between variables.

fig = px.scatter(
    df,
    x="StudyHours",
    y="Score",
    color="Class",
    size="Attendance",
    hover_data=[
        "StudentID"
    ],
    title="Multi-Variable Student Analysis"
)

fig.show()

This visualization represents several dimensions simultaneously.

29. Step 23 — Analyze Date and Time Variables

Dates should be converted to an appropriate datetime representation before time-based analysis.

df["Date"] = pd.to_datetime(
    df["Date"]
)

Extract useful components:

df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month
df["Day"] = df["Date"].dt.day
df["DayOfWeek"] = df["Date"].dt.day_name()

Then visualize trends:

fig = px.line(
    df,
    x="Date",
    y="Sales",
    title="Sales Over Time"
)

fig.show()

30. Look for Temporal Patterns

Ask:

• Is there an upward or downward trend?
• Are there seasonal patterns?
• Are there unusual spikes?
• Are some days or months consistently different?
• Are observations evenly distributed over time?

31. Step 24 — Check for Potential Data Leakage

In predictive modeling, data leakage occurs when information that would not legitimately be available at prediction time influences the model or analysis.

EDA should therefore include a review of whether every variable makes sense for the intended analytical stage.

32. Step 25 — Identify Identifier Columns

Columns such as:

StudentID
TransactionID
CustomerID
OrderID

may uniquely identify records but may not carry meaningful predictive information.

Do not automatically remove them. Determine their purpose first.

33. Step 26 — Analyze Cardinality

Cardinality refers to the number of distinct values in a variable.

df.nunique()

For example:

34. Step 27 — Explore Multiple Numerical Variables

For a smaller number of variables, pairwise visualizations can provide a broad overview of relationships.

import seaborn as sns

sns.pairplot(
    df[
        [
            "Attendance",
            "StudyHours",
            "Score"
        ]
    ]
)

Pair plots can help reveal:

• Relationships
• Clusters
• Distribution shapes
• Potential outliers

35. Automated EDA Tools

Automated profiling tools can generate broad summaries of datasets.

However, automated reports should complement—not replace—analytical reasoning.

36. Complete EDA Checklist

37. Reusable EDA Starter Template

The following template can be adapted to many tabular datasets.

import pandas as pd
import plotly.express as px

# 1. Load data
df = pd.read_csv("data.csv")

# 2. Basic structure
print("Shape:", df.shape)
print("\nColumns:")
print(df.columns.tolist())

# 3. First records
print("\nFirst five rows:")
print(df.head())

# 4. Data types and null counts
print("\nData information:")
df.info()

# 5. Statistical summary
print("\nStatistical summary:")
print(df.describe())

# 6. Missing values
print("\nMissing values:")
print(df.isnull().sum())

# 7. Duplicate records
print("\nDuplicate rows:")
print(df.duplicated().sum())

# 8. Unique values
print("\nUnique values:")
print(df.nunique())

# 9. Numerical columns
numeric_columns = df.select_dtypes(
    include="number"
).columns

print("\nNumerical columns:")
print(numeric_columns)

# 10. Categorical columns
categorical_columns = df.select_dtypes(
    include=["object", "category", "string"]
).columns

print("\nCategorical columns:")
print(categorical_columns)

# 11. Correlation
correlation = df.corr(
    numeric_only=True
)

print("\nCorrelation:")
print(correlation)

# 12. Example visualization
fig = px.histogram(
    df,
    x=numeric_columns[0],
    title="Distribution"
)

fig.show()

38. How to Document EDA Findings

A professional EDA should not end with charts alone. Record the important findings.

39. Think in Three Levels of EDA

40. EDA vs Data Cleaning

EDA and data cleaning are closely related but are not identical.

In practice, EDA and cleaning often form an iterative cycle.

Explore
   ↓
Find Problem
   ↓
Investigate
   ↓
Clean / Transform
   ↓
Explore Again
   ↓
Validate

41. Common EDA Mistakes

42. EDA Best Practices

• Start with the analytical objective.
• Preserve the original dataset before making transformations.
• Inspect structure before cleaning.
• Check both missingness and validity.
• Analyze numerical and categorical variables differently.
• Use visualizations to complement numerical summaries.
• Investigate outliers rather than automatically removing them.
• Examine relationships between important variables.
• Use group-based analysis to discover differences.
• Consider time-related patterns when dates are present.
• Check for potential data leakage before predictive modeling.
• Record findings and assumptions.
• Re-run EDA after major cleaning or transformation steps.

43. EDA Interview Questions

df.isnull().sum()

df.duplicated().sum()

44. Examination Questions — MCQs

45. Practical Examination Tasks

46. Mini Case Study — Student Performance Dataset

Assume the dataset contains:

StudentID
Class
Attendance
StudyHours
AssignmentScore
ExamScore

Step 1 — Structure

df.shape
df.columns
df.info()

Step 2 — Quality

df.isnull().sum()
df.duplicated().sum()

Step 3 — Statistics

df.describe()

Step 4 — Class Distribution

df["Class"].value_counts()

Step 5 — Score Distribution

fig = px.histogram(
    df,
    x="ExamScore",
    title="Examination Score Distribution"
)

fig.show()

Step 6 — Attendance Relationship

fig = px.scatter(
    df,
    x="Attendance",
    y="ExamScore",
    color="Class",
    title="Attendance vs Examination Score"
)

fig.show()

Step 7 — Class Comparison

class_summary = df.groupby(
    "Class"
)["ExamScore"].agg([
    "count",
    "mean",
    "median"
])

print(class_summary)

Step 8 — Final Questions

• Which class has the highest average score?
• Is attendance associated with examination score?
• Are there unusual scores?
• Are any important values missing?
• Are some classes much more variable than others?
• What additional data would improve the analysis?

47. The EDA Mindset

Good EDA is not simply about knowing pandas commands. It requires analytical thinking.

For example:

Observation:
Class A has a higher average score.

        ↓

Question:
Is Class A actually performing better?

        ↓

Investigation:
Compare distributions, sample sizes and variability.

        ↓

Validation:
Check whether the difference is consistent
and whether data quality issues exist.

        ↓

Explanation:
Report the evidence and limitations.

48. Quick-Fire EDA Revision

50. Final Takeaways

• EDA is a structured investigation of a dataset.
• Always begin by understanding the analytical objective.
• Inspect the dataset structure before modifying the data.
• Use shape, info(), dtypes, head() and describe() during the initial inspection.
• Missing values and duplicate records should be explicitly investigated.
• Non-null data can still be invalid, so domain rules matter.
• Analyze numerical and categorical variables using appropriate techniques.
• Use histograms and box plots to understand distributions.
• Use scatter plots and correlation analysis to investigate relationships.
• Group-based analysis can reveal patterns hidden by overall statistics.
• Outliers should be investigated rather than automatically removed.
• Correlation does not prove causation.
• Check for potential data leakage before predictive modeling.
• EDA is iterative: exploration can reveal problems that require cleaning, after which the data should be explored again.
• A professional EDA ends with documented findings, limitations and recommended next steps.