6.2 Project 1 — Data Analytics Pipeline

6.2 Project 1 — Data Analytics Pipeline

In the previous modules, you learned individual Python, NumPy, pandas, visualization, and automation techniques. This project combines those skills into one complete Data Analytics Pipeline.

You will work with a real-world public dataset available through Kaggle and follow a professional analytics workflow from raw data to actionable insights.

1. Complete Data Analytics Pipeline

Kaggle Dataset
      ↓
Dataset Acquisition
      ↓
Project Setup
      ↓
Data Loading
      ↓
Initial Inspection
      ↓
Data Cleaning
      ↓
Data Transformation
      ↓
Exploratory Data Analysis
      ↓
Statistical Analysis
      ↓
Visualization
      ↓
Insight Generation
      ↓
Report Creation
      ↓
Final Conclusions

This workflow is reusable across many data analytics projects.

2. What You Will Learn

• How to select a suitable public dataset.
• How to download and organize a dataset.
• How to load CSV data using pandas.
• How to inspect the structure of a dataset.
• How to identify missing values.
• How to identify duplicate records.
• How to detect inconsistent data.
• How to transform columns.
• How to perform descriptive statistics.
• How to identify trends and relationships.
• How to create meaningful visualizations.
• How to formulate analytical questions.
• How to communicate findings.
• How to build a reproducible analytics project.

3. Selecting a Kaggle Dataset

Kaggle hosts a large collection of public datasets that can be used for learning, experimentation, and portfolio projects.

For this tutorial, choose a dataset that contains enough records, multiple useful columns, and meaningful analytical questions.

Good Dataset Characteristics

• At least several hundred records.
• Multiple columns.
• A mixture of numerical and categorical variables where possible.
• A meaningful analytical context.
• Enough variation to discover patterns.
• Potentially some missing or imperfect data for cleaning practice.

4. Example Project Dataset

For demonstration, this project uses a generic Retail Sales Dataset structure.

The exact Kaggle dataset may differ, so students should adapt the column names in the code to match the dataset they download.

5. Define Analytical Questions Before Coding

Professional data analysis should not begin with random charts. Begin by defining questions that the data can answer.

Example Questions

1. What is the total sales value?
2. Which product categories generate the most revenue?
3. Which regions perform best?
4. How do sales change over time?
5. Which products have the highest sales?
6. What is the average order value?
7. Are there seasonal patterns?
8. Is quantity related to sales?
9. Which categories have the highest average selling price?
10. Which areas require further investigation?

6. Recommended Project Structure

retail_data_analysis/
│
├── data/
│   ├── raw/
│   │   └── retail_sales.csv
│   │
│   └── processed/
│       └── cleaned_sales.csv
│
├── notebooks/
│   └── retail_eda.ipynb
│
├── src/
│   ├── cleaning.py
│   └── analysis.py
│
├── reports/
│   └── analysis_report.html
│
├── visualizations/
│
├── logs/
│
├── requirements.txt
│
└── README.md

For a beginner project, a single Jupyter Notebook is acceptable. As the project becomes more advanced, separate reusable logic into Python modules.

7. Create the Python Environment

A dedicated environment helps prevent dependency conflicts.

python -m venv .venv

Activate the environment according to your operating system.

Then install the main project libraries:

pip install pandas numpy matplotlib seaborn jupyter

Save the installed dependencies:

pip freeze > requirements.txt

8. Start Jupyter Notebook

jupyter notebook

Create:

notebooks/retail_eda.ipynb

Organize the notebook into logical sections rather than writing code randomly from top to bottom.

9. Import Required Libraries

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

A clean import section makes the notebook easier to understand and reproduce.

10. Load the Kaggle Dataset

Suppose the downloaded file is:

data/raw/retail_sales.csv

Load it using pandas:

df = pd.read_csv(
    "../data/raw/retail_sales.csv"
)

df.head()

If your notebook location differs, adjust the path accordingly.

11. Understand Dataset Dimensions

df.shape

Example output:

(5000, 8)

This means the dataset contains:

• 5000 rows
• 8 columns

12. Examine Column Names

df.columns

You can also convert the column names to a list:

df.columns.tolist()

13. Preview the Dataset

df.head()

Display the final rows:

df.tail()

Display a larger sample:

df.sample(10)

14. Inspect Data Types and Missing Values

df.info()

This is one of the most important first inspection commands.

It helps identify:

• Number of records
• Column names
• Non-null counts
• Data types
• Memory usage

15. Descriptive Statistics

df.describe()

This provides statistics such as:

• Count
• Mean
• Standard deviation
• Minimum
• 25th percentile
• Median
• 75th percentile
• Maximum

16. Inspect Categorical Variables

df["Category"].value_counts()

This counts the number of records in each category.

For percentages:

df["Category"].value_counts(
    normalize=True
) * 100

17. Data Quality Assessment

Before performing analysis, inspect the dataset for common quality problems.

Check:

• Missing values
• Duplicate rows
• Incorrect data types
• Invalid values
• Inconsistent text
• Outliers
• Unexpected categories

18. Find Missing Values

df.isnull().sum()

Calculate missing percentages:

missing_percentage = (
    df.isnull().mean() * 100
)

print(
    missing_percentage
)

19. Find Duplicate Records

df.duplicated().sum()

View duplicates:

duplicates = df[
    df.duplicated()
]

duplicates.head()

20. Remove Duplicate Records

df = df.drop_duplicates()

Verify the new shape:

df.shape

21. Convert Date Columns

Date columns should generally be converted to pandas datetime objects before time-based analysis.

df["Order_Date"] = pd.to_datetime(
    df["Order_Date"],
    errors="coerce"
)

Check the result:

df["Order_Date"].dtype

22. Create Time-Based Features

df["Year"] = (
    df["Order_Date"].dt.year
)

df["Month"] = (
    df["Order_Date"].dt.month
)

df["Month_Name"] = (
    df["Order_Date"].dt.month_name()
)

df["Day"] = (
    df["Order_Date"].dt.day
)

These derived columns allow us to investigate temporal patterns.

23. Create Calculated Measures

If the dataset contains quantity and unit price, calculate sales independently when appropriate.

df["Calculated_Sales"] = (
    df["Quantity"] *
    df["Unit_Price"]
)

Compare it with an existing sales column:

df[
    ["Sales", "Calculated_Sales"]
].head()

This can help identify inconsistencies in the source data.

24. Clean Text Columns

df["Category"] = (
    df["Category"]
    .astype("string")
    .str.strip()
)

Standardize case when appropriate:

df["Category"] = (
    df["Category"]
    .str.title()
)

Be careful with automatic standardization because capitalization can sometimes carry meaningful information.

25. Validate Numerical Columns

df["Quantity"].describe()

df["Unit_Price"].describe()

df["Sales"].describe()

Look for impossible or suspicious values.

For example:

df[
    df["Quantity"] <= 0
]

26. Do Not Clean Data Blindly

Data cleaning is not simply a collection of commands such as dropna() and drop_duplicates().

Every transformation should have an analytical reason.

27. Save the Clean Dataset

df.to_csv(
    "../data/processed/cleaned_sales.csv",
    index=False
)

Keeping the cleaned dataset separate from the original raw data preserves reproducibility.

28. Calculate Key Performance Indicators

Begin the analytical stage by calculating high-level measures.

total_sales = df["Sales"].sum()

total_quantity = df["Quantity"].sum()

average_sales = df["Sales"].mean()

orders = df["Order_ID"].nunique()

average_order_value = (
    total_sales / orders
)

Display them:

print(
    "Total Sales:",
    total_sales
)

print(
    "Total Quantity:",
    total_quantity
)

print(
    "Average Sales:",
    average_sales
)

print(
    "Number of Orders:",
    orders
)

print(
    "Average Order Value:",
    average_order_value
)

29. Sales by Category

category_sales = (
    df.groupby("Category")["Sales"]
    .sum()
    .sort_values(
        ascending=False
    )
)

print(
    category_sales
)

This identifies the categories contributing the greatest sales.

30. Sales by Region

region_sales = (
    df.groupby("Region")["Sales"]
    .sum()
    .sort_values(
        ascending=False
    )
)

print(
    region_sales
)

31. Top Products

top_products = (
    df.groupby("Product")["Sales"]
    .sum()
    .sort_values(
        ascending=False
    )
    .head(10)
)

print(
    top_products
)

32. Sales Over Time

monthly_sales = (
    df.groupby(
        df["Order_Date"]
        .dt.to_period("M")
    )["Sales"]
    .sum()
)

print(
    monthly_sales
)

Convert the period index to timestamps when required for visualization:

monthly_sales.index = (
    monthly_sales.index
    .to_timestamp()
)

33. Visualization — Monthly Sales Trend

plt.figure(
    figsize=(12, 6)
)

plt.plot(
    monthly_sales.index,
    monthly_sales.values
)

plt.title(
    "Monthly Sales Trend"
)

plt.xlabel(
    "Month"
)

plt.ylabel(
    "Sales"
)

plt.xticks(
    rotation=45
)

plt.tight_layout()

plt.show()

A line chart is appropriate when the primary question concerns change over time.

34. Visualization — Sales by Category

plt.figure(
    figsize=(10, 6)
)

category_sales.plot(
    kind="bar"
)

plt.title(
    "Sales by Category"
)

plt.xlabel(
    "Category"
)

plt.ylabel(
    "Sales"
)

plt.xticks(
    rotation=45
)

plt.tight_layout()

plt.show()

35. Visualization — Top Products

top_products.sort_values().plot(
    kind="barh",
    figsize=(10, 6)
)

plt.title(
    "Top 10 Products by Sales"
)

plt.xlabel(
    "Sales"
)

plt.ylabel(
    "Product"
)

plt.tight_layout()

plt.show()

36. Visualization — Sales Distribution

plt.figure(
    figsize=(10, 6)
)

plt.hist(
    df["Sales"],
    bins=30
)

plt.title(
    "Distribution of Sales"
)

plt.xlabel(
    "Sales"
)

plt.ylabel(
    "Frequency"
)

plt.tight_layout()

plt.show()

Histograms help understand the distribution of a numerical variable.

37. Visualization — Detecting Outliers

plt.figure(
    figsize=(10, 5)
)

sns.boxplot(
    x=df["Sales"]
)

plt.title(
    "Sales Distribution and Potential Outliers"
)

plt.tight_layout()

plt.show()

38. Visualization — Quantity vs Sales

plt.figure(
    figsize=(10, 6)
)

sns.scatterplot(
    data=df,
    x="Quantity",
    y="Sales"
)

plt.title(
    "Quantity vs Sales"
)

plt.tight_layout()

plt.show()

Scatter plots can reveal relationships, clusters, and unusual observations.

39. Correlation Analysis

Select numerical variables:

numeric_columns = [
    "Quantity",
    "Unit_Price",
    "Sales"
]

correlation = (
    df[numeric_columns]
    .corr()
)

print(
    correlation
)

40. Correlation Heatmap

plt.figure(
    figsize=(8, 6)
)

sns.heatmap(
    correlation,
    annot=True,
    cmap="coolwarm",
    fmt=".2f"
)

plt.title(
    "Correlation Matrix"
)

plt.tight_layout()

plt.show()

41. Multi-Dimensional Analysis

Analyze category performance by region:

category_region = (
    df.groupby(
        ["Region", "Category"]
    )["Sales"]
    .sum()
    .reset_index()
)

print(
    category_region
)

42. Pivot Table Analysis

pivot_sales = pd.pivot_table(
    df,
    values="Sales",
    index="Region",
    columns="Category",
    aggfunc="sum",
    fill_value=0
)

print(
    pivot_sales
)

Pivot tables are useful for comparing multiple dimensions at once.

43. Structured EDA Questions

Now move beyond individual commands and ask structured questions.

Question A — Overall Performance

• How many records exist?
• How many unique orders exist?
• What is total sales?
• What is average order value?

Question B — Category Performance

• Which category has the highest sales?
• Which category has the lowest sales?
• How much does each category contribute?

Question C — Geographic Performance

• Which region performs best?
• Which region performs worst?
• Do categories behave differently across regions?

Question D — Time Trends

• Are sales increasing?
• Are there seasonal patterns?
• Are there unusual spikes or drops?

Question E — Product Performance

• Which products dominate sales?
• Are a small number of products responsible for a large proportion of revenue?

44. Calculate Category Contribution

category_sales_percentage = (
    category_sales /
    category_sales.sum()
) * 100

print(
    category_sales_percentage
)

This tells us what percentage of total sales comes from each category.

45. Identify Top Contributors

category_summary = (
    pd.DataFrame({
        "Sales": category_sales,
        "Contribution_%":
            category_sales_percentage
    })
)

print(
    category_summary
)

This creates a concise analytical summary suitable for a report.

46. Observation vs Insight

A strong data analyst does more than describe charts.

47. Outlier Investigation

A high-value transaction is not automatically an error.

First identify unusual observations:

Q1 = df["Sales"].quantile(
    0.25
)

Q3 = df["Sales"].quantile(
    0.75
)

IQR = Q3 - Q1

lower_bound = (
    Q1 - 1.5 * IQR
)

upper_bound = (
    Q3 + 1.5 * IQR
)

outliers = df[
    (df["Sales"] < lower_bound) |
    (df["Sales"] > upper_bound)
]

print(
    outliers
)

Investigate these observations before deciding whether they should be retained or excluded.

48. Final Data Validation

print(
    "Rows:",
    len(df)
)

print(
    "Missing values:"
)

print(
    df.isnull().sum()
)

print(
    "Duplicate rows:",
    df.duplicated().sum()
)

Also verify that calculated metrics are logically consistent.

49. Export Analytical Summary

category_summary.to_csv(
    "../reports/category_summary.csv"
)

region_sales.to_csv(
    "../reports/region_sales.csv"
)

top_products.to_csv(
    "../reports/top_products.csv"
)

50. Save Charts

plt.figure(
    figsize=(10, 6)
)

category_sales.plot(
    kind="bar"
)

plt.title(
    "Sales by Category"
)

plt.xlabel(
    "Category"
)

plt.ylabel(
    "Sales"
)

plt.tight_layout()

plt.savefig(
    "../visualizations/"
    "sales_by_category.png",
    dpi=300
)

plt.show()

Saving charts makes them reusable in reports and presentations.

51. Make the Project Reproducible

A professional analytics project should allow another person to understand how the result was produced.

Include:

• Dataset source.
• Dataset version or download date.
• Python version.
• Library versions.
• Cleaning decisions.
• Analysis methodology.
• Important assumptions.
• Generated reports.
• Visualization files.

52. Create requirements.txt

A project can record its Python dependencies in:

requirements.txt

Example:

numpy
pandas
matplotlib
seaborn
jupyter

Alternatively, an environment can generate a version-pinned file using:

pip freeze > requirements.txt

53. Create a README

A good README should explain the project before someone opens the notebook.

Recommended Sections

1. Project title
2. Project objective
3. Dataset source
4. Tools and technologies
5. Data cleaning process
6. EDA methodology
7. Key findings
8. Visualizations
9. Limitations
10. How to run the project

54. Suggested README Structure

# Retail Sales Data Analysis

## Project Overview

Brief description of the analytical problem.

## Dataset

Source and description of the dataset.

## Objectives

List the analytical questions.

## Technologies

- Python
- pandas
- NumPy
- Matplotlib
- Seaborn
- Jupyter

## Data Preparation

Describe cleaning and transformation.

## Exploratory Data Analysis

Describe the major analyses.

## Key Findings

Summarize evidence-based findings.

## Limitations

Explain what the data cannot establish.

## How to Run

Installation and execution instructions.

55. Document Analytical Limitations

Every real-world dataset has limitations.

Examples include:

• Missing observations.
• Limited time period.
• Potential sampling bias.
• Unknown data collection methodology.
• Unavailable variables.
• Potential measurement errors.

56. Turn EDA into a Data Story

Your final project should tell a logical story rather than present a random collection of charts.

Business / Analytical Question
          ↓
Dataset
          ↓
Data Quality
          ↓
Important Metrics
          ↓
Patterns
          ↓
Comparisons
          ↓
Insights
          ↓
Limitations
          ↓
Recommendations

57. Think Like a Dashboard Designer

Even if you are not building an interactive dashboard, think about what an executive or decision-maker would need to see first.

Example KPI Layer

58. Automate the Analytics Pipeline

Once the notebook works, convert repeatable parts into a Python script.

Raw Dataset
    ↓
Python Script
    ↓
Clean Data
    ↓
Analyze
    ↓
Generate Charts
    ↓
Save Report
    ↓
Email / Archive

This connects the project directly with the automation concepts from Module 5 and Lesson 6.1.

59. Convert EDA into a Python Script

from pathlib import Path

import pandas as pd


BASE_DIR = (
    Path(__file__)
    .resolve()
    .parent
)

DATA_FILE = (
    BASE_DIR
    / "data"
    / "raw"
    / "retail_sales.csv"
)


def load_data():

    return pd.read_csv(
        DATA_FILE
    )


def analyze_sales(df):

    return (
        df.groupby("Category")["Sales"]
        .sum()
        .sort_values(
            ascending=False
        )
    )


def main():

    df = load_data()

    print(
        df.head()
    )

    category_sales = (
        analyze_sales(df)
    )

    print(
        category_sales
    )


if __name__ == "__main__":

    main()

This is the beginning of converting exploratory notebook work into reusable application code.

60. Data Analytics Quality Checklist

• ☐ Dataset source is documented.
• ☐ Raw data is preserved.
• ☐ Data types have been checked.
• ☐ Missing values have been investigated.
• ☐ Duplicates have been investigated.
• ☐ Invalid values have been investigated.
• ☐ Date fields have been validated.
• ☐ Numerical distributions have been inspected.
• ☐ Outliers have been investigated.
• ☐ Analytical questions are clearly defined.
• ☐ Each important chart answers a question.
• ☐ Statistical results are interpreted correctly.
• ☐ Correlation is not presented as causation.
• ☐ Findings are supported by evidence.
• ☐ Limitations are documented.
• ☐ Code is reproducible.

61. Data Analytics Interview Questions

62. Examination Questions — MCQs

63. Practical Examination Tasks

64. Project Challenge

Select a Kaggle dataset of your choice and complete the entire pipeline independently.

Minimum Requirements

1. Public Kaggle dataset.
2. At least 500 records where practical.
3. At least five useful columns.
4. Documented analytical questions.
5. Data-quality assessment.
6. Data-cleaning process.
7. At least five EDA analyses.
8. At least five meaningful visualizations.
9. At least one grouped analysis.
10. At least one time-based analysis where dates are available.
11. At least one correlation analysis where appropriate.
12. Key findings.
13. Limitations.
14. Recommendations supported by the data.
15. README documentation.

65. Project Evaluation Rubric

66. Expected Final Deliverables

A complete submission should contain:

retail_data_analysis/
│
├── data/
│   ├── raw/
│   │   └── original_dataset.csv
│   │
│   └── processed/
│       └── cleaned_dataset.csv
│
├── notebooks/
│   └── EDA_Project.ipynb
│
├── visualizations/
│   ├── sales_trend.png
│   ├── category_sales.png
│   ├── region_sales.png
│   └── correlation.png
│
├── reports/
│   └── analysis_summary.csv
│
├── requirements.txt
│
└── README.md

67. Advanced Extension — Automate the Pipeline

Once the project works interactively, extend it into an automated analytics pipeline.

Kaggle / Public Dataset
          ↓
Scheduled Python Script
          ↓
Data Validation
          ↓
Cleaning
          ↓
EDA / Metrics
          ↓
Charts
          ↓
Report
          ↓
Email Notification
          ↓
Archive

You can schedule this pipeline using the techniques learned in Lesson 6.1.

68. Expert Tips for a Professional Data Project

1. Start with questions, not charts.
2. Understand the dataset before cleaning it.
3. Preserve the raw data.
4. Document every important cleaning decision.
5. Do not remove outliers automatically.
6. Use the appropriate chart for the question.
7. Do not confuse correlation with causation.
8. Distinguish observations from conclusions.
9. State analytical limitations.
10. Make the analysis reproducible.
11. Keep notebooks readable.
12. Move reusable logic into Python modules.
13. Use version-controlled project files.
14. Automate repeatable workflows.

69. Self-Assessment Checklist

Before moving to Project 2, make sure you can:

• ☐ Select a suitable public dataset.
• ☐ Create a data analytics project structure.
• ☐ Load CSV data using pandas.
• ☐ Inspect rows and columns.
• ☐ Analyze data types.
• ☐ Detect missing values.
• ☐ Investigate duplicate records.
• ☐ Validate numerical values.
• ☐ Convert date columns.
• ☐ Create derived columns.
• ☐ Group and aggregate data.
• ☐ Build pivot tables.
• ☐ Calculate KPIs.
• ☐ Analyze trends.
• ☐ Analyze distributions.
• ☐ Detect and investigate outliers.
• ☐ Calculate correlations.
• ☐ Build meaningful visualizations.
• ☐ Interpret analytical results.
• ☐ Distinguish correlation from causation.
• ☐ Document limitations.
• ☐ Create a reproducible project.
• ☐ Write a professional README.
• ☐ Convert exploratory work into reusable Python code.
• ☐ Connect the analytics pipeline with scheduled automation.