Model Performance Evaluation Metrics

Developing a Machine Learning model is only one part of an Artificial Intelligence project. After the model has been trained, it is important to determine how well it performs. A model that performs well on training data but fails on unseen data cannot be considered reliable.

To measure the effectiveness of a Machine Learning model, we use Model Performance Evaluation Metrics. These metrics compare the model's predictions with the actual values and help determine its accuracy, reliability, and overall performance.

Different evaluation metrics are used for different types of Machine Learning problems. For classification problems, metrics such as Accuracy, Precision, Recall, F1-Score, and Confusion Matrix are commonly used. For regression problems, Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) are widely used.

---

Learning Objectives

After studying this topic, you will be able to:

• Understand the purpose of model evaluation.
• Explain the Confusion Matrix.
• Calculate Accuracy, Precision, Recall, and F1-Score.
• Understand Mean Squared Error (MSE).
• Understand Root Mean Squared Error (RMSE).
• Select suitable evaluation metrics for different Machine Learning problems.

---

What are Model Performance Evaluation Metrics?

Model Performance Evaluation Metrics are mathematical measures used to evaluate how well a Machine Learning model performs by comparing its predictions with the actual outcomes.

---

Why are Evaluation Metrics Important?

Evaluation metrics help determine whether a Machine Learning model is suitable for deployment. They allow data scientists to compare multiple models and select the one that performs best.

• Measure prediction accuracy.
• Compare multiple models.
• Identify strengths and weaknesses.
• Detect prediction errors.
• Improve model performance.

---

Types of Evaluation Metrics

Machine Learning Problem	Common Metrics
Classification	Accuracy, Precision, Recall, F1-Score, Confusion Matrix
Regression	Mean Squared Error (MSE), Root Mean Squared Error (RMSE)

---

Confusion Matrix

A Confusion Matrix is a table used to evaluate the performance of a classification model. It compares the actual class labels with the predicted class labels.

---

Structure of a Confusion Matrix

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

---

Understanding the Terms

Term	Meaning
True Positive (TP)	Positive cases correctly predicted as positive.
True Negative (TN)	Negative cases correctly predicted as negative.
False Positive (FP)	Negative cases incorrectly predicted as positive.
False Negative (FN)	Positive cases incorrectly predicted as negative.

---

Real-Life Example

Suppose an AI model predicts whether an email is Spam or Not Spam.

• Spam correctly identified → True Positive.
• Normal email correctly identified → True Negative.
• Normal email identified as Spam → False Positive.
• Spam identified as Normal → False Negative.

---

Sample Confusion Matrix

	Predicted Positive	Predicted Negative
Actual Positive	80	10
Actual Negative	5	105

Therefore,

• TP = 80
• FN = 10
• FP = 5
• TN = 105

---

Accuracy

Accuracy measures the proportion of correct predictions made by the model out of all predictions.

Accuracy = (TP + TN)

           ───────────────

         TP + TN + FP + FN

---

Example

Using the above confusion matrix,

TP = 80

TN = 105

FP = 5

FN = 10

Accuracy

= (80 + 105)

  ─────────────

80 + 105 + 5 + 10

= 185 / 200

= 0.925

= 92.5%

---

Precision

Precision measures how many of the predicted positive cases are actually positive.

Precision = TP

           ───────

          TP + FP

---

Example

Precision

= 80

────────

80 + 5

= 80 / 85

= 0.941

= 94.1%

---

Why is Precision Important?

Precision is especially important when False Positives are costly.

Example:

• Email Spam Detection
• Fraud Detection
• Face Recognition

---

Think Like a Data Scientist

An email filtering system marks many genuine emails as spam.

Which evaluation metric should be improved?

Click to View Answer

The model should improve its Precision because it is producing many False Positives.

---

Recall (Sensitivity)

Recall measures how many of the actual positive cases are correctly identified by the Machine Learning model.

It focuses on minimizing False Negatives (FN). Recall becomes extremely important when missing a positive case can lead to serious consequences.

Recall = TP

         ───────

        TP + FN

---

Example

Using the same confusion matrix:

TP = 80

FN = 10

Recall

= 80

────────

80 + 10

= 80 / 90

= 0.889

= 88.9%

---

Why is Recall Important?

Recall is important when missing a positive case is more dangerous than making a wrong positive prediction.

Examples include:

• Disease detection.
• Cancer diagnosis.
• Fire detection systems.
• Earthquake warning systems.
• Fraud detection.

---

F1-Score

Sometimes, both Precision and Recall are equally important. In such situations, the F1-Score is used because it combines Precision and Recall into a single performance measure.

          2 × Precision × Recall

F1 Score = ──────────────────────

            Precision + Recall

---

Example

Precision = 94.1%

Recall = 88.9%

F1 Score

= 2 × 0.941 × 0.889

──────────────────────

0.941 + 0.889

= 0.914

= 91.4%

---

Why is F1-Score Important?

• Balances Precision and Recall.
• Useful for imbalanced datasets.
• Provides a single performance measure.
• Widely used in classification problems.

---

Mean Squared Error (MSE)

For Regression problems, the output is a continuous numerical value. Therefore, classification metrics such as Accuracy and Precision cannot be used.

Instead, we calculate the prediction error using Mean Squared Error (MSE).

---

Formula

          Σ (Actual − Predicted)²

MSE = ───────────────────────────

             Number of Records

---

Worked Example

Actual	Predicted	Error	Error²
50	48	2	4
60	62	-2	4
70	69	1	1
80	82	-2	4
90	88	2	4

Total Squared Error = 17

MSE = 17 / 5

MSE = 3.4

---

Root Mean Squared Error (RMSE)

RMSE is obtained by taking the square root of the Mean Squared Error. Since RMSE has the same unit as the original data, it is easier to interpret than MSE.

---

Formula

RMSE = √MSE

---

Worked Example

MSE = 3.4

RMSE = √3.4

RMSE = 1.84

---

Relationship Between MSE and RMSE

MSE	RMSE
Average squared error.	Square root of MSE.
Unit is squared.	Unit is same as original data.
Less intuitive.	Easier to interpret.

---

Comparison of Evaluation Metrics

Metric	Used For	Objective
Accuracy	Classification	Overall correct predictions.
Precision	Classification	Correct positive predictions.
Recall	Classification	Detect actual positive cases.
F1-Score	Classification	Balance Precision and Recall.
MSE	Regression	Average squared prediction error.
RMSE	Regression	Error in original units.

---

Classification vs Regression Metrics

Classification	Regression
Accuracy	MSE
Precision	RMSE
Recall	MAE (Advanced)
F1-Score	R² Score (Advanced)

---

Case Study

A hospital develops an AI system to detect pneumonia from X-ray images.

• High Recall ensures that very few pneumonia patients are missed.
• High Precision reduces false alarms for healthy patients.
• F1-Score balances both objectives.

Therefore, the hospital evaluates all three metrics before deploying the system.

---

Workflow of Model Evaluation

Train Model

      │

      ▼

Generate Predictions

      │

      ▼

Compare with Actual Values

      │

      ▼

Calculate Metrics

      │

      ▼

Evaluate Performance

---

Think Like a Data Scientist

An AI system predicts whether patients have a serious disease. Which metric should be given higher priority: Precision or Recall? Explain your answer.

Click to View Answer

Recall should be given higher priority because failing to identify a patient who actually has the disease (False Negative) can be life-threatening.

---

Competency-Based Question

A spam email detection system correctly identifies most spam emails but also marks many genuine emails as spam.

Which evaluation metric should be improved? Justify your answer.

---

Activity

Given the following confusion matrix, calculate:

• Accuracy
• Precision
• Recall
• F1-Score

	Predicted Positive	Predicted Negative
Actual Positive	45	5
Actual Negative	10	40

---

Common Beginner Mistakes

• Using Accuracy alone for imbalanced datasets.
• Confusing Precision with Recall.
• Applying MSE to classification problems.
• Ignoring False Positives and False Negatives.
• Selecting evaluation metrics without considering the application.

---

Quick Revision

---

Memory Trick

---

Exam Tips

• Memorize the formulas for Accuracy, Precision, Recall, F1-Score, MSE, and RMSE.
• Understand the meaning of TP, TN, FP, and FN.
• Practice numerical problems based on confusion matrices.
• Know when to use classification metrics and when to use regression metrics.
• Remember that lower MSE and RMSE values indicate better model performance.

---

Frequently Asked Questions (FAQs)

1. Which metric measures overall correctness?

Accuracy measures the overall percentage of correct predictions made by the model.

2. When is Precision more important than Recall?

Precision is more important when False Positives are costly, such as in spam email detection.

3. When is Recall more important than Precision?

Recall is more important when missing a positive case is dangerous, such as in disease diagnosis.

4. Why is F1-Score used?

F1-Score provides a balanced measure when both Precision and Recall are equally important.

5. Which metrics are used for regression problems?

MSE and RMSE are commonly used to evaluate regression models.

---

Summary

• Evaluation metrics measure the performance of Machine Learning models.
• Accuracy, Precision, Recall, and F1-Score are used for classification models.
• MSE and RMSE are used for regression models.
• Choosing the appropriate metric depends on the type of problem being solved.
• Proper evaluation ensures that AI models are accurate, reliable, and ready for deployment.

---

Next Topic: Practical: Calculate MSE and RMSE using Microsoft Excel