Class 10 Artificial Intelligence - Unit 3: Evaluating Models | Train-Test Split, Accuracy, Error & Classification Metrics
Class 10 · Artificial Intelligence
Unit 3: Evaluating Models
Complete Unit Syllabus
-
3.2 Splitting the Training Set Data for Evaluation
- What is Train-Test Split?
- Why do we need to do Train-Test Split?
-
3.3 What is Accuracy and Error?
- Accuracy
- Error
-
3.4 Evaluation Metrics for Classification
- What is Classification?
- Classification Metrics
- Confusion Matrix
- Accuracy from Confusion Matrix
- Precision from Confusion Matrix
- Recall from Confusion Matrix
- F1 Score
-
3.5 Ethical Concerns Around Model Evaluation
- Bias
- Transparency
- Accountability
Introduction to Model Evaluation
Model Evaluation is the process of checking how well an Artificial Intelligence or Machine Learning model performs.
Just as your academic performance is evaluated through marks, grades and ranks to understand your strengths and areas for improvement, an AI model is evaluated using different metrics to understand its performance and improve it.
3.2 Splitting the Training Set Data for Evaluation
What is Train-Test Split?
Train-Test Split is a technique used to check how well a Machine Learning algorithm is working. It involves taking the complete dataset and dividing it into two separate parts:
| Dataset | Purpose |
|---|---|
| Training Dataset | The portion of data used to teach or train the model. |
| Testing Dataset | The portion of data used to test how much the model has actually learned. |
The Train-Test Split technique is most effective when a sufficiently large dataset is available.
Why Do We Need Train-Test Split?
The main purpose of splitting data is to check whether the AI model can perform correctly on new and unseen data.
| Training Data | Testing Data |
|---|---|
| Used for learning. | Used for evaluation. |
| Contains examples the model learns from. | Contains examples the trained model has not seen before. |
| Helps the model learn patterns. | Checks whether the model can generalise those patterns. |
Overfitting
If we evaluate a model using exactly the same data on which it was trained, the model may simply memorise the answers. It may perform extremely well on the training data but perform poorly on new data.
Overfitting occurs when a model learns or memorises the training data too closely instead of learning general patterns that can be applied to new data.
Exam Strategy: Student vs AI Model
Imagine that your teacher gives you 50 solved mathematics problems to study. These are similar to the training data.
During the examination, the teacher gives you 5 completely new problems based on the same concepts. These represent the testing data.
- If you understood the concepts, you will solve the new problems correctly.
- If you only memorised the 50 solved answers, you may fail when faced with new problems.
This is similar to overfitting in Machine Learning.
Language App Example
Suppose you are developing a Student Translator App that identifies school slang.
- Training: The AI is trained using 1,000 sentences from students in Delhi.
- Testing: The application is tested using 200 sentences from students in Mumbai.
If the application is tested using exactly the same Delhi sentences used during training, it may appear highly accurate but may perform poorly when used with new sentences.
Case Study: Sarthak's Face Mask Detector
Sarthak developed a face mask detector. He collected a dataset of 500 images and used all 500 images to train the model.
When he used the same 500 images to evaluate the model, it correctly identified every image. However, when he showed the camera a new person's face, the model could not identify whether the person was wearing a mask.
The most likely reason is that Sarthak did not perform a Train-Test Split.
The model suffered from overfitting. It memorised the training images instead of learning general features of a face mask.
Sarthak could use 400 images for training and keep 100 separate images for testing.
Quick Revision
- Train-Test Split: Dividing a dataset into training and testing portions.
- Training Data: Data used to teach the model.
- Testing Data: Data used to evaluate the model on unseen examples.
- Overfitting: When a model memorises training data and performs poorly on new data.
3.3 What is Accuracy and Error?
Accuracy and Error are two important measures used to understand the performance of an AI model.
A higher score generally indicates better performance, while mistakes indicate areas where improvement is required.
The Story of Bob and Billy
Bob and Billy decided to go to a concert. The entry fee was ₹500.
- Bob brought ₹300.
- Billy brought ₹550.
Billy was more accurate because ₹550 was closer to ₹500 than ₹300 was.
| Person | Money Available | Difference from ₹500 |
|---|---|---|
| Bob | ₹300 | ₹200 |
| Billy | ₹550 | ₹50 |
Accuracy
Accuracy measures how many predictions made by an AI model are correct.
The better the performance of a model, the higher its accuracy generally tends to be.
If a teacher predicts that a student will score 90 marks in Mathematics and the student actually scores 90 marks, the prediction is completely accurate.
Error
Error represents a mistake made by the model. It is the difference between the predicted value and the actual value.
If an AI predicts that a student does not have a fever but the student actually has a fever, the prediction contains an error.
In AI, we generally try to minimise error and maximise accuracy.
Formulas
| Term | Formula | Easy Explanation |
|---|---|---|
| Absolute Error | Actual Value − Predicted Value | The difference between actual and predicted values, considering the magnitude of the difference. |
| Error Rate | Error ÷ Actual Value | Shows the error compared with the actual value. |
| Accuracy | 1 − Error Rate | Represents correctness as a decimal. |
| Accuracy % | Accuracy × 100 | Converts accuracy into percentage form. |
Practical Example: School Bus App
Suppose you build an application that predicts when the school bus will arrive.
- Predicted arrival: 7:30 AM
- Actual arrival: 7:35 AM
- Error: 5 minutes
If the application predicts 7:31 AM next time for an actual arrival of 7:35 AM, the error becomes smaller. Therefore, the prediction has improved.
Practical Example: Smart Attendance System
Suppose an AI-powered camera identifies students entering the school gate.
If there are 50 students and the AI correctly identifies 48 students:
Therefore, 2 students were incorrectly identified.
Case Study: Marks Predictor AI
A Computer Science student developed an AI model to predict final examination marks based on weekly test scores.
For Rahul:
- Predicted Marks: 85
- Actual Marks: 92
Absolute Error:
Error Rate:
Accuracy:
Accuracy Percentage:
3.4 Evaluation Metrics for Classification
What is Classification?
Classification is a Machine Learning task in which data is assigned to a particular category or class.
| Application | Possible Classes |
|---|---|
| School performance prediction | Pass / Fail |
| Email filtering | Spam / Inbox |
| Product classification | Vegetable / Grocery |
Confusion Matrix
A Confusion Matrix is a table used to evaluate the performance of a classification model. It compares the actual values with the values predicted by the AI model.
The Four Outcomes
| Term | Meaning | Example |
|---|---|---|
| True Positive (TP) | Actual value is Yes and the model predicts Yes. | AI predicts that a student has a fever and the student actually has a fever. |
| True Negative (TN) | Actual value is No and the model predicts No. | AI predicts that a student does not have a fever and the student actually does not have one. |
| False Positive (FP) | Actual value is No but the model predicts Yes. | AI identifies a genuine teacher's email as spam. |
| False Negative (FN) | Actual value is Yes but the model predicts No. | AI identifies a sick person as healthy. |
Accuracy from Confusion Matrix
Accuracy is the ratio of all correct predictions to the total number of predictions.
Accuracy may not always be reliable when the dataset is unbalanced.
For example, if there are 900 students who pass and 100 students who fail, a faulty model that predicts Pass for everyone will still show 90% accuracy, even though it completely fails to identify students who fail.
Precision
Precision tells us:
Precision is particularly useful when the cost of a False Positive is high.
Suppose an AI system decides whether weather conditions are suitable for launching a satellite. Missing a good day may be acceptable, but launching on a bad day because the AI incorrectly predicted suitable conditions could be extremely harmful.
Recall
Recall, also called Sensitivity, tells us:
Recall is particularly important when missing a positive case is dangerous.
In disease detection, identifying a sick person as healthy is dangerous because that person may not receive appropriate treatment.
F1 Score
The F1 Score provides a balance between Precision and Recall.
It is useful when the dataset is unbalanced and it is not possible to decide whether False Positives or False Negatives are more important.
Classification Metrics – Quick Comparison
| Metric | Main Focus | Formula |
|---|---|---|
| Accuracy | Overall correct predictions | (TP + TN) ÷ Total |
| Precision | Reducing False Positives | TP ÷ (TP + FP) |
| Recall | Reducing False Negatives | TP ÷ (TP + FN) |
| F1 Score | Balance between Precision and Recall | 2 × P × R ÷ (P + R) |
Case Study: School Pass-Fail Model
A school tested an AI model to predict whether students would Pass or Fail.
Out of 100 students:
- 40 were predicted to pass and actually passed.
- 30 were predicted to fail and actually failed.
- 20 were predicted to pass but actually failed.
- 10 were predicted to fail but actually passed.
Therefore:
- TP = 40
- TN = 30
- FP = 20
- FN = 10
Accuracy:
Total Wrong Predictions:
Which Metric Should We Prefer?
| Situation | Important Metric | Reason |
|---|---|---|
| Fire detection | Recall | Missing a real fire is extremely dangerous. |
| Disease detection | Recall | Missing a sick person can be dangerous. |
| High cost of false alarms | Precision | False positives need to be minimised. |
| Need balance between FP and FN | F1 Score | Balances Precision and Recall. |
Math Practice
A model has:
- TP = 150
- FP = 50
- TN = 750
- FN = 50
Calculate Precision.
Therefore, the Precision = 0.75.
3.5 Ethical Concerns Around Model Evaluation
Model evaluation is not only about mathematical scores and percentages. An AI model can have a high score and still be harmful if the evaluation process is unfair or inappropriate.
Three important ethical concerns are:
- Bias
- Transparency
- Accountability
1. Bias
Bias occurs when the evaluation process or selected metrics unintentionally favour one group over another.
Imagine an AI system that predicts which students will win a science fair. If the model is evaluated only using data from private schools, it may perform poorly for students from government schools.
The evaluation process may therefore introduce bias.
2. Transparency
Transparency means being open and honest about how an AI model was tested and how its performance scores were calculated.
Suppose a teacher uses an AI system to grade student essays. Transparency means the teacher should be able to explain why a student received a particular score instead of simply saying, "The AI gave this score."
Users should be able to understand how important evaluation results were obtained.
3. Accountability
Accountability means taking responsibility for the choices made during AI development and for the consequences of those choices.
If an automated attendance system marks a student absent even though the student was present, the school should not blindly accept the machine's decision.
The responsible people must investigate the error and correct the record.
Ethical Concerns – Quick Comparison
| Concern | Meaning | Simple Example |
|---|---|---|
| Bias | Unfair preference or disadvantage towards a group. | An AI performs well only for users with certain accents. |
| Transparency | Clearly explaining how the model was evaluated. | Explaining how an AI-generated score was calculated. |
| Accountability | Taking responsibility for the AI system and its consequences. | Correcting an incorrect automated attendance record. |
Case Study: Scholarship Selection AI
A school develops an AI system to shortlist students for a scholarship. The system is evaluated based on how quickly students complete an online form, assuming that speed shows confidence.
However, students who do not have computers at home or are not fluent in English take longer to complete the form. As a result, many deserving students are rejected.
Two major ethical concerns are present:
- Bias: The selected evaluation measure is unfairly disadvantageous to students with limited resources or language barriers.
- Accountability: The school is responsible for choosing an evaluation method that creates an unfair disadvantage.
Why Accuracy Alone May Not Be Enough
A high accuracy score does not automatically mean that an AI model is safe, fair or useful.
For example, if a dangerous disease is very rare, a model that always predicts "Healthy" may achieve very high accuracy while failing to identify any sick person.
Therefore, appropriate evaluation metrics such as Recall may be necessary in situations where missing positive cases is dangerous.
Unit 3 – Quick Revision
- Train-Test Split: Divides the dataset into training and testing portions.
- Training Data: Used to teach the AI model.
- Testing Data: Used to evaluate the model using unseen data.
- Overfitting: When a model memorises training data and performs poorly on new data.
- Accuracy: Measures correct predictions.
- Error: Represents the difference between predicted and actual values.
- Classification: Assigning data to predefined categories or classes.
- Confusion Matrix: Shows TP, TN, FP and FN.
- Precision: Focuses on reducing False Positives.
- Recall: Focuses on reducing False Negatives.
- F1 Score: Balances Precision and Recall.
- Bias: Unfair preference or disadvantage during evaluation.
- Transparency: Clearly explaining the evaluation process and results.
- Accountability: Taking responsibility for AI decisions and their consequences.