Evaluating Classification Models

Model evaluation is the process of measuring how well a trained model makes predictions.

The choice of evaluation metrics depends on the type of problem:

• Classification: Accuracy, Precision, Recall, F1-score
• Regression: R² (Coefficient of Determination), MSE, MAE

When evaluating classification models, these terms are commonly used:

• True Positive (TP): Correctly predicting positive cases (e.g., predicting a woman is pregnant when she actually is)
• True Negative (TN): Correctly predicting negative cases (e.g., predicting a woman is not pregnant when she actually isn’t)
• False Positive (FP): Incorrectly predicting positive cases (e.g., predicting a woman is pregnant when she isn’t)
• False Negative (FN): Incorrectly predicting negative cases (e.g., predicting a woman is not pregnant when she actually is)

Commonly used evaluation metrics for classification models include:

• Accuracy: The ratio of correct predictions to total predictions Formula: (TP + TN) / (TP + TN + FP + FN)
• Precision: The proportion of positive predictions that are actually correct Formula: TP / (TP + FP)
• Recall: The proportion of actual positives that are correctly identified Formula: TP / (TP + FN)
• F1-score: The harmonic mean of precision and recall Formula: 2 * (precision * recall) / (precision + recall)

Scikit-learn provides built-in functions to calculate these metrics easily.

The following example shows how to evaluate a classification model using accuracy:

Copy
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# Load dataset (Iris dataset)
X, y = load_iris(return_X_y=True)

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train the classifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)

# Make predictions
y_pred = knn.predict(X_test)

# Evaluate accuracy
acc = accuracy_score(y_test, y_pred)
print(f"Accuracy: {acc:.2f}")

In this example, the accuracy_score() function is used to compute the accuracy.

Similarly, Scikit-learn provides APIs to calculate other commonly used classification metrics.