Lesson 09 - Classification Metrics

The following topis are discussed in this notebook:

  • Precision, Recall, and Accurary
In [1]:
import numpy as np
from sklearn.metrics import classification_report, confusion_matrix

Binary Classification

In [2]:
#-----------------------------------------------------------
# Generate Data
#-----------------------------------------------------------

np.random.seed(2)
n = 1000
y_true = np.random.choice([0,1],size=n, p=[0.9, 0.1])

flip1s = np.random.choice([0,1], size=n, p=[0.9, 0.1])
y_pred = np.where( (y_true == 1) & (flip1s == 1), (y_true + 1)%2, y_true )

flip0s = np.random.choice([0,1], size=n, p=[0.8, 0.2])
y_pred = np.where( (y_true == 0) & (flip0s == 1), (y_true + 1)%2, y_pred )

Calculate Accuracy

In [3]:
acc = np.sum(y_pred == y_true) / len(y_true)
print(acc)

0.804

Confusion Matrix

  • TP (True Positives): The number of positive samples that were classified as positive.
  • FP (False Positives): The number of positive samples that were classified as negative.
  • TN (True Negatives): The number of negative samples that were classified as positive.
  • FN (False Negatives): The number of negative samples that were classified as positive.
Pred Neg Pred Pos
Actual Neg TN FP
Actual Pos FN TP
In [4]:
print(confusion_matrix(y_true, y_pred))

[[713 181]
 [ 15  91]]

Precision and Recall

  • Accuracy = $\Large\frac{TP + TN}{TP + TN + FP + FN}$

  • Recall = $\Large\frac{TP}{TP + FN} = \Large\frac{TP}{\textrm{Number of Positive Samples}}$

    • The proportion of positive instances that are correctly classified.
    • This is the proportion of instances correctly detected by the model.
    • Also called True Positive Rate (TPR), Sensitivity, Probability of Detection

  • Precision = $\Large\frac{TP}{TP + FP} = \Large\frac{TP}{\textrm{Number of Positive Predictions}}$

    • The proportion of instances classified as positive, that actually are positive.
    • Precision is the probability that an instance belongs to the positive class if it is actually classified as positive.
    • Also called Positive Predictive Value (PPV).

Each class can be viewed as having its own precision and recall values.

  • Positive Recall = $\Large\frac{TP}{TP + FN} = \Large\frac{TP}{\textrm{Number of Positive Samples}}$

  • Positive Precision = $\Large\frac{TP}{TP + FP} = \Large\frac{TP}{\textrm{Number of Positive Predictions}}$

  • Negative Recall = $\Large\frac{TN}{TN + FP} = \Large\frac{TN}{\textrm{Number of Negative Samples}}$

  • Netgative Precision = $\Large\frac{TN}{TN + FN} = \Large\frac{TN}{\textrm{Number of Negative Predictions}}$

Classification Report

In [5]:
print(classification_report(y_true, y_pred))

              precision    recall  f1-score   support

           0       0.98      0.80      0.88       894
           1       0.33      0.86      0.48       106

   micro avg       0.80      0.80      0.80      1000
   macro avg       0.66      0.83      0.68      1000
weighted avg       0.91      0.80      0.84      1000

Multiclass Classification

In [6]:
#-----------------------------------------------------------
# Generate Data
#-----------------------------------------------------------

np.random.seed(1)
n = 1000

y_true = np.random.choice([0,1,2],size=n, p=[0.4, 0.3, 0.3])

shift = np.random.choice([1,2], size=n)

flip0s = np.random.choice([0,1], size=n, p=[0.8, 0.2])
y_pred = np.where( (y_true == 0) & (flip0s == 1), (y_true + shift)%3, y_true )

flip1s = np.random.choice([0,1], size=n, p=[0.7, 0.3])
y_pred = np.where( (y_true == 1) & (flip1s == 1), (y_true + shift)%3, y_pred )

flip2s = np.random.choice([0,1], size=n, p=[0.94, 0.06])
y_pred = np.where( (y_true == 2) & (flip2s == 1), (y_true + shift)%3, y_pred )

Calculate Accuracy

In [7]:
acc = np.sum(y_pred == y_true) / len(y_true)
print(acc)

0.799

Confusion Matrix

In [8]:
print(confusion_matrix(y_true, y_pred))

[[309  40  40]
 [ 52 212  50]
 [  9  10 278]]

Classification Report

In [9]:
print(classification_report(y_true, y_pred))

              precision    recall  f1-score   support

           0       0.84      0.79      0.81       389
           1       0.81      0.68      0.74       314
           2       0.76      0.94      0.84       297

   micro avg       0.80      0.80      0.80      1000
   macro avg       0.80      0.80      0.80      1000
weighted avg       0.80      0.80      0.80      1000