The Confusion Matrix

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 21 of 58.

Accuracy counts mistakes, not their kind. For a flu test, sending a sick patient home is far worse than a false alarm. A confusion matrix counts each kind:

A 2 by 2 grid for a flu test on 100 patients. Rows are the truth, healthy then has flu; columns are the answer, said healthy then said flu. Blue right answers: 50 true negatives, 42 true positives. Orange mistakes: 3 false positives, 5 false negatives

Flu is the positive class, 1. The mistakes sit off the diagonal: false positives (false alarms) and false negatives (missed cases).

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

cm = confusion_matrix(y_test, pred)  # truth first
print(cm)
print(cm[1, 0])  # row 1: has flu, column 0: said healthy
ConfusionMatrixDisplay.from_predictions(y_test, pred, display_labels=["healthy", "flu"],
                                        ax=ax, colorbar=False)  # no color bar: the counts say it

This prints:

[[50  3]
 [ 5 42]]
5

This chapter is about shrinking that missed-cases cell. It ends with a screening test that misses almost no one.

Challenge

Medium

Lifeguards get a morning alert when a rip current may form. The starter trains the alert model and fills pred for the test mornings (1 means rip).

Make the confusion matrix and print the four counts as below.
Draw it on ax, labelled calm and rip, and show it.

For another model:

Rips caught: 30
Rips missed: 2
False alarms: 9
Calm days cleared: 59

Coddy describes your chart. It should look like this one, from a pipe leak alarm:

A confusion matrix titled Leak Alerts. Rows are the true label, dry then leak; columns the predicted label. Cells: 61 and 4 in the dry row, 7 and 28 in the leak row

Try it yourself

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

# One row per morning at a surf beach: 1 in rip means a rip current formed that day
beach = pd.read_csv("rips.csv")
features = ["wave_height_m", "wave_period_s", "wind_kmh", "tide_change_m"]
X = beach[features]
y = beach["rip"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

# The lifeguards' rip current alert
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
pred = model.predict(X_test)

fig, ax = plt.subplots()
ax.set_title("Rip Current Alerts")

# Write your code below