ROC Curves

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 25 of 58.

Every threshold gives its own recall and its own false alarms. The ROC curve shows them all at once:

A blue stepped curve climbing from the bottom left corner to the top right, far above a dashed gray coin toss diagonal. Orange dots mark thresholds 0.8, 0.5 and 0.2, moving up and right. The area under it is shaded: AUC = 0.914

Across: the false positive rate, the share of healthy patients flagged. Up: recall. Lower thresholds climb toward (1, 1).

from sklearn.metrics import roc_curve, roc_auc_score

proba = model.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, proba)  # tpr is recall
ax.plot(fpr, tpr)
print(f"{roc_auc_score(y_test, proba):.3f}")

This prints 0.914, the AUC: the area under the curve. 1 is perfect, 0.5 a coin toss. A higher curve catches more for the same false alarms.

Challenge

Medium

A cherry farm lights frost fires on nights that may freeze. The starter trains two frost models, logistic and knn, and sets up the chart.

Write draw_roc(model, name): it draws the model's ROC curve on ax, labelled name, and prints its AUC as below.
Call it for both models, add a dashed diagonal labelled coin toss and a legend, then show.

For landslide models:

logistic AUC: 0.869
k-NN AUC: 0.924

Then Coddy describes your chart. Theirs looks like this:

ROC curves titled Landslide Warnings: a stepped blue logistic curve and an orange k-NN curve with few corners, both well above a dashed green coin toss diagonal, with a legend

Try it yourself

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import roc_curve, roc_auc_score

# One row per spring night on a cherry farm: 1 in frost means frost formed before dawn
nights = pd.read_csv("frost.csv")
features = ["temp_6pm_c", "humidity_pct", "wind_kmh", "cloud_pct"]
X_train, X_test, y_train, y_test = train_test_split(
    nights[features], nights["frost"], test_size=0.25, random_state=0, stratify=nights["frost"])
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Two frost models
logistic = LogisticRegression(max_iter=1000)
logistic.fit(X_train, y_train)
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)

fig, ax = plt.subplots()
ax.set_title("Frost Warnings")
ax.set_xlabel("False positive rate")
ax.set_ylabel("Recall")

# Write your code below