The Decision Threshold

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 17 of 58.

predict says yes when the chance passes 0.5, the threshold. But a missed faulty screen may cost more than a second look. Slide the threshold down:

Fourteen phone screens on a line of their chance of being faulty: six faulty squares, eight fine dots. At threshold 0.5, 4 of 6 faults are caught with 1 false alarm; at 0.3, all 6 with 4 false alarms

A false alarm is a fine screen flagged. Pick your own threshold with a comparison:

proba = model.predict_proba(X_test)[:, 1]
flag = proba >= 0.3  # True: check this screen

Lower, it catches more real cases and raises more false alarms; higher, the reverse. The chances never change, only the cut.

Choose it by what each mistake costs.

Challenge

Medium

An airline rebooks passengers likely to miss their connection before they even land. The starter fits a model on connections.csv and reads a threshold.

Get the test passengers' chances of a missed connection.
For 0.5, then the threshold, count the passengers rebooked, the real misses caught (of all misses) and the needless rebookings.
Print a line per threshold.

For another airline and 0.25:

Threshold 0.5: 9 rebooked, 7 of 15 misses caught, 2 needless rebookings
Threshold 0.25: 21 rebooked, 13 of 15 misses caught, 8 needless rebookings

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# One row per passenger: layover and inbound delay in minutes, terminal change (1 or 0), checked bags; missed is 1 or 0
trips = pd.read_csv("connections.csv")
features = ["layover_min", "delay_min", "terminal_change", "checked_bags"]
X_train, X_test, y_train, y_test = train_test_split(trips[features], trips["missed"], test_size=0.25, random_state=0)
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
threshold = float(input())

# Write your code below