Imbalanced Classes

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 24 of 58.

When 95 of 100 patients are healthy, a model that always says healthy scores 95% accuracy and catches no one. On imbalanced classes, accuracy flatters:

Accuracy and recall bars for three models on data with 5% sick patients. Always healthy: accuracy 0.95, recall 0.00. Plain model: 0.98 and 0.62. Balanced model: 0.94 and 0.92

Two fixes. stratify=y keeps the 95 to 5 share in both sets, so the test set has sick patients to catch. class_weight="balanced" makes each rare row count more:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y)
model = LogisticRegression(max_iter=1000, class_weight="balanced")

The balanced model trades a little accuracy for many more caught cases. On rare cases, read recall and precision, not accuracy.

Challenge

Medium

About 5% of a factory's bearings fail early. The starter splits a test bench's records and scores a plain model, but its test set holds only 2 faulty bearings, so its 0.993 means little.

Fix the split so both sets keep the same share of faulty bearings.
Train a second model that weighs faulty rows up, and print its line as below.

For another bench:

Faulty bearings: train 8.1%, test 7.8%
Plain: accuracy 0.947, recall 0.417
Balanced: accuracy 0.900, recall 0.833

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, recall_score

# One row per bearing from the test bench: 1 in faulty means it failed soon after
parts = pd.read_csv("bearings.csv")
features = ["vibration_mm_s", "temp_c", "noise_db", "hours_k"]
X = parts[features]
y = parts["faulty"]

# Fix this split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
print(f"Faulty bearings: train {y_train.mean() * 100:.1f}%, test {y_test.mean() * 100:.1f}%")

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

plain = LogisticRegression(max_iter=1000)
plain.fit(X_train, y_train)
pred = plain.predict(X_test)
print(f"Plain: accuracy {accuracy_score(y_test, pred):.3f}, recall {recall_score(y_test, pred):.3f}")

# Write your code below