Precision and Recall

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 22 of 58.

Two questions grade a test's flu answers. Of its flu alarms, how many were right? Of the real flu cases, how many did it catch?

The flu test's confusion matrix drawn twice. Left, titled Precision 42 / 45 = 0.933, the said flu column is lit: 3 false alarms and 42 caught. Right, titled Recall 42 / 47 = 0.894, the flu row is lit: 5 missed and 42 caught

Precision answers the first: 42 of 45 alarms. Recall answers the second: 42 of 47 cases.

from sklearn.metrics import precision_score, recall_score

print(f"{precision_score(y_test, pred):.3f}")
print(f"{recall_score(y_test, pred):.3f}")

:.3f always shows 3 decimals, so 0.9 would print 0.900. This prints:

0.933
0.894

Choose by the costlier mistake. A missed flu case spreads, so this test needs recall. A spam filter that buries real mail needs precision.

Challenge

Medium

A shop's filter hides reviews that look fake. The starter makes two versions from one model, strict_pred and eager_pred, and reads which mistake hurts more: missed (a fake stays up) or hidden (a real review vanishes).

Print each filter's precision and recall, 3 decimals.
Print the filter that makes fewer of that mistake, as below (for missed: Fewer fakes missed: ...).

For another shop and hidden:

Strict: precision 0.900, recall 0.450
Eager: precision 0.640, recall 0.800
Fewer real reviews hidden: strict

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score

# Which mistake hurts more: missed (a fake review stays up) or hidden (a real review is hidden)
worse = input()

# One row per review: 1 in fake means the review was written to order
reviews = pd.read_csv("reviews.csv")
features = ["words", "exclamations", "account_days", "user_reviews", "stars", "photo"]
X_train, X_test, y_train, y_test = train_test_split(reviews[features], reviews["fake"], test_size=0.25, random_state=0)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

# Two filters from the same model: strict hides a review at a 0.7 chance of fake, eager at 0.3
strict_pred = proba >= 0.7
eager_pred = proba >= 0.3

# Write your code below