The F1 Score

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 23 of 58.

A model can max out precision or recall by giving up the other. The F1 score blends the two, and stays low unless both are high:

Bars for two models. Lopsided: precision 1, recall 0.1, plain average 0.55, F1 only 0.18. Balanced: precision 0.8, recall 0.7, plain average 0.75, F1 0.75

With P for precision and R for recall, F1 is 2 * P * R / (P + R). For a weather model:

from sklearn.metrics import f1_score, classification_report

y_test = ["sun", "sun", "rain", "rain", "snow", "sun", "rain", "snow"]
pred   = ["sun", "rain", "rain", "rain", "snow", "sun", "sun", "rain"]
print(f1_score(y_test, pred, average=None))  # one per class, sorted
print(classification_report(y_test, pred))

This prints:

[0.57142857 0.66666667 0.66666667]
              precision    recall  f1-score   support

        rain       0.50      0.67      0.57         3
        snow       1.00      0.50      0.67         2
         sun       0.67      0.67      0.67         3

    accuracy                           0.62         8
   macro avg       0.72      0.61      0.63         8
weighted avg       0.69      0.62      0.63         8

The report scores every class; support counts its test rows. Its weakest row tells you what to fix first.

Challenge

Medium

A recycling plant's camera sorts items into glass, metal, paper and plastic bins. The program reads k, and the starter trains the camera's k-NN, fills pred and lists the bins.

Print the classification report.
Find the bin with the lowest F1 and print it, then its weaker side, precision or recall, as below (3 decimals).

After the report, for another plant:

Weakest bin: metal, F1 0.667
Its weaker side: recall 0.571

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report, f1_score, precision_score, recall_score

k = int(input())

# One row per item on the belt, and the bin it belongs in
items = pd.read_csv("recycling.csv")
features = ["weight_g", "shine", "see_through", "bend"]
X_train, X_test, y_train, y_test = train_test_split(items[features], items["material"], test_size=0.25, random_state=0)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# The camera's model
model = KNeighborsClassifier(n_neighbors=k)
model.fit(X_train, y_train)
pred = model.predict(X_test)
# The bins, in the order the per-class scores come
bins = model.classes_

# Write your code below