Comparing Models Fairly

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 38 of 58.

Comparing models on one split is a coin toss: one may land a lucky test set. Give every model the same five folds instead. For 150 banknotes, real or fake:

Five fold scores per model as dots, with a black line at each mean. k-NN dots sit close together around 0.867; logistic is spread from 0.73 to 0.93 around 0.873; tree trails at 0.807

k-NN and logistic tie within their wobble, and k-NN is steadier. The tree trails.

A KFold splitter shuffles the rows once and hands each model identical folds:

from sklearn.model_selection import KFold

folds = KFold(n_splits=5, shuffle=True, random_state=0)
for model in [knn, logistic, tree]:
    scores = cross_val_score(model, X, y, cv=folds)
    print(f"{scores.mean():.3f} {scores.std():.3f}")

It prints the numbers in the chart:

0.867 0.042
0.873 0.074
0.807 0.077

Pick the highest mean; when means are close, the smaller std.

Challenge

Medium

A cooking site predicts which visitors cook a recipe they open. The starter scores three models on one split, and the tree wins. Check it fairly; the program reads the number of folds.

Make a KFold with that many splits, shuffle=True and random_state=0.
Cross-validate each model on those folds; print its mean and std, 3 decimals.
Print the model with the highest mean.

For other visitors and 5:

One split picks: k-NN
k-NN: mean 0.833, std 0.045
logistic: mean 0.860, std 0.044
tree: mean 0.787, std 0.029
Cross-validation picks: logistic

Try it yourself

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score, KFold
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

n_folds = int(input())

# Don't change the lines below: 300 visitors, 6 browsing habits; label 1 means they cooked the recipe
X, y = make_classification(n_samples=300, n_features=6, n_informative=4, n_redundant=0,
                           flip_y=0.06, class_sep=0.8, random_state=0)
names = ["k-NN", "logistic", "tree"]
models = [KNeighborsClassifier(), LogisticRegression(max_iter=1000),
          DecisionTreeClassifier(max_depth=4, random_state=0)]

# One split crowns a winner
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
single = []
for model in models:
    model.fit(X_train, y_train)
    single.append(model.score(X_test, y_test))
print(f"One split picks: {names[np.argmax(single)]}")

# Write your code below