K-Fold Validation

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 37 of 58.

This chapter makes your scores trustworthy. By its end, you will tune a model with charts like this:

Bars of mean accuracy over 5 folds for k from 1 to 41. The bars rise from 0.85 at k = 1 to a highlighted best of 0.91 at k = 15, then dip

One test split can be lucky. K-fold cross-validation cuts the rows into folds, here 5, and tests on each fold in turn, training on the rest:

Five rows of five cells. In each split one cell, a different one each time, is the test fold; the other four train. Iris tree scores beside them: 0.967, 0.967, 0.900, 0.967, 1.000

One call runs all five, here for a tree on the iris flowers:

from sklearn.model_selection import cross_val_score

tree = DecisionTreeClassifier(random_state=0)
scores = cross_val_score(tree, X, y, cv=5)
print(np.round(scores, 3))
print(f"{scores.mean():.3f} {scores.std():.3f}")

This prints:

[0.967 0.967 0.9   0.967 1.   ]
0.960 0.033

Report the mean as the score, and the std as how much it wobbles.

Challenge

Medium

A kombucha brewery predicts spoiled batches from 5 lab readings. How steady is its model? The program reads the number of folds; the starter makes the data and the model.

Cross-validate the model with that many folds.
Print each fold's accuracy, then the mean and std, all to 3 decimals.
Print the number of the weakest fold, counting from 1.

For other batches and 4:

Fold 1: 0.917
Fold 2: 0.933
Fold 3: 0.883
Fold 4: 0.917
Mean 0.912, std 0.018
Weakest fold: 3

Try it yourself

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

n_folds = int(input())

# Don't change the lines below: 240 kombucha batches, 5 lab readings each; label 1 means spoiled
X, y = make_classification(n_samples=240, n_features=5, n_informative=3, n_redundant=0,
                           flip_y=0.08, class_sep=0.9, random_state=2)
model = LogisticRegression(max_iter=1000)

# Write your code below