Tune the Winner

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 53 of 58.

k-NN won with its default of 5 neighbors. Would another k name penguins better? A grid search scores every k you list with cross-validation, like this for tumor scans:

Mean CV accuracy of k-NN on tumor scans for k from 1 to 29: it peaks at k = 3 with 0.974, marked by a star, then drifts down to 0.946 at k = 29

As in Tuning a Pipeline, k is named by its step:

{"kneighborsclassifier__n_neighbors": [3, 5, 7]}

A gain smaller than the folds' std may be luck. And tuning, too, uses only the training rows.

Challenge

Medium

Your k-NN pipeline is pipe, and default holds its CV accuracy with k = 5. The program reads the values of k to try into ks.

Grid search those values with 5 folds on the training rows.
Print the best k and its CV accuracy, to 3 decimals.
Print the gain over k = 5, to 3 decimals.

For another colony and the grid 3, 5, 9:

Best k: 3
CV accuracy: 0.969
Gain over k = 5: 0.009

Then try only big values, like 21 to 27. Does tuning always help?

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.neighbors import KNeighborsClassifier

# The colony from Meet the Penguins, its features and its label
penguins = pd.read_csv("penguins.csv")
numbers = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
text = ["island", "sex"]
X = penguins[numbers + text]
y = penguins["species"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

# From Prepare the Data: the two lanes
number_steps = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
text_steps = make_pipeline(SimpleImputer(strategy="constant", fill_value="unknown"), OneHotEncoder())
prep = ColumnTransformer([("numbers", number_steps, numbers), ("words", text_steps, text)])

# From Choose a Model: the winner, k-NN with its default k = 5
pipe = make_pipeline(prep, KNeighborsClassifier())
default = cross_val_score(pipe, X_train, y_train, cv=5).mean()

# The values of k to try
n = int(input())
ks = []
for i in range(n):
    ks.append(int(input()))

# Write your code below