PCA Before a Model

Part of the Unsupervised Learning section of Coddy's Machine Learning journey. Lesson 32 of 51.

Fewer features make a model faster and give it less noise to chase. PCA can sit in a pipeline, right before the model:

for n in [2, 5, 10, 20, 40, 64]:
    model = make_pipeline(PCA(n_components=n), KNeighborsClassifier())
    print(n, f"{cross_val_score(model, X, y, cv=5).mean():.3f}")

On the digits:

A line of k-NN accuracy on the digits by components kept: 0.595 with 2, 0.884 with 5, 0.940 with 10, then nearly flat, 0.958 with 20 up to 0.963 with all 64

Accuracy levels off near 20: the other 44 directions add almost nothing. Each fold fits PCA on its own training rows, so nothing leaks.

Mixed units? Put StandardScaler() first, as always.

Challenge

Medium

An olive oil lab grades oils by the light they absorb at 36 wavelengths, plus 4 lab tests. How few features would do? The program reads a slack.

For each n in sizes, cross-validate (5 folds) a pipeline: scaler, PCA with n components, LogisticRegression(max_iter=1000). Print each score.
Print the fewest components scoring within the slack of the best.

For another lab and 0.05:

2 components: 0.817
3 components: 0.833
4 components: 0.889
5 components: 0.928
10 components: 0.928
20 components: 0.917
40 components: 0.922
Fewest within 0.05 of the best: 4 components

Try it yourself

import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# 180 olive oils: light absorbed at 36 wavelengths, 4 lab tests, and the grade
oils = pd.read_csv("oils.csv")
X = oils.drop(columns=["oil", "grade"])
y = oils["grade"]
sizes = [2, 3, 4, 5, 10, 20, 40]
slack = float(input())

# Write your code below