StandardScaler

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 50 of 64.

A model must scale every new sample exactly as it scaled its training data. scikit-learn's StandardScaler standardizes and remembers how:

A diagram: a window cleaner's 4 past jobs, X_train, go through fit into a scaler that remembers mean_ 8 and 22 and scale_ 4.47 and 9.72. A new job, 8 and 30, goes through transform, using those numbers, and comes out as 0.00 and 0.82

In code, each job is windows cleaned and minutes taken:

from sklearn.preprocessing import StandardScaler

X_train = np.array([[2, 8], [10, 25], [6, 20], [14, 35]])
scaler = StandardScaler()
scaler.fit(X_train)
X_new = np.array([[8, 30]])  # one sample: a table with one row
print(np.round(scaler.transform(X_new), 2))

This prints:

[[0.   0.82]]

fit learns each column's mean and std (dividing by n, like np.std), kept in scaler.mean_ and scaler.scale_. transform applies them.

Never fit on new data: the job would be its own mean and become [[0. 0.]]. On training data, fit_transform does both in one call.

Challenge

Medium

A sorter learned from the avocados in X_train: weight, firmness and days since harvest. The program reads a new avocado into the one-row table new_avocado, but the starter's scaling prints [[0. 0. 0.]] for every avocado.

Fix it: scale the new avocado with the training means and stds.
Count its measurements more than 2 std from the training mean, either way, and print as below.

For another batch:

New avocado: [[2.98 0.   0.09]]
1 of 3 measurements are more than 2 std from the training avocados

Try it yourself

import numpy as np
from sklearn.preprocessing import StandardScaler

# Training avocados, one row each: weight in grams, firmness from 1 to 10, days since harvest
X_train = np.array([[210, 8, 2],
                    [185, 5, 6],
                    [240, 7, 3],
                    [170, 3, 9],
                    [225, 6, 5],
                    [195, 4, 7]])
# A new avocado on the belt
weight = float(input())
firmness = float(input())
days = float(input())
new_avocado = np.array([[weight, firmness, days]])

# This prints [[0. 0. 0.]] for every avocado: fix it
scaler = StandardScaler()
scaled = scaler.fit_transform(new_avocado)
print("New avocado:", np.round(scaled, 2))

# Write your code below

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

All lessons in Statistics for ML

3Probability by Simulation

Probability as Frequency The Law of Large Numbers Sums of Dice Expected Value Conditional Probability Recap: Dice Stall Odds

6Relationships

Reading Scatter Plots Correlation Correlation Matrix Correlation Heatmaps Correlation Is Not Cause Recap: Ice Cream and Sun

9Feature Scaling

Why Scale Features Min-Max Scaling StandardizationStandardScalerRecap: Climbing Partners

12Final Challenges

Train Delays Air Quality Wine Chemistry Clinic Survey

Practice on your own: Online Python compiler