Imputing in a Pipeline

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 47 of 58.

Real logs have gaps, and most models refuse them. A SimpleImputer fills each gap with its column's mean, median or most common value.

Fill before the split, and the test rows help choose that value: a leak. Seven dive depths and one gap:

Two copies of a column of 8 dive depths, the last two marked as test rows and one cell a gap. Left, fill then split: the gap gets 21.3, the mean of all 7 known depths, test rows counted, a leak. Right, split then pipeline: the gap gets 18.0, the mean of the training rows only

In a pipeline, fit learns the fills from training rows only:

from sklearn.impute import SimpleImputer

pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), model)
pipe.fit(X_train, y_train)
print(pipe[0].statistics_)  # the first step's fills

For a dive log (depth, minutes, water temperature), this prints:

[19. 44. 21.]

With words too, the numbers' steps become one lane of the ColumnTransformer:

lane = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
("numbers", lane, ["depth_m", "minutes"])

Challenge

Medium

A forest service predicts which lightning strikes start fires, but some readings are missing. The starter splits wildfire.csv and reads a strategy, mean or median.

Fit a pipeline of an imputer with that strategy, a scaler and logistic regression. Print its test accuracy.
For each column, print the pipeline's fill and what all rows of X would give, 1 decimal.

Another forest, with median:

Test accuracy: 0.840
humidity_pct: fill 46.0, all rows would give 46.0
wind_kmh: fill 29.0, all rows would give 32.0
temp_c: fill 25.0, all rows would give 26.0
days_without_rain: fill 23.0, all rows would give 23.0

Wherever the two differ, filling before the split would have let the test rows shape the training data.

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

# Lightning strikes in a dry forest: did the strike start a fire (1) or not (0)?
# Some weather readings are missing.
strikes = pd.read_csv("wildfire.csv")
columns = ["humidity_pct", "wind_kmh", "temp_c", "days_without_rain"]
X = strikes[columns]
y = strikes["fire"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
strategy = input()

# Write your code below