Underfit and Overfit

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 32 of 58.

You have met machine learning's biggest trap: k = 1 and deep trees memorized. This chapter names it and ends with charts like this:

Training and validation accuracy of a tree by depth: training climbs toward 1.0, validation peaks at depth 3 and sinks. A dashed line marks the chosen depth, a red star its one test score, 0.82

The trap: 15 basil plants, watered differently, and three curves fit to their harvest:

Leaf harvests of 15 training plants as gray dots and 5 test plants as black squares, against daily water. A straight blue line misses the hump, a green degree 3 curve follows it, a red degree 12 curve swings wildly between the dots

Degree 1 is too simple: underfitting. Degree 12 threads every training dot, swinging between them: overfitting, memorizing, not learning.

np.polyval(coefs, x) gives a curve's heights. Training and test RMSE, in grams:

for degree in [1, 3, 12]:
    coefs = np.polyfit(water_train, grams_train, degree)
    train_err = np.sqrt(mean_squared_error(grams_train, np.polyval(coefs, water_train)))
    test_err = np.sqrt(mean_squared_error(grams_test, np.polyval(coefs, water_test)))
    print(degree, round(train_err), round(test_err))

It prints:

1 156 138
3 15 19
12 9 173

The sign of overfitting: small training error, big test error.

Challenge

Medium

A wind farm models one turbine's power from wind speed. The starter splits its 24 readings and reads max_degree.

For each degree from 1 to max_degree, fit a curve on the training rows; print its train and test RMSE in whole kW.
Print the first degree whose test error is more than twice its train error, or No overfitting up to degree and max_degree.

For another turbine and 7:

Degree 1: train 117 kW, test 172 kW
Degree 2: train 115 kW, test 182 kW
Degree 3: train 64 kW, test 85 kW
Degree 4: train 63 kW, test 79 kW
Degree 5: train 47 kW, test 67 kW
Degree 6: train 47 kW, test 65 kW
Degree 7: train 23 kW, test 85 kW
Overfitting starts at degree 7

Try it yourself

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

max_degree = int(input())

# Don't change the lines below: wind speed (m/s) and power (kW) of one wind turbine
speed = np.array([3.8, 4.3, 5.1, 5.6, 5.6, 6.7, 6.8, 7.2, 7.8, 8.5, 8.9, 9.1,
                  10.2, 10.9, 11.4, 11.9, 12.2, 12.4, 12.6, 13.2, 13.9, 14.4, 15.5, 16.5])
power = np.array([0, 85, 179, 22, 17, 126, 255, 273, 419, 545, 612, 665,
                  913, 1120, 1114, 1227, 1299, 1400, 1293, 1329, 1396, 1493, 1461, 1552])
speed_train, speed_test, power_train, power_test = train_test_split(
    speed, power, test_size=0.25, random_state=0)

# Write your code below