Learning Curves

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 33 of 58.

Would more data make your model better? A learning curve shows it: test accuracy as the training set grows. Here, a deep tree on two-moons data:

Accuracy against training rows from 25 to 800. The training line sits at 1.0 throughout; the test line climbs from 0.80 to 0.94

The tree memorizes every training set, yet each extra row teaches it something real. Flexible models gain most from data.

Fit on the first n training rows, then score on the same test rows each time:

sizes = [25, 50, 100, 200, 400, 800]
test_scores = []
for n in sizes:
    tree = DecisionTreeClassifier(random_state=0)
    tree.fit(X_train[:n], y_train[:n])
    test_scores.append(tree.score(X_test, y_test))
ax.plot(sizes, test_scores, marker="o")

Still climbing at the right edge? More data will help.

Challenge

Medium

Would more recordings help tell two owl species apart? The starter splits 600 calls and readies both models and the chart.

For each n in sizes, fit both models on the first n training rows; keep their test accuracies.
Print each model's gain, last score minus first, 3 decimals.
Plot both against sizes, marker="o", labelled logistic and tree, with a legend.

Owls from another wood print:

Logistic gains 0.053 from more data
Tree gains 0.160 from more data

and draw this chart. Coddy then describes yours.

Owl Calls: Learning Curves. The logistic line rises from 0.64 to about 0.69 and stays flat; the tree line climbs steadily from 0.67 to 0.83

Try it yourself

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

# Don't change the lines below: 600 owl calls, pitch (Hz) and length (s)
rng = np.random.default_rng(23)
pitch = np.round(rng.uniform(300, 900, 600))
length = np.round(rng.uniform(0.2, 2.0, 600), 2)
# Species 1, the tawny owl, calls longer for its pitch; a few field notes are wrong
longer = length > 0.5 + 6.4 * ((pitch - 600) / 600) ** 2
wrong_note = rng.random(600) < 0.05
species = np.where(longer != wrong_note, 1, 0)
X = np.column_stack([pitch, length])
X_train, X_test, y_train, y_test = train_test_split(X, species, test_size=0.25, random_state=0)

sizes = [25, 50, 100, 200, 400]
logistic = LogisticRegression(max_iter=1000)
tree = DecisionTreeClassifier(random_state=0)

fig, ax = plt.subplots()
ax.set_title("Owl Calls: Learning Curves")
ax.set_xlabel("Training rows")
ax.set_ylabel("Test accuracy")

# Write your code below