Gradient Boosting

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 43 of 58.

A forest grows its trees side by side. Gradient boosting grows them one after another: each new tree learns what the trees before it still get wrong.

Here it fits a smoothie bar's sales, tree by tree:

Gray dots for 40 days of smoothie sales against temperature, rising from about 20 to 135. A dotted line after 1 tree is a flat rough step, a dashed line after 3 trees follows better, and a solid line after 15 trees runs through the dots

For classes, the model is GradientBoostingClassifier:

from sklearn.ensemble import GradientBoostingClassifier

boost = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, random_state=0)
boost.fit(X_train, y_train)

learning_rate sets how much of each fix to keep. Small steps are careful, so they need more trees.

Challenge

Medium

An orchid grower wants to know which plants will bloom. The starter splits orchids.csv, scores a forest and reads a number of trees.

For each rate in rates, train boosting with that many trees (random_state=0) and print its test accuracy.
Print the best rate. On a tie, keep the earlier one.

For another grower and 20:

Forest: 0.860
Boosting at rate 0.01: 0.600
Boosting at rate 0.1: 0.860
Boosting at rate 1.0: 0.900
Best rate for 20 trees: 1.0

Then try a few tree counts: when does boosting beat the forest?

Try it yourself

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Orchids on a grower's shelves: bloomed this year (1) or not (0)
orchids = pd.read_csv("orchids.csv")
X = orchids.drop(columns="bloomed")
y = orchids["bloomed"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
trees = int(input())
rates = [0.01, 0.1, 1.0]

# Don't change the lines below: the forest to compare with
forest = RandomForestClassifier(n_estimators=100, random_state=0)
forest.fit(X_train, y_train)
print(f"Forest: {forest.score(X_test, y_test):.3f}")

# Write your code below