Customer Churn

Part of the Supervised Learning section of Coddy's Machine Learning journey. Lesson 58 of 58.

A meal-kit service ships weekly boxes. subscribers.csv follows 300 customers and whether they cancelled within two months. Skipped boxes are one warning sign:

Bars of the share who cancelled by boxes skipped in the last 3 months: 15% for 0 skipped, 36% for 1, 60% for 2, 61% for 3 and 86% for 4 or more

The team wants to call customers at risk before they leave. One sign is not enough, so a forest will weigh them all.

Its depth needs tuning, and an ROC curve will show how well the risk scores rank who cancels.

Challenge

Hard

prep holds the lanes; the program reads tree depths into depths.

Split off a stratified test quarter, random_state=0.
Grid search prep plus a forest of 100 trees (random_state=0) over depths, 5 folds, on the training rows. Print the best depth and CV accuracy.
From the test chances of cancelling, print the AUC and the call list: chances of 0.5 or more.
Plot the ROC curve as forest, a dashed coin toss diagonal, a legend; show.

For another service and depths 3, 5:

Best depth: 3 (CV accuracy 0.761)
Test AUC: 0.802
Call list: 21 customers, 15 of them really cancelled

An ROC curve titled Who Will Cancel? with a solid forest curve climbing steeply above a dashed coin toss diagonal, and a legend

Then Coddy describes your curve.

Try it yourself

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_curve, roc_auc_score

subscribers = pd.read_csv("subscribers.csv")
numbers = ["months_subscribed", "boxes_skipped", "complaints", "monthly_price", "delivery_rating"]
text = ["plan", "channel"]
X = subscribers[numbers + text]
y = subscribers["churned"]

# The lanes, as in Prepare the Data
number_steps = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
text_steps = make_pipeline(SimpleImputer(strategy="constant", fill_value="unknown"), OneHotEncoder())
prep = ColumnTransformer([("numbers", number_steps, numbers), ("words", text_steps, text)])

# The tree depths to try
n = int(input())
depths = []
for i in range(n):
    depths.append(int(input()))

fig, ax = plt.subplots()
ax.set_title("Who Will Cancel?")
ax.set_xlabel("False positive rate")
ax.set_ylabel("Recall")

# Write your code below