Train Delays

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 61 of 64.

Two commuter lines, Maple and Birch, share a station. For 24 days it logged how late each line's rush-hour train was, in trains.csv:

Box plots of the delays of the two lines: Maple's box sits around 15 minutes with a dot at 91 marked signal failure and a low dot at 4, Birch's lower around 11 with dots at 63 and 74 marked signal failures

Three signal failures held trains an hour or more: real, but big enough to swamp any mean.

The manager asks: is Maple really later, or could the gap be chance? Tame the extremes, then let shuffling decide.

Challenge

Hard

The program reads the fence factor k; 1.5 is the box plot's rule.

Fence all delays at Q1 - k * IQR and Q3 + k * IQR, count the outliers, then cap every delay with np.clip.
Print each line's mean capped delay and the difference, Maple minus Birch.
Shuffle 2000 times: pool Maple's capped delays then Birch's, one rng.permutation per round, the first 24 as fake Maple.
Print the p-value, sign ignored, and the verdict: unlikely to be chance below 0.05, else could be chance.

Round to 2 decimals, p to 3. For k = 2:

Fences: 0.25 to 26.5 (3 outliers)
Maple mean: 15.27
Birch mean: 12.21
Difference: 3.06
p-value: 0.03
Verdict: unlikely to be chance

Does the verdict hold for a stricter or a looser k?

Try it yourself

import numpy as np
import pandas as pd

k = float(input())
df = pd.read_csv("trains.csv")
delays = df["delay_min"]

# Don't change the line below
rng = np.random.default_rng(5)

# Write your code below

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

All lessons in Statistics for ML

3Probability by Simulation

Probability as Frequency The Law of Large Numbers Sums of Dice Expected Value Conditional Probability Recap: Dice Stall Odds

6Relationships

Reading Scatter Plots Correlation Correlation Matrix Correlation Heatmaps Correlation Is Not Cause Recap: Ice Cream and Sun

9Feature Scaling

Why Scale Features Min-Max Scaling Standardization StandardScaler Recap: Climbing Partners

12Final Challenges

Train DelaysAir Quality Wine Chemistry Clinic Survey

Practice on your own: Online Python compiler