The t-Test

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 45 of 64.

Before computers, statistics had a shortcut: the t-test. A formula turns the means, spreads and group sizes into a p-value at once:

from scipy import stats

result = stats.ttest_ind(sans, serif)
print(np.round(result.pvalue, 3))
print(np.round(result.statistic, 2))

For the font study this prints 0.031 and -2.61. The 1000 shuffles gave 0.039: close, as usual.

The 1000 shuffled differences of the font study as a pale histogram, with the t-test's bell-shaped curve lying over it, and a dashed red line at the real gap of -6.6. The legend gives p = 0.039 for the shuffles and 0.031 for the formula

The t value is the gap in standard errors; its sign only says which group came first. The formula assumes bell-shaped data, so with small skewed groups, trust the shuffles.

It also checks whether one model really beats another.

Challenge

Medium

A paint maker timed three finishes drying. The starter puts each finish's minutes in times, lists three pairs, defines permutation_p(a, b) and reads level.

Print the t-test's p-value for matte vs satin beside the shuffles' one.
Test every pair, first finish first, and print the one with the smallest p-value.
Count the pairs below level.

Round p-values to 3 decimals. Another maker, level 0.05:

matte vs satin: t-test 0.012, simulation 0.023
Clearest difference: satin vs gloss (p = 0.0)
Pairs that differ: 3 of 3

Try it yourself

import numpy as np
import pandas as pd
from scipy import stats


# The permutation test of the last two lessons: returns its p-value for a against b
def permutation_p(a, b, shuffles=1000, seed=0):
    rng = np.random.default_rng(seed)
    pooled = np.concatenate([a, b])
    diffs = []
    for i in range(shuffles):
        shuffled = rng.permutation(pooled)
        diffs.append(shuffled[len(a):].mean() - shuffled[:len(a)].mean())
    real = b.mean() - a.mean()
    return (np.abs(np.array(diffs)) >= np.abs(real)).mean()


paint = pd.read_csv("paint.csv")
times = {
    "matte": paint[paint["finish"] == "matte"]["minutes"],
    "satin": paint[paint["finish"] == "satin"]["minutes"],
    "gloss": paint[paint["finish"] == "gloss"]["minutes"],
}
pairs = [("matte", "satin"), ("matte", "gloss"), ("satin", "gloss")]
level = float(input())

# Write your code below