P-Values

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 44 of 64.

The p-value turns the shuffles into one number: the share of fake gaps at least as large as the real one, either way.

The 1000 shuffled differences of the font study as a histogram. The bars at or beyond 6.6 seconds on both sides are red, 39 shuffles, and a label reads p = 39 / 1000 = 0.039

In code, for the font study:

real = sans.mean() - serif.mean()
rng = np.random.default_rng(4)
diffs = shuffled_diffs(serif, sans, 1000, rng)  # the last lesson's loop
p_value = (np.abs(diffs) >= np.abs(real)).mean()
print(p_value)

This prints 0.039. np.abs counts a gap of -7 as large as 7: the question is whether the fonts differ at all.

Below the significance level, often 0.05, chance is a poor explanation, for a font or a model's gain. Not proof: 4 shuffles in 100 still did it.

Challenge

Medium

A farm waters strawberries mornings or evenings. The starter splits strawberries.csv into morning and evening, lists columns, reads level, defines shuffled_diffs.

For each column, compute the real gap (evening minus morning mean), shuffled gaps (morning first) and p-value.
Print a line per column as below (gap 2 decimals, p 3): differs when p is below level.
Count those that differ.

Another farm, level 0.05:

berries: difference -5.4, p = 0.009, differs
weight_g: difference 0.2, p = 1.0, could be chance
sweetness_brix: difference 0.46, p = 0.022, differs
leaf_spots: difference 0.6, p = 0.397, could be chance
Columns that differ: 2 of 4

Try it yourself

import numpy as np
import pandas as pd


# Returns the shuffled differences of a permutation test (lesson 43's loop): fake b mean minus fake a mean
def shuffled_diffs(a, b, shuffles, rng):
    pooled = np.concatenate([a, b])
    diffs = []
    for i in range(shuffles):
        shuffled = rng.permutation(pooled)
        diffs.append(shuffled[len(a):].mean() - shuffled[:len(a)].mean())
    return np.array(diffs)


beds = pd.read_csv("strawberries.csv")
morning = beds[beds["watering"] == "morning"]
evening = beds[beds["watering"] == "evening"]
columns = ["berries", "weight_g", "sweetness_brix", "leaf_spots"]
level = float(input())
# Don't change the lines below
rng = np.random.default_rng(21)
shuffles = 1000

# Write your code below