Could It Be Chance?

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 43 of 64.

Five readers of a sans font beat five serif readers by 6.6 seconds. If the font changed nothing, any time could sit in either group. Shuffle to act that out:

Ten reading times as cells, five serif in blue and five sans in orange. An arrow labelled rng.permutation(pooled) leads to the same ten values in a mixed order, split into a fake serif group with mean 48.4 and a fake sans group with mean 49.8: a gap of +1.4 seconds made by chance

rng.permutation(a) returns a shuffled copy. Repeat it 1000 times and keep every fake gap:

serif = np.array([52, 47, 58, 50, 55])
sans = np.array([45, 49, 41, 50, 44])
pooled = np.concatenate([serif, sans])
rng = np.random.default_rng(4)
diffs = []
for i in range(1000):
    shuffled = rng.permutation(pooled)
    diffs.append(shuffled[5:].mean() - shuffled[:5].mean())

A histogram of the 1000 shuffled differences, from -7.8 to 7.8 seconds and piled around 0, with a red line for the real difference, -6.6, near the left edge

Chance gaps pile up around 0; the real one sits near the edge. This is a permutation test.

Before you trust that one model beats another, check that shuffling rarely makes a gap that big.

Challenge

Medium

Dogs learned to sit two ways: clicker and treats hold each dog's sessions. The starter makes rng and shuffles.

Compute the real gap: treats mean minus clicker mean.
Pool clicker then treats; shuffle shuffles times, split at len(clicker), keep each fake gap the same way round; no other draws.
Print the real gap, the first fake gap, the smallest and largest, to 2 decimals.

For other dogs:

Real difference: 3.0
First shuffle: -1.0
Smallest shuffled: -3.33
Largest shuffled: 3.67

Try it yourself

import numpy as np

# Training sessions each dog needed to learn "sit"
clicker = np.array([9, 12, 8, 11, 10, 7, 13])
treats = np.array([14, 11, 15, 12, 16, 13])
# Don't change the lines below
rng = np.random.default_rng(12)
shuffles = 1000

# Write your code below