Shapes of Data

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 8 of 64.

A histogram's outline is the data's shape, and it tells you which summary to trust:

Three histograms side by side. Left-skewed: a long tail to the left, mean line below the median line. Symmetric: mean and median together in the middle. Right-skewed: a long tail to the right, mean above the median

A long tail pulls the mean toward it, while the median stays with the crowd. Seconds eleven visitors spent on a page:

visits = np.array([8, 12, 15, 15, 18, 20, 22, 25, 40, 75, 140])
print(np.round(visits.mean(), 1), np.median(visits))

This prints:

35.5 20.0

Mean above median: right-skewed, a tail of big values. Mean below: left-skewed. Close together: symmetric.

Skew is a warning before training: a few extreme rows sit far from the rest, and the mean describes almost nobody.

Challenge

Easy

Before training on 150 players' logs, a game team checks the shape of each column of players.csv. For each column, in order, print its name, its mean (1 decimal), its median and a verdict:

right-skewed when the mean is more than 2 above the median,
left-skewed when it is more than 2 below,
symmetric otherwise.

For another game's log:

session_minutes: mean 14.3 above median 9.5, right-skewed
quiz_score: mean 83.3 below median 88.0, left-skewed
ping_ms: mean 46.4 close to median 46.0, symmetric

Try it yourself

import numpy as np
import pandas as pd

players = pd.read_csv("players.csv")

# Write your code below

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

All lessons in Statistics for ML

2Distributions

Density HistogramsShapes of DataThe Normal Curve The 68-95-99.7 Rule Z-Scores Recap: Battery Life

5Outliers

Box Plots The IQR Rule The Z-Score Rule Handling Outliers Recap: Sensor Glitches

8Comparing Groups

Box Plots by Group Difference in Means Could It Be Chance?P-Values The t-Test Recap: A/B Test

11Exploring a Dataset

A First Look Every Feature at Once Features Against the Label Two Best Features Recap: Diabetes Data

3Probability by Simulation

Probability as Frequency The Law of Large Numbers Sums of Dice Expected Value Conditional Probability Recap: Dice Stall Odds

6Relationships

Reading Scatter Plots Correlation Correlation Matrix Correlation Heatmaps Correlation Is Not Cause Recap: Ice Cream and Sun

9Feature Scaling

Why Scale Features Min-Max Scaling Standardization StandardScaler Recap: Climbing Partners

12Final Challenges

Train Delays Air Quality Wine Chemistry Clinic Survey

Practice on your own: Online Python compiler