Handling Outliers

Part of the Statistics for ML section of Coddy's Machine Learning journey. Lesson 27 of 64.

Ten flats used about 10 m³ of water each, but a leak pushed one to 64. You can drop it, or cap it at the nearest fence:

The water use of 10 flats in three rows. Keep: nine dots near 10 and the leak at 64, mean 15.2. Drop: the leak removed, mean 9.8. Cap: the leak pulled back to the fence at 14, mean 10.2

np.clip(a, low, high) caps a whole array:

water = np.array([8, 11, 9, 10, 12, 9, 64, 10, 11, 8])
low, high = 6.0, 14.0  # the IQR fences
dropped = water[(water >= low) & (water <= high)]
capped = np.clip(water, low, high)
print(len(dropped), len(capped))
print(capped)

This prints:

9 10
[ 8. 11.  9. 10. 12.  9. 14. 10. 11.  8.]

A model would chase that one flat, so decide before training. Fix or drop typos; keep or cap real extremes. Never delete silently: say how many you changed.

Challenge

Medium

A cycling app logged 15 rides' average speeds in speeds; two look odd. The starter put the IQR fences in low and high.

Build dropped, the speeds within the fences (both included), and capped.
Print one line each for keep, drop and cap: the rides, their mean and their .std(), rounded to 2 decimals.
Print how many rides capping changed.

For another rider's 10 rides:

Keep: 10 rides, mean 21.38, std 6.58
Drop: 9 rides, mean 19.2, std 0.77
Cap: 10 rides, mean 19.45, std 1.04
Capping changed 1 of 10 rides

Try it yourself

import numpy as np


# Returns the lower and upper IQR fences of an array
def iqr_fences(a):
    q1 = np.percentile(a, 25)
    q3 = np.percentile(a, 75)
    iqr = q3 - q1
    return q1 - 1.5 * iqr, q3 + 1.5 * iqr


speeds = np.array([24.5, 22.8, 26.1, 23.9, 3.1, 25.4, 21.7, 24.2, 27.3, 78.4, 23.1, 25.8, 22.4, 24.9, 26.6])
low, high = iqr_fences(speeds)

# Write your code below