Anomaly Scores

Part of the Unsupervised Learning section of Coddy's Machine Learning journey. Lesson 37 of 51.

A yes or no hides how odd a row is. score_samples gives every row a score: the lower, the stranger. For a pharmacy's prescriptions:

forest = IsolationForest(random_state=0).fit(X)
rx["score"] = forest.score_samples(X)
print(rx.sort_values("score").head(3))

This prints the three strangest:

     pills  refills  price     score
17     300        0  120.0 -0.881478
88      28       11   11.5 -0.708524
120     30        1   64.0 -0.700620

A histogram of 150 prescriptions' anomaly scores: most between -0.55 and -0.35, a few trailing left to -0.88, and a red dashed threshold at -0.6 with 4 to check

Where to cut is your choice: how many rows can a person check? Sort to rank, compare with a threshold to flag.

Challenge

Medium

An auditor has time for only a few expense claims. The starter fits an isolation forest; the program reads n and a threshold.

Score every claim into a new column score.
Print the n most unusual claims, strangest first, as below.
Print how many claims score below the threshold.

For another company, 2 and -0.6:

Most unusual claims:
  C003: 1960 EUR, score -0.763
  C040: 80 EUR, score -0.757
Below -0.6: 6 claims

Try it yourself

import pandas as pd
from sklearn.ensemble import IsolationForest

# One row per expense claim: amount, receipts, the hour it was filed and days after the trip
claims = pd.read_csv("claims.csv")
X = claims[["amount_eur", "items", "hour", "days_late"]]
n = int(input())
threshold = float(input())

# From Isolation Forest: the forest, fitted on every claim
forest = IsolationForest(random_state=0).fit(X)

# Write your code below