Two Kinds of Cells

Part of the Unsupervised Learning section of Coddy's Machine Learning journey. Lesson 51 of 51.

A lab measured 569 tissue samples, 30 cell measurements each. Too many to plot, and no eye can sort them:

A grid of 24 samples by 30 measurements, each cell colored from blue for low to red for high, a patchwork with no clear pattern

Each sample's diagnosis, benign or malignant, sits in a sealed envelope. Can the cells sort themselves into two kinds without it?

Shrink the 30 measurements with PCA, cluster what is left, and only then open the envelope to see how close the cells came.

Challenge

Hard

The program reads the share of variance to keep, 0.7 or more; the starter sets up a chart.

Scale, then keep the fewest components reaching that share. Print how many and what they keep.
Cluster those components with k-means: k = 2, n_init=10, random_state=0.
Open the envelope: per cluster, biggest first, print its diagnoses and majority. Then the agreement: the share matching their cluster's majority.
Scatter the first two components per cluster, labelled mostly benign or mostly malignant; legend, show.

For the first 300 samples and 0.8:

Components for 80% of the variance: 5 (they keep 0.844)
Cluster of 175: 146 benign, 29 malignant, so mostly benign
Cluster of 125: 8 benign, 117 malignant, so mostly malignant
Agreement with the diagnosis: 0.877

and their chart:

The first 300 samples on the first two components: a mostly benign cluster on the left and a mostly malignant cluster spreading to the right, with a legend

Then Coddy describes yours.

Try it yourself

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans

# 569 tissue samples, 30 cell measurements each
cancer = load_breast_cancer(as_frame=True)
cells = cancer.data
# The sealed envelope: benign or malignant per sample. Open it only at the end.
diagnosis = pd.Series(cancer.target_names[cancer.target])

# The share of the variance to keep, 0.7 or more
share = float(input())

fig, ax = plt.subplots()
ax.set_title("Cells on Two Components")
ax.set_xlabel("Component 1")
ax.set_ylabel("Component 2")

# Write your code below