Missing Values

Part of the Data and Plots section of Coddy's Machine Learning journey. Lesson 45 of 60.

Real data has gaps: a sensor fails, a quiz is missed. pandas marks each empty cell NaN, "not a number":

A table of four students and two quizzes with three empty cells marked NaN: Ben missed quiz1, Cy and Dee missed quiz2. Below, the gaps counted per column: 0, 1 and 2

Most models refuse NaN, so count the gaps first. .isna() marks each one True, and .sum() counts them:

scores = pd.DataFrame({
    "name": ["Ana", "Ben", "Cy", "Dee"],
    "quiz1": [8, np.nan, 7, 9],  # np.nan is a gap
    "quiz2": [6, 5, np.nan, np.nan],
})
print(scores.isna().sum())

This prints:

name     0
quiz1    1
quiz2    2
dtype: int64

Then choose: dropna() drops every row with a gap; fillna(0) writes 0 into each. Both return a new table, so store it.

Statistics skip NaN: the mean of quiz1 is 8.0, from its three known scores.

Challenge

Medium

A fitness tracker missed some readings in tracker.csv. Before training, see how each way of handling gaps moves the average sleep. The starter reads hours, a fill value.

Print the gaps in each column.
Print the mean sleep, rounded to 2 decimals, three ways: skipping gaps, over days with no gap in any column, and with gaps filled with hours.

For another tracker and 8:

day      0
steps    1
sleep    2
water    0
dtype: int64
Skipping gaps: 6.9
Complete days only: 7.0
Gaps filled with 8.0: 7.21

Try it yourself

import pandas as pd

tracker = pd.read_csv("tracker.csv")
hours = float(input())

# Write your code below

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

All lessons in Data and Plots

1Introduction

Hello, Machine Learning What Is a Model?Importing Libraries Your First Plot

4Shaping Data

Reshaping Arrays Joining Arrays Random Numbers Recap: Dice Simulation

7pandas DataFrames

Series DataFrames Reading CSV Files Inspecting Data Selecting Columns Recap: Movie Ratings

10Plotting DataFrames

Plotting a DataFrame Scatter from Columns One Color per Group Histogram of a Column Recap: Cafe Sales

2NumPy Arrays

Creating Arrays Shape and Size Ranges of Numbers Indexing and Slicing Recap: Temperature Grid

5Line and Scatter Plots

Labels and Titles Scatter Plots Several Lines and Legends Colors and Markers Figures and Axes Recap: Weekly Steps

8Filtering and Sorting

Selecting Rows Filtering Rows Sorting Values New Columns Recap: Top Products

11Datasets for ML

Built-in Datasets Features and Labels Exploring by Class Plotting the Classes Recap: Wine Dataset

3Array Math

Math on Whole Arrays Summary Statistics Statistics by Axis Broadcasting Boolean Masks Recap: Exam Scores

6Charts for Data

Bar Charts Histograms Several Charts in One Images as Arrays Recap: Rainfall Report

9Grouping and Cleaning

Counting Values Grouping Data Several SummariesMissing ValuesRecap: Survey Results

12Final Challenges

Sales Report Sensor Readings Digits at a Glance Penguin Explorer

Practice on your own: Online Python compiler