Data Cleaning - Missing data

Usually, the CSV files contain missing data, incorrect column names, duplicates, incorrect values, and incorrect data types. The Pandas module has all the tools to handle these problems.

Missing Data

To find out if a value is missing or not:

df.isna()

To know how many missing values each column has:

df.isna().sum()

The isna() function converts all values to true or false (missing or not), then applies. sum() will treat true as 1 and false as 0.

To iterate over the columns and their number of missing values, use the items() method just like in a dictionary (although it is a Pandas series object):

for column, missing_num in df.isna().sum().items():
	print(column, missing_num)

To fill in all those missing values for the whole dataframe, use the fillna() method:

df = df.fillna("some value")

To fill in missing values only for a column, use the fillna() function over a column:

df["some_column"] = df["some_column"].fillna("some value")

There is also an option to remove all rows with missing values:

df = df.dropna()