Identifying Missing Values

Before we can apply any strategy to handle missing data, we first need to reliably detect where it occurs within our dataset. Incomplete data can sabotage model training, leading to errors or biased results. Fortunately, the Pandas library provides efficient tools for locating these missing entries.

Standard Missing Data Representation in Pandas

Pandas primarily uses the special floating-point value NaN (Not a Number) to represent missing data in numerical arrays. For columns with an object dtype (often containing strings), Pandas might use None or NaN. Understanding this standard representation is the first step in identification.

Consider a simple DataFrame:

import pandas as pd
import numpy as np

data = {'col_a': [1, 2, np.nan, 4, 5],
        'col_b': [np.nan, 'x', 'y', 'z', 'x'],
        'col_c': [True, False, True, np.nan, False],
        'col_d': [10.1, 20.2, 30.3, np.nan, 50.5]}
df = pd.DataFrame(data)

print(df)
# Output:
#    col_a col_b  col_c  col_d
# 0    1.0   NaN   True   10.1
# 1    2.0     x  False   20.2
# 2    NaN     y   True   30.3
# 3    4.0     z    NaN    NaN
# 4    5.0     x  False   50.5

Detecting Missing Values with isnull() and isna()

Pandas offers two essentially identical methods, isnull() and isna(), to check for missing values. Both return a boolean DataFrame of the same shape as the original, where True indicates a missing value (NaN or None) and False indicates a present value.

# Check for missing values
missing_mask = df.isnull()
print(missing_mask)
# Output:
#    col_a  col_b  col_c  col_d
# 0  False   True  False  False
# 1  False  False  False  False
# 2   True  False  False  False
# 3  False  False   True   True
# 4  False  False  False  False

# isna() produces the same result
# missing_mask_alt = df.isna()
# print(missing_mask_alt)

Conversely, the notnull() and notna() methods return the opposite boolean mask, identifying where data is present.

# Check for present values
present_mask = df.notnull()
print(present_mask)
# Output:
#    col_a  col_b  col_c  col_d
# 0   True  False   True   True
# 1   True   True   True   True
# 2  False   True   True   True
# 3   True   True  False  False
# 4   True   True   True   True

Summarizing Missing Data Counts

While the boolean mask is informative, we usually need aggregate statistics. Since True evaluates to 1 and False to 0 in numerical contexts, we can chain the .sum() method after isnull() to count missing values per column.

# Count missing values per column
missing_counts = df.isnull().sum()
print(missing_counts)
# Output:
# col_a    1
# col_b    1
# col_c    1
# col_d    1
# dtype: int64

To understand the proportion of missing data, you can divide the counts by the total number of rows (the DataFrame's length):

# Calculate percentage of missing values per column
missing_percentage = (df.isnull().sum() / len(df)) * 100
print(missing_percentage)
# Output:
# col_a    20.0
# col_b    20.0
# col_c    20.0
# col_d    20.0
# dtype: float64

To get the total number of missing values across the entire DataFrame, you can sum the results twice:

# Count total missing values in the DataFrame
total_missing = df.isnull().sum().sum()
print(f"Total missing values: {total_missing}")
# Output:
# Total missing values: 4

You can also count missing values per row by specifying axis=1 in the sum() method: df.isnull().sum(axis=1). This can be useful for identifying rows that are particularly sparse.

Visualizing Missing Data Patterns

Visualizations can often provide a more intuitive understanding of missing data patterns than raw numbers. A common approach is to use a heatmap of the boolean mask generated by isnull(). Libraries like Matplotlib or Seaborn are typically used for this. Another effective visualization is a bar chart showing the count or percentage of missing values per feature.

Here's how you might generate a bar chart summarizing the missing counts using Plotly:

import plotly.express as px

# Assuming 'missing_counts' Series from the previous step
fig = px.bar(x=missing_counts.index, y=missing_counts.values,
             labels={'x':'Feature', 'y':'Number of Missing Values'},
             title="Missing Value Counts per Feature")
fig.update_layout(xaxis_tickangle=-45, title_x=0.5)
# To display the plot (in a notebook or environment supporting Plotly):
# fig.show()

# For inclusion in web content, you might export the JSON representation:
print(fig.to_json(pretty=False))

Bar chart displaying the number of missing entries found in each column of the example DataFrame.

Handling Non-Standard Missing Value Representations

Sometimes, missing data isn't encoded as NaN or None. Datasets might use placeholder strings like "?", "Unknown", "N/A", or unlikely numerical values like 999 or -1 to signify missing entries. These need explicit handling.

When loading data with Pandas (e.g., using pd.read_csv), you can specify these custom markers using the na_values parameter:

# Example: Assume 'data.csv' uses '?' and 'N/A' for missing values
# df = pd.read_csv('data.csv', na_values=['?', 'N/A'])

If the data is already loaded, you can use the .replace() method to convert these placeholders to np.nan, allowing isnull() to detect them correctly:

# Assume df_dirty contains '?' instead of NaN in 'col_b'
df_dirty = pd.DataFrame({
    'col_a': [1, 2, np.nan],
    'col_b': [10, '?', 30]
})

print("Before replacement:")
print(df_dirty)
print("\nMissing counts before:")
print(df_dirty.isnull().sum())

# Replace '?' with NaN
df_clean = df_dirty.replace('?', np.nan)

print("\nAfter replacement:")
print(df_clean)
print("\nMissing counts after:")
print(df_clean.isnull().sum())

# Output:
# Before replacement:
#    col_a col_b
# 0    1.0    10
# 1    2.0     ?
# 2    NaN    30
#
# Missing counts before:
# col_a    1
# col_b    0 # '?' is not recognized as missing yet
# dtype: int64
#
# After replacement:
#    col_a  col_b
# 0    1.0   10.0
# 1    2.0    NaN
# 2    NaN   30.0
#
# Missing counts after:
# col_a    1
# col_b    1 # '?' is now correctly identified as NaN
# dtype: int64

Accurately identifying all missing values, whether standard or non-standard, is the essential first step. Once identified, we can move on to understanding why the data might be missing and selecting appropriate methods to handle it, which are the topics of the subsequent sections in this chapter.