Pandas Debugging Guide

A systematic approach to debugging Pandas DataFrames and operations using the OILER framework (Orient, Investigate, Locate, Experiment, Reflect).

Common Error Patterns

1. SettingWithCopyWarning

Symptom: Warning message about setting values on a copy of a slice.

Cause: Modifying a view of a DataFrame rather than a copy. Pandas cannot guarantee whether the operation affects the original data.

Solution:

BAD - triggers warning

df_subset = df[df['col'] > 5] df_subset['new_col'] = 10 # Warning!

GOOD - explicit copy

df_subset = df[df['col'] > 5].copy() df_subset['new_col'] = 10 # Safe

GOOD - use .loc for in-place modification

df.loc[df['col'] > 5, 'new_col'] = 10

2. KeyError on Column Access

Symptom: KeyError: 'column_name'

Cause: Column doesn't exist due to typo, incorrect capitalization, or column was never created.

Solution:

Check available columns

print(df.columns.tolist())

Check for whitespace in column names

print([repr(c) for c in df.columns])

Strip whitespace from all column names

df.columns = df.columns.str.strip()

Case-insensitive column access

col_lower = {c.lower(): c for c in df.columns} actual_col = col_lower.get('mycolumn'.lower())

3. Merge/Join Mismatches

Symptom: Unexpected row counts after merge, NaN values, or MergeError .

Cause: Mismatched column names, different dtypes, or unexpected duplicates.

Solution:

Before merging - inspect both DataFrames

print(f"Left shape: {df1.shape}, Right shape: {df2.shape}") print(f"Left key dtype: {df1['key'].dtype}, Right: {df2['key'].dtype}") print(f"Left key unique: {df1['key'].nunique()}, Right: {df2['key'].nunique()}")

Check for duplicates in merge keys

print(f"Left duplicates: {df1['key'].duplicated().sum()}") print(f"Right duplicates: {df2['key'].duplicated().sum()}")

Explicit merge with indicator

result = df1.merge(df2, on='key', how='outer', indicator=True) print(result['_merge'].value_counts())

4. Memory Errors with Large DataFrames

Symptom: MemoryError or system becomes unresponsive.

Cause: DataFrame too large for available RAM.

Solution:

Check current memory usage

print(df.info(memory_usage='deep')) print(df.memory_usage(deep=True).sum() / 1024**2, 'MB')

Optimize dtypes

def optimize_dtypes(df): for col in df.select_dtypes(include=['int64']).columns: df[col] = pd.to_numeric(df[col], downcast='integer') for col in df.select_dtypes(include=['float64']).columns: df[col] = pd.to_numeric(df[col], downcast='float') for col in df.select_dtypes(include=['object']).columns: if df[col].nunique() / len(df) < 0.5: df[col] = df[col].astype('category') return df

Read in chunks

chunks = pd.read_csv('large_file.csv', chunksize=100000) for chunk in chunks: process(chunk)

Use PyArrow backend (Pandas 2.0+)

df = pd.read_csv('file.csv', dtype_backend='pyarrow')

5. dtype Conversion Issues

Symptom: ValueError during type conversion, unexpected NaN values.

Cause: Non-numeric strings in numeric columns, mixed types.

Solution:

Identify problematic values

def find_non_numeric(series): mask = pd.to_numeric(series, errors='coerce').isna() & series.notna() return series[mask].unique()

print(find_non_numeric(df['numeric_col']))

Safe conversion with error handling

df['numeric_col'] = pd.to_numeric(df['numeric_col'], errors='coerce')

Check for mixed types

print(df['col'].apply(type).value_counts())

Convert with explicit handling

df['date_col'] = pd.to_datetime(df['date_col'], errors='coerce', format='%Y-%m-%d')

6. Index Alignment Problems

Symptom: Unexpected NaN values after operations, incorrect calculations.

Cause: Pandas aligns operations by index, misaligned indices cause NaN.

Solution:

Check index alignment

print(f"Index 1: {df1.index[:5].tolist()}") print(f"Index 2: {df2.index[:5].tolist()}")

Reset index for array-like operations

result = df1.reset_index(drop=True) + df2.reset_index(drop=True)

Use .values for numpy-style operations (bypasses alignment)

result = df1['col'].values + df2['col'].values

Check for duplicate indices

print(f"Duplicate indices: {df.index.duplicated().sum()}")

7. TypeError: 'DataFrame' object is not callable

Symptom: TypeError when accessing DataFrame.

Cause: Using parentheses () instead of brackets [] .

Solution:

BAD

df('column_name') # TypeError!

GOOD

df['column_name'] df.loc[0, 'column_name']

8. AttributeError on Column Access

Symptom: AttributeError when using dot notation.

Cause: Column name contains spaces, special characters, or conflicts with DataFrame methods.

Solution:

BAD - fails for special names

df.my column # SyntaxError df.count # Returns method, not column named 'count'

GOOD - always works

df['my column'] df['count']

Debugging Tools

Essential Inspection Commands

Overview of DataFrame

df.info() # Columns, dtypes, non-null counts, memory df.describe() # Statistical summary df.shape # (rows, columns) df.dtypes # Column data types

Sample data

df.head(10) # First 10 rows df.tail(10) # Last 10 rows df.sample(10) # Random 10 rows

Column inspection

df.columns.tolist() # All column names as list df['col'].unique() # Unique values df['col'].value_counts() # Value frequency df['col'].isna().sum() # Missing value count

Memory usage

df.memory_usage(deep=True) # Per-column memory in bytes df.memory_usage(deep=True).sum() / 1024**2 # Total MB

Display Options

Show all columns

pd.set_option('display.max_columns', None) pd.set_option('display.width', None)

Show all rows (use carefully!)

pd.set_option('display.max_rows', 100)

Show full content of columns

pd.set_option('display.max_colwidth', None)

Float precision

pd.set_option('display.precision', 4)

Reset all options

pd.reset_option('all')

Pandas-Log for Chain Debugging

Install: pip install pandas-log

import pandas_log

Wrap operations with logging

with pandas_log.enable(): result = (df .query('col > 5') .groupby('category') .agg({'value': 'sum'}) )

Outputs: rows/columns affected at each step

The Four Phases (OILER Framework)

Phase 1: Orient

Understand the problem before diving in.

What is the error message?

What operation triggered it?

What is the expected vs actual behavior?

Quick state check

print(f"Shape: {df.shape}") print(f"Columns: {df.columns.tolist()}") print(f"Dtypes:\n{df.dtypes}") print(f"Head:\n{df.head(3)}")

Phase 2: Investigate

Gather information systematically.

Check data quality

def investigate_df(df): print("=== DataFrame Investigation ===") print(f"Shape: {df.shape}") print(f"\nMissing values:\n{df.isna().sum()}") print(f"\nDtypes:\n{df.dtypes}") print(f"\nDuplicate rows: {df.duplicated().sum()}") print(f"\nMemory: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")

# Check for mixed types in object columns
for col in df.select_dtypes(include=['object']).columns:
    types = df[col].apply(type).value_counts()
    if len(types) > 1:
        print(f"\nMixed types in '{col}':\n{types}")

investigate_df(df)

Phase 3: Locate

Narrow down the source of the problem.

For chained operations - break them apart

BAD - hard to debug

result = df.query('x > 5').groupby('cat').agg({'val': 'sum'}).reset_index()

GOOD - step by step

step1 = df.query('x > 5') print(f"After filter: {step1.shape}")

step2 = step1.groupby('cat') print(f"Groups: {step2.ngroups}")

step3 = step2.agg({'val': 'sum'}) print(f"After agg: {step3.shape}")

result = step3.reset_index()

Phase 4: Experiment & Reflect

Test fixes and document learnings.

Test fix on small sample first

sample = df.sample(100).copy()

Apply fix

sample['fixed_col'] = sample['col'].apply(fix_function)

Verify

assert sample['fixed_col'].isna().sum() == 0 assert sample['fixed_col'].dtype == expected_dtype

Apply to full DataFrame

df['fixed_col'] = df['col'].apply(fix_function)

Quick Reference Commands

Data Validation

Assert no missing values

assert df.notna().all().all(), f"Missing: {df.isna().sum()[df.isna().sum() > 0]}"

Assert unique index

assert not df.index.duplicated().any(), "Duplicate indices found"

Assert column exists

assert 'col' in df.columns, f"Column 'col' not found. Available: {df.columns.tolist()}"

Assert dtype

assert df['col'].dtype == 'int64', f"Wrong dtype: {df['col'].dtype}"

Common Fixes One-Liners

Remove duplicate rows

df = df.drop_duplicates()

Reset index

df = df.reset_index(drop=True)

Strip whitespace from string columns

df[str_cols] = df[str_cols].apply(lambda x: x.str.strip())

Fill missing values

df['col'] = df['col'].fillna(0) # or 'Unknown', df['col'].mean(), etc.

Convert to datetime

df['date'] = pd.to_datetime(df['date'], errors='coerce')

Rename columns

df = df.rename(columns={'old': 'new'})

Drop columns

df = df.drop(columns=['unwanted1', 'unwanted2'])

Debugging Merge Issues

def debug_merge(left, right, on, how='inner'): """Debug merge operation before executing.""" print(f"Left: {left.shape}, Right: {right.shape}")

# Check key columns
for key in (on if isinstance(on, list) else [on]):
    print(f"\nKey: '{key}'")
    print(f"  Left dtype: {left[key].dtype}, Right dtype: {right[key].dtype}")
    print(f"  Left unique: {left[key].nunique()}, Right unique: {right[key].nunique()}")
    print(f"  Left nulls: {left[key].isna().sum()}, Right nulls: {right[key].isna().sum()}")

    # Check overlap
    left_set = set(left[key].dropna())
    right_set = set(right[key].dropna())
    overlap = len(left_set & right_set)
    print(f"  Overlap: {overlap} ({overlap/len(left_set)*100:.1f}% of left)")

# Execute with indicator
result = left.merge(right, on=on, how=how, indicator=True)
print(f"\nResult: {result.shape}")
print(result['_merge'].value_counts())

return result.drop(columns=['_merge'])

Memory Optimization

def optimize_memory(df, verbose=True): """Reduce DataFrame memory usage.""" start_mem = df.memory_usage(deep=True).sum() / 1024**2

for col in df.columns:
    col_type = df[col].dtype

    if col_type == 'object':
        if df[col].nunique() / len(df) < 0.5:
            df[col] = df[col].astype('category')
    elif str(col_type).startswith('int'):
        df[col] = pd.to_numeric(df[col], downcast='integer')
    elif str(col_type).startswith('float'):
        df[col] = pd.to_numeric(df[col], downcast='float')

end_mem = df.memory_usage(deep=True).sum() / 1024**2

if verbose:
    print(f"Memory: {start_mem:.2f} MB -> {end_mem:.2f} MB ({(1-end_mem/start_mem)*100:.1f}% reduction)")

return df

Resources

• KDnuggets - Guide to Debugging Common Pandas Errors

• Debugging Chained Pandas Operations

• 10 Pandas Debugging Habits

• Pandas-Log for Debugging

• Common Error Messages in Pandas

• 50 Common Pandas Mistakes