YData Profiling Data Quality Skill

Master YData Profiling (formerly pandas-profiling) for automated data quality reports with comprehensive variable analysis, missing value patterns, correlation detection, and publication-ready HTML reports.

When to Use This Skill

USE YData Profiling when:

• Data quality assessment - Evaluating dataset health and completeness

• Initial data exploration - Understanding a new dataset quickly

• Missing value analysis - Detecting patterns in missing data

• Variable analysis - Understanding distributions and characteristics

• Data documentation - Creating shareable data quality reports

• Dataset comparison - Comparing training vs test data, or before/after

• Stakeholder reporting - Generating professional HTML reports

• Data validation - Checking data before ML model training

DON'T USE YData Profiling when:

• Real-time analysis - Need streaming data profiling

• Custom visualizations - Specific chart requirements

• Interactive dashboards - Use Streamlit or Dash instead

• Very large datasets - Over 10M rows (use sampling or minimal mode)

• Production pipelines - Need lightweight validation (use Great Expectations)

Prerequisites

Basic installation

pip install ydata-profiling

With all optional dependencies

pip install 'ydata-profiling[all]'

Using uv (recommended)

uv pip install ydata-profiling pandas numpy

Jupyter notebook support

pip install ydata-profiling ipywidgets notebook

Verify installation

python -c "from ydata_profiling import ProfileReport; print('YData Profiling ready!')"

Core Capabilities

1. Basic Profile Report Generation

Simplest Usage:

from ydata_profiling import ProfileReport import pandas as pd

Load data

df = pd.read_csv("data.csv")

Generate profile report

profile = ProfileReport(df, title="Data Quality Report")

Save to HTML file

profile.to_file("report.html")

Display in Jupyter notebook

profile.to_notebook_iframe()

With Configuration:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("sales_data.csv")

Customized profile report

profile = ProfileReport( df, title="Sales Data Quality Report", explorative=True, # Enable all analyses dark_mode=False, orange_mode=False, config_file=None, # Or path to custom config lazy=True # Defer computation )

Access specific sections

print(profile.description_set) # Variable descriptions print(profile.get_description()) # Full description

Save report

profile.to_file("sales_report.html")

From DataFrame with Sample Data:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np from datetime import datetime, timedelta

Create sample dataset

np.random.seed(42) n = 5000

df = pd.DataFrame({ "customer_id": range(1, n + 1), "name": [f"Customer_{i}" for i in range(n)], "age": np.random.normal(40, 15, n).astype(int), "income": np.random.exponential(50000, n), "category": np.random.choice(["A", "B", "C", "D"], n, p=[0.4, 0.3, 0.2, 0.1]), "registration_date": [ datetime(2020, 1, 1) + timedelta(days=int(d)) for d in np.random.uniform(0, 1825, n) ], "is_active": np.random.choice([True, False], n, p=[0.8, 0.2]), "score": np.random.uniform(0, 100, n), "email": [f"customer_{i}@example.com" for i in range(n)] })

Add some missing values

df.loc[np.random.choice(n, 200), "income"] = np.nan df.loc[np.random.choice(n, 150), "age"] = np.nan

Generate report

profile = ProfileReport(df, title="Customer Data Profile") profile.to_file("customer_profile.html")

2. Variable Analysis

Understanding Variable Types:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

Dataset with various variable types

df = pd.DataFrame({ # Numeric variables "integer_col": np.random.randint(1, 100, 1000), "float_col": np.random.randn(1000) * 100,

# Categorical variables
"category_high_card": [f"cat_{i}" for i in np.random.randint(1, 100, 1000)],
"category_low_card": np.random.choice(["A", "B", "C"], 1000),

# Boolean
"boolean_col": np.random.choice([True, False], 1000),

# Date/Time
"date_col": pd.date_range("2020-01-01", periods=1000, freq="H"),

# Text
"text_col": ["Sample text " * np.random.randint(1, 10) for _ in range(1000)],

# URL
"url_col": [f"https://example.com/page/{i}" for i in range(1000)],

# Constant
"constant_col": ["constant"] * 1000,

# Unique
"unique_col": range(1000)

})

profile = ProfileReport( df, title="Variable Types Analysis", explorative=True )

The report will automatically detect:

- Numeric: integer_col, float_col

- Categorical: category_high_card, category_low_card

- Boolean: boolean_col

- DateTime: date_col

- Text: text_col

- URL: url_col

- Constant: constant_col

- Unique: unique_col (potentially ID column)

profile.to_file("variable_types_report.html")

Detailed Variable Statistics:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

df = pd.DataFrame({ "revenue": np.random.exponential(1000, 5000), "quantity": np.random.randint(1, 100, 5000), "discount": np.random.uniform(0, 0.5, 5000), "category": np.random.choice(["Electronics", "Clothing", "Food"], 5000) })

profile = ProfileReport(df, title="Sales Variables Analysis")

Access variable-level statistics programmatically

description = profile.get_description()

Numeric variable statistics

for var_name, var_data in description.variables.items(): print(f"\n{var_name}:") print(f" Type: {var_data['type']}") if "mean" in var_data: print(f" Mean: {var_data['mean']:.2f}") print(f" Std: {var_data['std']:.2f}") print(f" Min: {var_data['min']:.2f}") print(f" Max: {var_data['max']:.2f}")

3. Missing Value Analysis

Detecting Missing Patterns:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

Create dataset with various missing patterns

np.random.seed(42) n = 5000

df = pd.DataFrame({ "complete": np.random.randn(n), # No missing

"random_missing": np.where(
    np.random.random(n) < 0.1,
    np.nan,
    np.random.randn(n)
),

"conditional_missing": np.where(
    np.random.randn(n) > 1.5,
    np.nan,
    np.random.randn(n)
),

"block_missing": np.concatenate([
    np.random.randn(4000),
    np.full(1000, np.nan)
]),

"highly_missing": np.where(
    np.random.random(n) < 0.7,
    np.nan,
    np.random.randn(n)
)

})

Profile with missing value analysis

profile = ProfileReport( df, title="Missing Value Analysis", missing_diagrams={ "bar": True, "matrix": True, "heatmap": True } )

profile.to_file("missing_analysis.html")

Programmatic access to missing info

description = profile.get_description() print("\nMissing Value Summary:") for var_name, var_data in description.variables.items(): missing_count = var_data.get("n_missing", 0) missing_pct = var_data.get("p_missing", 0) * 100 print(f" {var_name}: {missing_count} ({missing_pct:.1f}%)")

Missing Value Configuration:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("data_with_missing.csv")

Detailed missing value analysis

profile = ProfileReport( df, title="Missing Value Deep Dive", missing_diagrams={ "bar": True, # Bar chart of missing values per variable "matrix": True, # Nullity matrix (pattern visualization) "heatmap": True # Nullity correlation heatmap }, # Treat certain values as missing vars={ "num": { "low_categorical_threshold": 0 } } )

profile.to_file("missing_deep_dive.html")

4. Correlation Analysis

Multiple Correlation Methods:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

Create correlated dataset

np.random.seed(42) n = 2000

x1 = np.random.randn(n) x2 = np.random.randn(n)

df = pd.DataFrame({ "x1": x1, "x2": x2, "y_strong": x1 * 2 + np.random.randn(n) * 0.5, # Strong correlation "y_moderate": x1 + np.random.randn(n) * 2, # Moderate correlation "y_weak": x1 * 0.5 + np.random.randn(n) * 3, # Weak correlation "y_negative": -x1 + np.random.randn(n) * 0.5, # Negative correlation "y_nonlinear": x1 ** 2 + np.random.randn(n), # Non-linear relationship "y_independent": np.random.randn(n), # No correlation "category": np.random.choice(["A", "B", "C"], n) # Categorical })

Profile with all correlation methods

profile = ProfileReport( df, title="Correlation Analysis", correlations={ "pearson": {"calculate": True, "warn_high_correlations": 0.9}, "spearman": {"calculate": True, "warn_high_correlations": 0.9}, "kendall": {"calculate": True, "warn_high_correlations": 0.9}, "phi_k": {"calculate": True, "warn_high_correlations": 0.9}, "cramers": {"calculate": True, "warn_high_correlations": 0.9} } )

profile.to_file("correlation_analysis.html")

Correlation Thresholds:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("features.csv")

Custom correlation thresholds

profile = ProfileReport( df, title="Feature Correlation Report", correlations={ "pearson": { "calculate": True, "warn_high_correlations": 0.8, # Warn above this "threshold": 0.3 # Minimum to display }, "spearman": { "calculate": True, "warn_high_correlations": 0.8 } } )

profile.to_file("feature_correlations.html")

5. Large Dataset Handling

Minimal Mode for Speed:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

Large dataset

large_df = pd.DataFrame({ f"col_{i}": np.random.randn(1000000) for i in range(50) }) large_df["category"] = np.random.choice(["A", "B", "C"], 1000000)

print(f"Dataset size: {large_df.shape}")

Minimal mode - fast but less detailed

profile = ProfileReport( large_df, title="Large Dataset Profile", minimal=True # Enables minimal mode )

profile.to_file("large_dataset_minimal.html")

Sampling for Large Datasets:

from ydata_profiling import ProfileReport import pandas as pd import numpy as np

def profile_large_dataset( df: pd.DataFrame, sample_size: int = 100000, title: str = "Sampled Profile" ) -> ProfileReport: """ Profile large dataset using sampling.

Args:
    df: Input DataFrame
    sample_size: Number of rows to sample
    title: Report title

Returns:
    ProfileReport object
"""
if len(df) > sample_size:
    df_sampled = df.sample(n=sample_size, random_state=42)
    print(f"Sampled {sample_size} rows from {len(df)}")
else:
    df_sampled = df
    print(f"Using full dataset: {len(df)} rows")

return ProfileReport(
    df_sampled,
    title=f"{title} (n={len(df_sampled):,})",
    minimal=len(df_sampled) > 50000  # Auto minimal for large samples
)

Usage

large_df = pd.read_parquet("huge_dataset.parquet")

profile = profile_large_dataset(large_df, sample_size=50000)

profile.to_file("sampled_profile.html")

Explorative vs Minimal Configuration:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("data.csv")

EXPLORATIVE MODE - Full analysis (slower)

profile_full = ProfileReport( df, title="Full Explorative Report", explorative=True, # Enable all analyses correlations={ "pearson": {"calculate": True}, "spearman": {"calculate": True}, "kendall": {"calculate": True}, "phi_k": {"calculate": True} }, missing_diagrams={ "bar": True, "matrix": True, "heatmap": True }, interactions={ "continuous": True # Scatter plots for numeric pairs } )

MINIMAL MODE - Quick overview (faster)

profile_minimal = ProfileReport( df, title="Minimal Quick Report", minimal=True, # Disable expensive computations correlations=None, # Skip correlations missing_diagrams={"bar": False, "matrix": False, "heatmap": False}, interactions={"continuous": False} )

6. Comparison Reports

Comparing Two Datasets:

from ydata_profiling import ProfileReport, compare import pandas as pd import numpy as np

Create two related datasets

np.random.seed(42)

Training data

df_train = pd.DataFrame({ "feature_1": np.random.randn(5000), "feature_2": np.random.exponential(100, 5000), "feature_3": np.random.choice(["A", "B", "C"], 5000), "target": np.random.randint(0, 2, 5000) })

Test data (slightly different distribution)

df_test = pd.DataFrame({ "feature_1": np.random.randn(2000) * 1.2, # Different variance "feature_2": np.random.exponential(120, 2000), # Different mean "feature_3": np.random.choice(["A", "B", "C", "D"], 2000), # New category "target": np.random.randint(0, 2, 2000) })

Generate individual profiles

profile_train = ProfileReport(df_train, title="Training Data") profile_test = ProfileReport(df_test, title="Test Data")

Compare profiles

comparison = compare([profile_train, profile_test])

Save comparison report

comparison.to_file("train_test_comparison.html")

Before/After Comparison:

from ydata_profiling import ProfileReport, compare import pandas as pd import numpy as np

Original data

df_before = pd.DataFrame({ "value": np.concatenate([ np.random.randn(900), np.array([100, -50, 200, 150, -100]) # Outliers ]), "category": np.random.choice(["A", "B", "C"], 905), "score": np.random.uniform(0, 100, 905) })

Add missing values

df_before.loc[np.random.choice(905, 50), "value"] = np.nan

Cleaned data (after preprocessing)

df_after = df_before.copy()

Remove outliers

Q1 = df_after["value"].quantile(0.25) Q3 = df_after["value"].quantile(0.75) IQR = Q3 - Q1 df_after = df_after[ (df_after["value"] >= Q1 - 1.5 * IQR) & (df_after["value"] <= Q3 + 1.5 * IQR) ]

Fill missing values

df_after["value"] = df_after["value"].fillna(df_after["value"].median())

Compare before and after

profile_before = ProfileReport(df_before, title="Before Cleaning") profile_after = ProfileReport(df_after, title="After Cleaning")

comparison = compare([profile_before, profile_after]) comparison.to_file("cleaning_comparison.html")

7. HTML Report Customization

Custom Report Configuration:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("data.csv")

Customized report

profile = ProfileReport( df, title="Custom Styled Report", dataset={ "description": "This is a sample dataset for analysis", "creator": "Data Team", "copyright_holder": "Company Inc.", "copyright_year": "2025", "url": "https://company.com/data" }, variables={ "descriptions": { "revenue": "Total revenue in USD", "units": "Number of units sold", "category": "Product category" } }, html={ "style": { "full_width": True }, "navbar_show": True, "minify_html": True }, progress_bar=True )

profile.to_file("custom_report.html")

Report Sections Control:

from ydata_profiling import ProfileReport import pandas as pd

df = pd.read_csv("data.csv")

Control which sections appear

profile = ProfileReport( df, title="Selective Report", samples={ "head": 10, # Show first 10 rows "tail": 10 # Show last 10 rows }, duplicates={ "head": 10 # Show first 10 duplicate rows }, correlations={ "pearson": {"calculate": True}, "spearman": {"calculate": False}, # Skip Spearman "kendall": {"calculate": False}, # Skip Kendall "phi_k": {"calculate": False} # Skip Phi-K }, missing_diagrams={ "bar": True, "matrix": False, # Skip matrix "heatmap": False # Skip heatmap } )

profile.to_file("selective_report.html")

Export Options:

from ydata_profiling import ProfileReport import pandas as pd import json

df = pd.read_csv("data.csv") profile = ProfileReport(df, title="Export Demo")

Export to HTML

profile.to_file("report.html")

Export to JSON

profile.to_file("report.json")

Get JSON as string

json_output = profile.to_json()

Get as dictionary

description_dict = profile.get_description()

Save widgets for notebook

profile.to_widgets()

Complete Examples

Example 1: Data Quality Pipeline

from ydata_profiling import ProfileReport import pandas as pd import numpy as np from datetime import datetime import os import json

def data_quality_pipeline( df: pd.DataFrame, output_dir: str, dataset_name: str = "dataset", target_column: str = None ) -> dict: """ Complete data quality assessment pipeline.

Args:
    df: Input DataFrame
    output_dir: Output directory for reports
    dataset_name: Name for the dataset
    target_column: Target variable (optional)

Returns:
    Dictionary with quality metrics
"""
os.makedirs(output_dir, exist_ok=True)

print(f"Starting data quality assessment for: {dataset_name}")
print(f"Dataset shape: {df.shape}")

# Generate profile report
profile = ProfileReport(
    df,
    title=f"Data Quality Report: {dataset_name}",
    explorative=True,
    correlations={
        "pearson": {"calculate": True, "warn_high_correlations": 0.9},
        "spearman": {"calculate": True},
        "phi_k": {"calculate": True}
    },
    missing_diagrams={
        "bar": True,
        "matrix": True,
        "heatmap": True
    }
)

# Save HTML report
html_path = os.path.join(output_dir, f"{dataset_name}_report.html")
profile.to_file(html_path)

# Extract quality metrics
description = profile.get_description()

# Calculate quality scores
quality_metrics = {
    "dataset_name": dataset_name,
    "timestamp": datetime.now().isoformat(),
    "rows": len(df),
    "columns": len(df.columns),
    "memory_mb": df.memory_usage(deep=True).sum() / 1024**2
}

# Missing value analysis
total_cells = df.shape[0] * df.shape[1]
missing_cells = df.isnull().sum().sum()
quality_metrics["completeness_score"] = (1 - missing_cells / total_cells) * 100

# Variable analysis
variable_summary = []
for var_name, var_data in description.variables.items():
    var_info = {
        "name": var_name,
        "type": str(var_data.get("type", "unknown")),
        "missing_count": var_data.get("n_missing", 0),
        "missing_pct": var_data.get("p_missing", 0) * 100,
        "unique_count": var_data.get("n_distinct", 0)
    }

    if "mean" in var_data:
        var_info["mean"] = var_data["mean"]
        var_info["std"] = var_data["std"]

    variable_summary.append(var_info)

quality_metrics["variables"] = variable_summary

# Duplicate analysis
duplicate_rows = df.duplicated().sum()
quality_metrics["duplicate_rows"] = duplicate_rows
quality_metrics["uniqueness_score"] = (1 - duplicate_rows / len(df)) * 100

# Constant columns
constant_cols = [col for col in df.columns if df[col].nunique() <= 1]
quality_metrics["constant_columns"] = constant_cols

# High cardinality columns
high_card_threshold = 0.9
high_card_cols = [
    col for col in df.columns
    if df[col].nunique() / len(df) > high_card_threshold
]
quality_metrics["high_cardinality_columns"] = high_card_cols

# Overall quality score
quality_score = (
    quality_metrics["completeness_score"] * 0.4 +
    quality_metrics["uniqueness_score"] * 0.3 +
    (100 - len(constant_cols) / len(df.columns) * 100) * 0.3
)
quality_metrics["overall_quality_score"] = quality_score

# Save metrics as JSON
metrics_path = os.path.join(output_dir, f"{dataset_name}_metrics.json")
with open(metrics_path, "w") as f:
    json.dump(quality_metrics, f, indent=2, default=str)

print(f"\nQuality Assessment Complete:")
print(f"  Completeness Score: {quality_metrics['completeness_score']:.1f}%")
print(f"  Uniqueness Score: {quality_metrics['uniqueness_score']:.1f}%")
print(f"  Overall Quality Score: {quality_metrics['overall_quality_score']:.1f}%")
print(f"\nReports saved to: {output_dir}")

return quality_metrics

Usage example

def generate_sample_data(): """Generate sample data for testing.""" np.random.seed(42) n = 10000

df = pd.DataFrame({
    "customer_id": range(1, n + 1),
    "revenue": np.random.exponential(500, n),
    "quantity": np.random.randint(1, 50, n),
    "category": np.random.choice(["Electronics", "Clothing", "Food"], n),
    "region": np.random.choice(["North", "South", "East", "West"], n),
    "date": pd.date_range("2024-01-01", periods=n, freq="H"),
    "score": np.random.uniform(0, 100, n)
})

# Add some issues
df.loc[np.random.choice(n, 500), "revenue"] = np.nan
df.loc[np.random.choice(n, 300), "score"] = np.nan

# Add duplicates
df = pd.concat([df, df.sample(200)], ignore_index=True)

return df

sample_df = generate_sample_data()

metrics = data_quality_pipeline(sample_df, "quality_output", "sales_data")

Example 2: ML Dataset Profiling

from ydata_profiling import ProfileReport, compare import pandas as pd import numpy as np from sklearn.model_selection import train_test_split import os

def ml_dataset_profiling( X: pd.DataFrame, y: pd.Series, output_dir: str, test_size: float = 0.2 ) -> dict: """ Profile ML dataset with train/test comparison.

Args:
    X: Feature DataFrame
    y: Target Series
    output_dir: Output directory
    test_size: Test set proportion

Returns:
    Profiling results
"""
os.makedirs(output_dir, exist_ok=True)

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=test_size, random_state=42
)

# Combine features and target
df_train = X_train.copy()
df_train["target"] = y_train.values

df_test = X_test.copy()
df_test["target"] = y_test.values

print(f"Training set: {df_train.shape}")
print(f"Test set: {df_test.shape}")

# Generate individual profiles
print("\nProfiling training set...")
profile_train = ProfileReport(
    df_train,
    title="Training Set Profile",
    explorative=True
)
profile_train.to_file(os.path.join(output_dir, "train_profile.html"))

print("Profiling test set...")
profile_test = ProfileReport(
    df_test,
    title="Test Set Profile",
    explorative=True
)
profile_test.to_file(os.path.join(output_dir, "test_profile.html"))

# Generate comparison report
print("Generating comparison report...")
comparison = compare([profile_train, profile_test])
comparison.to_file(os.path.join(output_dir, "train_test_comparison.html"))

# Check for data drift
drift_metrics = {}
numeric_cols = X.select_dtypes(include=[np.number]).columns

for col in numeric_cols:
    train_mean = df_train[col].mean()
    test_mean = df_test[col].mean()
    train_std = df_train[col].std()

    # Calculate drift as standardized difference
    drift = abs(train_mean - test_mean) / train_std if train_std > 0 else 0
    drift_metrics[col] = {
        "train_mean": train_mean,
        "test_mean": test_mean,
        "drift_score": drift
    }

# Identify significant drift
significant_drift = [
    col for col, metrics in drift_metrics.items()
    if metrics["drift_score"] > 0.5
]

if significant_drift:
    print(f"\nWarning: Significant drift detected in: {significant_drift}")

return {
    "train_profile": profile_train,
    "test_profile": profile_test,
    "comparison": comparison,
    "drift_metrics": drift_metrics,
    "significant_drift": significant_drift,
    "output_dir": output_dir
}

Generate sample ML dataset

def create_ml_dataset(n_samples=10000): """Create sample ML dataset.""" np.random.seed(42)

X = pd.DataFrame({
    "feature_1": np.random.randn(n_samples),
    "feature_2": np.random.exponential(10, n_samples),
    "feature_3": np.random.uniform(0, 100, n_samples),
    "feature_4": np.random.randn(n_samples) * 5,
    "category": np.random.choice(["A", "B", "C"], n_samples)
})

# Target based on features
y = (X["feature_1"] + X["feature_2"] / 10 > 1).astype(int)
y = pd.Series(y, name="target")

return X, y

X, y = create_ml_dataset(10000)

results = ml_dataset_profiling(X, y, "ml_profiling_output")

Example 3: Time Series Data Profiling

from ydata_profiling import ProfileReport import pandas as pd import numpy as np from datetime import datetime, timedelta import os

def profile_time_series( df: pd.DataFrame, date_column: str, value_columns: list, output_dir: str ) -> dict: """ Profile time series data with temporal analysis.

Args:
    df: Time series DataFrame
    date_column: Date/time column name
    value_columns: List of value columns to analyze
    output_dir: Output directory

Returns:
    Profiling results
"""
os.makedirs(output_dir, exist_ok=True)

# Ensure datetime type
df = df.copy()
df[date_column] = pd.to_datetime(df[date_column])

# Sort by date
df = df.sort_values(date_column)

# Add temporal features
df["year"] = df[date_column].dt.year
df["month"] = df[date_column].dt.month
df["day_of_week"] = df[date_column].dt.dayofweek
df["hour"] = df[date_column].dt.hour
df["is_weekend"] = df["day_of_week"].isin([5, 6])

# Generate main profile
profile = ProfileReport(
    df,
    title="Time Series Data Profile",
    tsmode=True,  # Enable time series mode
    sortby=date_column,
    explorative=True
)

profile.to_file(os.path.join(output_dir, "time_series_profile.html"))

# Analyze temporal patterns
temporal_analysis = {}

for col in value_columns:
    # Monthly statistics
    monthly = df.groupby("month")[col].agg(["mean", "std", "min", "max"])
    temporal_analysis[f"{col}_monthly"] = monthly.to_dict()

    # Day of week statistics
    dow = df.groupby("day_of_week")[col].agg(["mean", "std"])
    temporal_analysis[f"{col}_day_of_week"] = dow.to_dict()

# Profile by time period
# Most recent period
recent_cutoff = df[date_column].max() - timedelta(days=30)
df_recent = df[df[date_column] >= recent_cutoff]

if len(df_recent) > 100:
    profile_recent = ProfileReport(
        df_recent,
        title="Recent 30 Days Profile",
        minimal=True
    )
    profile_recent.to_file(os.path.join(output_dir, "recent_30d_profile.html"))

print(f"Time series profiling complete!")
print(f"Date range: {df[date_column].min()} to {df[date_column].max()}")
print(f"Total records: {len(df):,}")

return {
    "main_profile": profile,
    "temporal_analysis": temporal_analysis,
    "output_dir": output_dir
}

Generate sample time series

def create_time_series_data(): """Create sample time series data.""" np.random.seed(42)

# Generate hourly data for 1 year
dates = pd.date_range("2024-01-01", "2024-12-31", freq="H")
n = len(dates)

# Trend + seasonality + noise
trend = np.linspace(100, 150, n)
daily_seasonality = 20 * np.sin(2 * np.pi * np.arange(n) / 24)
weekly_seasonality = 10 * np.sin(2 * np.pi * np.arange(n) / (24 * 7))
noise = np.random.randn(n) * 5

return pd.DataFrame({
    "timestamp": dates,
    "value": trend + daily_seasonality + weekly_seasonality + noise,
    "volume": np.random.exponential(1000, n),
    "category": np.random.choice(["A", "B", "C"], n)
})

ts_df = create_time_series_data()

results = profile_time_series(

ts_df,

date_column="timestamp",

value_columns=["value", "volume"],

output_dir="time_series_output"

)

Integration Examples

YData Profiling with Streamlit

import streamlit as st from ydata_profiling import ProfileReport import pandas as pd from streamlit_pandas_profiling import st_profile_report

st.set_page_config(page_title="Data Profiler", layout="wide") st.title("Interactive Data Profiler")

uploaded_file = st.file_uploader("Upload CSV", type=["csv"])

if uploaded_file: df = pd.read_csv(uploaded_file)

st.subheader("Data Preview")
st.dataframe(df.head(100))

# Profile options
with st.sidebar:
    st.header("Profile Options")
    minimal = st.checkbox("Minimal Mode", value=False)
    explorative = st.checkbox("Explorative Mode", value=True)

if st.button("Generate Profile"):
    with st.spinner("Generating report..."):
        profile = ProfileReport(
            df,
            title="Data Profile",
            minimal=minimal,
            explorative=explorative
        )

        st_profile_report(profile)

YData Profiling with Polars

from ydata_profiling import ProfileReport import polars as pl import pandas as pd

def profile_polars_df( lf: pl.LazyFrame, title: str = "Polars Data Profile", **kwargs ) -> ProfileReport: """ Profile Polars LazyFrame using YData Profiling.

Args:
    lf: Polars LazyFrame
    title: Report title
    **kwargs: Additional ProfileReport arguments

Returns:
    ProfileReport object
"""
# Collect and convert to pandas
df_polars = lf.collect()
df_pandas = df_polars.to_pandas()

return ProfileReport(df_pandas, title=title, **kwargs)

Usage

lf = pl.scan_parquet("data.parquet")

profile = profile_polars_df(lf, title="Polars Data Profile")

profile.to_file("profile.html")

Best Practices

1. Use Minimal Mode for Large Datasets

GOOD: Minimal mode for large data

profile = ProfileReport(large_df, minimal=True)

AVOID: Full explorative on large data

profile = ProfileReport(large_df, explorative=True) # Slow!

2. Sample for Initial Exploration

GOOD: Sample first, then full profile

sample = df.sample(n=10000, random_state=42) profile = ProfileReport(sample, title="Sample Profile")

If interesting, profile full data

profile_full = ProfileReport(df, minimal=True)

3. Customize for Your Needs

GOOD: Disable unnecessary computations

profile = ProfileReport( df, correlations={"pearson": {"calculate": True}}, # Only Pearson missing_diagrams={"bar": True, "matrix": False, "heatmap": False} )

4. Use Lazy Evaluation

GOOD: Lazy profile, compute when needed

profile = ProfileReport(df, lazy=True)

... do other work ...

profile.to_file("report.html") # Computes here

Troubleshooting

Common Issues

Issue: Memory error with large dataset

Solution 1: Use minimal mode

profile = ProfileReport(df, minimal=True)

Solution 2: Sample data

profile = ProfileReport(df.sample(50000))

Issue: Slow report generation

Solution: Disable expensive computations

profile = ProfileReport( df, correlations=None, interactions={"continuous": False}, missing_diagrams={"matrix": False, "heatmap": False} )

Issue: Report too large

Solution: Limit samples shown

profile = ProfileReport( df, samples={"head": 5, "tail": 5}, duplicates={"head": 5} )

Issue: DateTime not recognized

Solution: Convert explicitly

df["date_col"] = pd.to_datetime(df["date_col"]) profile = ProfileReport(df)

Version History

• 1.0.0 (2026-01-17): Initial release

• Basic profile report generation

• Variable analysis and type detection

• Missing value analysis patterns

• Correlation methods (Pearson, Spearman, Kendall, Phi-K)

• Large dataset handling (minimal mode, sampling)

• Comparison reports for datasets

• HTML report customization

• Time series profiling

• Complete pipeline examples

• Integration with Streamlit and Polars

• Best practices and troubleshooting

Resources

• Official Documentation: https://docs.profiling.ydata.ai/

• GitHub: https://github.com/ydataai/ydata-profiling

• PyPI: https://pypi.org/project/ydata-profiling/

• Migration from pandas-profiling: https://docs.profiling.ydata.ai/latest/migration/

Generate comprehensive data quality reports automatically with YData Profiling!