Vaex

Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

When to Use This Skill

Use Vaex when:

• Processing tabular datasets larger than available RAM (gigabytes to terabytes)

• Performing fast statistical aggregations on massive datasets

• Creating visualizations and heatmaps of large datasets

• Building machine learning pipelines on big data

• Converting between data formats (CSV, HDF5, Arrow, Parquet)

• Needing lazy evaluation and virtual columns to avoid memory overhead

• Working with astronomical data, financial time series, or other large-scale scientific datasets

Core Capabilities

Vaex provides six primary capability areas, each documented in detail in the references directory:

1. DataFrames and Data Loading

Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference references/core_dataframes.md for:

• Opening large files efficiently

• Converting from pandas/NumPy/Arrow

• Working with example datasets

• Understanding DataFrame structure

2. Data Processing and Manipulation

Perform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference references/data_processing.md for:

• Filtering and selections

• Virtual columns and expressions

• Groupby operations and aggregations

• String operations and datetime handling

• Working with missing data

3. Performance and Optimization

Leverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference references/performance.md for:

• Understanding lazy evaluation

• Using delay=True for batching operations

• Materializing columns when needed

• Caching strategies

• Asynchronous operations

4. Data Visualization

Create interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference references/visualization.md for:

• Creating 1D and 2D plots

• Heatmap visualizations

• Working with selections

• Customizing plots and subplots

5. Machine Learning Integration

Build ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference references/machine_learning.md for:

• Feature scaling and encoding

• PCA and dimensionality reduction

• K-means clustering

• Integration with scikit-learn/XGBoost/CatBoost

• Model serialization and deployment

6. I/O Operations

Efficiently read and write data in various formats with optimal performance. Reference references/io_operations.md for:

• File format recommendations

• Export strategies

• Working with Apache Arrow

• CSV handling for large files

• Server and remote data access

Quick Start Pattern

For most Vaex tasks, follow this pattern:

import vaex

1. Open or create DataFrame

df = vaex.open('large_file.hdf5') # or .csv, .arrow, .parquet

OR

df = vaex.from_pandas(pandas_df)

2. Explore the data

print(df) # Shows first/last rows and column info df.describe() # Statistical summary

3. Create virtual columns (no memory overhead)

df['new_column'] = df.x ** 2 + df.y

4. Filter with selections

df_filtered = df[df.age > 25]

5. Compute statistics (fast, lazy evaluation)

mean_val = df.x.mean() stats = df.groupby('category').agg({'value': 'sum'})

6. Visualize

df.plot1d(df.x, limits=[0, 100]) df.plot(df.x, df.y, limits='99.7%')

7. Export if needed

df.export_hdf5('output.hdf5')

Working with References

The reference files contain detailed information about each capability area. Load references into context based on the specific task:

• Basic operations: Start with references/core_dataframes.md and references/data_processing.md

• Performance issues: Check references/performance.md

• Visualization tasks: Use references/visualization.md

• ML pipelines: Reference references/machine_learning.md

• File I/O: Consult references/io_operations.md

Best Practices

• Use HDF5 or Apache Arrow formats for optimal performance with large datasets

• Leverage virtual columns instead of materializing data to save memory

• Batch operations using delay=True when performing multiple calculations

• Export to efficient formats rather than keeping data in CSV

• Use expressions for complex calculations without intermediate storage

• Profile with df.stat() to understand memory usage and optimize operations

Common Patterns

Pattern: Converting Large CSV to HDF5

import vaex

Open large CSV (processes in chunks automatically)

df = vaex.from_csv('large_file.csv')

Export to HDF5 for faster future access

df.export_hdf5('large_file.hdf5')

Future loads are instant

df = vaex.open('large_file.hdf5')

Pattern: Efficient Aggregations

Use delay=True to batch multiple operations

mean_x = df.x.mean(delay=True) std_y = df.y.std(delay=True) sum_z = df.z.sum(delay=True)

Execute all at once

results = vaex.execute([mean_x, std_y, sum_z])

Pattern: Virtual Columns for Feature Engineering

No memory overhead - computed on the fly

df['age_squared'] = df.age ** 2 df['full_name'] = df.first_name + ' ' + df.last_name df['is_adult'] = df.age >= 18

Resources

This skill includes reference documentation in the references/ directory:

• core_dataframes.md

• DataFrame creation, loading, and basic structure

• data_processing.md

• Filtering, expressions, aggregations, and transformations

• performance.md

• Optimization strategies and lazy evaluation

• visualization.md

• Plotting and interactive visualizations

• machine_learning.md

• ML pipelines and model integration

• io_operations.md

• File formats and data import/export