Databricks Cost Tuning

Overview

Reduce Databricks spending by optimizing cluster configurations, leveraging spot instances, right-sizing SQL warehouses, and implementing cost governance policies. Databricks charges per DBU (Databricks Unit) with rates varying by workload type: Jobs Compute ($0.15/DBU), All-Purpose Compute ($0.40/DBU), SQL Compute ($0.22/DBU), and Serverless ($0.07/DBU).

Prerequisites

• Databricks Premium or Enterprise workspace

• Access to system.billing.usage tables

• Workspace admin for cluster policy creation

Instructions

Step 1: Identify Top Cost Drivers

-- Top 10 most expensive clusters this month SELECT cluster_id, cluster_name, sku_name, SUM(usage_quantity) AS total_dbus, ROUND(SUM(usage_quantity * list_price), 2) AS cost_usd FROM system.billing.usage WHERE usage_date >= date_trunc('month', current_date()) GROUP BY cluster_id, cluster_name, sku_name ORDER BY cost_usd DESC LIMIT 10;

Step 2: Enforce Cluster Policies

{ "name": "cost-optimized-interactive", "definition": { "autotermination_minutes": { "type": "range", "minValue": 10, "maxValue": 60, "defaultValue": 20 }, "num_workers": { "type": "range", "minValue": 1, "maxValue": 8 }, "node_type_id": { "type": "allowlist", "values": ["m5.xlarge", "m5.2xlarge"] }, "aws_attributes.availability": { "type": "fixed", "value": "SPOT_WITH_FALLBACK" }, "spark_conf.spark.databricks.cluster.profile": { "type": "fixed", "value": "singleNode" } } }

Step 3: Use Spot Instances for Jobs

jobs/cost_optimized_job.py

job_config = { "name": "nightly-etl", "new_cluster": { "num_workers": 4, "node_type_id": "m5.2xlarge", "aws_attributes": { "availability": "SPOT_WITH_FALLBACK", "first_on_demand": 1, # Driver on-demand, workers spot "spot_bid_price_percent": 100, }, "autotermination_minutes": 0, # Jobs clusters auto-terminate on completion }, }

Spot instances save 60-90% on worker node costs

Step 4: Right-Size SQL Warehouses

-- Check warehouse utilization (queries/hour and queue time) SELECT warehouse_id, warehouse_name, COUNT(*) AS query_count, AVG(total_duration_ms) AS avg_duration_ms, MAX(queue_duration_ms) AS max_queue_ms FROM system.query.history WHERE start_time > current_timestamp() - INTERVAL 7 DAYS GROUP BY warehouse_id, warehouse_name; -- If max_queue_ms is near 0, warehouse is oversized. Reduce cluster size. -- If max_queue_ms > 30000, warehouse needs more capacity or auto-scaling. # 30000: 30 seconds in ms

Step 5: Schedule Auto-Stop for Development Clusters

Set aggressive auto-termination for all dev clusters

databricks clusters list --output JSON |
jq -r '.[] | select(.cluster_name | test("dev|sandbox|test")) | .cluster_id' |
xargs -I{} databricks clusters edit {} --json '{"autotermination_minutes": 15}'

Error Handling

Issue Cause Solution

Spot instance interruption Cloud provider reclaiming capacity Use SPOT_WITH_FALLBACK, checkpoint long jobs

Cluster policy too restrictive Workers can't handle workload Increase max_workers or allow larger instance types

SQL warehouse idle but running No auto-stop configured Enable auto-stop with 10-minute timeout

Billing data not available System tables not enabled Enable system table access in account settings

Examples

Basic usage: Apply databricks cost tuning to a standard project setup with default configuration options.

Advanced scenario: Customize databricks cost tuning for production environments with multiple constraints and team-specific requirements.

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale

Resources

• Official monitoring documentation

• Community best practices and patterns

• Related skills in this plugin pack