Turbo Pipeline Lifecycle Reference

CLI commands for managing pipeline lifecycle. Covers streaming and job-mode differences. For interactive pipeline troubleshooting, use /turbo-doctor instead.

Quick Reference

Pipeline States

Streaming vs Job Mode Lifecycle

Streaming and job-mode pipelines behave differently for lifecycle operations:

Job-Mode Pipeline Lifecycle

Job-mode pipelines (job: true in YAML) are one-time batch processes. They:

1. Start — process data from start_at (or earliest) to end_block (or chain tip)
2. Run — process the bounded data range
3. Complete — automatically stop when the range is fully processed
4. Auto-cleanup — ~1 hour after completion, the pipeline is automatically removed

Key rules for job-mode pipelines:

• Cannot pause, resume, or restart — these commands return an error for job pipelines
• Cannot update in place — must delete the old job, then apply a new one
• Redeploying a job — if you need to re-run:

  goldsky turbo delete my-job-pipeline
  goldsky turbo apply my-job-pipeline.yaml
  

• If the job errors, it still auto-deletes ~1 hour after termination — same as successful jobs.

Pause, Resume, and Restart

Pause a Pipeline

Temporarily stop processing without deleting:

goldsky turbo pause <pipeline-name>
# or by YAML:
goldsky turbo pause -f <pipeline.yaml>

This sets deployment replicas to 0, preserving all state for later resumption.

Resume a Pipeline

Restore a paused pipeline to its running state:

goldsky turbo resume <pipeline-name>
# or by YAML:
goldsky turbo resume -f <pipeline.yaml>

You can only resume a paused pipeline. Attempting to resume an already running pipeline returns an error.

Restart a Pipeline

Trigger a pod restart for a running or paused pipeline:

goldsky turbo restart <pipeline-name>

To clear all checkpoints and reprocess from the beginning:

goldsky turbo restart <pipeline-name> --clear-state

Restart vs Resume: Use resume to restore a paused pipeline without restarting pods. Use restart when you need a fresh pod restart (e.g., after configuration changes or to recover from issues). Restart is not supported for Job-mode pipelines — use delete + apply instead.

Important Notes

Deletion is Permanent

• All checkpoints are lost
• Pipeline configuration is removed
• Cannot be undone

Data in Sinks is Preserved

• Data already written to PostgreSQL, ClickHouse, etc. remains
• Only the pipeline itself is deleted
• You may need to manually clean up sink data if desired

Checkpoints and Restarts

• Deleting a pipeline removes its checkpoints
• If you recreate a pipeline with the same name, it starts fresh
• To preserve checkpoints, use goldsky turbo apply to update instead of delete/recreate

Job-Mode Pipelines Cannot Be Updated In Place

• Job pipelines (job: true) must be deleted before redeploying
• Attempting goldsky turbo apply on an existing job returns an error: pipeline already exists
• Jobs auto-cleanup ~1 hour after termination regardless of success or failure
• Always delete the old job first if redeploying: goldsky turbo delete <name>, then goldsky turbo apply <file.yaml>

Project Scope

• goldsky turbo list shows pipelines in your current project only
• Use goldsky project list to see available projects
• Pipelines are isolated per project

Common Patterns

Delete and Recreate

If you need to restart a pipeline from scratch:

1. Delete the existing pipeline:

   goldsky turbo delete my-pipeline
   

2. Deploy fresh:

   goldsky turbo apply my-pipeline.yaml
   

This resets all checkpoints and starts processing from the configured start_at position.

Rename Instead of Delete

To keep checkpoints but change the pipeline name:

1. Update the name field in your YAML

2. Apply the new configuration:

   goldsky turbo apply my-pipeline-v2.yaml
   

3. Delete the old pipeline:

   goldsky turbo delete my-pipeline
   

Note: This creates a new pipeline; checkpoints don't transfer between names.

Clean Up Test Pipelines

For development, use a naming convention like test-* or dev-*:

name: test-usdc-transfers # Easy to identify for cleanup

Then clean up all test pipelines when done.

Troubleshooting

Error: Pipeline Not Found

Error: Pipeline 'wrong-name' not found

Fix: Run goldsky turbo list to see exact pipeline names. Names are case-sensitive.

Error: Permission Denied

Error: Permission denied

Fix: You need Editor or Admin role. Contact a project Owner to upgrade your role.

Related

• /turbo-doctor — Interactive diagnostic skill for pipeline issues
• /turbo-builder — Build and deploy new pipelines