Seedance Video Generation

Overview

This skill enables AI video generation using ByteDance's Seedance 1.5 Pro model through the Replicate API. Generate videos from text prompts, animate still images, and create content with perfectly synchronized audio and lip-syncing.

Core Capabilities

1. Text-to-Video Generation

Generate videos from natural language descriptions.

Example requests:

• "Generate a video of waves crashing on a beach at sunset"

• "Create a video of a cat playing with a ball of yarn"

• "Make a 10-second video of a city skyline timelapse"

Process:

• Extract the scene description from the user's request

• Determine appropriate duration and aspect ratio

• Execute scripts/generate_video.py with the prompt

• Save the generated video to the user's working directory

2. Image-to-Video Animation

Animate still images into dynamic videos.

Example requests:

• "Animate this portrait photo with subtle movements"

• "Turn this landscape image into a video with moving clouds"

• "Make this product image come alive with gentle motion"

Process:

• Ensure the input image is accessible (local file or URL)

• Upload image to Replicate if needed

• Execute scripts/generate_video.py with the image and prompt

• Save the animated video to the user's working directory

3. Audio-Synchronized Video

Generate videos with perfectly synchronized audio and lip movements.

Key features:

• Audio and video generated simultaneously (not sequentially)

• Millisecond-precision lip-sync across multiple languages

• Supported languages: English, Mandarin, Japanese, Korean, Spanish, Portuguese, Indonesian

Example requests:

• "Generate a video of a person saying 'Hello, welcome to my channel'"

• "Create a talking head video explaining quantum physics"

• "Make a video with a character singing a song"

4. Camera Controls

Apply cinematic camera movements to generated videos.

Available movements:

• Pan - Horizontal camera sweep

• Tilt - Vertical camera movement

• Zoom - Move closer or farther from subject

• Tracking - Follow a moving subject

• Orbital - Circle around the subject

Example requests:

• "Generate a video with a slow zoom into the subject"

• "Create a video with the camera panning across a landscape"

• "Make a video with orbital camera movement around the character"

To fix the camera in place, specify "fixed camera" or use the camera_fixed parameter.

Parameters

Parameter Type Default Description

prompt

string required Scene description

duration

int 5 Video length in seconds (2-12)

aspect_ratio

string 16:9 Output dimensions

image

file/URL none Input image for image-to-video

last_frame_image

file/URL none Target end frame

camera_fixed

bool false Lock camera position

generate_audio

bool true Include synchronized audio

seed

int random For reproducible results

Aspect Ratios

• 16:9

• Widescreen (YouTube, presentations)

• 9:16

• Vertical (TikTok, Instagram Reels, Stories)

• 1:1

• Square (Instagram posts)

• 4:3

• Standard (traditional video)

• 3:4

• Portrait

• 21:9

• Ultra-wide (cinematic)

• 9:21

• Ultra-tall

Using the Generation Script

The scripts/generate_video.py script handles all Replicate API interactions.

Basic text-to-video:

python scripts/generate_video.py "a sunset over mountains" --output sunset.mp4

With duration and aspect ratio:

python scripts/generate_video.py "a cat playing"
--duration 8
--aspect-ratio 9:16
--output cat_playing.mp4

Image-to-video:

python scripts/generate_video.py "gentle wind blowing through hair"
--image portrait.jpg
--output animated_portrait.mp4

Fixed camera:

python scripts/generate_video.py "person talking to camera"
--camera-fixed
--output talking_head.mp4

Without audio:

python scripts/generate_video.py "abstract visuals"
--no-audio
--output silent_video.mp4

Reproducible output:

python scripts/generate_video.py "a forest scene"
--seed 42
--output forest.mp4

Output Format

• Format: MP4 (H.264 video, AAC audio)

• Frame rate: 24 fps (fixed)

• Audio: Synchronized stereo audio (when enabled)

• Resolution: Varies by aspect ratio (typically 1280x720 or similar)

Environment Setup

Ensure the REPLICATE_API_KEY environment variable is set:

export REPLICATE_API_KEY="your-api-key-here"

Get an API key from https://replicate.com/account/api-tokens

Performance Considerations

• Video generation takes 30 seconds to 3 minutes depending on duration and complexity

• Longer videos (10-12 seconds) require more generation time

• Image-to-video may take slightly longer due to image processing

• The script displays progress updates during generation

Best Practices

Detailed prompts: More specific descriptions produce better results

• Good: "A golden retriever running through a sunlit meadow with wildflowers, slow motion"

• Less effective: "A dog running"

Match aspect ratio to platform: Use 9:16 for TikTok/Reels, 16:9 for YouTube

Duration selection: Start with shorter durations (5s) to iterate quickly, then increase

Camera guidance: Include camera movement in prompt or use camera_fixed for stability

Audio considerations: Disable audio (--no-audio ) for videos where you'll add custom music/voiceover

Character consistency: For multi-clip narratives, use similar prompts and seed values

Common Use Cases

Social Media Content

TikTok/Reels vertical video

python scripts/generate_video.py "trendy dance moves in neon-lit studio"
--aspect-ratio 9:16 --duration 8 --output tiktok_content.mp4

Product Showcase

Animate product image

python scripts/generate_video.py "elegant rotation revealing product details"
--image product.jpg --duration 6 --output product_showcase.mp4

Talking Head / Avatar

Generate speaking video

python scripts/generate_video.py "professional presenter explaining a concept, direct eye contact"
--camera-fixed --duration 10 --output presenter.mp4

Cinematic B-Roll

Atmospheric footage

python scripts/generate_video.py "aerial view of misty mountains at dawn, slow pan"
--aspect-ratio 21:9 --duration 12 --output broll.mp4

Limitations

• Maximum duration: 12 seconds per generation

• Fixed frame rate: 24 fps (cannot be changed)

• Audio is generated, not from external source

• Character consistency across clips requires careful prompting

• Complex scenes with multiple characters may have reduced quality

Error Handling

The script handles common errors:

• Missing API key: Clear error message with setup instructions

• Invalid image URL/path: Helpful error with troubleshooting steps

• Rate limiting: Automatic retry with backoff

• Generation failures: Detailed error from Replicate API

Troubleshooting

Video generation stuck:

• Check API key is valid and has credits

• Try a shorter duration first

• Simplify the prompt

Poor lip-sync:

• Ensure prompt clearly describes speaking/dialogue

• Use camera_fixed for talking head videos

• Keep subject centered in frame description

Unexpected camera movement:

• Add camera_fixed flag for stable shots

• Or specify desired movement explicitly in prompt

Image-to-video not working:

• Ensure image is accessible (valid URL or local path)

• Check image format (JPG, PNG supported)

• Try with a simpler, clearer image