Multimodal Models

Pre-trained models for vision, audio, and cross-modal tasks.

Model Overview

Model Modality Task

CLIP Image + Text Zero-shot classification, similarity

Whisper Audio → Text Transcription, translation

Stable Diffusion Text → Image Image generation, editing

CLIP (Vision-Language)

Zero-shot image classification without training on specific labels.

CLIP Use Cases

Task How

Zero-shot classification Compare image to text label embeddings

Image search Find images matching text query

Content moderation Classify against safety categories

Image similarity Compare image embeddings

CLIP Models

Model Parameters Trade-off

ViT-B/32 151M Recommended balance

ViT-L/14 428M Best quality, slower

RN50 102M Fastest, lower quality

CLIP Concepts

Concept Description

Dual encoder Separate encoders for image and text

Contrastive learning Trained to match image-text pairs

Normalization Always normalize embeddings before similarity

Descriptive labels Better labels = better zero-shot accuracy

Key concept: CLIP embeds images and text in same space. Classification = find nearest text embedding.

CLIP Limitations

• Not for fine-grained classification

• No spatial understanding (whole image only)

• May reflect training data biases

Whisper (Speech Recognition)

Robust multilingual transcription supporting 99 languages.

Whisper Use Cases

Task Configuration

Transcription Default transcribe task

Translation to English task="translate"

Subtitles Output format SRT/VTT

Word timestamps word_timestamps=True

Whisper Models

Model Size Speed Recommendation

turbo 809M Fast Recommended

large 1550M Slow Maximum quality

small 244M Medium Good balance

base 74M Fast Quick tests

tiny 39M Fastest Prototyping only

Whisper Concepts

Concept Description

Language detection Auto-detects, or specify for speed

Initial prompt Improves technical terms accuracy

Timestamps Segment-level or word-level

faster-whisper 4× faster alternative implementation

Key concept: Specify language when known—auto-detection adds latency.

Whisper Limitations

• May hallucinate on silence/noise

• No speaker diarization (who said what)

• Accuracy degrades on >30 min audio

• Not suitable for real-time captioning

Stable Diffusion (Image Generation)

Text-to-image generation with various control methods.

SD Use Cases

Task Pipeline

Text-to-image DiffusionPipeline

Style transfer Image2Image

Fill regions Inpainting

Guided generation ControlNet

Custom styles LoRA adapters

SD Models

Model Resolution Quality

SDXL 1024×1024 Best

SD 1.5 512×512 Good, faster

SD 2.1 768×768 Middle ground

Key Parameters

Parameter Effect Typical Value

num_inference_steps Quality vs speed 20-50

guidance_scale Prompt adherence 7-12

negative_prompt Avoid artifacts "blurry, low quality"

strength (img2img) How much to change 0.5-0.8

seed Reproducibility Fixed number

Control Methods

Method Input Use Case

ControlNet Edge/depth/pose Structural guidance

LoRA Trained weights Custom styles

Img2Img Source image Style transfer

Inpainting Image + mask Fill regions

Memory Optimization

Technique Effect

CPU offload Reduces VRAM usage

Attention slicing Trades speed for memory

VAE tiling Large image support

xFormers Faster attention

DPM scheduler Fewer steps needed

Key concept: Use SDXL for quality, SD 1.5 for speed. Always use negative prompts.

SD Limitations

• GPU strongly recommended (CPU very slow)

• Large VRAM requirements for SDXL

• May generate anatomical errors

• Prompt engineering matters

Common Patterns

Embedding and Similarity

All three models use embeddings:

• CLIP: Image/text embeddings for similarity

• Whisper: Audio embeddings for transcription

• SD: Text embeddings for image conditioning

GPU Acceleration

Model VRAM Needed

CLIP ViT-B/32 ~2 GB

Whisper turbo ~6 GB

SD 1.5 ~6 GB

SDXL ~10 GB

Best Practices

Practice Why

Use recommended model sizes Best quality/speed balance

Cache embeddings (CLIP) Expensive to recompute

Specify language (Whisper) Faster than auto-detect

Use negative prompts (SD) Avoid common artifacts

Set seeds for reproducibility Consistent results

Resources

• CLIP: https://github.com/openai/CLIP

• Whisper: https://github.com/openai/whisper

• Diffusers: https://huggingface.co/docs/diffusers