SAM 3 (facebookresearch/sam3)

Overview

Build, integrate, fine-tune, and evaluate Meta SAM 3 with reproducible commands and minimal setup friction.

Quick Routing

User intent Action

Install SAM 3 and run first inference Follow setup in references/setup-and-inference.md

Add SAM 3 to an existing Python app Generate starter code with scripts/create_inference_starter.py and adapt API calls

Verify environment before setup/inference Run scripts/sam3_preflight_check.py

Fine-tune on custom data Use references/training-and-eval.md training flow and config guidance

Run SA-Co benchmarks or eval custom predictions Use eval commands in references/training-and-eval.md and upstream scripts/eval/* docs

Debug runtime failures Run the troubleshooting checklist in references/setup-and-inference.md

Core Workflow

• Confirm objective and modality.

• Set up environment and checkpoint access.

• Run a smoke test.

• Execute the task path: inference, training, or evaluation.

• Return reproducible commands and file paths.

1. Confirm objective and modality

• Identify whether the user needs image inference, video inference, fine-tuning, or benchmark evaluation.

• Confirm whether CUDA is available and which GPU memory budget applies.

• Confirm whether Hugging Face access to facebook/sam3 is already approved.

2. Set up environment and checkpoint access

Use a clean environment:

conda create -n sam3 python=3.12 -y conda activate sam3 pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126 git clone https://github.com/facebookresearch/sam3.git cd sam3 pip install -e .

Authenticate with Hugging Face before first model load:

hf auth login

Optionally run a preflight check before model download:

python scripts/sam3_preflight_check.py --strict

For full setup and verification commands, read references/setup-and-inference.md .

3. Run a smoke test

From this skill folder, generate a starter script:

python scripts/create_inference_starter.py --mode image --output ./sam3_smoke.py

Edit placeholders and run from a SAM3 checkout.

4. Execute the task path

• Image and video inference: Use references/setup-and-inference.md to apply Sam3Processor and build_sam3_video_predictor patterns.

• Fine-tuning: Use references/training-and-eval.md and start from a provided config in sam3/train/configs .

• Benchmark evaluation: Use references/training-and-eval.md plus upstream dataset notes under scripts/eval/gold , scripts/eval/silver , and scripts/eval/veval .

5. Return reproducible output

• Report exact commands run and any config overrides.

• Include checkpoint source and authentication assumptions.

• Include prompt text, frame index, and confidence threshold when reporting inference outputs.

Guardrails

• Do not assume checkpoint access is granted; verify login and permission first.

• Prefer official sam3.model_builder and predictor APIs over custom re-implementations.

• Keep generated scripts editable and avoid machine-specific absolute paths.

• If running on CPU, explicitly note expected performance limits before large jobs.

Resources

• Setup and inference guide: references/setup-and-inference.md

• Training and evaluation guide: references/training-and-eval.md

• Starter generator: scripts/create_inference_starter.py

• Preflight checker: scripts/sam3_preflight_check.py