LLM Training

Frameworks and techniques for training and finetuning large language models.

Framework Comparison

Framework Best For Multi-GPU Memory Efficient

Accelerate Simple distributed Yes Basic

DeepSpeed Large models, ZeRO Yes Excellent

PyTorch Lightning Clean training loops Yes Good

Ray Train Scalable, multi-node Yes Good

TRL RLHF, reward modeling Yes Good

Unsloth Fast LoRA finetuning Limited Excellent

Accelerate (HuggingFace)

Minimal wrapper for distributed training. Run accelerate config for interactive setup.

Key concept: Wrap model, optimizer, dataloader with accelerator.prepare() , use accelerator.backward() for loss.

DeepSpeed (Large Models)

Microsoft's optimization library for training massive models.

ZeRO Stages:

• Stage 1: Optimizer states partitioned across GPUs

• Stage 2: + Gradients partitioned

• Stage 3: + Parameters partitioned (for largest models, 100B+)

Key concept: Configure via JSON, higher stages = more memory savings but more communication overhead.

TRL (RLHF/DPO)

HuggingFace library for reinforcement learning from human feedback.

Training types:

• SFT (Supervised Finetuning): Standard instruction tuning

• DPO (Direct Preference Optimization): Simpler than RLHF, uses preference pairs

• PPO: Classic RLHF with reward model

Key concept: DPO is often preferred over PPO - simpler, no reward model needed, just chosen/rejected response pairs.

Unsloth (Fast LoRA)

Optimized LoRA finetuning - 2x faster, 60% less memory.

Key concept: Drop-in replacement for standard LoRA with automatic optimizations. Best for 7B-13B models.

Memory Optimization Techniques

Technique Memory Savings Trade-off

Gradient checkpointing ~30-50% Slower training

Mixed precision (fp16/bf16) ~50% Minor precision loss

4-bit quantization (QLoRA) ~75% Some quality loss

Flash Attention ~20-40% Requires compatible GPU

Gradient accumulation Effective batch↑ No memory cost

Decision Guide

Scenario Recommendation

Simple finetuning Accelerate + PEFT

7B-13B models Unsloth (fastest)

70B+ models DeepSpeed ZeRO-3

RLHF/DPO alignment TRL

Multi-node cluster Ray Train

Clean code structure PyTorch Lightning

Resources

• Accelerate: https://huggingface.co/docs/accelerate

• DeepSpeed: https://www.deepspeed.ai/

• TRL: https://huggingface.co/docs/trl

• Unsloth: https://github.com/unslothai/unsloth