Receiving: Signature Verification

• Always verify HMAC signature—payload can be forged; don't trust without signature
• Common pattern: HMAC-SHA256(secret, raw_body) compared to header value
• Use raw body bytes—parsed JSON may reorder keys, breaking signature
• Timing-safe comparison—prevent timing attacks on signature check
• Reject missing or invalid signature with 401—log for investigation

Receiving: Replay Prevention

• Check timestamp in payload or header—reject if too old (>5 minutes)
• Combine with signature—timestamp without signature can be forged
• Store processed event IDs—reject duplicates even within time window
• Clock skew tolerance: allow 1-2 minutes past—but not hours

Receiving: Idempotency (Critical)

• Webhooks can arrive multiple times—sender retries on timeout, network issues
• Use event ID for deduplication—store processed IDs in database/Redis
• Make handlers idempotent—same event twice should have same effect
• Idempotency window: keep IDs for 24-72h—balance storage vs protection

Receiving: Fast Response

• Return 200/202 immediately—process asynchronously in queue
• Senders timeout (5-30s typical)—slow processing = retry = duplicates
• Minimal validation before 200—signature check, then queue
• Background job for actual processing—failures don't affect acknowledgment

Receiving: Error Handling

• 2xx = success, sender won't retry
• 4xx = permanent failure, sender may stop retrying—use for bad signature, unknown event type
• 5xx = temporary failure, sender will retry—use for downstream issues
• Log full payload on error—helps debugging; redact sensitive fields

Sending: Retry Strategy

• Exponential backoff: 1min, 5min, 30min, 2h, 8h—then give up or alert
• Cap retries (5-10 attempts)—don't retry forever
• Record delivery attempts—show status to user
• Different retry for 4xx vs 5xx—4xx often means stop retrying

Sending: Signature Generation

• Include timestamp in signature—prevents replay of captured webhooks
• Sign raw JSON body—document exact signing algorithm
• Header format: t=timestamp,v1=signature—allows versioned signatures
• Provide verification code examples—reduce integration friction

Sending: Timeouts

• 5-10 second timeout—don't wait forever for slow receivers
• Treat timeout as failure—retry later
• Don't follow redirects—or limit to 1-2; prevents redirect loops
• Validate HTTPS certificate—don't skip verification

Event Design

• Include event type: {"type": "order.created", ...}—receivers filter by type
• Include timestamp: ISO 8601 with timezone—for ordering and freshness
• Include full resource or ID—prefer full data; saves receiver a lookup
• Version events: api_version field—allows breaking changes

Delivery Tracking

• Log every attempt: URL, status code, response time, response body
• Dashboard for retry queue—let users see pending/failed deliveries
• Manual retry button—for stuck webhooks after receiver fix
• Webhook logs retention: 7-30 days—balance debugging vs storage

Security Checklist

• HTTPS only—never send webhooks to HTTP endpoints
• Rotate secrets periodically—support multiple active secrets during rotation
• IP allowlisting optional—document your IP ranges if offered
• Don't include secrets in payload—webhook URL should be secret enough
• Rate limit per endpoint—one slow receiver shouldn't affect others

Common Mistakes

• No signature verification—anyone can POST fake events to your endpoint
• Processing before responding—timeout causes retries, duplicate processing
• No idempotency handling—double charges, duplicate records
• Trusting event data blindly—always verify by fetching from source API for critical actions