Deployment Pipeline Design

Architecture patterns for multi-stage CI/CD pipelines with approval gates and deployment strategies.

Purpose

Design robust, secure deployment pipelines that balance speed with safety through proper stage organization and approval workflows.

When to Use

• Design CI/CD architecture

• Implement deployment gates

• Configure multi-environment pipelines

• Establish deployment best practices

• Implement progressive delivery

Pipeline Stages

Standard Pipeline Flow

┌─────────┐ ┌──────┐ ┌─────────┐ ┌────────┐ ┌──────────┐ │ Build │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│ └─────────┘ └──────┘ └─────────┘ └────────┘ └──────────┘

Detailed Stage Breakdown

• Source - Code checkout

• Build - Compile, package, containerize

• Test - Unit, integration, security scans

• Staging Deploy - Deploy to staging environment

• Integration Tests - E2E, smoke tests

• Approval Gate - Manual approval required

• Production Deploy - Canary, blue-green, rolling

• Verification - Health checks, monitoring

• Rollback - Automated rollback on failure

Approval Gate Patterns

Pattern 1: Manual Approval

GitHub Actions

production-deploy: needs: staging-deploy environment: name: production url: https://app.example.com runs-on: ubuntu-latest steps: - name: Deploy to production run: | # Deployment commands

Pattern 2: Time-Based Approval

GitLab CI

deploy:production: stage: deploy script: - deploy.sh production environment: name: production when: delayed start_in: 30 minutes only: - main

Pattern 3: Multi-Approver

Azure Pipelines

stages:

• stage: Production dependsOn: Staging jobs:
  • deployment: Deploy environment: name: production resourceType: Kubernetes strategy: runOnce: preDeploy: steps: - task: ManualValidation@0 inputs: notifyUsers: 'team-leads@example.com' instructions: 'Review staging metrics before approving'

Reference: See assets/approval-gate-template.yml

Deployment Strategies

1. Rolling Deployment

apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 10 strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1

Characteristics:

• Gradual rollout

• Zero downtime

• Easy rollback

• Best for most applications

2. Blue-Green Deployment

Blue (current)

kubectl apply -f blue-deployment.yaml kubectl label service my-app version=blue

Green (new)

kubectl apply -f green-deployment.yaml

Test green environment

kubectl label service my-app version=green

Rollback if needed

kubectl label service my-app version=blue

Characteristics:

• Instant switchover

• Easy rollback

• Doubles infrastructure cost temporarily

• Good for high-risk deployments

3. Canary Deployment

apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: replicas: 10 strategy: canary: steps: - setWeight: 10 - pause: {duration: 5m} - setWeight: 25 - pause: {duration: 5m} - setWeight: 50 - pause: {duration: 5m} - setWeight: 100

Characteristics:

• Gradual traffic shift

• Risk mitigation

• Real user testing

• Requires service mesh or similar

4. Feature Flags

from flagsmith import Flagsmith

flagsmith = Flagsmith(environment_key="API_KEY")

if flagsmith.has_feature("new_checkout_flow"): # New code path process_checkout_v2() else: # Existing code path process_checkout_v1()

Characteristics:

• Deploy without releasing

• A/B testing

• Instant rollback

• Granular control

Pipeline Orchestration

Multi-Stage Pipeline Example

name: Production Pipeline

on: push: branches: [ main ]

jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Build application run: make build - name: Build Docker image run: docker build -t myapp:${{ github.sha }} . - name: Push to registry run: docker push myapp:${{ github.sha }}

test: needs: build runs-on: ubuntu-latest steps: - name: Unit tests run: make test - name: Security scan run: trivy image myapp:${{ github.sha }}

deploy-staging: needs: test runs-on: ubuntu-latest environment: name: staging steps: - name: Deploy to staging run: kubectl apply -f k8s/staging/

integration-test: needs: deploy-staging runs-on: ubuntu-latest steps: - name: Run E2E tests run: npm run test:e2e

deploy-production: needs: integration-test runs-on: ubuntu-latest environment: name: production steps: - name: Canary deployment run: | kubectl apply -f k8s/production/ kubectl argo rollouts promote my-app

verify: needs: deploy-production runs-on: ubuntu-latest steps: - name: Health check run: curl -f https://app.example.com/health - name: Notify team run: | curl -X POST ${{ secrets.SLACK_WEBHOOK }}
-d '{"text":"Production deployment successful!"}'

Pipeline Best Practices

• Fail fast - Run quick tests first

• Parallel execution - Run independent jobs concurrently

• Caching - Cache dependencies between runs

• Artifact management - Store build artifacts

• Environment parity - Keep environments consistent

• Secrets management - Use secret stores (Vault, etc.)

• Deployment windows - Schedule deployments appropriately

• Monitoring integration - Track deployment metrics

• Rollback automation - Auto-rollback on failures

• Documentation - Document pipeline stages

Rollback Strategies

Automated Rollback

deploy-and-verify: steps: - name: Deploy new version run: kubectl apply -f k8s/

- name: Wait for rollout
  run: kubectl rollout status deployment/my-app

- name: Health check
  id: health
  run: |
    for i in {1..10}; do
      if curl -sf https://app.example.com/health; then
        exit 0
      fi
      sleep 10
    done
    exit 1

- name: Rollback on failure
  if: failure()
  run: kubectl rollout undo deployment/my-app

Manual Rollback

List revision history

kubectl rollout history deployment/my-app

Rollback to previous version

kubectl rollout undo deployment/my-app

Rollback to specific revision

kubectl rollout undo deployment/my-app --to-revision=3

Monitoring and Metrics

Key Pipeline Metrics

• Deployment Frequency - How often deployments occur

• Lead Time - Time from commit to production

• Change Failure Rate - Percentage of failed deployments

• Mean Time to Recovery (MTTR) - Time to recover from failure

• Pipeline Success Rate - Percentage of successful runs

• Average Pipeline Duration - Time to complete pipeline

Integration with Monitoring

• name: Post-deployment verification run: |

  Wait for metrics stabilization

  sleep 60

  Check error rate

  ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')

  if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then echo "Error rate too high: $ERROR_RATE" exit 1 fi

Reference Files

• references/pipeline-orchestration.md

• Complex pipeline patterns

• assets/approval-gate-template.yml

• Approval workflow templates

Related Skills

• github-actions-templates

• For GitHub Actions implementation

• gitlab-ci-patterns

• For GitLab CI implementation

• secrets-management

• For secrets handling