CI/CD Pipelines

GitHub Actions Workflows

Core Concepts

• Workflows: YAML files defining automation processes

• Events: Triggers for workflow execution (push, pull_request, schedule)

• Jobs: Collections of steps that run on the same runner

• Steps: Individual tasks within a job

• Actions: Reusable components for common tasks

Best Practices

Workflow Organization

• Use separate workflows for different concerns (CI, CD, release)

• Use workflow templates for consistency

• Use composite actions for reusable steps

• Use reusable workflows for shared pipeline logic

• Use matrix strategy for testing across multiple configurations

Job Optimization

• Use caching for dependencies and build artifacts

• Use parallel jobs for faster execution

• Use job dependencies for sequential execution

• Use artifacts for passing data between jobs

• Use self-hosted runners for custom environments

Security

• Use secrets for sensitive data

• Use environments for deployment protection

• Use required reviews for critical deployments

• Use OIDC for cloud provider authentication

• Use branch protection rules

GitHub Actions Examples

name: CI/CD Pipeline

on: push: branches: [main, develop] pull_request: branches: [main]

jobs: test: runs-on: ubuntu-latest strategy: matrix: node-version: [14.x, 16.x, 18.x] steps: - uses: actions/checkout@v3 - name: Use Node.js ${{ matrix.node-version }} uses: actions/setup-node@v3 with: node-version: ${{ matrix.node-version }} cache: 'npm' - run: npm ci - run: npm test - run: npm run build

deploy: needs: test runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' steps: - uses: actions/checkout@v3 - name: Deploy to production run: | # Deployment commands

GitLab CI/CD Pipelines

Core Concepts

• Pipelines: Series of jobs executed in stages

• Stages: Logical grouping of jobs

• Jobs: Individual tasks that execute scripts

• Runners: Agents that execute jobs

• Artifacts: Files passed between jobs

Best Practices

Pipeline Configuration

• Use .gitlab-ci.yml for pipeline definition

• Use stages for logical job grouping

• Use only/except rules for job execution control

• Use rules for more complex conditions

• Use needs for job dependencies

Job Optimization

• Use artifacts for passing data between jobs

• Use cache for dependency caching

• Use parallel jobs for faster execution

• Use retry for transient failures

• Use timeout for job duration limits

Security

• Use CI/CD variables for sensitive data

• Use protected variables for protected branches

• Use masked variables for hiding values

• Use environments for deployment control

• Use approval rules for critical deployments

GitLab CI/CD Examples

stages:

• test
• build
• deploy

test: stage: test image: node:18 script: - npm ci - npm test parallel: matrix: - NODE_VERSION: [14, 16, 18] cache: paths: - node_modules/

build: stage: build image: node:18 script: - npm ci - npm run build artifacts: paths: - dist/

deploy: stage: deploy image: node:18 script: - npm install -g serverless - serverless deploy only: - main when: manual

Jenkins Pipelines

Core Concepts

• Pipelines: Groovy scripts defining automation

• Stages: Logical grouping of steps

• Steps: Individual operations

• Agents: Nodes where pipeline runs

• Credentials: Secure storage for secrets

Best Practices

Pipeline Design

• Use Declarative Pipeline syntax

• Use shared libraries for reusable code

• Use pipeline stages for logical organization

• Use when conditions for conditional execution

• Use post sections for cleanup and notifications

Agent Management

• Use labels for agent selection

• Use Docker agents for consistent environments

• Use Kubernetes agents for dynamic scaling

• Use agent none for lightweight pipelines

• Use stash/unstash for passing data

Security

• Use credentials binding for secrets

• Use credential scopes for access control

• Use approval steps for manual gates

• Use input steps for user interaction

• Use role-based strategy for permissions

Jenkins Pipeline Examples

pipeline { agent any

stages {
    stage('Test') {
        parallel {
            stage('Node 14') {
                agent { docker { image 'node:14' } }
                steps {
                    sh 'npm ci'
                    sh 'npm test'
                }
            }
            stage('Node 16') {
                agent { docker { image 'node:16' } }
                steps {
                    sh 'npm ci'
                    sh 'npm test'
                }
            }
        }
    }
    
    stage('Build') {
        agent { docker { image 'node:18' } }
        steps {
            sh 'npm ci'
            sh 'npm run build'
            archiveArtifacts artifacts: 'dist/**'
        }
    }
    
    stage('Deploy') {
        when {
            branch 'main'
        }
        steps {
            input message: 'Deploy to production?', ok: 'Deploy'
            sh 'npm install -g serverless'
            sh 'serverless deploy'
        }
    }
}

post {
    always {
        cleanWs()
    }
    success {
        emailext subject: 'Build Success',
            body: 'The build completed successfully.',
            to: 'team@example.com'
    }
    failure {
        emailext subject: 'Build Failed',
            body: 'The build failed.',
            to: 'team@example.com'
    }
}

}

Azure DevOps Pipelines

Core Concepts

• Pipelines: YAML or GUI-defined automation

• Stages: Major divisions in a pipeline

• Jobs: Units of work within a stage

• Tasks: Pre-built units of work

• Agents: Machines that run jobs

Best Practices

Pipeline Configuration

• Use YAML pipelines for version control

• Use templates for reusable pipeline logic

• Use parameters for pipeline customization

• Use variables for configuration values

• Use stages for environment separation

Job Optimization

• Use parallel jobs for faster execution

• Use artifacts for passing data between jobs

• Use caching for dependency caching

• Use matrix strategy for multiple configurations

• Use demands for agent selection

Security

• Use secret variables for sensitive data

• Use variable groups for organization

• Use key vault for secret management

• Use environment checks for deployment control

• Use approval gates for manual intervention

Azure DevOps Pipeline Examples

trigger:

• main
• develop

pool: vmImage: 'ubuntu-latest'

stages:

• stage: Test jobs:

  • job: Test strategy: matrix: Node_14: node_version: 14.x Node_16: node_version: 16.x Node_18: node_version: 18.x steps:
    • task: UseNode@1 inputs: versionSpec: $(node_version)
    • script: | npm ci npm test displayName: 'Install dependencies and run tests'

• stage: Build dependsOn: Test jobs:

  • job: Build steps:
    • task: UseNode@1 inputs: versionSpec: '18.x'
    • script: | npm ci npm run build displayName: 'Build application'
    • publish: dist artifact: dist

• stage: Deploy dependsOn: Build condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main')) jobs:

  • deployment: Deploy environment: 'production' strategy: runOnce: deploy: steps: - script: | npm install -g serverless serverless deploy displayName: 'Deploy to production'

Build and Test Strategies

Build Strategies

• Incremental Builds: Build only changed components

• Parallel Builds: Build multiple components simultaneously

• Matrix Builds: Build across multiple configurations

• Cached Builds: Use cache for faster builds

• Docker Builds: Use Docker for consistent build environments

Test Strategies

• Unit Tests: Test individual components in isolation

• Integration Tests: Test component interactions

• End-to-End Tests: Test complete user flows

• Performance Tests: Test system performance

• Security Tests: Test for security vulnerabilities

Quality Gates

• Code Coverage: Minimum code coverage threshold

• Linting: Code style and quality checks

• Static Analysis: Security and vulnerability scanning

• Dependency Scanning: Check for vulnerable dependencies

• License Compliance: Check for license compliance

Deployment Patterns

Blue-Green Deployment

• Concept: Maintain two identical production environments

• Benefits: Zero downtime, instant rollback

• Implementation: Switch traffic between blue and green environments

• Best Practices: Use load balancers for traffic switching, test green environment before switching

Canary Deployment

• Concept: Gradually roll out changes to a subset of users

• Benefits: Reduced risk, gradual feedback

• Implementation: Route small percentage of traffic to new version

• Best Practices: Monitor metrics closely, have rollback plan ready

Rolling Deployment

• Concept: Replace instances one at a time

• Benefits: Gradual rollout, minimal downtime

• Implementation: Update instances in batches

• Best Practices: Use health checks, monitor for failures

Feature Flags

• Concept: Enable/disable features without deployment

• Benefits: Gradual rollout, instant rollback

• Implementation: Use feature flag management system

• Best Practices: Keep flags temporary, clean up old flags

Immutable Infrastructure

• Concept: Replace rather than modify infrastructure

• Benefits: Consistency, easier rollback, no configuration drift

• Implementation: Use infrastructure as code, replace instances on updates

• Best Practices: Version all infrastructure, use golden images

Pipeline Orchestration

Pipeline Dependencies

• Sequential Execution: Jobs run in sequence

• Parallel Execution: Jobs run simultaneously

• Conditional Execution: Jobs run based on conditions

• Artifact Passing: Pass data between jobs

• Environment Promotion: Promote through environments

Pipeline Triggers

• Push Triggers: Run on code push

• Pull Request Triggers: Run on pull requests

• Schedule Triggers: Run on schedule

• Manual Triggers: Run on demand

• Webhook Triggers: Run on webhook events

Pipeline Notifications

• Email Notifications: Send email on pipeline events

• Slack Notifications: Send Slack messages on pipeline events

• Microsoft Teams Notifications: Send Teams messages on pipeline events

• Custom Webhooks: Send custom webhook notifications

• Status Badges: Display pipeline status as badges

Pipeline Security

Secrets Management

• Secret Variables: Store secrets as pipeline variables

• Secret Stores: Use secret stores (Vault, AWS Secrets Manager)

• OIDC Authentication: Use OIDC for cloud provider authentication

• Key Vault: Use key vault for secret management

• Secret Rotation: Rotate secrets regularly

Access Control

• Branch Protection: Protect branches from direct pushes

• Pull Request Reviews: Require reviews for merges

• Environment Approvals: Require approvals for deployments

• Role-Based Access: Use RBAC for pipeline access

• Audit Logging: Enable audit logging for compliance

Supply Chain Security

• Dependency Scanning: Scan for vulnerable dependencies

• Container Scanning: Scan container images for vulnerabilities

• SBOM Generation: Generate software bill of materials

• Code Signing: Sign artifacts for integrity

• Policy Enforcement: Enforce policies with tools like OPA