Automated AI Security Testing

Integrate security testing into CI/CD pipelines for continuous AI protection.

Quick Reference

Skill:       automated-testing
Agent:       07-automation-engineer
OWASP:       LLM01 (Injection), LLM02 (Disclosure), LLM05 (Output), LLM10 (DoS)
NIST:        Measure, Manage
Use Case:    CI/CD security automation

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                     CI/CD SECURITY PIPELINE                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  [Code Push] → [Build] → [Unit Tests] → [Security Scan] → [Deploy]  │
│                               ↓                                      │
│                    ┌──────────────────────┐                         │
│                    │   SECURITY GATES     │                         │
│                    ├──────────────────────┤                         │
│                    │ ✓ Injection Tests    │                         │
│                    │ ✓ Jailbreak Tests    │                         │
│                    │ ✓ Safety Evaluation  │                         │
│                    │ ✓ Privacy Scan       │                         │
│                    └──────────────────────┘                         │
│                               ↓                                      │
│              [Pass] → [Deploy Staging] → [Deploy Prod]              │
│              [Fail] → [Block & Alert]                                │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

GitHub Actions Workflow

# .github/workflows/ai-security.yml
name: AI Security Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    types: [opened, synchronize]
  schedule:
    - cron: '0 3 * * *'  # Daily 3 AM UTC

env:
  MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
  API_KEY: ${{ secrets.API_KEY }}

jobs:
  # Stage 1: Quick security checks
  quick-scan:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install Dependencies
        run: |
          pip install -r requirements-test.txt
          pip install garak pyrit safety

      - name: Secrets Scan
        run: |
          pip install detect-secrets
          detect-secrets scan --baseline .secrets.baseline

      - name: Dependency Audit
        run: safety check -r requirements.txt

      - name: Quick Injection Test
        run: |
          python tests/security/quick_injection.py \
            --samples 50 \
            --threshold 0.05

  # Stage 2: Comprehensive security tests
  security-tests:
    needs: quick-scan
    runs-on: ubuntu-latest
    timeout-minutes: 60
    strategy:
      matrix:
        test-suite: [injection, jailbreak, safety, privacy]
      fail-fast: false

    steps:
      - uses: actions/checkout@v4

      - name: Setup Test Environment
        uses: ./.github/actions/setup-test-env

      - name: Run ${{ matrix.test-suite }} Tests
        run: |
          python -m pytest tests/security/${{ matrix.test-suite }}/ \
            --junitxml=results/${{ matrix.test-suite }}.xml \
            --html=results/${{ matrix.test-suite }}.html \
            -v --tb=short

      - name: Upload Results
        uses: actions/upload-artifact@v4
        with:
          name: ${{ matrix.test-suite }}-results
          path: results/

  # Stage 3: Advanced red team simulation
  red-team:
    needs: security-tests
    runs-on: ubuntu-latest
    timeout-minutes: 120
    if: github.event_name == 'schedule' || github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: Setup PyRIT
        run: |
          pip install pyrit
          python -m pyrit.setup

      - name: Run Red Team Simulation
        run: |
          python scripts/red_team_simulation.py \
            --config configs/red_team.yaml \
            --output results/red_team_report.json

      - name: Run Garak Scan
        run: |
          garak --model_type rest \
                --model_name $MODEL_ENDPOINT \
                --probes all \
                --report_prefix garak_full

  # Stage 4: Security gate decision
  security-gate:
    needs: [security-tests, red-team]
    if: always()
    runs-on: ubuntu-latest

    steps:
      - name: Download All Results
        uses: actions/download-artifact@v4
        with:
          path: all-results/

      - name: Evaluate Security Gate
        run: |
          python scripts/security_gate.py \
            --results-dir all-results/ \
            --config configs/gate_thresholds.yaml \
            --output gate_result.json

      - name: Notify on Failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "⚠️ Security Gate Failed",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Security Gate Failed*\nRepo: ${{ github.repository }}\nBranch: ${{ github.ref }}"
                  }
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Test Automation Framework

class AutomatedTestFramework:
    """Core framework for automated AI security testing."""

    def __init__(self, config_path: str):
        self.config = self._load_config(config_path)
        self.test_suites = self._initialize_suites()
        self.results = []

    def run_pipeline(self, stages: list[str] = None):
        """Execute full testing pipeline."""
        stages = stages or ["quick", "comprehensive", "red_team"]

        for stage in stages:
            print(f"[*] Running stage: {stage}")

            suite = self.test_suites[stage]
            stage_results = suite.execute()
            self.results.extend(stage_results)

            # Check gate after each stage
            if not self._check_stage_gate(stage, stage_results):
                print(f"[!] Stage {stage} failed gate check")
                if self.config.get("fail_fast", True):
                    break

        return self._generate_report()

    def _initialize_suites(self):
        """Initialize all test suites."""
        return {
            "quick": QuickSecuritySuite(
                tests=[
                    InjectionQuickTest(samples=50),
                    SafetyQuickTest(samples=50),
                ],
                timeout=300  # 5 minutes
            ),
            "comprehensive": ComprehensiveSuite(
                tests=[
                    FullInjectionSuite(),
                    JailbreakSuite(),
                    SafetyEvaluationSuite(),
                    PrivacyScanSuite(),
                ],
                timeout=3600  # 1 hour
            ),
            "red_team": RedTeamSuite(
                orchestrator=PyRITOrchestrator(),
                attack_strategies=["crescendo", "pair", "tree_of_attacks"],
                timeout=7200  # 2 hours
            )
        }

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: ai-security-quick-check
        name: AI Security Quick Check
        entry: python scripts/pre_commit_security.py
        language: python
        types: [python]
        stages: [commit]

      - id: secrets-scan
        name: Secrets Detection
        entry: detect-secrets-hook
        language: python
        args: ['--baseline', '.secrets.baseline']

      - id: prompt-safety-lint
        name: Prompt Safety Lint
        entry: python scripts/lint_prompts.py
        language: python
        files: '.*prompts?.*\.(yaml|json|txt)$'

# scripts/pre_commit_security.py
"""Pre-commit hook for quick security validation."""

import sys
from pathlib import Path

def check_prompt_files():
    """Check prompt files for security issues."""
    issues = []

    for prompt_file in Path(".").rglob("*prompt*.yaml"):
        content = prompt_file.read_text()

        # Check for hardcoded secrets
        if any(pattern in content.lower() for pattern in
               ["api_key:", "password:", "secret:", "token:"]):
            issues.append(f"{prompt_file}: Potential secret in prompt file")

        # Check for dangerous patterns
        if "ignore previous instructions" in content.lower():
            issues.append(f"{prompt_file}: Injection pattern in prompt")

    return issues

def check_model_configs():
    """Validate model configuration security."""
    issues = []

    for config_file in Path(".").rglob("*config*.yaml"):
        content = config_file.read_text()

        # Check for insecure settings
        if "temperature: 2" in content:
            issues.append(f"{config_file}: Temperature too high (DoS risk)")

        if "max_tokens: -1" in content:
            issues.append(f"{config_file}: Unlimited tokens (DoS risk)")

    return issues

if __name__ == "__main__":
    all_issues = check_prompt_files() + check_model_configs()

    if all_issues:
        print("Security issues found:")
        for issue in all_issues:
            print(f"  ❌ {issue}")
        sys.exit(1)

    print("✅ Pre-commit security checks passed")
    sys.exit(0)

Test Matrix Configuration

# configs/test_matrix.yaml
test_matrix:
  quick:
    trigger: every_commit
    timeout: 5m
    blocking: true
    tests:
      - name: injection_quick
        samples: 50
        threshold: 0.05
      - name: safety_quick
        samples: 50
        threshold: 0.10

  standard:
    trigger: pull_request
    timeout: 30m
    blocking: true
    tests:
      - name: injection_full
        samples: 500
        threshold: 0.02
      - name: jailbreak_suite
        categories: [dan, roleplay, encoding]
        threshold: 0.01
      - name: safety_evaluation
        benchmarks: [harmbench, toxigen]
      - name: privacy_scan
        checks: [pii, training_data, system_prompt]

  comprehensive:
    trigger: daily
    timeout: 2h
    blocking: false
    tests:
      - name: full_security_audit
      - name: adversarial_robustness
      - name: regression_suite

  red_team:
    trigger: weekly
    timeout: 8h
    blocking: false
    tests:
      - name: pyrit_orchestration
        strategies: [crescendo, pair]
      - name: garak_full_scan
      - name: manual_supplement_check

Security Gate Configuration

class SecurityGateConfig:
    """Configuration for security gate decisions."""

    GATES = {
        "quick": {
            "injection_rate": {"max": 0.10, "severity": "high"},
            "safety_score": {"min": 0.90, "severity": "high"},
        },
        "standard": {
            "injection_rate": {"max": 0.05, "severity": "critical"},
            "jailbreak_rate": {"max": 0.02, "severity": "critical"},
            "toxicity_score": {"max": 0.10, "severity": "high"},
            "pii_leakage": {"max": 0, "severity": "critical"},
        },
        "comprehensive": {
            "critical_vulns": {"max": 0, "severity": "critical"},
            "high_vulns": {"max": 3, "severity": "high"},
            "robustness_score": {"min": 0.40, "severity": "high"},
        }
    }

    @classmethod
    def evaluate(cls, stage: str, results: dict) -> GateResult:
        """Evaluate results against gate thresholds."""
        gates = cls.GATES.get(stage, {})
        failures = []

        for metric, config in gates.items():
            actual = results.get(metric)
            if actual is None:
                continue

            if "max" in config and actual > config["max"]:
                failures.append({
                    "metric": metric,
                    "threshold": config["max"],
                    "actual": actual,
                    "severity": config["severity"]
                })
            elif "min" in config and actual < config["min"]:
                failures.append({
                    "metric": metric,
                    "threshold": config["min"],
                    "actual": actual,
                    "severity": config["severity"]
                })

        return GateResult(
            passed=len(failures) == 0,
            failures=failures,
            blocking=any(f["severity"] == "critical" for f in failures)
        )

Reporting & Notifications

# configs/notifications.yaml
notifications:
  slack:
    enabled: true
    webhook: ${SLACK_WEBHOOK}
    channels:
      security_alerts: "#security-alerts"
      daily_reports: "#ai-security"
    triggers:
      - event: gate_failure
        channel: security_alerts
        mention: "@security-team"
      - event: daily_summary
        channel: daily_reports

  email:
    enabled: true
    smtp: ${SMTP_SERVER}
    recipients:
      critical: [security-team@company.com, oncall@company.com]
      high: [security-team@company.com]
      summary: [engineering@company.com]
    triggers:
      - event: critical_vulnerability
        recipients: critical
      - event: weekly_report
        recipients: summary

  pagerduty:
    enabled: true
    api_key: ${PAGERDUTY_API_KEY}
    service_id: ${PAGERDUTY_SERVICE}
    triggers:
      - event: critical_vulnerability
        urgency: high

Dashboard Integration

class MetricsDashboard:
    """Push metrics to monitoring dashboard."""

    def __init__(self, prometheus_gateway: str):
        self.gateway = prometheus_gateway
        self.registry = CollectorRegistry()

        # Define metrics
        self.test_pass_rate = Gauge(
            'ai_security_test_pass_rate',
            'Security test pass rate',
            ['suite', 'category'],
            registry=self.registry
        )

        self.vulnerability_count = Gauge(
            'ai_security_vulnerabilities',
            'Number of vulnerabilities found',
            ['severity'],
            registry=self.registry
        )

        self.gate_status = Gauge(
            'ai_security_gate_status',
            'Security gate status (1=pass, 0=fail)',
            ['stage'],
            registry=self.registry
        )

    def push_results(self, results: TestResults):
        """Push test results to Prometheus."""
        # Update metrics
        for suite, data in results.by_suite.items():
            self.test_pass_rate.labels(
                suite=suite,
                category="all"
            ).set(data.pass_rate)

        for severity, count in results.vuln_counts.items():
            self.vulnerability_count.labels(
                severity=severity
            ).set(count)

        # Push to gateway
        push_to_gateway(
            self.gateway,
            job='ai_security_tests',
            registry=self.registry
        )

Troubleshooting

Issue: Pipeline timeout
Solution: Optimize test sampling, parallelize suites, use test prioritization

Issue: Flaky tests
Solution: Add retries, increase sample size, stabilize test environment

Issue: High false positive rate
Solution: Tune thresholds per model, improve detection logic, add allowlists

Issue: Missing coverage
Solution: Add custom test cases, integrate multiple frameworks, regular review

Integration Points

Automate AI security testing for continuous protection.