Managing Secrets
Secure storage, rotation, and delivery of secrets (API keys, database credentials, TLS certificates) for applications and infrastructure.
When to Use This Skill
Use when:
-
Storing API keys, database credentials, or encryption keys
-
Implementing secret rotation (manual or automatic)
-
Syncing secrets from external stores to Kubernetes
-
Setting up dynamic secrets (database, cloud providers)
-
Scanning code for leaked secrets
-
Implementing zero-knowledge patterns
-
Meeting compliance requirements (SOC 2, ISO 27001, PCI DSS)
Quick Decision Frameworks
Framework 1: Choosing a Secret Store
Scenario Primary Choice Alternative
Kubernetes + Multi-Cloud Vault + ESO Cloud Secret Manager + ESO
Kubernetes + Single Cloud Cloud Secret Manager + ESO Vault + ESO
Serverless (AWS Lambda) AWS Secrets Manager AWS Parameter Store
Multi-Cloud Enterprise HashiCorp Vault Doppler (SaaS)
Small Team (<10 apps) Doppler, Infisical 1Password Secrets Automation
GitOps-Centric SOPS (git-encrypted) Sealed Secrets (K8s-only)
Decision Tree:
-
Kubernetes? → External Secrets Operator (ESO) with chosen backend
-
Single cloud? → Cloud-native (AWS/GCP/Azure)
-
Multi-cloud/on-prem? → HashiCorp Vault
-
GitOps? → SOPS or Sealed Secrets
Framework 2: Static vs. Dynamic Secrets
Secret Type Use Dynamic? TTL Solution
Database credentials YES 1 hour Vault DB engine
Cloud IAM (AWS/GCP) YES 15 min Vault cloud engine
SSH/RDP access YES 5 min Vault SSH engine
TLS certificates YES 24 hours Vault PKI / cert-manager
Third-party API keys NO Quarterly Vault KV v2 (manual rotation)
Framework 3: Kubernetes Secret Delivery
Method Use Case Rotation Restart Required
External Secrets Operator Static secrets, periodic sync Polling (1h) Yes
Secrets Store CSI Driver File-based, watch rotation inotify No
Vault Secrets Operator Vault-specific, dynamic Automatic renewal Optional
HashiCorp Vault Fundamentals
Core Components
-
Secrets Engines: KV v2 (static), Database (dynamic), AWS, PKI, SSH
-
Auth Methods: Kubernetes, JWT/OIDC, AppRole, LDAP
-
Policies: HCL-based access control (least privilege)
-
Leases: TTL for secrets, auto-renewal, auto-revocation
Static Secrets (KV v2)
Create secret
vault kv put secret/myapp/config api_key=sk_live_EXAMPLE
Read secret
vault kv get secret/myapp/config
List versions
vault kv metadata get secret/myapp/config
Dynamic Database Credentials
Configure PostgreSQL
vault write database/config/postgres
plugin_name=postgresql-database-plugin
connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb"
Create role
vault write database/roles/app-role
db_name=postgres
creation_statements="CREATE ROLE "{{name}}"..."
default_ttl="1h"
Generate credentials
vault read database/creds/app-role
For detailed Vault architecture, see references/vault-architecture.md .
Kubernetes Integration
External Secrets Operator (ESO)
Syncs secrets from 30+ providers to Kubernetes Secrets.
apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-backend spec: provider: vault: server: "https://vault.example.com" auth: kubernetes: role: "app-role"
apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: database-credentials spec: refreshInterval: 1h secretStoreRef: name: vault-backend target: name: db-credentials data:
- secretKey: password remoteRef: key: secret/data/database/config
Vault Secrets Operator (VSO)
Kubernetes-native Vault integration with automatic lease renewal.
apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultDynamicSecret metadata: name: postgres-creds spec: vaultAuthRef: vault-auth mount: database path: creds/app-role renewalPercent: 67 # Renew at 67% of TTL destination: name: dynamic-db-creds
For ESO vs CSI vs VSO comparison, see references/kubernetes-integration.md .
Secret Rotation Patterns
Pattern 1: Versioned Static Secrets (Blue/Green)
-
Create new secret version in Vault
-
Update staging environment
-
Monitor for errors (24-48 hours)
-
Gradual production rollout (10% → 50% → 100%)
-
Revoke old secret (after 7 days)
Pattern 2: Dynamic Database Credentials
Vault auto-generates credentials with short TTL:
-
App fetches credentials from Vault
-
Vault automatically renews lease (at 67% of TTL)
-
On expiration, Vault revokes access
-
On renewal failure, app requests new credentials
Pattern 3: TLS Certificate Rotation
Using cert-manager + Vault PKI:
-
cert-manager requests certificate from Vault
-
Automatically renews before expiration (default: 67% of duration)
-
Updates Kubernetes Secret on renewal
-
Optional pod restart (via Reloader)
For detailed rotation workflows, see references/rotation-patterns.md .
Multi-Language Integration
Python (hvac)
import hvac
client = hvac.Client(url='https://vault.example.com') client.auth.kubernetes(role='app-role', jwt=jwt)
Fetch dynamic credentials
response = client.secrets.database.generate_credentials(name='postgres-role') username = response['data']['username'] password = response['data']['password']
Go (Vault API)
import vault "github.com/hashicorp/vault/api"
client, _ := vault.NewClient(vault.DefaultConfig()) k8sAuth, _ := auth.NewKubernetesAuth("app-role") client.Auth().Login(context.Background(), k8sAuth)
secret, _ := client.Logical().Read("database/creds/postgres-role")
TypeScript (node-vault)
import vault from 'node-vault';
const client = vault({ endpoint: 'https://vault.example.com' }); await client.kubernetesLogin({ role: 'app-role', jwt });
const response = await client.read('database/creds/postgres-role');
For complete examples, see examples/dynamic-db-credentials/ .
Secret Scanning
Pre-Commit Hooks (Gitleaks)
Install Gitleaks
brew install gitleaks
Run on staged files
gitleaks protect --staged --verbose
Pre-commit hook prevents secrets from being committed. For setup, see examples/secret-scanning/pre-commit .
CI/CD Integration
GitHub Actions
- name: Run Gitleaks uses: gitleaks/gitleaks-action@v2
Remediation Workflow
When a secret is leaked:
-
Rotate immediately (within 1 hour)
-
Revoke at provider
-
Remove from Git history (BFG Repo-Cleaner)
-
Force push (notify team)
-
Audit access (who had access during leak window)
-
Document incident
For detailed remediation, see references/secret-scanning.md .
Zero-Knowledge Patterns
Client-Side Encryption (E2EE)
User password → PBKDF2 → encryption key → encrypt secret → send to server
Server stores only encrypted blobs (cannot decrypt).
Shamir's Secret Sharing
Split secret into N shares, require M to reconstruct (e.g., 3 of 5).
Initialize Vault with Shamir shares
vault operator init -key-shares=5 -key-threshold=3
Unseal requires 3 of 5 key shares
vault operator unseal <KEY_1> vault operator unseal <KEY_2> vault operator unseal <KEY_3>
For implementations, see references/zero-knowledge.md .
Library Recommendations (2025)
Secret Stores
Library Use Case Trust Score
HashiCorp Vault Enterprise, multi-cloud High (73.3/100)
External Secrets Operator Kubernetes integration High (85.0/100)
AWS Secrets Manager AWS workloads High
GCP Secret Manager GCP workloads High
Azure Key Vault Azure workloads High
Secret Scanning
Library Use Case Trust Score
Gitleaks Pre-commit, CI/CD High (89.9/100)
TruffleHog Git history scanning Medium
Client Libraries
Language Library Version
Python hvac
2.2.0+
Go vault/api
Latest
TypeScript node-vault
0.10.2+
Rust vaultrs
0.7+
Common Workflows
Workflow 1: Vault + ESO on Kubernetes
-
Install Vault (Helm chart)
-
Initialize and unseal Vault
-
Enable Kubernetes auth
-
Install External Secrets Operator
-
Create SecretStore (Vault connection)
-
Create ExternalSecret (secret mapping)
For step-by-step guide, see examples/vault-eso-setup/ .
Workflow 2: Dynamic Database Credentials
-
Enable database secrets engine
-
Configure database connection
-
Create role with TTL
-
App fetches credentials
-
Vault auto-renews lease
For implementation, see examples/dynamic-db-credentials/ .
Workflow 3: Secret Scanning Remediation
-
Gitleaks detects secret
-
Block commit (pre-commit hook)
-
Developer removes secret
-
Developer stores in Vault
-
Developer references Vault path
For setup, see examples/secret-scanning/ .
Integration with Related Skills
-
auth-security: OAuth client secrets, JWT signing keys
-
databases-*: Dynamic database credentials
-
deploying-applications: Container registry credentials
-
observability: Grafana/Datadog API keys
-
infrastructure-as-code: Cloud provider credentials
Security Best Practices
-
Never commit secrets to Git (use Gitleaks pre-commit hook)
-
Use dynamic secrets where possible
-
Rotate secrets regularly (quarterly for static, hourly for dynamic)
-
Implement least privilege (Vault policies, RBAC)
-
Enable audit logging
-
Encrypt at rest (Vault storage, etcd encryption)
-
Use short TTLs (< 24 hours for dynamic secrets)
-
Monitor failed access attempts
Common Pitfalls
Secrets in Environment Variables
Environment variables visible in process lists. Solution: Use file-based secrets (Kubernetes volumes, CSI driver).
Hardcoded Secrets in Manifests
Base64 is not encryption. Solution: Use External Secrets Operator.
No Secret Rotation
Stale credentials increase breach risk. Solution: Use dynamic secrets or automate rotation.
Root Token in Production
Unlimited permissions. Solution: Use auth methods with least privilege policies.
For Detailed Information, See
-
references/vault-architecture.md
-
Vault internals, HA setup, policies
-
references/kubernetes-integration.md
-
ESO, CSI driver, VSO comparison
-
references/rotation-patterns.md
-
Detailed rotation workflows
-
references/secret-scanning.md
-
Gitleaks, remediation procedures
-
references/zero-knowledge.md
-
E2EE, Shamir's secret sharing
-
references/cloud-providers.md
-
AWS, GCP, Azure secret managers
-
examples/vault-eso-setup/
-
Complete Kubernetes setup
-
examples/dynamic-db-credentials/
-
Multi-language examples
-
examples/secret-scanning/
-
Pre-commit hooks, CI/CD
-
scripts/setup_vault.sh
-
Automated Vault installation