Secrets Management

Secure secret synchronization using External Secrets Operator (ESO) and OpenBao.

Architecture Overview

┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ OpenBao │────►│ External Secrets │────►│ K8s Secrets │ │ (Vault) │ │ Operator │ │ (auto-synced) │ └─────────────┘ └──────────────────┘ └─────────────────┘

OpenBao Setup

OpenBao is the secrets vault (HashiCorp Vault fork, MPL 2.0 licensed).

Initialize and Unseal

Initialize (first time only)

kubectl exec -n openbao openbao-0 -- bao operator init

Unseal (required after pod restart)

kubectl exec -n openbao openbao-0 -- bao operator unseal <key1> kubectl exec -n openbao openbao-0 -- bao operator unseal <key2> kubectl exec -n openbao openbao-0 -- bao operator unseal <key3>

Verify status

kubectl exec -n openbao openbao-0 -- bao status

Enable KV Secrets Engine

kubectl exec -n openbao openbao-0 -- bao secrets enable -path=secret kv-v2

Store a Secret

kubectl exec -n openbao openbao-0 -- bao kv put secret/myapp/db
username=myuser
password=mypassword

External Secrets Operator

ESO syncs secrets from OpenBao to Kubernetes Secrets.

ClusterSecretStore

Cluster-wide connection to OpenBao:

apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: openbao spec: provider: vault: server: "http://openbao.openbao.svc:8200" path: "secret" version: "v2" auth: tokenSecretRef: name: openbao-token namespace: external-secrets key: token

ExternalSecret

Sync specific secrets to a namespace:

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: myapp-db-credentials namespace: myapp spec: refreshInterval: 1h secretStoreRef: name: openbao kind: ClusterSecretStore

target: name: myapp-db-credentials # K8s Secret name creationPolicy: Owner

data: - secretKey: username # Key in K8s Secret remoteRef: key: secret/myapp/db # Path in OpenBao property: username # Field in OpenBao secret

- secretKey: password
  remoteRef:
    key: secret/myapp/db
    property: password

Template Secrets

Transform secrets during sync:

spec: target: name: myapp-connection-string template: type: Opaque data: DATABASE_URL: | postgresql://{{ .username }}:{{ .password }}@db.svc:5432/myapp

data: - secretKey: username remoteRef: key: secret/myapp/db property: username - secretKey: password remoteRef: key: secret/myapp/db property: password

Common Patterns

Database Credentials

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: postgres-credentials namespace: databases spec: refreshInterval: 1h secretStoreRef: name: openbao kind: ClusterSecretStore target: name: postgres-credentials dataFrom: - extract: key: secret/databases/postgres

API Keys

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: api-keys namespace: myapp spec: refreshInterval: 30m secretStoreRef: name: openbao kind: ClusterSecretStore data: - secretKey: OPENAI_API_KEY remoteRef: key: secret/apis/openai property: api_key - secretKey: GITHUB_TOKEN remoteRef: key: secret/apis/github property: token

Validation Commands

Check ExternalSecret sync status

kubectl get externalsecrets -A kubectl describe externalsecret <name> -n <namespace>

Check synced K8s Secret

kubectl get secret <name> -n <namespace> -o yaml

Check ClusterSecretStore status

kubectl get clustersecretstores

OpenBao status

kubectl exec -n openbao openbao-0 -- bao status

Troubleshooting

ExternalSecret not syncing

Check ESO controller logs

kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

Check ExternalSecret status

kubectl describe externalsecret <name> -n <namespace>

OpenBao sealed after restart

Check seal status

kubectl exec -n openbao openbao-0 -- bao status

Unseal (need 3 of 5 keys typically)

kubectl exec -n openbao openbao-0 -- bao operator unseal

Best Practices

• Never commit secrets - Use ExternalSecrets for all credentials

• Use refresh intervals - 1h for stable secrets, 30m for rotating

• Scope secrets narrowly - One ExternalSecret per application

• Template connection strings - Avoid exposing raw credentials

• Monitor sync status - Alert on ExternalSecret failures

• Backup unseal keys - Store securely outside cluster