Use Vault and External Secrets in Kubernetes

This post records how I use Vault and External Secrets Operator in my home Kubernetes cluster.

The main idea is:

Vault KV v2 -> ClusterSecretStore -> ExternalSecret -> Kubernetes Secret -> Pod

For consistency with the other posts in this series, the examples use these fake apps:

example-api: public API at api.example.com
example-worker: public worker UI at worker.example.com
example-admin: internal admin app, no public route

I want Kubernetes manifests to stay in Git, but I do not want real secret values in Git. So Git keeps only the secret mapping, and Vault keeps the real values.

Series

This post is part of my home Kubernetes GitOps series:

Why I use this

Kubernetes Secrets are not a good source of truth for me. They are useful inside the cluster, but I do not want to copy base64 values into YAML and commit them.

With this setup:

Vault stores the real secret values.
External Secrets Operator reads Vault.
Kubernetes receives normal Secret objects.
Deployments reference Kubernetes Secrets as usual.
Git contains only manifests and non-secret config.

This is easier to operate than manually recreating Secrets after every cluster rebuild.

Install External Secrets Operator

In my GitOps repo, External Secrets Operator is installed by Argo CD from the Helm chart.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-10"
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    chart: external-secrets
    targetRevision: 2.5.0
    helm:
      releaseName: external-secrets
      valuesObject:
        installCRDs: true
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

The sync wave is important. External Secrets Operator and its CRDs must exist before applications create ClusterSecretStore and ExternalSecret resources.

Put secret data into Vault

My Vault uses KV v2 under the secret mount.

Vault KV v2 secret engine

For an example API service, I store the .env file content in Vault:

vault kv put secret/example-api/env-file dotenv=@.env

For an example worker service:

vault kv put secret/example-worker/config-file config.json=@config.json

For an example admin app:

vault kv put secret/example-admin/config-file config=@config.yml

The important part is the path and property name. The ExternalSecret manifest will reference both.

For example:

Vault path: secret/example-api/env-file
Property: dotenv
K8s Secret: example-api-env-file
Key: .env

Configure Vault Kubernetes auth

External Secrets Operator authenticates to Vault with its Kubernetes service account.

In my cluster, the service account is:

namespace: external-secrets
name: external-secrets

Vault needs to trust the Kubernetes API for token review. For a new cluster, I create a reviewer service account with only the token review permission it needs.

kubectl create namespace vault-auth --dry-run=client -o yaml | kubectl apply -f -
kubectl -n vault-auth create serviceaccount vault-auth
kubectl create clusterrolebinding vault-auth-tokenreview \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault-auth:vault-auth

Create vault-auth-token.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  namespace: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

Then export the reviewer JWT and cluster CA:

kubectl apply -f vault-auth-token.yaml
TOKEN_REVIEWER_JWT=$(kubectl -n vault-auth get secret vault-auth-token -o jsonpath='{.data.token}' | base64 -d)
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt

Configure Vault:

export VAULT_ADDR=https://vault.example.internal:8200
vault login

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://rke-api.example.internal:6443" \
  kubernetes_ca_cert=@ca.crt \
  token_reviewer_jwt="$TOKEN_REVIEWER_JWT"

If the Kubernetes auth mount already exists, I do not recreate it. I only update auth/kubernetes/config with the new cluster API, CA, and reviewer JWT.

I do not disable issuer validation in the default example. If the cluster has an issuer mismatch, I would fix the Vault Kubernetes auth issuer configuration first, and only disable issuer validation when I understand why it is needed.

The reviewer token is sensitive. It should not be committed to Git, and it should be rotated when the cluster or Vault auth configuration is rebuilt.

Create policies and roles

Vault policy controls which KV paths External Secrets Operator can read.

For an example API service:

vault policy write example-api - <<'EOF'
path "secret/data/example-api/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/external-secrets-example-api \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  audience=https://kubernetes.default.svc.cluster.local \
  policies=example-api \
  ttl=1h

For an example worker service:

vault policy write example-worker - <<'EOF'
path "secret/data/example-worker/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/external-secrets-example-worker \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  audience=https://kubernetes.default.svc.cluster.local \
  policies=example-worker \
  ttl=1h

For an example admin app:

vault policy write example-admin - <<'EOF'
path "secret/data/example-admin/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/external-secrets-example-admin \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  audience=https://kubernetes.default.svc.cluster.local \
  policies=example-admin \
  ttl=1h

Notice that the policy path uses secret/data/... because this is Vault KV v2. The ExternalSecret remote key uses example-api/env-file, but the policy still needs the KV v2 API path.

Create ClusterSecretStore

For the example API service, the ClusterSecretStore points to Vault. Before applying it, I put the internal CA public certificate in a ConfigMap.

kubectl -n external-secrets create configmap vault-ca \
  --from-file=ca.crt=./vault-ca.crt \
  --dry-run=client -o yaml | kubectl apply -f -

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-example-api
spec:
  provider:
    vault:
      server: "https://vault.example.internal:8200"
      path: "secret"
      version: "v2"
      caProvider:
        type: ConfigMap
        name: vault-ca
        namespace: external-secrets
        key: ca.crt
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets-example-api
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

The example above uses an internal hostname, HTTPS, and a CA ConfigMap named vault-ca. I avoid plain HTTP for Vault because Kubernetes auth tokens and secret values move across this connection. Vault should be served over HTTPS with an internal CA, and ClusterSecretStore should trust that CA using caProvider or caBundle.

The trust flow should be:

internal CA -> Vault server certificate -> Kubernetes CA ConfigMap/Secret -> ClusterSecretStore -> HTTPS Vault URL

If I use an internal DNS name like vault.example.internal, the Vault certificate must include that name in the SAN. If I use the IP directly, the certificate must include that IP in the SAN.

Create ExternalSecret

After the store is ready, I create an ExternalSecret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example-api-env-file
  namespace: example-api
spec:
  refreshInterval: 2m
  secretStoreRef:
    name: vault-example-api
    kind: ClusterSecretStore
  target:
    name: example-api-env-file
    creationPolicy: Owner
  data:
    - secretKey: .env
      remoteRef:
        key: example-api/env-file
        property: dotenv

This tells External Secrets Operator:

Read Vault path secret/example-api/env-file.
Get the dotenv property.
Create a Kubernetes Secret named example-api-env-file.
Put the value into the .env key.
Refresh it every 2 minutes.

The Deployment can then mount or load example-api-env-file like a normal Kubernetes Secret.

Validate

First check the operator.

kubectl -n external-secrets get pods
kubectl get crd | grep external-secrets

Then check the store.

kubectl get clustersecretstore
kubectl describe clustersecretstore vault-example-api

Check the generated Secret.

kubectl -n example-api get secret example-api-env-file

For other apps:

kubectl -n example-worker get secret example-worker-config-file
kubectl -n example-admin get secret example-admin-config-file

If the store is not ready, I usually check Vault Kubernetes auth first.

kubectl -n external-secrets logs deploy/external-secrets --tail=120

Common issues:

Vault auth points to an old Kubernetes API server.
Vault has the wrong cluster CA.
The token reviewer JWT is expired or from the wrong cluster.
The Vault role is bound to the wrong service account or namespace.
The Vault policy path uses the wrong KV v2 path.
The expected property does not exist in Vault.

SOPS or Vault

For this design, I do not need SOPS as the main secret source.

Vault is the source of truth. External Secrets Operator only syncs values into Kubernetes. Git does not contain encrypted secret values; it contains manifests that describe where the values should come from.

SOPS would make sense if I wanted encrypted files in Git to become the source of truth, or if I needed an offline bootstrap path without Vault. For my current flow, adding SOPS would be another secret system to operate, not a direct improvement.

Conclusion

This setup gives me a clean split:

Git: Kubernetes manifests and secret references
Vault: real secret values
External Secrets Operator: sync controller
Kubernetes: runtime Secret objects

It is still important to secure the Vault transport. HTTPS with an internal CA is the safer default, even for a private cluster.