Use Mozilla SOPS with GitOps for encrypted Kubernetes Secrets

This post records another way to manage Kubernetes Secrets in a GitOps repo: encrypt the secret files before committing them.

In the Vault and External Secrets post, my rule was:

Git mapping -> Vault values -> ExternalSecret -> Kubernetes Secret

With SOPS, the flow is different:

encrypted Secret in Git -> GitOps controller decrypts -> Kubernetes Secret -> Pod

Both patterns can be safe, but they solve different problems. Vault is a runtime secret source. SOPS is encrypted configuration in Git. If I choose SOPS, I treat the Git repo as the source of truth for encrypted secret values, and I protect the decryption key like production infrastructure.

Series

This post is part of my home Kubernetes GitOps series:

When I would use SOPS

SOPS is useful when I want the whole application state to live in Git, including encrypted secret values.

I would use it for:

A small cluster where running Vault is too much operational weight.
Bootstrap secrets needed before Vault or External Secrets exists.
App config that should be reviewed and versioned together with manifests.
Disaster recovery where a fresh cluster can be rebuilt from Git plus one protected decryption key.

I would not use SOPS as an excuse to relax secret handling. Encrypted values are still sensitive. The repo should not contain the private key, CI should not print decrypted manifests, and only the GitOps controller should decrypt in normal operation.

The safety model

The safe version has a simple boundary:

Git contains encrypted secret values.
Developers encrypt with public recipients.
The cluster stores the private decryption key.
The GitOps controller decrypts only while applying manifests.
Pods receive normal Kubernetes Secret objects.

The dangerous version is also simple:

the private key is committed to Git
CI decrypts and uploads logs
every developer has the same long-lived private key
decrypted files are written back into the repo
Argo CD repo-server or Redis is reachable by other workloads

SOPS protects data at rest in Git. After decryption, the normal Kubernetes Secret risks still exist. Anyone who can read Secrets in the namespace can read the value. Anyone who can exec into a pod may be able to read mounted files or environment variables. SOPS does not replace RBAC, namespace isolation, or runtime secret hygiene.

Pick age for local GitOps

SOPS supports several key backends: age, OpenPGP, AWS KMS, GCP KMS, Azure Key Vault, and Vault transit.

For a home or small platform cluster, I usually prefer age because the model is easy to reason about:

public recipient: safe to put in .sops.yaml
private identity: must be protected
multiple recipients: useful for key rotation or break-glass access

Install the tools locally:

brew install age sops

Generate a cluster identity:

age-keygen -o cluster-age.agekey

The output contains a public recipient like this:

# public key: age1h3examplepublicrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AGE-SECRET-KEY-1EXAMPLEPRIVATEIDENTITYXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Only the public recipient belongs in Git. The private identity must be stored outside the repo.

Store the private key in the cluster

If Flux will decrypt the files, I put the age private key in the flux-system namespace.

kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=./cluster-age.agekey \
  --dry-run=client -o yaml | kubectl apply -f -

Then I remove the local plaintext key from the working directory or move it to a password manager:

shred -u cluster-age.agekey

On macOS, shred may not be available and secure deletion depends on the storage layer. The more important rule is operational: do not leave cluster-age.agekey inside the Git repo, shell history, shared folders, or CI artifacts.

For production, I prefer a cloud KMS or a hardware-backed secret store when it is available. With KMS, SOPS can encrypt to an identity that does not require copying a raw private key into every admin laptop.

Configure .sops.yaml

I keep the encryption policy at the root of the GitOps repo.

creation_rules:
  - path_regex: apps/.*/secrets/.*\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: age1h3examplepublicrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

This rule says:

Only files under apps/*/secrets/ are matched.
Only data and stringData values are encrypted.
Kubernetes metadata, names, labels, and Secret keys stay readable in Git.

I avoid encrypting the entire YAML file for Kubernetes manifests. GitOps diffs are more useful when reviewers can still see that the file is a Secret named example-api-env-file in namespace example-api, even though the values are encrypted.

For stricter repos, I add one rule per environment:

creation_rules:
  - path_regex: clusters/prod/.*/secrets/.*\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: age1prodrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

  - path_regex: clusters/staging/.*/secrets/.*\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: age1stagingrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

That prevents the staging controller from decrypting production secrets.

Create an encrypted Secret

I start with a normal Kubernetes Secret manifest using stringData. This keeps the source file readable before encryption and avoids manually base64-encoding values.

apiVersion: v1
kind: Secret
metadata:
  name: example-api-env-file
  namespace: example-api
type: Opaque
stringData:
  .env: |
    DATABASE_URL=postgres://example-api:[email protected]:5432/example_api
    REDIS_URL=redis://:[email protected]:6379/0
    JWT_SECRET=change-me

Then encrypt it:

sops --encrypt --in-place apps/example-api/secrets/env-file.yaml

After encryption, the file still looks like Kubernetes YAML, but the secret values are encrypted.

apiVersion: v1
kind: Secret
metadata:
  name: example-api-env-file
  namespace: example-api
type: Opaque
stringData:
  .env: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
  age:
    - recipient: age1h3examplepublicrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...
        -----END AGE ENCRYPTED FILE-----
  encrypted_regex: ^(data|stringData)$
  version: 3.x.x

The important check is that stringData is encrypted and the plaintext values are gone.

rg "change-me|DATABASE_URL|JWT_SECRET" apps/example-api/secrets

This command should return nothing.

Apply with Flux

Flux has native SOPS support in the kustomize controller. The Kustomization only needs to know which Kubernetes Secret contains the decryption key.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: example-api
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/example-api
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Then the application can reference the generated Kubernetes Secret normally:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-api
  namespace: example-api
spec:
  template:
    spec:
      containers:
        - name: api
          image: ghcr.io/example/example-api:1.0.0
          envFrom:
            - secretRef:
                name: example-api-env-file

With Flux, the controller reads the encrypted file from Git, decrypts it in the controller process, and applies the Secret to the cluster.

If I use Argo CD

Argo CD does not make SOPS decryption a built-in first-class feature in the same way Flux does. The common route is a config management plugin, Helm Secrets, or KSOPS.

That can work, but I treat it as a larger security decision because decryption happens in the Argo CD manifest-generation path. The decrypted manifests can touch repo-server and cache layers. If I use this pattern, I want:

Repo-server isolated with NetworkPolicy.
Redis protected and unreachable from application namespaces.
Plugin image pinned and owned by the platform team.
No decrypted manifests printed to logs.
Argo CD RBAC locked down so not everyone can read generated manifests.

For a small cluster where SOPS is the main secret workflow, Flux is the cleaner fit. For a cluster already standardized on Argo CD, I would compare the plugin risk against the Vault and External Secrets pattern from the earlier post.

Rotate a secret value

Changing a secret value is normal GitOps:

sops apps/example-api/secrets/env-file.yaml

Edit the plaintext in the SOPS editor, save, and commit the encrypted diff.

Then verify:

rg "new-plain-value" apps/example-api/secrets
kubectl -n example-api get secret example-api-env-file
kubectl -n example-api rollout restart deploy/example-api

Whether the workload needs a restart depends on how it reads the Secret. If the value is injected as environment variables, the pod must restart. If the Secret is mounted as files, Kubernetes updates the mounted files eventually, but the application still needs to reload them.

Rotate the age key

Key rotation is different from secret value rotation.

First add a new recipient to .sops.yaml:

creation_rules:
  - path_regex: apps/.*/secrets/.*\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: >-
      age1oldrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,
      age1newrecipientxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Then update encrypted files so both recipients can decrypt:

sops updatekeys apps/example-api/secrets/env-file.yaml

After the cluster has the new private key and reconciliation succeeds, remove the old recipient from .sops.yaml and run sops updatekeys again.

The safe order is:

Add new recipient.
Re-encrypt files for both old and new recipients.
Install the new private key in the cluster.
Confirm GitOps can still reconcile.
Remove the old recipient.
Re-encrypt again.
Delete the old private key from the cluster and admin storage.

Do not remove the old recipient before the cluster can decrypt with the new one. That turns a key rotation into an outage.

Repository guardrails

I add a few boring checks because mistakes here are expensive.

.gitignore:

*.agekey
*.dec.yaml
*.decrypted.yaml
.env

Pre-commit or CI check:

rg "AGE-SECRET-KEY|DATABASE_URL=|JWT_SECRET=|BEGIN OPENSSH PRIVATE KEY" .

SOPS structure check:

sops --decrypt apps/example-api/secrets/env-file.yaml >/dev/null

Encrypted value check:

yq '.stringData[".env"]' apps/example-api/secrets/env-file.yaml

That should show an ENC[...] value, not plaintext.

I also prefer branch protection for secret changes. A reviewer does not need to see the plaintext value, but they can still review:

which namespace receives the Secret
which Secret name changes
which workloads consume it
whether a new recipient was added
whether the file matches the expected .sops.yaml rule

SOPS or Vault

I think about it this way:

Vault is better when secrets are dynamic, centrally audited, shared across systems, or rotated by an external process.
SOPS is better when encrypted files in Git are the desired source of truth.
External Secrets is better when Kubernetes should sync values from an external secret manager.
Sealed Secrets is better when I want a Kubernetes-only asymmetric encryption workflow and do not need SOPS formats or KMS integrations.

For my GitOps cluster, Vault and External Secrets remain the cleaner runtime pattern. But SOPS is a very good bootstrap and small-cluster pattern as long as the private key is treated like production access.

Conclusion

The safe SOPS rule is not “encrypted means harmless.” The rule is:

public recipients in Git, private keys outside Git, decryption only in the controller, decrypted values never in logs

Used that way, SOPS gives GitOps a practical encrypted source of truth without committing raw Kubernetes Secret values.