P Peter Lee - Blog
cloud

Persist Vault audit logs on the host: permissions, HUP, and 90-day rotation

Configure the audit device, container storage, and logrotate separately, then verify that new requests reach the new file after rotation.

peterlee
peterlee
Sep 04, 2026
English繁中

A Vault server that delivers secrets successfully does not necessarily leave the evidence needed to explain who performed an API operation. Container server logs describe process state and errors. Audit devices record API requests and responses. Seeing output from docker logs vault does not prove that both exist.

My Vault runs outside Kubernetes in Docker Compose. The repository records a bind mount from /var/log/vault on the host to /vault/logs in the container, with logrotate managing audit.jsonl. This article focuses on persistence and rotation rather than repeating the Vault and External Secrets delivery flow.

Restarting is different from recreating a container

Files in an ordinary container writable layer normally survive stopping and restarting that same container. Removing and recreating it loses that layer. Volumes and bind mounts can persist across container lifecycles; tmpfs is a separate, non-persistent case. The Docker storage guide explains these distinctions.

Finding logs after restarting the same container is not enough to prove reliable retention. Check where the audit device writes, whether that location is mounted persistently, and whether backups cover it.

Vault file audit device
  → /vault/logs/audit.jsonl       container view
  → /var/log/vault/audit.jsonl    host view of the same file
  → logrotate rename + create
  → HUP tells Vault to reopen

A bind mount fits this setup because both Vault and host-side logrotate need access to the directory. It does not protect against host disk failure or replace an off-host backup.

Prepare the directory for the actual writer

Match the UID and GID of the Vault process, not simply a host account named vault. Inspect the process with docker top vault -eo pid,uid,gid,args. Rootless Docker and user namespace remapping also require checking the host ID mapping.

The following example assumes that you have verified the Vault process maps to UID 100 and GID 1000 on the host. These are not guaranteed identities for every Vault image:

sudo install -d -m 0700 -o 100 -g 1000 /var/log/vault

Merge a directory mount into the existing Compose service, preserving the config and data volumes and other security settings:

services:
  vault:
    volumes:
      - ./config:/vault/config:ro
      - ./data:/vault/data:rw
      - /var/log/vault:/vault/logs:rw

Mount the directory rather than a single audit.jsonl file. Rotation renames the old file and creates a new one; a directory mount lets the container see the new pathname and inode.

Applying the mount change normally requires recreating the container. If an existing audit file remains in its writable layer, plan preservation and cutover before covering the path or deleting the container. With Shamir unseal, also prepare the maintenance procedure for unsealing after recreation. Do not restart Vault merely to add a log mount without a recovery path.

A mounted directory does not enable auditing

With Vault initialized and unsealed, use an identity authorized to manage audit devices to inspect existing configuration:

vault audit list -detailed

Only if the target device does not already exist, enable a file device using the container-visible path:

vault audit enable -path=host-file file \
  file_path=/vault/logs/audit.jsonl \
  mode=0600

host-file is the device name; file is its type. The device emits JSON records suitable for line-oriented processing. Do not disable and re-enable an existing device merely to rotate its file: that interrupts recording and changes its audit hashing key. The Vault audit documentation distinguishes devices, operational logs, and hashing behavior.

Generate a non-destructive request that is actually audited, then inspect newly appended metadata. A logged-in session with self-lookup permission can use:

vault token lookup >/dev/null
sudo tail -n 4 /var/log/vault/audit.jsonl | \
  jq -c '{time, type, request_id: .request.id, operation: .request.operation}'

Time, record type, request ID, and operation are enough to check delivery without displaying the full token lookup response or audit body. Do not rely on /sys/health for this test: it is exempt from auditing, so a successful health response does not demonstrate a file write.

Rotate by renaming, creating, and reopening

This example for /etc/logrotate.d/vault-audit retains the repository’s daily rotation and 90-archive policy, adding delaycompress so the newest archive is compressed on the next cycle:

/var/log/vault/audit.jsonl {
    daily
    rotate 90
    maxage 90
    missingok
    notifempty
    compress
    delaycompress
    dateext
    create 0600 100 1000
    sharedscripts
    postrotate
        /usr/bin/docker kill --signal=HUP vault
    endscript
}

Verify the Docker binary path, container name, and UID/GID before installing the root-owned configuration. Ordinary users must not be able to modify it. This assumes the system logrotate service can reach the relevant Docker daemon; another user’s rootless daemon needs a different arrangement.

Vault holds a file descriptor rather than repeatedly looking up the filename. Renaming audit.jsonl alone can leave it writing to the old inode. HUP makes file audit devices close and reopen their files, as documented by the Vault file audit device.

docker kill --signal=HUP sends a selected signal rather than the default SIGKILL. Confirm that Vault is the process receiving it or that the entrypoint forwards signals correctly. A successful Docker command does not prove that Vault reopened the file. The Docker kill reference calls out shell-form entrypoints in particular.

Because Vault supports reopening, copytruncate is unnecessary here. Writes can be lost between its copy and truncate operations; it is not a fix for a missing HUP. delaycompress gives reopening some breathing room, but it does not compensate for a failed signal.

What “90 days” does and does not mean

The settings have distinct meanings:

  • daily is a rotation condition, not a scheduler; cron or a timer must run logrotate.
  • rotate 90 counts archives rather than measuring every event’s age.
  • maxage 90 removes old archives, but checks age only when the active log is due for rotation.
  • notifempty and missed scheduler runs affect the actual rotation timeline.

This is neither a guarantee of 90 complete days of evidence nor a precise expiry-deletion mechanism. See the logrotate manual for these semantics and the copytruncate limitation. Legal or contractual retention requirements need a separate design for retention, deletion, access control, and tamper resistance.

Daily rotation is not a disk-space cap either. A burst of API traffic can fill the disk before the next scheduled run. Adding size-based rotation requires revisiting archive counts, names, and retention duration rather than simply inserting maxsize.

Dry-run first, then prove that the new file receives requests

Start with non-mutating checks on the host:

sudo logrotate --debug /etc/logrotate.d/vault-audit
systemctl status logrotate.timer
sudo ls -ln /var/log/vault
df -h /var/log/vault

On a cron-based host, inspect the actual cron schedule instead. Debug mode neither rotates files nor runs the postrotate script, so it cannot validate HUP delivery or write permissions.

In a test environment or maintenance window, perform a real rotation:

sudo logrotate --verbose --force /etc/logrotate.d/vault-audit

This changes files, sends HUP, and may delete old archives under the retention policy. Check preservation requirements first. Repeated forced rotations on the same day can also collide with dateext filenames; do not ignore that error.

Compare inodes with ls -li before and after rotation. Confirm the new active file’s owner and permissions, then issue another vault token lookup >/dev/null. Its request and response should appear in the new audit.jsonl, rather than only growing the old archive. Check Vault operational logs and logrotate results for permission or reopen errors.

Audit storage is also an availability dependency

When auditing is enabled and Vault cannot write to any enabled audit device, the corresponding API request can be refused. With one file device, a full disk or permission error becomes a service problem. HashiCorp’s audit best practices recommend at least two devices of different types, including a remote destination. Two files on one disk are not independent failure domains.

Default HMAC protection does not anonymize the entire log. Metadata, some value types, and fields retained by configuration can still be sensitive. Restrict archive access, protect backups, and monitor rotation failures, disk usage, and audit write failures. Do not enable raw logging merely to make searching easier.

The final acceptance test is not that compressed files exist. New requests must continue reaching the active file after rotation, older records must remain usable, and the storage path must not silently undermine Vault availability. That is the difference between having auditing enabled and being able to rely on it.

#vault#docker#security#observability