P Peter Lee - Blog
cloud

Build an OpenTelemetry stack for Kubernetes apps

Keep the observability backend outside Kubernetes and collect application telemetry, Pod stdout, and host logs through separate collectors.

peterlee
peterlee
Jun 02, 2026
English繁中
Build an OpenTelemetry stack for Kubernetes apps

My application workloads run in Kubernetes, but the observability backend does not. OpenTelemetry Collector, Prometheus, Loki, Tempo, and Grafana run on the NUC with Docker Compose. This keeps dashboards and historical telemetry available even when I am rebuilding the cluster.

Application telemetry, Pod stdout, and host logs enter through separate collectors:

application OTLP -> OpenTelemetry Collector -> Prometheus / Loki / Tempo
Pod stdout       -> Alloy DaemonSet         -> Loki
host file        -> host Alloy              -> Loki
                                                |
                                             Grafana

That distinction prevents a common source of confusion. A healthy Collector does not prove that Pod stdout is being collected, and a healthy Loki does not prove that any of these sources has delivered a log line.

This post uses the same fictional applications as the other examples. The OpenTelemetry snippets focus on example-api, while the same pattern can be repeated for example-worker and example-admin.

  • example-api: service.name=example-api, service.namespace=example
  • example-worker: service.name=example-worker, service.namespace=example
  • example-admin: service.name=example-admin, service.namespace=example

Why the backend stays outside Kubernetes

Running the backend in Docker Compose is not inherently better than running it in Kubernetes. It fits this cluster because it separates two failure domains. If Kubernetes networking, storage, or Argo CD is unhealthy, I can still open Grafana and inspect the evidence already stored on the host.

The tradeoff is operational responsibility. Compose volumes need backups and retention limits, the host must remain reachable from the cluster, and the Collector endpoints need the same network protection as any other internal service.

Services

The Compose stack has five core services:

  • otel-collector: receives application OTLP
  • prometheus: scrapes collector-exported metrics
  • loki: stores OTLP logs and Pod stdout forwarded by Alloy
  • tempo: stores traces
  • grafana: browses metrics, logs, and traces

Argo CD also installs Grafana Alloy as a DaemonSet inside Kubernetes. Alloy is the node-local collector for Pod stdout; it is not part of the Compose project. The Compose project has a separate, tightly scoped Alloy service for one host log source and an init container that prepares Tempo’s data volume.

The collector exposes common OTLP ports:

  • 4317: OTLP gRPC
  • 4318: OTLP HTTP
  • 9464: Prometheus scrape endpoint

Docker Compose shape

These are service fragments for an existing Compose project. Merge the services under one services key, choose compatible pinned image versions, and provide the backend configuration files and persistent volumes before starting the stack. The snippets omit Prometheus scrape configuration, Loki and Tempo storage configuration, and Grafana provisioning.

The Collector no longer mounts /var/log/pods. It only receives OTLP, so it can run as a non-root user with a read-only configuration file. The service also has explicit CPU, memory, and PID limits.

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:<pinned-version>
    restart: unless-stopped
    user: "10001:10001"
    cpus: "1.0"
    mem_limit: 768m
    pids_limit: 256
    command:
      - --config=/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"
      - "9464:9464"
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro

The container limit is higher than the Collector’s internal memory limit. This leaves room for the process to shed load before the runtime terminates it.

Prometheus scrapes the collector:

prometheus:
  image: prom/prometheus:<pinned-version>
  command:
    - --config.file=/etc/prometheus/prometheus.yml
    - --storage.tsdb.path=/prometheus
    - --storage.tsdb.retention.time=14d
    - --storage.tsdb.retention.size=20GB
  ports:
    - "9090:9090"

Loki and Tempo store logs and traces:

loki:
  image: grafana/loki:<pinned-version>
  ports:
    - "3100:3100"

tempo:
  image: grafana/tempo:<pinned-version>
  command:
    - -target=all
    - -config.file=/etc/tempo.yaml
  ports:
    - "3200:3200"
    - "4319:4317"

Configure Grafana with an admin password or secret file and disable anonymous access:

grafana:
  image: grafana/grafana:<pinned-version>
  ports:
    - "3000:3000"
  environment:
    GF_SECURITY_ADMIN_USER: admin
    GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set a password}
    GF_AUTH_ANONYMOUS_ENABLED: "false"

Collector receivers

The collector receives OTLP data from applications:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Defining a receiver or processor does not activate it. A component is used only after it is added to a pipeline under service. The Collector configuration documentation is useful when a valid-looking component appears to do nothing.

Resource attributes

I add a stable deployment environment so metrics, logs, and traces can line up. Before Prometheus turns resource attributes into labels, I remove process details that are not needed for these dashboards while retaining the identity of each metric producer.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert

  resource/metrics:
    attributes:
      - key: process.pid
        action: delete
      - key: process.command_args
        action: delete

  batch:
    timeout: 5s

Keep service.instance.id, or another attribute set that uniquely identifies each producer. Removing it without aggregation can make replicas emit counters into the same series, violating the metrics single-writer principle. Use query-time aggregation for service totals; deleting identity labels is not aggregation. Ensure the application SDK supplies a distinct instance identity.

Request IDs, arbitrary URL path parameters, queried domain names, and client IPs belong in log bodies or trace attributes when needed, rather than unbounded metric or Loki labels. A fixed environment or a controlled set of service names is a different case.

Pipelines

The collector has separate pipelines for traces, metrics, and logs.

exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
    resource_to_telemetry_conversion:
      enabled: true
  otlp_grpc/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp_http/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, tail_sampling, batch]
      exporters: [otlp_grpc/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, resource/metrics, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http/loki]

The Tempo and Loki endpoints are inside the Docker network, so plain internal service names are enough for this compose stack.

Keep errors and slow traces with tail sampling

Storing every trace is unnecessary in this home environment, but keeping only a random sample makes failures easy to miss. The policy below selects traces containing an ERROR span, traces slower than one second, and a 10 percent baseline of the remaining traffic that reaches the Collector:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 5000
    expected_new_traces_per_sec: 20
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

The decision uses spans available within the configured wait. Head sampling, late spans, buffer limits, or delivery failures can prevent an error trace from being retained; this is a selection policy, not a guarantee of complete capture. See OpenTelemetry sampling.

Tail sampling waits for spans before deciding whether to retain a trace, so it uses memory and needs capacity planning. Merge tail_sampling into the earlier processors mapping rather than creating a second top-level key. If this Collector is later scaled to multiple replicas, spans from one trace must reach a consistent sampling tier; the Collector scaling guide explains why simply adding replicas can produce incomplete decisions.

Collect Pod stdout with Alloy

Pod stdout belongs to a node-local collector. Argo CD installs Grafana Alloy as a DaemonSet, mounts the node’s /var/log, discovers Pods through the Kubernetes API, and keeps only targets scheduled on the same node.

The filter below requires HOSTNAME to contain the node name, not the Pod’s default hostname. Grafana’s Alloy Helm chart sets it from the Downward API field spec.nodeName; a custom DaemonSet must provide the same environment variable.

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    action        = "keep"
    regex         = sys.env("HOSTNAME")
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    separator     = "/"
    target_label  = "__path__"
    replacement   = "/var/log/pods/*$1/*.log"
  }

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
}

local.file_match "pod_logs" {
  path_targets = discovery.relabel.pod_logs.output
}

loki.source.file "pod_logs" {
  targets    = local.file_match.pod_logs.targets
  forward_to = [loki.process.pod_logs.receiver]
}

loki.process "pod_logs" {
  stage.cri {}
  stage.decolorize {}
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://<observability-host>:3100/loki/api/v1/push"
  }
}

Here, local.file_match expands the __path__ glob into files before loki.source.file reads them, following Grafana’s file discovery example. The relabel rules preserve namespace and container as queryable labels; discovery metadata beginning with __ does not become a Loki label automatically. The complete configuration also adds controlled app and job labels. Grafana’s Kubernetes log collection guide explains why a DaemonSet should be limited to the local node, while the loki.source.file reference documents file discovery and position tracking.

This path collects whatever a container writes to stdout or stderr. An application can also emit structured logs over OTLP, but enabling both paths for the same event creates duplicates. I decide the owner per log source instead of assuming that more collectors always produce better coverage.

Collect one host log without indexing unbounded fields

AdGuard Home writes its DNS query log to a host file rather than Kubernetes stdout. A separate Alloy container receives read-only access to that single file and stores its position in a named volume. It starts at the end of the existing file, so a restart continues from the saved position instead of backfilling the whole history.

The container drops all Linux capabilities except the one needed to traverse the root-owned log path. It has no Docker socket and runs with a read-only root filesystem. In Loki, query type and protocol are bounded labels; queried domain names and client IP addresses stay in the JSON body because indexing them would create unbounded cardinality.

App environment

From inside a Kubernetes pod, localhost means the pod itself. The OTLP endpoint must be a host reachable from the cluster.

For OTLP HTTP:

  • OTEL_SERVICE_NAME=example-api
  • OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.namespace=example
  • OTEL_EXPORTER_OTLP_ENDPOINT=http://otel.example.internal:4318
  • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  • OTEL_LOGS_EXPORTER=otlp
  • OTEL_METRICS_EXPORTER=otlp
  • OTEL_TRACES_EXPORTER=otlp

For OTLP gRPC:

  • OTEL_SERVICE_NAME=example-api
  • OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.namespace=example
  • OTEL_EXPORTER_OTLP_ENDPOINT=http://otel.example.internal:4317
  • OTEL_EXPORTER_OTLP_PROTOCOL=grpc
  • OTEL_LOGS_EXPORTER=otlp
  • OTEL_METRICS_EXPORTER=otlp
  • OTEL_TRACES_EXPORTER=otlp

If the app receives its .env from Vault and External Secrets, I add these values to Vault instead of committing them into Git.

The examples above use plain HTTP OTLP inside an internal network. If telemetry crosses an untrusted network, I would put TLS in front of the collector or use an OTLP endpoint that supports TLS directly.

Put limits on storage and dashboards

The Compose stack retains Loki logs and Tempo traces for 60 days. Prometheus retains metrics for 14 days or until its configured size limit is reached. A time limit alone does not protect the host from a sudden increase in series or log volume, so Prometheus also has a storage cap and every container has a resource limit.

Dashboards are provisioned from version-controlled JSON and provider files. Kubernetes dashboards query the in-cluster Prometheus data source, while application dashboards correlate Prometheus metrics, Loki logs, and Tempo traces. Keeping dashboards in Git makes a rebuilt Grafana useful without manually importing them again.

Start and check

After assembling and validating the complete Compose and backend configuration, start the stack:

cd opentelemetry
docker compose up -d

Check service readiness:

docker compose ps
curl http://localhost:9090/-/ready
curl http://localhost:3100/ready
curl http://localhost:3200/ready

Then verify data, not only service health:

  • In Prometheus, query up and confirm that application metric series are changing rather than merely present.
  • In Loki, use {service_name=~".+"} for OTLP logs and a namespace selector for Alloy-collected Pod logs.
  • In Tempo, search for service.name = example-api, then deliberately generate one error and one slow request to test the sampling policies.
  • In Kubernetes, confirm that one Alloy Pod runs on each expected node and that its logs show successful writes to Loki.

If Loki is ready but empty, check the query’s labels, then inspect the producer: Alloy discovery and host paths for Pod stdout, or the application’s OTLP exporter for structured logs. This narrows the missing segment instead of treating every empty panel as a dashboard problem.

Tempo config can change between major versions. If Tempo crash-loops after an upgrade, check the config shape before debugging Docker networking.

Host storage and network limits

The backend is still a single-host dependency. Docker volumes need backups, the cluster-to-host OTLP and Loki paths need firewall rules, and retention is not a substitute for capacity monitoring. Plain internal HTTP is also a choice for this network, not a general recommendation.

#kubernetes#cloud#opentelemetry#grafana