cloud

Deploy Apache Airflow on Kubernetes

A short deployment note for running Apache Airflow on top of a Kubernetes cluster.

Deploy Apache Airflow on Kubernetes

Apache Airflow is useful when you need to orchestrate scheduled data pipelines, operational tasks, or repeatable workflows. Running it on Kubernetes keeps the scheduler, webserver, workers, and supporting services easier to scale and isolate.

This post is a placeholder for my Kubernetes Airflow deployment notes. The key pieces to document are:

  1. Choosing an executor that matches the cluster size.
  2. Keeping DAGs, logs, and metadata storage persistent.
  3. Separating webserver, scheduler, and worker resources.
  4. Managing secrets and connection strings outside the DAG code.

I will expand this into a full deployment walkthrough with manifests, storage choices, and operational notes.