Workload · Scheduling data pipelines

Apache Airflow, self hosted on a server of your own

To run Apache Airflow self hosted, give it a Linux machine with at least 4 GB of memory for its containers (the project suggests 8 GB), a PostgreSQL or MySQL metadata database, and an executor that matches how many machines will run your tasks. The official Docker Compose file starts the scheduler, the API server and the web UI in a few minutes, but its own header says it is meant for local development. This page covers the handful of changes that make it safe to leave running.

The short version
  1. Floor from the docs: 4 GB for the containers, ideally 8 GB, 2 CPUs, 10 GB of disk
  2. Metadata database: PostgreSQL or MySQL, never SQLite or MariaDB
  3. One machine: LocalExecutor is the documented single-server choice
  4. Bind port 8080 to localhost and reach the UI over SSH

What Airflow asks of the machine

Airflow's published minimums are modest; what your own tasks do sets the real size. These are the figures the project states:

ItemWhat the Airflow docs sayWhere it comes from
Memory for the containersAt least 4 GB, ideally 8 GBDocker Compose quick start
ProcessorsAt least 2 CPUs recommendedResource check in the compose file's init step
Free diskAt least 10 GB recommendedThe same check
Operating systemLinux for production; Windows only through WSL2 or containersPrerequisites page
Python, if you skip Docker3.11, 3.12, 3.13 or 3.14Prerequisites page
Metadata databasePostgreSQL 14 to 18, or MySQL 8.0, 8.4 or InnovationDatabase setup page

The prerequisites page itself says the real requirement depends heavily on the deployment. On a single server, every task runs on the same box as the scheduler and the database, so add the memory your busiest overlapping tasks use: a task that loads a large file or runs a transformation does it right there. Task logs land in ./logs on the host and keep growing until you clean them up.

Pick the executor first, then the box

The executor decides where tasks run, so it decides whether you need one machine or several.

  • LocalExecutor runs tasks inside the scheduler process. It is Airflow's default, and the docs describe it for small production installs on one machine.
  • CeleryExecutor puts tasks on a central queue that workers pull from, so workers can sit on other machines. The compose file ships with it, using Redis as the queue, plus an optional Flower monitor on port 5555; it has no login, so if you turn it on, bind it to 127.0.0.1 and [::1] the same way as 8080.
  • KubernetesExecutor starts each task in its own pod, which means you need a cluster underneath.

On one dedicated server, Local has the fewest moving parts. Celery earns its extra services when you expect to add worker machines later. Airflow 3 dropped the hybrid LocalKubernetes and CeleryKubernetes executors, so older guides that recommend them no longer apply.

Why a whole machine suits a scheduler

A scheduler is only useful if its runs start on time, and that depends on memory and processor time nobody else is using. A dedicated server is one physical computer that runs your work and nobody else's: all of its cores, memory and disks are yours.

  • Root access, so you install Docker, pin the Airflow image you want and change any setting.
  • An unmetered port. Pipelines that pull exports from APIs or push files to storage never create a transfer bill, because there is no allowance to run over.
  • A console when SSH stops answering, on rack servers, blades and the m510 and m710x cartridges: the screen and the power button, even when the system will not boot.
  • Reinstalls from the control panel, as many as you like, each taking about ten minutes, for when a first attempt goes sideways.
  • Month to month. Try Airflow for a month and cancel when any month ends if it is not for you.

The server that fits

Every class on the dedicated servers page can run Airflow; the question is how much your DAGs do at once.

  • Compact cartridges suit one job: a learning install or a few light pipelines. Choose one with 8 GB or more so the containers have the headroom the docs suggest.
  • Everyday servers, one processor with plenty of headroom, cover most teams running Airflow and its database together.
  • Two-processor servers fit many heavy tasks in parallel, or Airflow beside a busy database on the same machine.
  • Disks matter for the metadata database and the log folder. If you want them on fast storage, look at NVMe dedicated servers.

A ready server is handed over within 30 minutes of payment approval; the payment and fraud checks before that are the one wait without a fixed figure.

Up and running in an evening

  1. Install Docker Engine with the Compose plugin, version 2.14.0 or newer, using Docker's guide for your distribution.
  2. Make a project folder, fetch the compose file and create the folders and .env file Airflow mounts (first block below). Every line runs only if the one before it worked.
  3. Before the first start, set your own secrets. Generate a Fernet key with docker run --rm apache/airflow:3.4.0 python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())", then open .env in an editor and add FERNET_KEY= with that key, AIRFLOW__API_AUTH__JWT_SECRET= with a long random string, and _AIRFLOW_WWW_USER_PASSWORD= with the admin password. Run chmod 600 .env afterwards. The init step creates the admin account on its first run, so without this it is created as airflow with the password airflow.
  4. Keep the web UI off the public internet. In docker-compose.yaml, replace the ports: entry of airflow-apiserver (the ports: line and the "8080:8080" line under it) with the second block. Docker publishes ports ahead of ufw rules, so a host firewall alone does not hide port 8080; binding it to localhost on both IPv4 and IPv6 does. While you are in the file, set AIRFLOW__CORE__LOAD_EXAMPLES to 'false' if you do not want the example DAGs.
  5. Initialise and start (third block). The first command migrates the database and creates the account; it should end with the init container exiting with code 0. Every Airflow service in the file waits for that init to complete successfully, so the second command will not start Airflow on top of a failed one.
  6. From your own computer, open a tunnel with ssh -L 8080:localhost:8080 you@your-server and browse to http://localhost:8080. Sign in with the password you set.
mkdir -p ~/airflow && cd ~/airflow && \
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.4.0/docker-compose.yaml' && \
mkdir -p ./dags ./logs ./plugins ./config && \
echo -e "AIRFLOW_UID=$(id -u)" > .env
    ports:
      - "127.0.0.1:8080:8080"
      - "[::1]:8080:8080"
docker compose up airflow-init
docker compose up -d

The containers read .env only when they are created. After you change it, run docker compose up -d --force-recreate, or the running services keep the old values. Avoid the cleanup command in the quick start, docker compose down --volumes --remove-orphans, on a server you care about: it deletes the PostgreSQL volume and every run history with it.

Keeping it healthy

  • Back up the metadata database. docker compose exec -T postgres pg_dump -U airflow airflow > airflow-$(date +%F).sql writes a dated dump; copy it off the server.
  • Keep a copy of .env. Connection and variable passwords are encrypted with the Fernet key, and the docs say they cannot be read without it.
  • Trim old history. The database docs point to airflow db clean for pruning old records before the tables grow large.
  • Watch the health endpoint. The compose file probes /api/v2/monitor/health on port 8080; the production guide asks for a check that notices when the scheduler stops sending heartbeats.
  • Add Python packages properly. The compose file says _PIP_ADDITIONAL_REQUIREMENTS is only for quick checks, and the image docs add that it reinstalls the packages every time a container starts; build an extended image instead.
  • Upgrade with a dump in hand. Take a fresh dump, test the new image tag on a copy, then switch. The init step runs the schema migration, which the docs say is safe to repeat.

Tracking experiments as well as pipelines? Our guide to a self-hosted MLflow tracking server can share the same machine.

When one server is the wrong answer

Be honest with yourself about three limits before you order.

  • The compose file is not a production design. The quick start says it gives none of the security guarantees a production system needs and points to the official Helm chart on Kubernetes. The steps above close the obvious holes; they do not add high availability.
  • One machine is one point of failure. If the scheduler and its database live on one server, a hardware fault pauses every pipeline until it is fixed. Teams that cannot wait run Airflow across several machines, often on Kubernetes: managed Kubernetes on bare metal in New York, or Kubernetes you run yourself.
  • You run Airflow; we run the hardware. We keep the machine and the network working. Your DAGs, upgrades and the database are yours to look after.

Questions people ask

Is Airflow's Docker Compose file good enough for production?
Not as shipped. Its header says it is for local development, it starts with a default airflow/airflow login and a placeholder JWT secret, and it publishes port 8080 on every interface. Setting your own secrets and binding the UI to localhost fixes the worst of it, but the project still recommends Kubernetes with its Helm chart for production.
Can I keep the metadata in SQLite or MariaDB?
No. The database docs say SQLite is for development only and should never be used in production, and that MariaDB is not supported. Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or Innovation. For PostgreSQL in production, the docs also recommend PgBouncer in front, because Airflow opens many connections.
How much memory does a self-hosted Airflow need?
The quick start asks for at least 4 GB for the Docker containers and suggests 8 GB. That covers Airflow itself, not your work: every task that runs on the same server adds its own memory, so size for the heaviest tasks that overlap.
Should I use LocalExecutor or CeleryExecutor on one server?
On a single machine, LocalExecutor is the simpler fit and the one the docs describe for small single-machine production installs. CeleryExecutor, which the compose file uses, adds Redis and a worker service; it pays off when you plan to add worker machines later.
How soon can the server be ready?
A ready server is handed over within 30 minutes of payment approval, with your chosen operating system installed and the login sent by email. The approval checks come first, and an order held for an identity check waits longest. Installing Docker and Airflow afterwards is the evening described above.

Get a server for your Airflow pipelines

A whole machine with root access and an unmetered port, handed over within 30 minutes of approval, with no contract.

Choose your server

Read next