What Airflow asks of the machine
Airflow's published minimums are modest; what your own tasks do sets the real size. These are the figures the project states:
| Item | What the Airflow docs say | Where it comes from |
|---|---|---|
| Memory for the containers | At least 4 GB, ideally 8 GB | Docker Compose quick start |
| Processors | At least 2 CPUs recommended | Resource check in the compose file's init step |
| Free disk | At least 10 GB recommended | The same check |
| Operating system | Linux for production; Windows only through WSL2 or containers | Prerequisites page |
| Python, if you skip Docker | 3.11, 3.12, 3.13 or 3.14 | Prerequisites page |
| Metadata database | PostgreSQL 14 to 18, or MySQL 8.0, 8.4 or Innovation | Database setup page |
The prerequisites page itself says the real requirement depends heavily on the deployment. On a single server, every task runs on the same box as the scheduler and the database, so add the memory your busiest overlapping tasks use: a task that loads a large file or runs a transformation does it right there. Task logs land in ./logs on the host and keep growing until you clean them up.
Pick the executor first, then the box
The executor decides where tasks run, so it decides whether you need one machine or several.
- LocalExecutor runs tasks inside the scheduler process. It is Airflow's default, and the docs describe it for small production installs on one machine.
- CeleryExecutor puts tasks on a central queue that workers pull from, so workers can sit on other machines. The compose file ships with it, using Redis as the queue, plus an optional Flower monitor on port 5555; it has no login, so if you turn it on, bind it to 127.0.0.1 and [::1] the same way as 8080.
- KubernetesExecutor starts each task in its own pod, which means you need a cluster underneath.
On one dedicated server, Local has the fewest moving parts. Celery earns its extra services when you expect to add worker machines later. Airflow 3 dropped the hybrid LocalKubernetes and CeleryKubernetes executors, so older guides that recommend them no longer apply.
Why a whole machine suits a scheduler
A scheduler is only useful if its runs start on time, and that depends on memory and processor time nobody else is using. A dedicated server is one physical computer that runs your work and nobody else's: all of its cores, memory and disks are yours.
- Root access, so you install Docker, pin the Airflow image you want and change any setting.
- An unmetered port. Pipelines that pull exports from APIs or push files to storage never create a transfer bill, because there is no allowance to run over.
- A console when SSH stops answering, on rack servers, blades and the m510 and m710x cartridges: the screen and the power button, even when the system will not boot.
- Reinstalls from the control panel, as many as you like, each taking about ten minutes, for when a first attempt goes sideways.
- Month to month. Try Airflow for a month and cancel when any month ends if it is not for you.
The server that fits
Every class on the dedicated servers page can run Airflow; the question is how much your DAGs do at once.
- Compact cartridges suit one job: a learning install or a few light pipelines. Choose one with 8 GB or more so the containers have the headroom the docs suggest.
- Everyday servers, one processor with plenty of headroom, cover most teams running Airflow and its database together.
- Two-processor servers fit many heavy tasks in parallel, or Airflow beside a busy database on the same machine.
- Disks matter for the metadata database and the log folder. If you want them on fast storage, look at NVMe dedicated servers.
A ready server is handed over within 30 minutes of payment approval; the payment and fraud checks before that are the one wait without a fixed figure.
Up and running in an evening
- Install Docker Engine with the Compose plugin, version 2.14.0 or newer, using Docker's guide for your distribution.
- Make a project folder, fetch the compose file and create the folders and
.envfile Airflow mounts (first block below). Every line runs only if the one before it worked. - Before the first start, set your own secrets. Generate a Fernet key with
docker run --rm apache/airflow:3.4.0 python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())", then open.envin an editor and addFERNET_KEY=with that key,AIRFLOW__API_AUTH__JWT_SECRET=with a long random string, and_AIRFLOW_WWW_USER_PASSWORD=with the admin password. Runchmod 600 .envafterwards. The init step creates the admin account on its first run, so without this it is created as airflow with the password airflow. - Keep the web UI off the public internet. In
docker-compose.yaml, replace theports:entry ofairflow-apiserver(theports:line and the"8080:8080"line under it) with the second block. Docker publishes ports ahead of ufw rules, so a host firewall alone does not hide port 8080; binding it to localhost on both IPv4 and IPv6 does. While you are in the file, setAIRFLOW__CORE__LOAD_EXAMPLESto'false'if you do not want the example DAGs. - Initialise and start (third block). The first command migrates the database and creates the account; it should end with the init container exiting with code 0. Every Airflow service in the file waits for that init to complete successfully, so the second command will not start Airflow on top of a failed one.
- From your own computer, open a tunnel with
ssh -L 8080:localhost:8080 you@your-serverand browse to http://localhost:8080. Sign in with the password you set.
mkdir -p ~/airflow && cd ~/airflow && \
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.4.0/docker-compose.yaml' && \
mkdir -p ./dags ./logs ./plugins ./config && \
echo -e "AIRFLOW_UID=$(id -u)" > .env
ports:
- "127.0.0.1:8080:8080"
- "[::1]:8080:8080"
docker compose up airflow-init
docker compose up -d
The containers read .env only when they are created. After you change it, run docker compose up -d --force-recreate, or the running services keep the old values. Avoid the cleanup command in the quick start, docker compose down --volumes --remove-orphans, on a server you care about: it deletes the PostgreSQL volume and every run history with it.
Keeping it healthy
- Back up the metadata database.
docker compose exec -T postgres pg_dump -U airflow airflow > airflow-$(date +%F).sqlwrites a dated dump; copy it off the server. - Keep a copy of
.env. Connection and variable passwords are encrypted with the Fernet key, and the docs say they cannot be read without it. - Trim old history. The database docs point to
airflow db cleanfor pruning old records before the tables grow large. - Watch the health endpoint. The compose file probes
/api/v2/monitor/healthon port 8080; the production guide asks for a check that notices when the scheduler stops sending heartbeats. - Add Python packages properly. The compose file says
_PIP_ADDITIONAL_REQUIREMENTSis only for quick checks, and the image docs add that it reinstalls the packages every time a container starts; build an extended image instead. - Upgrade with a dump in hand. Take a fresh dump, test the new image tag on a copy, then switch. The init step runs the schema migration, which the docs say is safe to repeat.
Tracking experiments as well as pipelines? Our guide to a self-hosted MLflow tracking server can share the same machine.
When one server is the wrong answer
Be honest with yourself about three limits before you order.
- The compose file is not a production design. The quick start says it gives none of the security guarantees a production system needs and points to the official Helm chart on Kubernetes. The steps above close the obvious holes; they do not add high availability.
- One machine is one point of failure. If the scheduler and its database live on one server, a hardware fault pauses every pipeline until it is fixed. Teams that cannot wait run Airflow across several machines, often on Kubernetes: managed Kubernetes on bare metal in New York, or Kubernetes you run yourself.
- You run Airflow; we run the hardware. We keep the machine and the network working. Your DAGs, upgrades and the database are yours to look after.
Questions people ask
- Is Airflow's Docker Compose file good enough for production?
- Not as shipped. Its header says it is for local development, it starts with a default airflow/airflow login and a placeholder JWT secret, and it publishes port 8080 on every interface. Setting your own secrets and binding the UI to localhost fixes the worst of it, but the project still recommends Kubernetes with its Helm chart for production.
- Can I keep the metadata in SQLite or MariaDB?
- No. The database docs say SQLite is for development only and should never be used in production, and that MariaDB is not supported. Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or Innovation. For PostgreSQL in production, the docs also recommend PgBouncer in front, because Airflow opens many connections.
- How much memory does a self-hosted Airflow need?
- The quick start asks for at least 4 GB for the Docker containers and suggests 8 GB. That covers Airflow itself, not your work: every task that runs on the same server adds its own memory, so size for the heaviest tasks that overlap.
- Should I use LocalExecutor or CeleryExecutor on one server?
- On a single machine, LocalExecutor is the simpler fit and the one the docs describe for small single-machine production installs. CeleryExecutor, which the compose file uses, adds Redis and a worker service; it pays off when you plan to add worker machines later.
- How soon can the server be ready?
- A ready server is handed over within 30 minutes of payment approval, with your chosen operating system installed and the login sent by email. The approval checks come first, and an order held for an identity check waits longest. Installing Docker and Airflow afterwards is the evening described above.