What you are setting up
This is the shared place where every training run lands: its parameters, its metric curves, the model it produced. Training scripts set MLFLOW_TRACKING_URI to your server and log over HTTP; your team opens the same address in a browser to compare runs.
- The server (
mlflow server) serves the web UI and the API, on 127.0.0.1 port 5000 unless told otherwise. - The backend store is the database for run details and the model registry. MLflow switched its default to SQLite in 3.7.0; you will use PostgreSQL.
- The artifact store is where files go: weights, plots, datasets. Uploads pass through the server by default, so your scripts never need storage keys.
- The login app (
basic-auth) keeps users and permissions, here in that same PostgreSQL database.
Without the login app, whoever reaches the port can read, edit and delete every run.
Will one machine be enough?
For one team, usually yes: the tracking server trains nothing, it keeps what training produces. MLflow's self-hosting docs give no hardware minimums, so plan around what you log.
| What you log | Where it ends up | What to watch |
|---|---|---|
| Params, metrics, tags | PostgreSQL rows | Many small writes; quick drives keep the UI snappy |
| Weights, checkpoints, images | The artifact folder | Disk space, above all |
| Runs you delete | Still on disk | Space returns only after mlflow gc |
Fine-tuning large language models? Checkpoints will decide your drive size.
The server that fits
An NVMe dedicated server suits this job because the database and the model files both sit on drives wired to the processor over PCIe, and nobody else shares the machine. Every NVMe build is an HPE Moonshot cartridge on a 1 Gbps port that nothing meters, so pulling old checkpoints back to a GPU box adds nothing to your bill.
- Xeon D-1500: 8 or 16 cores, memory up to 128 GB.
- Xeon E3-1585L v5: four cores, memory up to 64 GB.
- Xeon E-2286M: eight cores, memory up to 64 GB, built in Bucharest.
- Drives each between 240 GB and 3.84 TB. A pair in RAID 1 turns a dead drive into a swap instead of lost runs; keep backups anyway.
- How fast: a racked machine is yours about 30 minutes after your order is approved; one you configure needs 4 to 24 hours if the parts are here, longer if a part must be ordered.
Expecting more model files than one drive holds? Keep MLflow here and send artifacts to an S3 bucket on your own MinIO server, pointing MLFLOW_S3_ENDPOINT_URL at it as the artifact store docs describe.
Step 1: install MLflow and PostgreSQL
As root, on a fresh Ubuntu 24.04 server, paste this. It switches the firewall on with SSH alone, installs PostgreSQL and MLflow (in its own virtualenv, run by an mlflow user) and makes three random secrets, among them your first admin password. They land in files readable by root and the mlflow group alone, and the main file appears last, so a second paste stops at line one.
cd /tmp && C=/etc/mlflow && if [ -e $C/server.env ]; then echo "STOP - installed already"; false; fi &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
apt-get update && apt-get install -y python3-venv postgresql curl &&
{ getent passwd mlflow >/dev/null || useradd -r -m -d /var/lib/mlflow -s /usr/sbin/nologin mlflow; } &&
install -d -o mlflow -g mlflow -m 750 /var/lib/mlflow/artifacts &&
python3 -m venv /opt/mlflow/venv && /opt/mlflow/venv/bin/pip install 'mlflow[auth]==3.17.2' psycopg2-binary &&
PW=$(openssl rand -hex 24) && CSRF=$(openssl rand -hex 32) && FIRST=$(openssl rand -hex 16) &&
[ ${#PW} = 48 ] && [ ${#CSRF} = 64 ] && [ ${#FIRST} = 32 ] &&
sudo -u postgres createuser mlflow &&
printf "ALTER ROLE mlflow PASSWORD '%s';\n" "$PW" | sudo -u postgres psql -v ON_ERROR_STOP=1 &&
sudo -u postgres createdb --owner=mlflow mlflow &&
URI="postgresql://mlflow:$PW@127.0.0.1:5432/mlflow" && install -d -m 750 -g mlflow $C &&
install -m 640 -g mlflow /dev/null $C/auth.ini && install -m 640 -g mlflow /dev/null $C/server.env.tmp &&
printf '[mlflow]\ndefault_permission = READ\ndatabase_uri = %s\nadmin_username = admin\n' "$URI" > $C/auth.ini &&
printf 'MLFLOW_BACKEND_STORE_URI=%s\nMLFLOW_FLASK_SERVER_SECRET_KEY=%s\nMLFLOW_AUTH_CONFIG_PATH=%s/auth.ini\nMLFLOW_AUTH_ADMIN_PASSWORD=%s\n' \
"$URI" "$CSRF" "$C" "$FIRST" > $C/server.env.tmp &&
mv $C/server.env.tmp $C/server.env && echo "Done - admin password $FIRST" || echo 'STOP - see above'
Copy the printed admin password into your password manager. If the paste stops part way, nothing uses the secrets yet: open a postgres shell with sudo -iu postgres, run dropdb --if-exists mlflow and dropuser --if-exists mlflow, exit, and paste again.
Step 2: start it with a login in front
MLflow 3.17.2 has no default admin password. First started with --app-name basic-auth, it reads MLFLOW_AUTH_ADMIN_PASSWORD to create the admin user, and it will not start if that, or MLFLOW_FLASK_SERVER_SECRET_KEY, is missing (mlflow/server/auth/__init__.py). Passwords need at least 12 characters.
cat > /etc/systemd/system/mlflow.service <<'EOF'
[Unit]
Description=MLflow tracking server
Requires=postgresql.service
After=postgresql.service network-online.target
[Service]
User=mlflow
Group=mlflow
WorkingDirectory=/var/lib/mlflow
EnvironmentFile=/etc/mlflow/server.env
ExecStart=/opt/mlflow/venv/bin/mlflow server --app-name basic-auth --host 127.0.0.1 --port 5000 --artifacts-destination /var/lib/mlflow/artifacts
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now mlflow &&
up=no; for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && { up=yes; break; }; sleep 2; done
[ "$up" = yes ] &&
code=$(curl -s -o /dev/null -w '%{http_code}' 'http://127.0.0.1:5000/api/2.0/mlflow/experiments/search?max_results=1') &&
[ "$code" = 401 ] && sed -i '/^MLFLOW_AUTH_ADMIN_PASSWORD=/d' /etc/mlflow/server.env &&
! grep -q ADMIN_PASSWORD /etc/mlflow/server.env && systemctl restart mlflow &&
sleep 5 && for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && break; sleep 2; done &&
curl -fs http://127.0.0.1:5000/health >/dev/null && echo "Login is on and the first-start password is gone" || echo 'STOP - look at journalctl -u mlflow -n 50'
This waits for /health, confirms the API answers 401 when you send no password, and only then strips the first-start password from the env file and restarts. The docs say to unset it after the first start; deleting the line also takes it off the disk. Runs, users and permissions share one database, which the docs confirm is safe because the login tables track their own migrations.
Step 3: let your team in without opening the port
Tunnel over SSH with your own account first: ssh -N -L 15000:127.0.0.1:5000 you@server.example.com. Visit http://localhost:15000 and log in as admin. Create accounts for colleagues on the /signup page, and set a fresh admin password through the auth client's update_user_password().
New users can read every experiment, as default_permission is READ; to change that, set NO_PERMISSIONS in /etc/mlflow/auth.ini and restart.
For access without tunnels, put HTTPS in front with Caddy:
- Point a DNS name such as
mlflow.example.comat the server. - Install Caddy with
apt-get -y install caddy(Ubuntu 24.04 carries it in universe). Open the web ports before it asks for a certificate:ufw allow 80,443/tcp. - Write
/etc/caddy/Caddyfilewith a site block formlflow.example.comholdingreverse_proxy 127.0.0.1:5000, thensystemctl reload caddy. - MLflow's DNS-rebinding guard accepts only localhost and private addresses unless told otherwise, and its CORS guard refuses changes sent from a web origin it does not know (your browser sends one even when you save in MLflow's own UI). Put your name in
H=mlflow.example.com, then allow both:[[ $H && $H != mlflow.example.com ]] && printf 'MLFLOW_SERVER_ALLOWED_HOSTS=%s\nMLFLOW_SERVER_CORS_ALLOWED_ORIGINS=%s\n' "localhost:*,127.0.0.1:*,$H" "https://$H" >> /etc/mlflow/server.env && echo written || echo STOP - check H. After written,systemctl restart mlflow. - Point clients at
https://mlflow.example.com. Leave port 5000 closed; the login sends a password on every request, so keep it inside HTTPS.
Keeping your runs safe
Your history lives in two places, the PostgreSQL database and the artifact folder, and this nightly job saves both. It dumps the database and reads the dump back, archives the files and lists them, and prunes sets past seven days once both succeed. Tar may exit with code 1 when an upload changes a file mid-read; the job allows that one case and still checks the archive.
install -d -m 700 /var/backups/mlflow &&
cat > /usr/local/sbin/mlflow-backup <<'EOF'
out=/var/backups/mlflow; stamp=$(date +%F-%H%M%S)
if ! { sudo -u postgres pg_dump --format=custom mlflow > "$out/mlflow-db-$stamp.dump" &&
pg_restore --list "$out/mlflow-db-$stamp.dump" > /dev/null; }; then
echo "STOP - database dump not usable, see $out"; exit 1; fi
tar -czf "$out/mlflow-files-$stamp.tar.gz" -C / var/lib/mlflow/artifacts etc/mlflow etc/systemd/system/mlflow.service
rc=$?
if [ $rc -gt 1 ] || ! tar -tzf "$out/mlflow-files-$stamp.tar.gz" > /dev/null; then
echo "STOP - file archive not usable, see $out"; exit 1; fi
find "$out" -name 'mlflow-*' -mtime +7 -delete
echo "backup ok $stamp"
EOF
printf '40 2 * * * root umask 077; bash /usr/local/sbin/mlflow-backup >> /var/log/mlflow-backup.log 2>&1\n' > /etc/cron.d/mlflow-backup &&
(umask 077; bash /usr/local/sbin/mlflow-backup)
Pull the backup folder to a second machine with rsync -a over SSH, using a key that may read this root-only folder, and skip --delete, so a wiped server cannot empty your copies. Restoring the set stamped STAMP, as root:
- Check the set before anything stops:
rm -f /run/m1 /run/m2; b=/var/backups/mlflow; pg_restore --list $b/mlflow-db-STAMP.dump >/dev/null && tar -tzf $b/mlflow-files-STAMP.tar.gz var/lib/mlflow/artifacts >/dev/null && touch /run/m1 && echo checked || echo STOP - the set is damaged. Then check for leftovers:rm /run/m1 && [ ! -e /var/lib/mlflow/artifacts.old ] && [ -z "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && touch /run/m2 && echo free || echo STOP - earlier restore leftovers. Then stop MLflow and move today's state aside:rm /run/m2 && systemctl stop mlflow && echo 'ALTER DATABASE mlflow RENAME TO previous_mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.old && echo moved || echo STOP - not fully moved. Each needs the marker the last one left in/run. After a STOP from the first two, MLflow is still running: clear the cause and start over. After one from the third, see what moved first. - Load the database with
sudo -u postgres createdb --owner=mlflow mlflow, thensudo -u postgres pg_restore --exit-on-error --role=mlflow --no-owner --dbname=mlflow < /var/backups/mlflow/mlflow-db-STAMP.dump. - Bring back the files with
[ ! -e /var/lib/mlflow/artifacts ] && tar -xzf /var/backups/mlflow/mlflow-files-STAMP.tar.gz -C / var/lib/mlflow/artifacts && chown -R mlflow: /var/lib/mlflow/artifacts && echo unpacked || echo STOP - artifacts folder still there, do not start MLflow. Only after unpacked prints, runsystemctl start mlflowand log in to check. - Changed your mind? Check first:
rm -f /run/r0 /run/r1; [ -d /var/lib/mlflow/artifacts.old ] && [ ! -e /var/lib/mlflow/artifacts.restored ] && touch /run/r0 && echo found || echo STOP - check artifacts.old and artifacts.restored. Then check the old database:rm /run/r0 && [ -n "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && touch /run/r1 && echo ready || echo STOP - no previous_mlflow database. After ready, set the restore aside:rm /run/r1 && systemctl stop mlflow && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.restored && sudo -u postgres dropdb mlflow && touch /run/r2 && echo dropped || echo STOP - not set aside. After dropped, bring the old state back:rm /run/r2 && echo 'ALTER DATABASE previous_mlflow RENAME TO mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts.old /var/lib/mlflow/artifacts && systemctl start mlflow && echo rolled back || echo STOP - not done. A STOP after ready can mean a half-done rollback: see what moved. Happy?rm -f /run/r1; sudo -u postgres dropdb previous_mlflowand removeartifacts.old(andartifacts.restoredif a rollback left one). - Moving to a new server? Do steps 1 and 2 there, then this list. Taking only the artifact folder from the archive keeps the new server's own database password. The restored database brings its users, so log in with the old server's accounts, not the admin password the new server printed.
Updating MLflow without losing data
A new release can change the database layout, and the server will not start until you migrate it. MLflow's CLI docs warn that migrations can run long and may not be transactional, so every upgrade goes: back up, stop, upgrade, migrate, start.
- Run
bash /usr/local/sbin/mlflow-backupand wait forbackup ok. - Read the release notes, then
systemctl stop mlflow && /opt/mlflow/venv/bin/pip install --upgrade 'mlflow[auth]==X.Y.Z'. - Run the schema migration as the mlflow user with
sudo -u mlflow sh -c '. /etc/mlflow/server.env; u=$MLFLOW_BACKEND_STORE_URI; /opt/mlflow/venv/bin/mlflow db upgrade "$u"'. Because the URI is a command argument,psshows it while the command runs. systemctl start mlflow. The login tables update themselves at startup (sqlalchemy_store.py).- Something broke? Put the previous version back with pip and restore the backup from the first step.
The licence, and when to choose something else
MLflow's licence is Apache 2.0 (see LICENSE.txt). As we read it, in general terms and not as legal advice: use it, change it and build paid services on it; when you pass copies on, include the licence and notices and mark changed files. You need not release your own code.
- One server can fail, and this setup has no standby.
- The repo's
docker-compose/folder runs MLflow, PostgreSQL and RustFS together, butdocker-compose.ymlpublishes their ports on every address with no login app, and.env.dev.exampleships stock passwords. Treat it as a laptop setup. - Training belongs on GPU hardware: a Jupyter GPU server logs into this one. If the database becomes the bottleneck, move it to its own PostgreSQL server.
Questions people ask
- How do my notebooks start logging to it?
- Set
MLFLOW_TRACKING_URIto your server's address and export your login asMLFLOW_TRACKING_USERNAMEandMLFLOW_TRACKING_PASSWORD, or callmlflow.set_tracking_uri()in code.mlflow.start_run()and the logging calls then go to your server. - What happens to the runs in my old mlruns folder?
mlflow migrate-filestore, in 3.10 and later, copies a file store into a new, empty SQLite database without changing IDs or timestamps. It cannot target PostgreSQL, because file-store experiment IDs are larger than PostgreSQL's 32-bit limit. Your artifact files stay where they are.- Can teammates see each other's experiments?
- By default, yes: everyone gets READ. With
default_permission = NO_PERMISSIONSpeople see only what they create or are granted; an experiment's creator gets MANAGE on it and can share it.