Workload · Tracking ML experiments

Run your own MLflow tracking server, with a login and backups

Self-hosting an MLflow tracking server means running mlflow server somewhere your whole team can reach, with a database for run details, a folder for model files and a login in front. As shipped it has no login, which is fine on a laptop and risky anywhere else. Here you build it on one NVMe server with PostgreSQL, the built-in basic-auth app and a private port, then add HTTPS, nightly backups and safe upgrades, all for MLflow 3.17.2 on Ubuntu 24.04.

The short version
  1. One address for every run, metric and model your team logs
  2. PostgreSQL holds run details; model files sit on NVMe
  3. Login from the first start, with no default admin password
  4. Never listens publicly: SSH tunnel, or HTTPS through Caddy
  5. Checked nightly backups you can actually restore

What you are setting up

This is the shared place where every training run lands: its parameters, its metric curves, the model it produced. Training scripts set MLFLOW_TRACKING_URI to your server and log over HTTP; your team opens the same address in a browser to compare runs.

  • The server (mlflow server) serves the web UI and the API, on 127.0.0.1 port 5000 unless told otherwise.
  • The backend store is the database for run details and the model registry. MLflow switched its default to SQLite in 3.7.0; you will use PostgreSQL.
  • The artifact store is where files go: weights, plots, datasets. Uploads pass through the server by default, so your scripts never need storage keys.
  • The login app (basic-auth) keeps users and permissions, here in that same PostgreSQL database.

Without the login app, whoever reaches the port can read, edit and delete every run.

Will one machine be enough?

For one team, usually yes: the tracking server trains nothing, it keeps what training produces. MLflow's self-hosting docs give no hardware minimums, so plan around what you log.

What you logWhere it ends upWhat to watch
Params, metrics, tagsPostgreSQL rowsMany small writes; quick drives keep the UI snappy
Weights, checkpoints, imagesThe artifact folderDisk space, above all
Runs you deleteStill on diskSpace returns only after mlflow gc

Fine-tuning large language models? Checkpoints will decide your drive size.

The server that fits

An NVMe dedicated server suits this job because the database and the model files both sit on drives wired to the processor over PCIe, and nobody else shares the machine. Every NVMe build is an HPE Moonshot cartridge on a 1 Gbps port that nothing meters, so pulling old checkpoints back to a GPU box adds nothing to your bill.

  • Xeon D-1500: 8 or 16 cores, memory up to 128 GB.
  • Xeon E3-1585L v5: four cores, memory up to 64 GB.
  • Xeon E-2286M: eight cores, memory up to 64 GB, built in Bucharest.
  • Drives each between 240 GB and 3.84 TB. A pair in RAID 1 turns a dead drive into a swap instead of lost runs; keep backups anyway.
  • How fast: a racked machine is yours about 30 minutes after your order is approved; one you configure needs 4 to 24 hours if the parts are here, longer if a part must be ordered.

Expecting more model files than one drive holds? Keep MLflow here and send artifacts to an S3 bucket on your own MinIO server, pointing MLFLOW_S3_ENDPOINT_URL at it as the artifact store docs describe.

Step 1: install MLflow and PostgreSQL

As root, on a fresh Ubuntu 24.04 server, paste this. It switches the firewall on with SSH alone, installs PostgreSQL and MLflow (in its own virtualenv, run by an mlflow user) and makes three random secrets, among them your first admin password. They land in files readable by root and the mlflow group alone, and the main file appears last, so a second paste stops at line one.

cd /tmp && C=/etc/mlflow && if [ -e $C/server.env ]; then echo "STOP - installed already"; false; fi &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
apt-get update && apt-get install -y python3-venv postgresql curl &&
{ getent passwd mlflow >/dev/null || useradd -r -m -d /var/lib/mlflow -s /usr/sbin/nologin mlflow; } &&
install -d -o mlflow -g mlflow -m 750 /var/lib/mlflow/artifacts &&
python3 -m venv /opt/mlflow/venv && /opt/mlflow/venv/bin/pip install 'mlflow[auth]==3.17.2' psycopg2-binary &&
PW=$(openssl rand -hex 24) && CSRF=$(openssl rand -hex 32) && FIRST=$(openssl rand -hex 16) &&
[ ${#PW} = 48 ] && [ ${#CSRF} = 64 ] && [ ${#FIRST} = 32 ] &&
sudo -u postgres createuser mlflow &&
printf "ALTER ROLE mlflow PASSWORD '%s';\n" "$PW" | sudo -u postgres psql -v ON_ERROR_STOP=1 &&
sudo -u postgres createdb --owner=mlflow mlflow &&
URI="postgresql://mlflow:$PW@127.0.0.1:5432/mlflow" && install -d -m 750 -g mlflow $C &&
install -m 640 -g mlflow /dev/null $C/auth.ini && install -m 640 -g mlflow /dev/null $C/server.env.tmp &&
printf '[mlflow]\ndefault_permission = READ\ndatabase_uri = %s\nadmin_username = admin\n' "$URI" > $C/auth.ini &&
printf 'MLFLOW_BACKEND_STORE_URI=%s\nMLFLOW_FLASK_SERVER_SECRET_KEY=%s\nMLFLOW_AUTH_CONFIG_PATH=%s/auth.ini\nMLFLOW_AUTH_ADMIN_PASSWORD=%s\n' \
  "$URI" "$CSRF" "$C" "$FIRST" > $C/server.env.tmp &&
mv $C/server.env.tmp $C/server.env && echo "Done - admin password $FIRST" || echo 'STOP - see above'

Copy the printed admin password into your password manager. If the paste stops part way, nothing uses the secrets yet: open a postgres shell with sudo -iu postgres, run dropdb --if-exists mlflow and dropuser --if-exists mlflow, exit, and paste again.

Step 2: start it with a login in front

MLflow 3.17.2 has no default admin password. First started with --app-name basic-auth, it reads MLFLOW_AUTH_ADMIN_PASSWORD to create the admin user, and it will not start if that, or MLFLOW_FLASK_SERVER_SECRET_KEY, is missing (mlflow/server/auth/__init__.py). Passwords need at least 12 characters.

cat > /etc/systemd/system/mlflow.service <<'EOF'
[Unit]
Description=MLflow tracking server
Requires=postgresql.service
After=postgresql.service network-online.target

[Service]
User=mlflow
Group=mlflow
WorkingDirectory=/var/lib/mlflow
EnvironmentFile=/etc/mlflow/server.env
ExecStart=/opt/mlflow/venv/bin/mlflow server --app-name basic-auth --host 127.0.0.1 --port 5000 --artifacts-destination /var/lib/mlflow/artifacts
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now mlflow &&
up=no; for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && { up=yes; break; }; sleep 2; done
[ "$up" = yes ] &&
code=$(curl -s -o /dev/null -w '%{http_code}' 'http://127.0.0.1:5000/api/2.0/mlflow/experiments/search?max_results=1') &&
[ "$code" = 401 ] && sed -i '/^MLFLOW_AUTH_ADMIN_PASSWORD=/d' /etc/mlflow/server.env &&
! grep -q ADMIN_PASSWORD /etc/mlflow/server.env && systemctl restart mlflow &&
sleep 5 && for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && break; sleep 2; done &&
curl -fs http://127.0.0.1:5000/health >/dev/null && echo "Login is on and the first-start password is gone" || echo 'STOP - look at journalctl -u mlflow -n 50'

This waits for /health, confirms the API answers 401 when you send no password, and only then strips the first-start password from the env file and restarts. The docs say to unset it after the first start; deleting the line also takes it off the disk. Runs, users and permissions share one database, which the docs confirm is safe because the login tables track their own migrations.

Step 3: let your team in without opening the port

Tunnel over SSH with your own account first: ssh -N -L 15000:127.0.0.1:5000 you@server.example.com. Visit http://localhost:15000 and log in as admin. Create accounts for colleagues on the /signup page, and set a fresh admin password through the auth client's update_user_password().

New users can read every experiment, as default_permission is READ; to change that, set NO_PERMISSIONS in /etc/mlflow/auth.ini and restart.

For access without tunnels, put HTTPS in front with Caddy:

  1. Point a DNS name such as mlflow.example.com at the server.
  2. Install Caddy with apt-get -y install caddy (Ubuntu 24.04 carries it in universe). Open the web ports before it asks for a certificate: ufw allow 80,443/tcp.
  3. Write /etc/caddy/Caddyfile with a site block for mlflow.example.com holding reverse_proxy 127.0.0.1:5000, then systemctl reload caddy.
  4. MLflow's DNS-rebinding guard accepts only localhost and private addresses unless told otherwise, and its CORS guard refuses changes sent from a web origin it does not know (your browser sends one even when you save in MLflow's own UI). Put your name in H=mlflow.example.com, then allow both: [[ $H && $H != mlflow.example.com ]] && printf 'MLFLOW_SERVER_ALLOWED_HOSTS=%s\nMLFLOW_SERVER_CORS_ALLOWED_ORIGINS=%s\n' "localhost:*,127.0.0.1:*,$H" "https://$H" >> /etc/mlflow/server.env && echo written || echo STOP - check H. After written, systemctl restart mlflow.
  5. Point clients at https://mlflow.example.com. Leave port 5000 closed; the login sends a password on every request, so keep it inside HTTPS.

Keeping your runs safe

Your history lives in two places, the PostgreSQL database and the artifact folder, and this nightly job saves both. It dumps the database and reads the dump back, archives the files and lists them, and prunes sets past seven days once both succeed. Tar may exit with code 1 when an upload changes a file mid-read; the job allows that one case and still checks the archive.

install -d -m 700 /var/backups/mlflow &&
cat > /usr/local/sbin/mlflow-backup <<'EOF'
out=/var/backups/mlflow; stamp=$(date +%F-%H%M%S)
if ! { sudo -u postgres pg_dump --format=custom mlflow > "$out/mlflow-db-$stamp.dump" &&
       pg_restore --list "$out/mlflow-db-$stamp.dump" > /dev/null; }; then
  echo "STOP - database dump not usable, see $out"; exit 1; fi
tar -czf "$out/mlflow-files-$stamp.tar.gz" -C / var/lib/mlflow/artifacts etc/mlflow etc/systemd/system/mlflow.service
rc=$?
if [ $rc -gt 1 ] || ! tar -tzf "$out/mlflow-files-$stamp.tar.gz" > /dev/null; then
  echo "STOP - file archive not usable, see $out"; exit 1; fi
find "$out" -name 'mlflow-*' -mtime +7 -delete
echo "backup ok $stamp"
EOF
printf '40 2 * * * root umask 077; bash /usr/local/sbin/mlflow-backup >> /var/log/mlflow-backup.log 2>&1\n' > /etc/cron.d/mlflow-backup &&
(umask 077; bash /usr/local/sbin/mlflow-backup)

Pull the backup folder to a second machine with rsync -a over SSH, using a key that may read this root-only folder, and skip --delete, so a wiped server cannot empty your copies. Restoring the set stamped STAMP, as root:

  1. Check the set before anything stops: rm -f /run/m1 /run/m2; b=/var/backups/mlflow; pg_restore --list $b/mlflow-db-STAMP.dump >/dev/null && tar -tzf $b/mlflow-files-STAMP.tar.gz var/lib/mlflow/artifacts >/dev/null && touch /run/m1 && echo checked || echo STOP - the set is damaged. Then check for leftovers: rm /run/m1 && [ ! -e /var/lib/mlflow/artifacts.old ] && [ -z "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && touch /run/m2 && echo free || echo STOP - earlier restore leftovers. Then stop MLflow and move today's state aside: rm /run/m2 && systemctl stop mlflow && echo 'ALTER DATABASE mlflow RENAME TO previous_mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.old && echo moved || echo STOP - not fully moved. Each needs the marker the last one left in /run. After a STOP from the first two, MLflow is still running: clear the cause and start over. After one from the third, see what moved first.
  2. Load the database with sudo -u postgres createdb --owner=mlflow mlflow, then sudo -u postgres pg_restore --exit-on-error --role=mlflow --no-owner --dbname=mlflow < /var/backups/mlflow/mlflow-db-STAMP.dump.
  3. Bring back the files with [ ! -e /var/lib/mlflow/artifacts ] && tar -xzf /var/backups/mlflow/mlflow-files-STAMP.tar.gz -C / var/lib/mlflow/artifacts && chown -R mlflow: /var/lib/mlflow/artifacts && echo unpacked || echo STOP - artifacts folder still there, do not start MLflow. Only after unpacked prints, run systemctl start mlflow and log in to check.
  4. Changed your mind? Check first: rm -f /run/r0 /run/r1; [ -d /var/lib/mlflow/artifacts.old ] && [ ! -e /var/lib/mlflow/artifacts.restored ] && touch /run/r0 && echo found || echo STOP - check artifacts.old and artifacts.restored. Then check the old database: rm /run/r0 && [ -n "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && touch /run/r1 && echo ready || echo STOP - no previous_mlflow database. After ready, set the restore aside: rm /run/r1 && systemctl stop mlflow && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.restored && sudo -u postgres dropdb mlflow && touch /run/r2 && echo dropped || echo STOP - not set aside. After dropped, bring the old state back: rm /run/r2 && echo 'ALTER DATABASE previous_mlflow RENAME TO mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts.old /var/lib/mlflow/artifacts && systemctl start mlflow && echo rolled back || echo STOP - not done. A STOP after ready can mean a half-done rollback: see what moved. Happy? rm -f /run/r1; sudo -u postgres dropdb previous_mlflow and remove artifacts.old (and artifacts.restored if a rollback left one).
  5. Moving to a new server? Do steps 1 and 2 there, then this list. Taking only the artifact folder from the archive keeps the new server's own database password. The restored database brings its users, so log in with the old server's accounts, not the admin password the new server printed.

Updating MLflow without losing data

A new release can change the database layout, and the server will not start until you migrate it. MLflow's CLI docs warn that migrations can run long and may not be transactional, so every upgrade goes: back up, stop, upgrade, migrate, start.

  1. Run bash /usr/local/sbin/mlflow-backup and wait for backup ok.
  2. Read the release notes, then systemctl stop mlflow && /opt/mlflow/venv/bin/pip install --upgrade 'mlflow[auth]==X.Y.Z'.
  3. Run the schema migration as the mlflow user with sudo -u mlflow sh -c '. /etc/mlflow/server.env; u=$MLFLOW_BACKEND_STORE_URI; /opt/mlflow/venv/bin/mlflow db upgrade "$u"'. Because the URI is a command argument, ps shows it while the command runs.
  4. systemctl start mlflow. The login tables update themselves at startup (sqlalchemy_store.py).
  5. Something broke? Put the previous version back with pip and restore the backup from the first step.

The licence, and when to choose something else

MLflow's licence is Apache 2.0 (see LICENSE.txt). As we read it, in general terms and not as legal advice: use it, change it and build paid services on it; when you pass copies on, include the licence and notices and mark changed files. You need not release your own code.

  • One server can fail, and this setup has no standby.
  • The repo's docker-compose/ folder runs MLflow, PostgreSQL and RustFS together, but docker-compose.yml publishes their ports on every address with no login app, and .env.dev.example ships stock passwords. Treat it as a laptop setup.
  • Training belongs on GPU hardware: a Jupyter GPU server logs into this one. If the database becomes the bottleneck, move it to its own PostgreSQL server.

Questions people ask

How do my notebooks start logging to it?
Set MLFLOW_TRACKING_URI to your server's address and export your login as MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD, or call mlflow.set_tracking_uri() in code. mlflow.start_run() and the logging calls then go to your server.
What happens to the runs in my old mlruns folder?
mlflow migrate-filestore, in 3.10 and later, copies a file store into a new, empty SQLite database without changing IDs or timestamps. It cannot target PostgreSQL, because file-store experiment IDs are larger than PostgreSQL's 32-bit limit. Your artifact files stay where they are.
Can teammates see each other's experiments?
By default, yes: everyone gets READ. With default_permission = NO_PERMISSIONS people see only what they create or are granted; an experiment's creator gets MANAGE on it and can share it.

Give your experiments a home of their own

Pick an NVMe cartridge, mirror its drives if you like, and give every run one place to land. No contract, and the port is never metered.

Get a server for MLflow

Read next