Run your own MLflow tracking server, with a login and backups

What you are setting up

You are building the shared place where every training run lands: its parameters, its metric curves, the model it produced. Training scripts set MLFLOW_TRACKING_URI to your server and log over HTTP; your team opens the same address in a browser to compare runs.

• The server (mlflow server) serves the web UI and the API, on 127.0.0.1 port 5000 unless told otherwise.
• The backend store is the database for run details and the model registry. MLflow switched its default to SQLite in 3.7.0; you will use PostgreSQL.
• The artifact store is where files go: weights, plots, datasets. Uploads pass through the server by default, so your scripts never need storage keys.
• The login app (basic-auth) keeps users and permissions, here in that same PostgreSQL database.

Leave the login app off and whoever reaches the port can read, edit and delete every run.

Will one machine be enough?

For one team, usually yes: the tracking server trains nothing, it keeps what training produces. We found no hardware minimums in MLflow's self-hosting docs, so plan around what you log.

If your team is fine-tuning large language models, checkpoints will decide your drive size long before anything else does.

The server that fits

An NVMe dedicated server suits this job because the database and the model files both sit on drives wired to the processor over PCIe, and nobody else shares the machine. Every NVMe build is an HPE Moonshot cartridge on a 1 Gbps port that nothing meters, so pulling old checkpoints back to a GPU box adds nothing to your bill.

• Xeon D-1500: 8 or 16 cores, memory up to 128 GB.
• Xeon E3-1585L v5: four cores, memory up to 64 GB.
• Xeon E-2286M: eight cores, memory up to 64 GB, built in Bucharest.
• Drives each between 240 GB and 3.84 TB. A pair in RAID 1 turns a dead drive into a swap instead of lost runs; keep backups anyway.
• How fast: a racked machine is yours about 30 minutes after your order is approved. A build you configure takes 4 to 24 hours when we hold the parts, and longer when one has to be bought in.

Expecting more model files than one drive holds? Keep MLflow here and send artifacts to an S3 bucket on your own MinIO server, pointing MLFLOW_S3_ENDPOINT_URL at it as the artifact store docs describe.

Step 1: install MLflow and PostgreSQL

As root, on a newly installed Ubuntu 24.04 server, paste this. It switches the firewall on with SSH alone, installs PostgreSQL and MLflow (in its own virtualenv, run by an mlflow user) and makes three random secrets, among them your first admin password. They land in files readable by root and the mlflow group alone, and the main file appears last, so a second paste stops at line one.

cd /tmp && C=/etc/mlflow && if [ -e $C/server.env ]; then echo "STOP - installed already"; false; fi &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
apt-get update && apt-get install -y python3-venv postgresql curl &&
{ getent passwd mlflow >/dev/null || useradd -r -m -d /var/lib/mlflow -s /usr/sbin/nologin mlflow; } &&
install -d -o mlflow -g mlflow -m 750 /var/lib/mlflow/artifacts &&
python3 -m venv /opt/mlflow/venv && /opt/mlflow/venv/bin/pip install 'mlflow[auth]==3.17.2' psycopg2-binary &&
PW=$(openssl rand -hex 24) && CSRF=$(openssl rand -hex 32) && FIRST=$(openssl rand -hex 16) &&
[ ${#PW} = 48 ] && [ ${#CSRF} = 64 ] && [ ${#FIRST} = 32 ] &&
sudo -u postgres createuser mlflow &&
printf "ALTER ROLE mlflow PASSWORD '%s';\n" "$PW" | sudo -u postgres psql -v ON_ERROR_STOP=1 &&
sudo -u postgres createdb --owner=mlflow mlflow &&
URI="postgresql://mlflow:$PW@127.0.0.1:5432/mlflow" && install -d -m 750 -g mlflow $C &&
install -m 640 -g mlflow /dev/null $C/auth.ini && install -m 640 -g mlflow /dev/null $C/server.env.tmp &&
printf '[mlflow]\ndefault_permission = READ\ndatabase_uri = %s\nadmin_username = admin\n' "$URI" > $C/auth.ini &&
printf 'MLFLOW_BACKEND_STORE_URI=%s\nMLFLOW_FLASK_SERVER_SECRET_KEY=%s\nMLFLOW_AUTH_CONFIG_PATH=%s/auth.ini\nMLFLOW_AUTH_ADMIN_PASSWORD=%s\n' \
  "$URI" "$CSRF" "$C" "$FIRST" > $C/server.env.tmp &&
mv $C/server.env.tmp $C/server.env && echo "Done - admin password $FIRST" || echo 'STOP - see above'

Copy the printed admin password into your password manager. If the paste stops part way, nothing uses the secrets yet: open a postgres shell with sudo -iu postgres, run dropdb --if-exists mlflow and dropuser --if-exists mlflow, exit, and paste again.

Step 2: start it with a login in front

MLflow 3.17.2 has no default admin password. Started with --app-name basic-auth for the first time, it reads MLFLOW_AUTH_ADMIN_PASSWORD to create the admin user, and it will not start at all if that, or MLFLOW_FLASK_SERVER_SECRET_KEY, is missing (mlflow/server/auth/__init__.py). Passwords need at least 12 characters.

cat > /etc/systemd/system/mlflow.service <<'EOF'
[Unit]
Description=MLflow tracking server
Requires=postgresql.service
After=postgresql.service network-online.target

[Service]
User=mlflow
Group=mlflow
WorkingDirectory=/var/lib/mlflow
EnvironmentFile=/etc/mlflow/server.env
ExecStart=/opt/mlflow/venv/bin/mlflow server --app-name basic-auth --host 127.0.0.1 --port 5000 --artifacts-destination /var/lib/mlflow/artifacts
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now mlflow &&
up=no; for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && { up=yes; break; }; sleep 2; done
[ "$up" = yes ] &&
code=$(curl -s -o /dev/null -w '%{http_code}' 'http://127.0.0.1:5000/api/2.0/mlflow/experiments/search?max_results=1') &&
[ "$code" = 401 ] && sed -i '/^MLFLOW_AUTH_ADMIN_PASSWORD=/d' /etc/mlflow/server.env &&
! grep -q ADMIN_PASSWORD /etc/mlflow/server.env && systemctl restart mlflow &&
sleep 5 && for i in $(seq 60); do curl -fs http://127.0.0.1:5000/health >/dev/null && break; sleep 2; done &&
curl -fs http://127.0.0.1:5000/health >/dev/null && echo "Login is on and the first-start password is gone" || echo 'STOP - look at journalctl -u mlflow -n 50'

This waits for /health to answer, confirms the API turns you away with 401 when you send no password, and only then strips the first-start password from the env file and restarts. The docs tell you to unset it once the first start has worked; deleting the line, rather than blanking it, also takes the password off the disk. Runs, users and permissions share one database, and the docs confirm that is safe since the login tables track their own migrations.

Step 3: let your team in without opening the port

Start by tunnelling over SSH with your own account: ssh -N -L 15000:127.0.0.1:5000 you@server.example.com. Visit http://localhost:15000 and log in with the admin account. While logged in as admin, create accounts for colleagues on the /signup page, and set a fresh admin password through the auth client's update_user_password().

Each new user can read every experiment, because default_permission is READ. Want people to see only what they are given? Change it to NO_PERMISSIONS in /etc/mlflow/auth.ini and restart.

When the team needs it without tunnels, put HTTPS in front with Caddy:

1. Create DNS records for a name such as mlflow.example.com pointing at the server.
2. Install Caddy with apt-get -y install caddy (Ubuntu 24.04 carries it in universe). Open the web ports before it asks for a certificate: ufw allow 80,443/tcp.
3. Write /etc/caddy/Caddyfile with a site block for mlflow.example.com holding reverse_proxy 127.0.0.1:5000, then systemctl reload caddy.
4. MLflow's DNS-rebinding guard accepts only localhost and private addresses unless told otherwise, and its CORS guard refuses changes sent from a web origin it does not know (your browser sends one even when you save in MLflow's own UI). Put your name in H and allow both with `H=mlflow.example.com; if [ "$H" = mlflow.example.com ]; then echo 'STOP - set H to your own name'; else printf 'MLFLOW_SERVER_ALLOWED_HOSTS=%s\nMLFLOW_SERVER_CORS_ALLOWED_ORIGINS=%s\n' "localhost:*,127.0.0.1:*,$H" "https://$H" >> /etc/mlflow/server.env && systemctl restart mlflow; fi`.
5. Point clients at https://mlflow.example.com. Leave port 5000 closed; the login sends a password on every request and should only ever travel inside HTTPS.

Keeping your runs safe

Your history lives in two places, the PostgreSQL database and the artifact folder, and this nightly job saves both. It dumps the database and checks the dump reads back, archives the files and lists the archive, and prunes sets past seven days once both have succeeded. Tar may exit with code 1 when an upload changes a file mid-read; the job allows that one case and still checks the archive.

install -d -m 700 /var/backups/mlflow &&
cat > /usr/local/sbin/mlflow-backup <<'EOF'
out=/var/backups/mlflow; stamp=$(date +%F-%H%M%S)
if ! { sudo -u postgres pg_dump --format=custom mlflow > "$out/mlflow-db-$stamp.dump" &&
       pg_restore --list "$out/mlflow-db-$stamp.dump" > /dev/null; }; then
  echo "STOP - database dump not usable, see $out"; exit 1; fi
tar -czf "$out/mlflow-files-$stamp.tar.gz" -C / var/lib/mlflow/artifacts etc/mlflow etc/systemd/system/mlflow.service
rc=$?
if [ $rc -gt 1 ] || ! tar -tzf "$out/mlflow-files-$stamp.tar.gz" > /dev/null; then
  echo "STOP - file archive not usable, see $out"; exit 1; fi
find "$out" -name 'mlflow-*' -mtime +7 -delete
echo "backup ok $stamp"
EOF
printf '40 2 * * * root umask 077; bash /usr/local/sbin/mlflow-backup >> /var/log/mlflow-backup.log 2>&1\n' > /etc/cron.d/mlflow-backup &&
(umask 077; bash /usr/local/sbin/mlflow-backup)

Copy the backup folder to a second machine: pull it with rsync -a over SSH, using a key that may read this root-only folder, and skip --delete, so a wiped server cannot empty your copies. Restoring the set stamped STAMP, as root:

1. Check the set and the leftovers before anything stops, then move today's state aside: `b=/var/backups/mlflow; pg_restore --list $b/mlflow-db-STAMP.dump >/dev/null && tar -tzf $b/mlflow-files-STAMP.tar.gz var/lib/mlflow/artifacts >/dev/null && [ ! -e /var/lib/mlflow/artifacts.old ] && [ -z "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && systemctl stop mlflow && echo 'ALTER DATABASE mlflow RENAME TO previous_mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.old && echo moved. No moved` means the set is damaged or an earlier restore's leftovers are in the way; MLflow keeps running until every check passes.
2. Load the database with sudo -u postgres createdb --owner=mlflow mlflow, then sudo -u postgres pg_restore --exit-on-error --role=mlflow --no-owner --dbname=mlflow < /var/backups/mlflow/mlflow-db-STAMP.dump.
3. Bring back the files with tar -xzf /var/backups/mlflow/mlflow-files-STAMP.tar.gz -C / var/lib/mlflow/artifacts && chown -R mlflow: /var/lib/mlflow/artifacts, then systemctl start mlflow and log in to check.
4. Changed your mind? Roll back with `[ -d /var/lib/mlflow/artifacts.old ] && [ ! -e /var/lib/mlflow/artifacts.restored ] && [ -n "$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='previous_mlflow'")" ] && systemctl stop mlflow && mv /var/lib/mlflow/artifacts /var/lib/mlflow/artifacts.restored && sudo -u postgres dropdb mlflow && echo 'ALTER DATABASE previous_mlflow RENAME TO mlflow;' | sudo -u postgres psql --set ON_ERROR_STOP=on && mv /var/lib/mlflow/artifacts.old /var/lib/mlflow/artifacts && systemctl start mlflow && echo 'rolled back'. Happy? sudo -u postgres dropdb previous_mlflow and remove artifacts.old (and artifacts.restored` if a rollback left one).
5. Moving to a new server? Do steps 1 and 2 there, then this list. Taking only the artifact folder from the archive keeps the new server's own database password. The restored database brings its users with it, so log in with the old server's accounts; the admin password the new server printed no longer applies.

Updating MLflow without losing data

A new release can change the database layout, and the server will not start on the old layout until you migrate it. MLflow's CLI docs warn that migrations can run long and may not be transactional, so every upgrade goes: back up, stop, upgrade, migrate, start.

1. Run bash /usr/local/sbin/mlflow-backup and wait for backup ok.
2. Read the release notes, then systemctl stop mlflow && /opt/mlflow/venv/bin/pip install --upgrade 'mlflow[auth]==X.Y.Z'.
3. Run the schema migration as the mlflow user with sudo -u mlflow sh -c '. /etc/mlflow/server.env; u=$MLFLOW_BACKEND_STORE_URI; /opt/mlflow/venv/bin/mlflow db upgrade "$u"'. Because the URI is a command argument, ps shows it while the command runs.
4. systemctl start mlflow. The login tables update themselves at startup (sqlalchemy_store.py).
5. Something broke? Put the previous version back with pip and restore the backup from the first step.

The licence, and when to choose something else

MLflow's licence is Apache 2.0 (see LICENSE.txt). How we read it, as general information and not legal advice: use it, change it and build paid services on it freely; when you pass copies on, include the licence and notices and mark changed files. Nothing obliges you to release your own code.

• A single server is a single place things can fail, and this setup has no standby.
• The repo's docker-compose/ folder runs MLflow, PostgreSQL and RustFS together, but docker-compose.yml publishes their ports on every address with no login app, and .env.dev.example ships stock passwords. Treat it as a laptop setup.
• Training belongs on GPU hardware: a Jupyter GPU server logs into this one. When the database itself becomes the bottleneck, move it to its own PostgreSQL server.

Questions people ask

Do I have to pay for MLflow itself?

No. MLflow is free to download and use; it is Apache 2.0 licensed, and the registry, the tracking UI and the login app all install with one pip command. Your only cost is the server.

How do my notebooks start logging to it?

Set MLFLOW_TRACKING_URI to your server's address, export your username into MLFLOW_TRACKING_USERNAME and your password into MLFLOW_TRACKING_PASSWORD, or call mlflow.set_tracking_uri() in code. After that, mlflow.start_run() and the logging calls send everything to your server.

What happens to the runs in my old mlruns folder?

mlflow migrate-filestore, in 3.10 and later, copies a file store into a new, empty SQLite database without changing IDs or timestamps. It cannot target PostgreSQL, because file-store experiment IDs are larger than PostgreSQL's 32-bit limit. Your artifact files stay where they are.

Can teammates see each other's experiments?

By default, yes: every user gets READ on everything. Set default_permission = NO_PERMISSIONS and people see only what they create or are granted; whoever creates an experiment gets MANAGE on it and can share it.