Workload · Running a Loki log server

A Grafana Loki Server for Your Logs, Set Up on One Machine

A Grafana Loki server for one team runs best as a single process in monolithic mode on one machine with plenty of disk, which Loki's documentation recommends for up to roughly 20 GB of logs a day. Alloy ships the logs in, Grafana reads them out, and a retention rule decides how long they stay. Below: sizing, the install, and the two defaults that catch people out.

The short version
  1. Monolithic mode: one process, up to about 20 GB of logs a day
  2. Retention is off by default: switch it on or the disk fills
  3. Loki has no login of its own, so port 3100 stays private
  4. Whole server, drives and RAID you choose, unmetered port
  5. Monthly term, no contract

How much can one Loki machine take?

Loki's documentation says monolithic mode, where every component runs in one process started with -target=all, suits small read and write volumes of up to approximately 20 GB of logs per day. Beyond that the project points you to microservices mode. The in-between simple scalable mode is due to be removed in Loki 4.0, so do not build anything new on it.

One machine copes because Loki does not index the text of your lines, only the labels on each stream (a stream is every line sharing the same labels), and it stores the lines compressed. The catch: every new label value creates new streams. Keep labels for fixed things such as host, application and environment, and any changing label to tens of values at most. Request IDs and trace IDs belong in the line, where a filter like |= "text" finds them.

Sizing the disks, memory and processor

Grafana publishes resource figures only for large distributed clusters, so size a single box from your own numbers.

ResourceWhat uses itHow to size it
DiskCompressed log data and the label index, kept until retention deletes themMeasure a week of growth, divide by seven, multiply by your retention days, then leave generous headroom
MemoryRecent lines held before they are written out, the query results cache (100 MB in the sample config) and running queriesGrows with the number of active streams and with how far back your queries reach
ProcessorMostly queries: long time ranges and regular expressions over many streamsGrafana warns that unoptimised queries can need up to 10 times the expected query resources

Disk is the one that bites. With the filesystem store, Loki never deletes data because the disk is filling; only retention removes it.

Why a whole machine suits a log store

A log server writes all day and gets read hardest when something else is already broken, the worst moment to find you share disks with a stranger.

  • Disks you choose. On our storage dedicated servers the drives are set up for your own balance of speed, capacity and safety, and a storage consultant on chat helps you pick the RAID level. Loki's filesystem store keeps one copy, so RAID is your defence against a dead drive.
  • A port nobody meters. The port is yours alone at the access switch and traffic is never billed, so ten more senders add nothing to the invoice.
  • Room to grow in place. Memory, disks and port speed can be added to the machine you already have, with no migration and no new IP address.
  • A console of your own. The blade and rack lines carry an iLO, iDRAC or BMC console for a machine that will not boot.

The server that fits

Start with the storage servers: enterprise HPE hardware, with Dell and Supermicro on some lines, hardware RAID on the blade and rack lines, and Ubuntu or Debian picked at order on most lines. The deciding questions are how many drives the chassis takes and how much memory your long-range queries want.

If your volume is modest but your queries scan a lot of it, faster drives pay off more than extra capacity; look at the NVMe dedicated servers. For hot-swap bays and a failed part replaced within four hours of the report, see the enterprise dedicated servers. A racked machine is ready in about 30 minutes once the order is approved, and every line with its current price is on the dedicated server catalogue.

Loki, Alloy and Grafana in one evening

All three go on one Ubuntu or Debian server from Grafana's package repository, with Loki answering only on the machine itself.

  1. Add the Grafana repository and install the three packages with the first block below. The Loki package runs as the loki user and reads /etc/loki/config.yml, which starts as the project's local sample config.
  2. Make a data directory on your RAID volume: sudo mkdir -p /srv/loki, then sudo chown loki:loki /srv/loki. The sample config keeps everything under /tmp/loki, which is no place for logs you mean to keep.
  3. Edit /etc/loki/config.yml so its blocks match the second block below, and leave schema_config as shipped. The edits bind ports 3100 and 9096 to the machine, move the data, keep logs 28 days, and switch off the anonymous usage report sent to Grafana Labs.
  4. Start Loki and check it: sudo systemctl enable --now loki, then sudo systemctl restart loki so it reads your edit, then curl http://127.0.0.1:3100/ready until it reports ready.
  5. Replace /etc/alloy/config.alloy with the third block, then run sudo systemctl enable --now alloy and sudo systemctl reload alloy. It tails every .log file in /var/log.
  6. Start Grafana with sudo systemctl daemon-reload and sudo systemctl enable --now grafana-server. Sign in straight away on port 3000 as admin with the password admin and set a new password at the prompt; until you do, anyone who reaches the page can take the account.
  7. In Grafana open Connections, then Add new connection, choose Loki and enter http://127.0.0.1:3100 as the URL, with no API path after it. In Explore, pick a file from the label browser and add |= "error".
sudo apt install gpg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install loki
sudo apt-get install alloy
sudo apt-get install grafana
auth_enabled: false

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 3100
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9096
  log_level: info
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /srv/loki
  storage:
    filesystem:
      chunks_directory: /srv/loki/chunks
      rules_directory: /srv/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

compactor:
  working_directory: /srv/loki/compactor
  delete_request_store: filesystem
  retention_enabled: true

limits_config:
  metric_aggregation_enabled: true
  enable_multi_variant_queries: true
  retention_period: 672h

analytics:
  reporting_enabled: false
local.file_match "local_files" {
  path_targets = [{"__path__" = "/var/log/*.log"}]
  sync_period = "5s"
}

loki.source.file "log_scrape" {
  targets    = local.file_match.local_files.targets
  forward_to = [loki.write.grafana_loki.receiver]
  tail_from_end = true
}

loki.write "grafana_loki" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
  }
}

Grafana listens on every interface by default. Set http_addr under [server] in /etc/grafana/grafana.ini to keep it on one, and leave allow_sign_up at its default of false so nobody can create an account. If Alloy shows nothing, check journalctl -u alloy for permission errors: its service user needs read access to every file it tails.

Keeping it healthy once the logs pour in

  • Watch free space, not just dashboards. The compactor marks expired data and deletes it after retention_delete_delay, two hours by default, so free space trails your setting. Alert on the data volume yourself.
  • Shorter lives for noisy streams. A retention_stream rule in limits_config keeps chosen streams for less time than the global period. Each such rule needs a period of at least 24h, and retention only works with the 24h index period the sample config already uses.
  • Senders on other machines. Loki has no login of its own, and its documentation says to put an authenticating reverse proxy in front of it. To take logs from other servers, run a proxy with TLS and a password on a public port, forward it to 127.0.0.1:3100, and fill in basic_auth in each sender's loki.write block. Never open 3100 itself.
  • Copies of what you cannot rebuild. Keep /etc/loki/config.yml, /etc/alloy/config.alloy and /etc/grafana/grafana.ini in version control, and copy /srv/loki to another machine if losing the history would hurt. RAID covers a dead drive, not a deleted directory.
  • Upgrade on purpose. apt-get upgrade moves all three packages; read Loki's upgrade guide before a major version.

When one box is the wrong answer

If you already produce well over 20 GB of logs a day, or the log store must survive losing a whole machine, a single monolithic server is the wrong design. Loki's answer there is microservices mode on an object store, and even its documented high-availability setup for monolithic mode runs two instances over a shared object store with a replication factor of 3.

Loki is label-first: you narrow by stream, then filter lines. If your team wants to search any word without choosing labels first, read our Graylog server page before you commit.

Questions people ask

Do I need S3 or another object store to run Loki?
Not on one machine. Loki's filesystem store is common for single-binary installs and is what the sample config uses. It keeps a single copy, so put it on a RAID volume and keep a copy elsewhere. Loki also speaks the S3 API, including MinIO, if you outgrow one set of disks.
Can my other servers ship their logs to this one?
Yes. Install Alloy on each of them and point its loki.write block at your Loki server. Because Loki has no authentication of its own, put an authenticating reverse proxy with TLS in front of it and give each sender the proxy's address and credentials.
How long does Loki keep logs if I change nothing?
Forever. Retention is off by default, and a retention_period of 0s means logs are kept indefinitely. Turn on retention_enabled in the compactor block and set retention_period in limits_config; 672h keeps 28 days.
Does Loki index the full text of my logs?
No. It indexes only the labels on each stream and stores the lines compressed. A query must start with a stream selector such as {job="nginx"}, and line filters like |= "error" then scan only the streams you picked.
Can I run Loki in Docker instead of from packages?
Yes. Grafana documents a docker run of the grafana/loki image that mounts your config directory. The same rules apply: keep storage in a host directory on your RAID volume so logs outlive the container, switch on retention, and never publish port 3100 to the internet.

Get a server for your Loki log store

Pick the drives and RAID level for your retention window, install Loki, Alloy and Grafana from Grafana's packages, and have logs flowing the same evening.

Choose your storage server

Read next