Workload · Running your own search cluster

Elasticsearch on a dedicated server, sized and set up so it stays healthy

An Elasticsearch dedicated server works when the JVM heap gets no more than half the memory, the rest goes to the filesystem cache, the data disk stays under 85% full and snapshots leave the box. Elastic's own guidance is that Elasticsearch should ideally run alone on a host and use all of it, which is what a whole rented machine gives you. Below: sizing, installing the 9.x package on Debian or Ubuntu, the settings that trip people up, and when one machine is not enough.

The short version
  1. Heap: automatic by default, never above 50% of RAM, ideally under about 26 GB
  2. Disk: allocation stops at 85% full, indices go read-only at 95%
  3. Security, TLS and the elastic user are switched on at first start
  4. Snapshots are the only supported backup, never a copy of the data folder

Where the memory and disk actually go

Elasticsearch splits memory between the JVM heap, which holds the node's working structures, and the filesystem cache, which holds the index files every search reads. Elastic wants at least half of system memory left to that cache, so the heap is capped at half.

ResourceWhat Elastic's documentation saysWhat it means for you
JVM heapSized automatically from node roles and total memory; if you set it, minimum and maximum must match and stay at or below 50% of RAMLeave the default unless you have a reason; on a 32 GB build that is at most 16 GB
Heap ceilingStay under the compressed object pointer limit: about 26 GB is safe on most systems, up to 30 GB on someOn 64 GB or 128 GB, more RAM goes to the cache, not a bigger heap
Filesystem cacheAt least half of system memoryWhatever the heap does not use is serving your searches
DiskLow watermark 85%, high 90%, flood stage 95%Plan the drive so your data lives below 85% of usable space
Kernelvm.max_map_count of 1048576The package tries to set it; pin it in a file under /etc/sysctl.d/ anyway

Storage speed matters too: Elastic notes that indexing writes several files at once with mixed random and sequential reads, so SSDs tend to beat spinning disks.

Why a whole machine suits it

Elastic recommends running Elasticsearch on a dedicated host, because features such as automatic heap sizing assume it is the only heavy application there. A shared virtual machine breaks that assumption quietly: the RAM you sized against may not all be yours, and a neighbour's disk queue becomes your search latency.

On an NVMe dedicated server the processor, memory, drives and port are yours alone, and no other customer's virtual disks share the drive. Root access lets you set sysctl values, swap and limits yourself. The 1 Gbps port is unmetered, so bulk indexing and reindexing never become a transfer bill, and it can be raised to 10 Gbps. There is no contract, and a racked machine can go by the hour while you test a mapping.

The server that fits your index

Every NVMe build is an HPE Moonshot cartridge with 32 GB to 128 GB of memory and one or two NVMe drives from 240 GB to 3.84 TB, in New York, Miami, San Francisco, Amsterdam and Bucharest. Pick by memory first, since that decides how much of your index stays in cache.

  • Xeon E3-1585L v5, 4 cores, 32 to 64 GB: a site or product search, or a modest log index.
  • Xeon D-1548 or D-1587, 8 or 16 cores, 32 to 128 GB: the line to choose when the working set is large or many queries run at once.
  • Xeon E-2286M, 8 cores, 32 to 64 GB, built in Bucharest only: an option when your users and your data should stay in Europe.

Two drives can be mirrored in software RAID 1, so a dead drive costs a swap rather than your index; usable space is then one drive. Elastic's speed advice is RAID 0 with replicas on other nodes, but on a single node the mirror is the safer trade. For many terabytes rather than speed, a storage server takes up to 24 drives behind hardware RAID. A racked machine is yours about 30 minutes after the order is approved; a build you configure to order takes 4 to 24 hours when we hold the parts.

Up and running on Debian or Ubuntu

  1. Install Debian or Ubuntu from the image list, log in as root or a sudo user, and allow SSH in your firewall before you enable it (the firewall steps below do this in the right order).
  2. Pin the kernel setting: put vm.max_map_count=1048576 in a file of its own such as /etc/sysctl.d/99-elasticsearch.conf (Debian 13 no longer reads /etc/sysctl.conf at boot), then run sudo sysctl --system.
  3. Turn swap off, which Elastic prefers over every alternative: sudo swapoff -a, then comment out any line containing swap in /etc/fstab so it stays off after a reboot.
  4. Add Elastic's signing key and the 9.x APT repository, then install the package (first block below).
  5. Enable and start the service, set the elastic password, and check the node answers (second block). Run the reset tool right after the first start: until you do, nobody, including you, can log in as the superuser.
sudo apt-get install apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt-get update && sudo apt-get install elasticsearch
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

The start command prints nothing either way; if curl fails, read the log files in /var/log/elasticsearch/ (the service does not write to the systemd journal by default). Given only a user name, curl asks for the password rather than taking it on the command line, so it stays out of your shell history. A healthy node answers with a short JSON document naming the cluster and version.

On first start the package turns security on: TLS certificates for the HTTP and transport layers, the settings written to elasticsearch.yml, and a Kibana enrollment token valid for 30 minutes. HTTP listens on all addresses on port 9200, so fence it before loading real data, and never set xpack.security.enabled: false on a public machine.

  1. Close port 9200 to the world and open it only to the machines that query it. The chain below adds the SSH rule first and enables the firewall only if both rules were accepted; replace the example address with your app server's. On Debian, install the firewall first with sudo apt-get install ufw.
  2. Run sudo ufw status and check that port 22 appears twice, once marked (v6), and that the 9200 rule names only your app server's address. If your app server reaches Elasticsearch over IPv6, add a second 9200 rule with its IPv6 address.
sudo ufw allow 22 && sudo ufw allow proto tcp from 203.0.113.10 to any port 9200 && sudo ufw enable

Keeping it healthy once it is live

  • Heap. Leave it automatic. If you must set it, create a file such as /etc/elasticsearch/jvm.options.d/heap.options with matching -Xms and -Xmx lines, for example -Xms16g and -Xmx16g on a 32 GB machine, and restart.
  • Disk. Watch usage against the watermarks. Past 85% the node takes no new shards except the primaries of brand-new indices, past 90% it tries to move shards away, and at 95% every index with a shard there gets a read-only block that lifts by itself once usage drops below 90%.
  • Yellow is normal on one node. Elasticsearch never puts a replica on the same node as its primary. Set index.number_of_replicas to 0 for a green status, knowing you then rely entirely on snapshots.
  • Snapshots. Add a mounted directory to path.repo in elasticsearch.yml, restart, then register it as an fs repository with the snapshot API. Mount something that is not this server's own drive, such as a share on a second machine, or the backup dies with the disk it protects.
  • Updates. New 9.x releases, with the bundled JVM, arrive through Elastic's APT repository. Take a snapshot before you upgrade.

Do not back up by copying /var/lib/elasticsearch. Elastic is explicit that a copy of the data directory is not a consistent point-in-time image, even with the node stopped, and that restoring one can fail or silently lose data. Snapshots are the only supported backup.

If hardware fails, the service agreement has a failed drive replaced within four hours of your call or chat, and our chat assistant brings in a person whenever you ask.

When one server is the wrong answer

Elastic's resilience guide says a one-node cluster is not resilient, keeps no redundant copy, and is not recommended for production; a resilient cluster needs at least three master-eligible nodes. One machine suits a site search, a staging copy of production, or logs you can restore from last night's snapshot. If an hour of dark search costs you money, plan three nodes from the start.

The NVMe cartridges have no hardware RAID controller and no RAID 5, 6 or 10, and Elastic advises giving a heavy Logstash deployment its own host. If you are really building log collection with a web interface, our Graylog server guide covers that setup.

Questions people ask

Is one machine enough to run Elasticsearch in production?
For many small jobs it works, but Elastic does not recommend a one-node cluster for production because it has no redundancy: if the node fails, search stops until you restore. A resilient cluster needs at least three master-eligible nodes. If you start with one, take regular snapshots to storage on another machine.
How much heap should I give Elasticsearch?
Usually none by hand: Elasticsearch sizes the heap automatically from the node's roles and total memory, and Elastic recommends that default for most production setups. If you override it, keep minimum and maximum equal, at or below half of RAM, and under roughly 26 GB so compressed object pointers stay on. Extra memory beyond that helps through the filesystem cache.
Why did my indices suddenly refuse writes?
Most often the disk reached the flood-stage watermark, 95% by default. Elasticsearch then puts a read-only block on every index with a shard on that node. Free space or grow the disk; the block is released automatically once usage falls below the 90% high watermark.
Why is my cluster health yellow on a single server?
Replicas are never placed on the same node as their primary, so on one node they stay unassigned and health reads yellow. Setting the replica count to 0 on every index turns it green. That changes the label, not the risk: your protection is then your snapshots.
Will reindexing or shipping logs to the server cost extra in traffic?
Not on our side. The port on every NVMe server is unmetered, with no transfer allowance or overage bill, and can be raised from 1 Gbps to 10 Gbps.

Give your search index a machine of its own

Pick an NVMe server with the memory your index needs, month to month, on a port nobody meters.

Get a server for your Elasticsearch

Read next