Workload · Running an Apache Cassandra cluster

Apache Cassandra server: one node to learn on, three to rely on

A production Apache Cassandra server is really three machines acting as one: the project's hardware guidance sets a floor of 2 cores and 8 GB of ECC memory per node, calls 8 or more cores and 32 GB typical, and a keyspace stored three times keeps answering while one node is down. Here is how to size each node, install Cassandra 5.0, join the nodes and keep them healthy.

The short version
  1. Per node: 2 cores and 8 GB ECC at minimum, 8+ cores and 32 GB typical
  2. Java heap between 2 GB and half the machine's memory
  3. Three copies of each row, so QUORUM keeps working with a node down
  4. Fence port 7000 to the other nodes and 9042 to your applications
  5. Whole NVMe machines on an unmetered port, month to month

What each node needs from its machine

Each Cassandra node wants several cores, plenty of error-correcting memory and a local disk it does not share. The project's hardware page puts the numbers like this:

ResourceProduction minimumTypical production nodeWorth knowing
Processor2 cores8 cores or moreMore cores raise both read and write throughput
Memory8 GB32 GB or moreECC memory at all times: Cassandra has few guards against flipped bits
Java heap2 GBNo more than half the RAMAbove 12 GB of heap the docs suggest G1GC
StorageSSD or spinning diskEither performs wellLocal disks: keep data off NFS and SANs

Two disk rules follow. On spinning disks, put the commit log and the data directories on separate physical drives. And the docs prefer RAID 0 or plain separate disks (JBOD) to RAID 1 or RAID 5, because the cluster already keeps copies of every row on other nodes.

Why a real cluster starts at three nodes

Three nodes is the smallest cluster that can lose a machine and keep serving both reads and writes. With a replication factor of 3, every row lives on three nodes. A request at QUORUM needs a majority of those replicas, which is 2 of the 3, so one node can be down for a reboot, an upgrade or a failed part while your application carries on.

  • One node suits development and learning CQL; in production every restart is an outage.
  • Two nodes with two copies need both for a majority, so losing either stops QUORUM traffic.
  • Three nodes with three copies is where the arithmetic first works for you.

The production guidance adds two rules. Create keyspaces with NetworkTopologyStrategy, never SimpleStrategy, even in one datacenter, so a second can be added later. And raise the system_auth keyspace, which holds logins, from one copy to 3 to 5 per datacenter, or a down node can stop people signing in.

The servers that fit a Cassandra node

Our NVMe dedicated servers are whole machines whose SSDs reach the processor over PCIe, with no other customer on the drive, memory or port. For Cassandra, look at the Intel Xeon D-1500 line: the D-1548 has 8 cores and the D-1587 has 16, with 32 GB to 128 GB of memory, which covers the typical node above. The 4-core Xeon E3-1585L v5 suits a development or light cluster.

  • Drives: one NVMe drive from 240 GB to 3.84 TB, or two of the same size in software RAID 1. Since the cluster keeps the copies, one larger drive per node usually beats a mirror.
  • Port: 1 Gbps included and never metered, so a repair or a new node streaming its data is not a traffic bill.
  • Cities: New York, Miami, San Francisco, Amsterdam and Bucharest, which gives NetworkTopologyStrategy real datacenters to place replicas in.
  • Trying it first: racked machines are yours about 30 minutes after the order is approved and can be rented by the hour, handy for a trial cluster.

The NVMe page lists DDR4 memory without saying whether it is ECC, which the Cassandra docs insist on. Ask on chat before you order, or choose the blade and rack lines on enterprise servers, which carry registered ECC memory. With a build to order you can also put many drives behind a controller set to RAID 0 or JBOD.

Your first node in a few commands

Cassandra 5.0 is the current GA series, and it runs on Java 11 or Java 17; cqlsh needs Python 3.8 to 3.13. On Debian or Ubuntu:

  1. Install the latest Java 17 (OpenJDK is fine) and check it with java -version.
  2. Add the Apache repository for the 5.0 series and install the package, as below. Run it as root, since the curl line writes into /etc/apt/keyrings.
  3. Watch system.log until it says Cassandra is listening for CQL clients on port 9042.
  4. Run nodetool status: a node that is up and normal shows UN. Then try cqlsh to open a CQL prompt.
echo "deb [signed-by=/etc/apt/keyrings/apache-cassandra.asc] https://debian.cassandra.apache.org 50x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
curl -o /etc/apt/keyrings/apache-cassandra.asc https://downloads.apache.org/cassandra/KEYS
sudo apt-get update
sudo apt-get install cassandra

Settings live in /etc/cassandra, and the new node listens on localhost only.

Joining the three nodes together

Nodes find each other through cassandra.yaml, so the same few settings change on every machine. The Debian package starts Cassandra the moment it is installed, as a one-node cluster called Test Cluster, and a node will not start again once its cluster_name differs from the name it saved. So on each newly installed node that has never held data, stop it with sudo service cassandra stop, clear the system tables from that first start with sudo rm -rf /var/lib/cassandra/data/system/* (never on a node that holds data you need), then edit /etc/cassandra/cassandra.yaml:

  1. Give every node the same cluster_name.
  2. Under seed_provider, list the addresses of your seed nodes. The list is identical on every node.
  3. Set listen_address to this node's own address, the one the other nodes will connect to. The docs call 0.0.0.0 always wrong here.
  4. Set rpc_address to the address your applications will reach; its default is localhost.
  5. Set endpoint_snitch: GossipingPropertyFileSnitch, which the docs call the most flexible choice for your own hardware. Decide your racks now: changing them after the cluster is built is unsupported.
  6. Before starting any node on its public address, put up the firewall described in the next section: allow SSH, allow 7000 only from the other nodes and 9042 only from your application servers, over IPv4 and IPv6, then sudo ufw enable. Then start a seed node first and wait for UN in nodetool status, then the next node, and the next, waiting for each to show UN.
cluster_name: 'orders'
listen_address: 203.0.113.11
rpc_address: 203.0.113.11
endpoint_snitch: GossipingPropertyFileSnitch
authenticator: PasswordAuthenticator
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "203.0.113.11:7000,203.0.113.12:7000"

Lock it down before clients connect

Out of the box Cassandra checks no passwords at all: its default AllowAllAuthenticator lets anyone who reaches port 9042 in. The authenticator: PasswordAuthenticator line above closes that, but it opens a short window of its own, because the built-in cassandra superuser's password is cassandra. With the firewall up, only your own machines can reach port 9042, so close that window as soon as all three nodes show UN, before any application connects:

  1. Log in once on a node with cqlsh 203.0.113.11 -u cassandra (use that node's own address: cqlsh connects to 127.0.0.1 unless you give it one, and the node no longer listens there), and type the default password cassandra when asked.
  2. Create your own superuser with CREATE ROLE dba WITH SUPERUSER = true AND LOGIN = true AND PASSWORD = '...';, choosing a long password of your own.
  3. Log in as that role with cqlsh 203.0.113.11 -u dba and switch the default one off with ALTER ROLE cassandra WITH SUPERUSER = false AND LOGIN = false;.
  4. Still logged in as that role, create your keyspace with three copies, for example CREATE KEYSPACE orders WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor' : 3};, and do the same for system_auth with ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': 3};, using the datacenter name nodetool status prints. Follow a replication change with nodetool repair --full.
  5. If the nodes reach each other over public addresses, set internode_encryption: all under server_encryption_options, and turn on client_encryption_options for applications. The docs advise setting up both when the cluster is first built, because changing them later risks downtime.

The firewall you put up before the first start fences the ports: 7000, the traffic between nodes, should answer only the other nodes, and 9042 only your application servers. JMX, which nodetool uses, listens on localhost by default; leave it there. With ufw, allow SSH first with sudo ufw allow 22, add one rule per peer such as sudo ufw allow proto tcp from 203.0.113.12 to any port 7000, repeat it for the peer's IPv6 address (for example 2001:db8::12), add the 9042 rules, and only then run sudo ufw enable.

Keeping the cluster healthy

  • Repair on a schedule. Repairing within the grace period for deletions, 10 days by default, stops deleted rows coming back. The docs suggest an incremental nodetool repair every 1 to 3 days and nodetool repair --full every 1 to 3 weeks.
  • Snapshots, then copies. nodetool snapshot --tag nightly orders hard-links the keyspace's files into a snapshots folder beside the data, on the same drive. We sell no backup product, so copy each snapshot to another machine, check the copy is complete, and only then free the space with nodetool clearsnapshot -t nightly orders.
  • Watch the ring. nodetool status on any node shows every peer; anything but UN deserves a look.
  • Upgrades. The 5.0 series is maintained until 8.0 is released, so patch releases for it keep coming; after updating a node, wait for it to show UN again before moving to the next.
  • Hardware. Under the service agreement a failed drive or other part is replaced within four hours of your call or chat, and our chat assistant brings in a person whenever you ask.

When this is not the right home

These are whole machines with root access, not a managed database: installing, repairing and upgrading Cassandra is your job, and a single node is not a production cluster however fast its drive. If one node must hold more than a 3.84 TB NVMe drive, a rack server with a shelf of drives from build to order fits better. And if your data fits on one machine and you mainly need flexible queries, MongoDB on a dedicated server or MariaDB is less to run.

Questions people ask

How much memory should each Cassandra node have?
The project's guidance puts the production floor at 8 GB and calls 32 GB or more typical, always ECC. Give the Java heap at least 2 GB and no more than half the machine's memory. Above a 12 GB heap, the docs suggest looking at the G1 garbage collector.
Is one Cassandra server enough to start with?
For development, yes. For anything people depend on, plan three nodes with a replication factor of 3, so that QUORUM reads and writes still succeed while one node is down. A one-node cluster stops answering every time you restart it.
Which ports does a Cassandra cluster use?
Nodes talk to each other on port 7000 and clients connect with CQL on port 9042. JMX, which nodetool uses, stays on localhost unless you change it. Open 7000 only between your nodes and 9042 only to your applications, over IPv4 and IPv6 alike.
Should I mirror the NVMe drives in each node?
Usually not. The Cassandra docs prefer RAID 0 or separate disks to RAID 1, because every row already has copies on other nodes. On the NVMe cartridges that points to one larger drive per node.
Which Cassandra and Java versions should I install?
Cassandra 5.0 is the current GA series, and its install guide asks for the latest Java 11 or Java 17, plus Python 3.8 to 3.13 for cqlsh. The older 4.1 and 4.0 series are still maintained.

Build your Cassandra cluster on NVMe

Pick three whole NVMe machines in the city your users are in, each on an unmetered port, with no contract to sign.

Choose your Cassandra nodes

Read next