Your own self hosted Kafka server, sized and locked down

Will one server carry your event stream?

For development, testing and a modest pipeline, yes. Producers write events into named streams called topics; consumers read them back. The docs set no minimum machine but name what matters:

• Memory: recent data sits in the page cache, the spare memory Linux uses to hold files. The Hardware and OS page suggests room for 30 seconds of writes, so your write rate times 30. The Java Version page's typical settings give Kafka a 6 GB heap.
• Disks: speed is usually the limit, more drives help, and Kafka's data should not share a drive with the system or other logs. XFS or ext4 both work (XFS did better in the project's tests), mounted with noatime.
• Open files: allow the broker at least 100,000 to start with.
• Partitions: topics are split into partitions, and every log segment takes two memory-map areas. With about 65,535 allowed on many Linux systems, tens of thousands of partitions mean raising vm.max_map_count.
• Java: 17, 21 and 25 are fully supported. Use Linux; Windows is not well supported.

Our sums: 20 MB/s of writes needs about 600 MB of cache, plus heap. Disk follows how long you keep data, 168 hours by default: at that rate a week fills about 12 TB.

The server that fits

Kafka does best when nothing else competes for its memory and disks, so a dedicated server suits it: every core, all the memory and every drive are yours, you log in as root, and with an unmetered port a dozen consumers each reading the whole stream never add a line to your bill. A ready server is yours within 30 minutes of approval. You manage the server; we keep the hardware and the network running.

• First broker, staging or a side project: an everyday single-processor server. Rack servers and blades give you a console screen and power button of your own, even when the system will not boot.
• Heavier traffic and more partitions: a two-processor machine from the serious-workloads group, for the extra memory.
• Separate data drives: a build to order is where you choose drives and RAID or JBOD; it takes 4-24 hours when parts we hold are fitted, 5-10 working days when they must be bought in.
• Cluster controllers: the docs expect roughly 5 GB of RAM plus 5 GB of disk apiece, so small machines do, such as an NVMe cartridge. Only two cartridge models, m510 and m710x, come with a console; any other cartridge can just be powered on and off. The ARM cartridge installs Ubuntu 16.04 or 18.04 only, too old for this guide.

Busier than a gigabit? See 10 Gbps servers.

Why your apps keep connecting to localhost

The broker told them to. listeners is where Kafka opens its ports; advertised.listeners is the address it hands out. The protocol guide explains that a client contacts any address you gave it, asks for the cluster's metadata, then talks to whatever host and port each partition's leader advertises. The sample config/server.properties shipped with Kafka advertises localhost:9092, so an app elsewhere gets in, is sent to localhost, its own machine, and stalls. And 0.0.0.0, the broker configs reference adds, cannot be advertised.

So this setup advertises the hostname in NAME, puts that same name in the TLS certificate (clients check it by default) and splits the ports:

• 9092, CLIENT: your apps, over TLS with a SCRAM password. It listens on every address, IPv4 and IPv6, and the firewall lets in only the app servers you name.
• 9093, CONTROLLER: the controller quorum, which holds the cluster's records. The listener guide says to isolate it, since clients have no use for it, so it answers on 127.0.0.1 only.
• 9094, INTERNAL: traffic between brokers and your admin commands on the server itself, 127.0.0.1 only. Treat anything running on this machine as a Kafka administrator.

Kafka's documentation supplies the setting names. Three listeners, port 9094 and a local ANONYMOUS super user are decisions we made for this guide.

Step one: install Kafka and its certificate

Create the DNS record first. This step installs Java 21 (supported in full, though the docs prefer 25), downloads Kafka 4.4.0, refuses it unless it matches Apache's SHA-512 checksum (the downloads page also links KEYS for a signature check), unpacks it under /opt and creates a certificate for NAME lasting 825 days, our choice. Its password goes into /root/kafka-secrets, readable by root alone.

# As root on Ubuntu 24.04. NAME = the hostname your apps will use; DNS must point here.
NAME=kafka.example.net; VER=4.4.0; PKG=kafka_2.13-$VER.tgz; SEC=/root/kafka-secrets; KST=/etc/kafka/broker.p12
if [ "$NAME" = kafka.example.net ] || ! getent ahostsv4 "$NAME" >/dev/null; then echo "STOP - set NAME"
elif [ -e /opt/kafka ] || [ -e $SEC ]; then echo "STOP - installed already"
else
apt-get update && apt-get install -y openjdk-21-jre-headless curl ufw && cd /root &&
curl -fsSLO https://dlcdn.apache.org/kafka/$VER/$PKG && curl -fsSLO https://downloads.apache.org/kafka/$VER/$PKG.sha512 &&
SUM=$(cut -d: -f2- $PKG.sha512 | tr -dc 0-9A-Fa-f | tr A-F a-f) && [ ${#SUM} -eq 128 ] &&
[ "$SUM" = "$(sha512sum $PKG | cut -d' ' -f1)" ] && { id kafka || useradd -r -M -s /usr/sbin/nologin kafka; } &&
mkdir /opt/kafka-$VER && tar -xzf $PKG -C /opt/kafka-$VER --strip-components=1 && ln -s /opt/kafka-$VER /opt/kafka &&
install -d -m 750 -o kafka -g kafka /var/lib/kafka /var/log/kafka && install -d -m 750 -g kafka /etc/kafka &&
install -m 600 /dev/null $SEC && PW=$(openssl rand -hex 24) && [ -n "$PW" ] && printf 'H=%s\nP=%s\n' $NAME $PW > $SEC &&
KP=$PW keytool -genkeypair -keyalg RSA -validity 825 -dname "CN=$NAME" -ext "SAN=DNS:$NAME" \
 -storetype pkcs12 -keystore $KST -storepass:env KP && chgrp kafka $KST && chmod 640 $KST &&
KP=$PW keytool -exportcert -rfc -keystore $KST -storepass:env KP -file /etc/kafka/broker-cert.pem &&
echo "step one done" || echo "STOP - look at the message above"
fi

Did its first run end with STOP - look at the message above? Only then, never after installed already, clean up with rm -rf /root/kafka-secrets /etc/kafka /opt/kafka-4.4.0 /opt/kafka and start again.

Step two: tell Kafka who may talk to it

The server is broker and controller at once, combined mode in the KRaft docs, which advise against it for critical systems. Passwords use SCRAM, which the SASL guide wants only over TLS. With the StandardAuthorizer on, a topic nobody has been granted is open to super users alone. A one-server cluster keeps one copy of its internal topics, like the shipped sample, and topics are never created by accident.

# Run as root once step one is done. Creates /etc/kafka/kraft.properties and will not overwrite it.
. /root/kafka-secrets && CONF=/etc/kafka/kraft.properties
if [ -e $CONF ]; then echo "STOP - $CONF is already there"
else
install -m 640 -g kafka /dev/null $CONF && cat > $CONF <<EOF &&
process.roles=broker,controller
node.id=1
controller.quorum.bootstrap.servers=127.0.0.1:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=INTERNAL
listeners=CLIENT://0.0.0.0:9092,CONTROLLER://127.0.0.1:9093,INTERNAL://127.0.0.1:9094
advertised.listeners=CLIENT://$H:9092,CONTROLLER://127.0.0.1:9093,INTERNAL://127.0.0.1:9094
listener.security.protocol.map=CLIENT:SASL_SSL,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.client.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
ssl.keystore.type=PKCS12
ssl.keystore.location=/etc/kafka/broker.p12
ssl.keystore.password=$P
ssl.key.password=$P
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:ANONYMOUS
auto.create.topics.enable=false
log.dirs=/var/lib/kafka
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
share.coordinator.state.topic.replication.factor=1
share.coordinator.state.topic.min.isr=1
EOF
echo "step two done" || echo "STOP - look at the message above"
fi

If the first run halts with STOP - look at the message above, delete /etc/kafka/kraft.properties and repeat it.

Step three: switch it on with the doors shut

Storage is formatted first; --standalone makes this server the one voting controller, the starting point the docs recommend. The firewall goes up, allowing only SSH, before Kafka opens a port. The service file allows at least 100,000 open files and a 6 GB heap; left unset, the start script picks 1 GB. Finally it checks that 9093 and 9094 answer on 127.0.0.1 and nowhere else.

# Run as root once step two is done: format, firewall on, service up, then a check of the private ports.
BIN=/opt/kafka/bin; LOCAL=127.0.0.1:9094; UNIT=/etc/systemd/system/kafka.service
if [ -e $UNIT ] || [ -e /var/lib/kafka/meta.properties ]; then echo "STOP - this step has run before"
else
CID=$($BIN/kafka-storage.sh random-uuid) && [ -n "$CID" ] && $BIN/kafka-storage.sh format --standalone -t "$CID" \
 -c /etc/kafka/kraft.properties && chown -R kafka:kafka /var/lib/kafka &&
sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable &&
printf '%s\n' '[Unit]' 'Description=Apache Kafka' 'After=network-online.target' 'Wants=network-online.target' \
 '[Service]' 'User=kafka' 'Environment="KAFKA_HEAP_OPTS=-Xmx6g -Xms6g" LOG_DIR=/var/log/kafka' \
 'LimitNOFILE=100000' 'ExecStart=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/kraft.properties' \
 'Restart=on-failure' '[Install]' 'WantedBy=multi-user.target' > $UNIT &&
systemctl daemon-reload && systemctl enable --now kafka &&
for n in $(seq 40); do $BIN/kafka-metadata-quorum.sh --bootstrap-server $LOCAL describe --status >/dev/null 2>&1 && break; sleep 3; done &&
$BIN/kafka-metadata-quorum.sh --bootstrap-server $LOCAL describe --status && ss -Hltn | grep -q '127.0.0.1]*:9093 ' &&
! ss -Hltn | grep -E ':909[34] ' | grep -qv '127\.0\.0\.1' && echo "running; 9093 and 9094 only answer locally" ||
echo "STOP - see journalctl -u kafka -n 50"
fi

Did the first run halt at the journalctl STOP line? Then no data exists yet (never do this after has run before): run systemctl disable --now kafka, then rm -rf /var/lib/kafka/* /etc/systemd/system/kafka.service, and try once more.

Give each app its own key

1. Generate a password nobody else can read: umask 077; openssl rand -hex 24 > /root/shop.pw.
2. Create the user (other local accounts could spot the password with ps while it runs): /opt/kafka/bin/kafka-configs.sh --bootstrap-server 127.0.0.1:9094 --alter --add-config "SCRAM-SHA-512=[iterations=8192,password=$(cat /root/shop.pw)]" --entity-type users --entity-name shop.
3. Let it write to topics whose names begin with shop.: /opt/kafka/bin/kafka-acls.sh --bootstrap-server 127.0.0.1:9094 --add --allow-principal User:shop --producer --topic shop. --resource-pattern-type prefixed. To read too, repeat it with --consumer --topic shop. --group shop- instead of --producer --topic shop..
4. Make a topic with three partitions: /opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9094 --create --topic shop.orders --partitions 3 --replication-factor 1.
5. That server also needs Java and the Kafka tarball for this test. Copy /etc/kafka/broker-cert.pem there and create shop.properties (install -m 600 /dev/null shop.properties) with ssl.truststore.type=PEM, ssl.truststore.location pointing at that copy, security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512 and sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="shop" password="..."; holding the password.
6. Open the door for that server alone: ufw allow proto tcp from 198.51.100.7 to any port 9092, with its real address. Send a test event from it: kafka-console-producer.sh --command-config shop.properties --bootstrap-server kafka.example.net:9092 --topic shop.orders, swapping in your NAME.

When you outgrow one machine

A cluster you trust with data runs three or more controllers, in processes apart from the brokers: N failures need 2N + 1 members. The first controller is formatted --standalone; later ones use --no-initial-controllers and join through the add-controller command of kafka-metadata-quorum.sh; brokers need no extra flag. Every format gets the same cluster ID through -t, and every server lists the controllers in controller.quorum.bootstrap.servers. Kafka's docs offer no way to grow this combined server into that shape: build the cluster new and point producers at it.

Across machines, the controller and broker-to-broker ports can no longer sit on 127.0.0.1. Use SASL_SSL, or SSL plus ssl.client.auth=required: plain SSL makes every peer User:ANONYMOUS, so take that super user out and put the servers' own principals in super.users. Let the firewall admit 9093 from the cluster's own servers and the replication port from the other brokers, by address. Keep three copies of each topic and set min.insync.replicas, the count of in-sync copies an acks=all write must reach.

Keeping it healthy

Disk is what runs out. Data goes after 168 hours (log.retention.hours); size limits are off by default, and each topic can set its own. Check free space on /var/lib/kafka with df -h, and the controller with kafka-metadata-quorum.sh and its describe --replication command.

To upgrade, unpack the new version beside the old, check its checksum, repoint the link (ln -sfn) and restart. Once it runs well, finish with /opt/kafka/bin/kafka-features.sh --bootstrap-server 127.0.0.1:9094 upgrade --release-version plus the new number; that raises the metadata version, which cannot always be lowered again, so keep the old folder until then. From 4.4 on, the broker must be at least as new as whatever version of kafka-storage formatted its disk. Your certificate runs out after 825 days; openssl x509 -in /etc/kafka/broker-cert.pem -noout -enddate tells you when, so make a new one and send apps the new broker-cert.pem in good time.

One server keeps one copy of your events; Kafka's safety comes from replicas. What you can save is the setup apps rely on: umask 077; k=/root/$(date +%F-%H%M%S)-kafka-setup.tgz; tar -czf "$k" -C / root/kafka-secrets etc/kafka && tar -tzf "$k" > /dev/null && echo "saved $k", then move it to another machine and keep it private, since the keystore password is inside. Topics, users and permissions are not. On a replacement with the same NAME: step one, unpack the archive over it as root (tar -xzf FILE -C /), skip step two, step three, then recreate topics, users and permissions.

Kafka's licence is the Apache License 2.0 (its repository's LICENSE file); bundled parts listed in LICENSE-binary keep their own licences, such as MIT, EPL 2.0 and CDDL 1.1. This is our reading and not legal advice. Running it for any purpose, a paid service included, needs no source release. Give Kafka to someone else, changed or as downloaded, and you include the licence, keep the NOTICE text and copyright notices, flag files you changed and respect the bundled licences. Sue anyone claiming Kafka itself infringes a patent and your patent licence under it ends; it gives no right to the Kafka trademarks.

Questions people ask

Do I still have to run ZooKeeper?

No. Since 4.0, Kafka supports KRaft mode only, so its own controllers hold the cluster's records. An old ZooKeeper-based cluster first moves to KRaft on the 3.9 bridge release, as the 3.9 docs explain.

How much memory should a Kafka server have?

There is no official minimum. Take 30 seconds of your peak write rate for the cache, add the 6 GB heap from the typical settings, plus a margin of our own if it also runs the controller (the KRaft page's 5 GB estimate for a controller).

My app reaches the server but never sends anything. Why?

Check advertised.listeners first. After first contact, a client connects to the address the broker advertises. If that is localhost, an internal address or a hostname the certificate doesn't list, the client cannot follow it.