Kafka Architecture: A Founder's Guide (2026)

The Weekend I Ran a Three-Broker Cluster on My Laptop

I already told the story of the 1 AM night I almost put Apache Kafka behind Clickly, my little Bitly clone with zero users. That was in what a Kafka topic actually is, and the short version is: I closed the tabs and went to bed.

But a few months later, in 2024, the itch came back. Not to ship Kafka — to understand it. So one Saturday I did the nerd thing. I wrote a Docker Compose file, spun up three Kafka brokers on my laptop, pointed a producer at them, and watched my fan spin up like the machine was trying to take off. Three JVMs, a couple of gigs of RAM gone, and a clicks topic that nobody was clicking.

It was the most useful weekend I spent that year, because I finally saw the shape of the thing instead of the marketing. Kafka isn’t one program. It’s a small distributed system with maybe five moving parts, and every one of them exists to solve a problem you have to genuinely, painfully have before it makes sense. So let me walk you through the whole machine — brokers, partitions, consumer groups, replication, and the big 2026 change — the way I wish someone had drawn it for me before my fan screamed. And then, like always, I’ll tell you why you shouldn’t run it yet.

---

The Cluster: Brokers Are the Machine

Start here, because everything else hangs off it. A broker is just one Kafka server — one process running on one machine. A cluster is several brokers working together. When people say “we run Kafka,” they mean “we run a cluster of brokers,” and that plural is the whole reason Kafka is heavy.

A single broker holds some of your data and serves some of your traffic. When you add a second and a third, the cluster spreads the load across all of them. That’s the point: no single machine has to hold everything or answer every request. This is also why my laptop groaned — I was running three of these servers on a machine built to run a browser.

If you’ve read the topic post, you know a topic is a named, append-only log. Here’s the part that post didn’t cover: a topic doesn’t live on one broker. It gets sliced up and scattered across the whole cluster. The slices are partitions, and they’re where the real architecture begins.

---

Partitions: Where the Parallelism Actually Lives

Everyone thinks the topic is the important unit. It isn’t. The partition is. A topic is really just a name stamped on a bunch of partitions, and each partition is an independent append-only log that lives on a broker.

I covered partitions from the “how a topic stays ordered” angle before, so here I want to hit the thing that matters for architecture: a partition is the atom of parallelism. Everything Kafka does to go fast, it does by having more partitions.

Say your clicks topic has six partitions spread across your three brokers — two each. Now six producers can write at once, to six different logs, on three different machines. There’s no single bottleneck, no one file everyone is fighting over. That’s how Kafka pushes toward a million messages a second when it’s tuned, versus a traditional broker sitting in the thousands-to-tens-of-thousands range. It’s not magic. It’s just many logs, spread out, written in parallel.

The catch, which bit me: order is only guaranteed inside one partition. Kafka routes messages with the same key — say, a user ID — to the same partition every time, so one user’s events stay ordered while different users scatter and get processed at once. That partition count is the single most consequential number in your architecture, because it also caps how many readers can work in parallel. Which brings us to the readers.

---

Consumer Groups: How Kafka Splits the Work

This is the part that finally made Kafka click for me, and it’s the part most “intro to Kafka” posts fumble.

A single consumer reading a busy topic will fall behind. Obviously. So Kafka lets you run several copies of the same consumer and label them with a shared group ID — that’s a consumer group. The group as a whole reads the entire topic, but Kafka hands each partition to exactly one member of the group. Six partitions, three consumers in the group? Kafka gives each consumer two partitions. Scale up to six consumers, and each gets one. That’s how you process a firehose in parallel without two workers grabbing the same message.

Two things fall out of this that are worth burning into memory:

• A partition maps to at most one consumer in a group. So if you have six partitions, a seventh consumer in that group just sits there idle — there’s nothing left to give it. Your partition count is the ceiling on your parallelism. Pick too few partitions early and you can’t scale readers later without a painful reshuffle.
• Different groups are completely independent. Your billing service (group A) and your analytics service (group B) each read the whole orders topic without stepping on each other, because each group tracks its own offsets — its own bookmarks. That “many independent readers of one durable log” property is the entire reason Kafka exists, and it’s what a queue like RabbitMQ fundamentally can’t do (I put those two head-to-head in RabbitMQ vs Kafka).

When a consumer joins or dies, the group rebalances — Kafka pauses briefly and reassigns partitions across the surviving members. It’s automatic, it’s clever, and in production it’s also a well-known source of latency spikes that people write long blog posts about tuning. File that under “problems you have to earn.”

---

Replication: Why a Dead Broker Doesn’t Lose Your Data

Spread your data across three machines and you’ve created a new problem: machines die. If a broker holding a partition falls over and that partition only existed there, your data is gone. Kafka’s answer is replication, and it’s genuinely elegant.

Every partition is copied to multiple brokers. The number of copies is the replication factor — the default for auto-created topics is 3. Of those copies, one is the leader and the rest are followers. All reads and writes go to the leader; the followers do nothing but frantically copy the leader’s log, staying as caught-up as they can.

The followers that are fully caught up are called the in-sync replicas (the ISR). Here’s the durability guarantee that matters: a write isn’t considered committed until every in-sync replica has it. So when the leader broker dies, Kafka just promotes one of the in-sync followers to leader, and not a single committed message is lost. With a replication factor of 3, two of your brokers can be on fire and you’re still serving data.

That’s beautiful engineering. It’s also three copies of everything, on three machines, all running full-time, all needing memory and disk and network — for a redirect service with no users. Hold that thought.

---

KRaft: The Year Kafka Finally Killed ZooKeeper

If you spun up Kafka before 2025 — like I did that Saturday — there was a fourth thing running that nobody enjoyed: Apache ZooKeeper. For most of Kafka’s life, the brokers stored the data, but a totally separate ZooKeeper cluster stored the metadata — which broker leads which partition, who’s in the cluster, what topics exist. Two distributed systems to operate instead of one. It was the single most complained-about part of running Kafka, and it’s why my laptop was juggling even more processes than the three brokers alone.

That era is over. Kafka 4.0, released March 18, 2025, removed ZooKeeper entirely. All that metadata now lives inside Kafka itself, managed by a built-in consensus protocol called KRaft (Kafka Raft) running on a set of controller brokers. The 4.x line — current in 2026 — is KRaft-only; there’s no ZooKeeper option to fall back to.

Why should a founder care about an internal plumbing change? Because it’s the clearest signal in years that even Kafka’s own maintainers knew the operational weight was too much. Killing ZooKeeper makes a Kafka cluster meaningfully simpler to run: one system, faster failovers, faster startup, no second thing to babysit at 3 AM. It’s a real improvement. It just doesn’t change the fact that you’re still running a cluster.

---

So Do You Actually Need Any of This? (No)

Read back over what we just built. Brokers plural. Partitions tuned to a parallelism ceiling you have to guess in advance. Consumer groups that rebalance and spike. Three replicated copies of every message across three machines. A KRaft controller quorum electing leaders. This is a genuinely gorgeous distributed system — and it is aimed squarely at a problem you do not have.

Let me put the real number on it, the same one from the topic post: a self-hosted production cluster on cloud VMs runs somewhere around $850–$1,500 a month just for infrastructure, before you count the human hours to keep it alive. Managed softens the ops pain, not the concept — Confluent Cloud’s serverless tier bills roughly $0.11 per GB in, $0.11 per GB out, and about $0.10 per GB-month stored, and a moderate workload lands near $200/month. Amazon MSK is in the same neighborhood.

Now remember who you are. You’re the person from the survival guide who felt physically ill paying $90/month with $0 MRR. This architecture — the thing I ran on my laptop just to see it — is not on your critical path to your first ten users.

You don’t have a partitioning strategy problem. You have a “nobody uses my product yet” problem. No amount of brokers fixes that.

Use a Postgres table instead. If you’re already running Postgres — and you should be — you have a queue hiding in it. A plain table plus SELECT ... FOR UPDATE SKIP LOCKED gives you an atomic, race-free job queue where every worker grabs a different row, in the same transaction as your data. Libraries like pg-boss wrap it up so you don’t hand-roll it. That’s your “consumer group.” That’s your parallelism. And it handles far more load than you think, comfortably into tens of thousands of jobs before you’d feel it. When Postgres finally groans, a $5 Redis box is the next rung, and RabbitMQ the one after. Kafka is rungs beyond all of them.

The whole of Clickly’s tracking ran on a Postgres table. It was fine. The three-broker cluster taught me exactly one thing worth having: what I was choosing not to build.

---

The Bottom Line

Kafka’s architecture is a cluster of brokers holding topics that are sliced into partitions — the true unit of parallelism — read by consumer groups that split those partitions one-to-one, kept safe by replication across in-sync copies, and coordinated since Kafka 4.0 by KRaft instead of ZooKeeper. Every piece exists for a reason, and every reason is “we are moving a firehose of events that many independent systems need, forever, at massive scale.” That’s a wonderful problem. It’s a problem you earn by winning.

If you can’t name three separate services that each need to independently read and replay the same stream of events, you don’t need this machine. You need a table, a couple of workers, and all the energy you just saved poured into the only thing that decides whether you make it: getting someone to actually use the thing you built.

Draw the architecture on the back of an electricity bill if you want. Run three brokers on your laptop one Saturday and watch the fan scream — honestly, do, it’s a great weekend. Then close the tabs and ship with the boring stack that costs you a dollar.

This is the Broken Engineer Guide. I over-engineer everything, fail at business, and share the scars so you don’t have to earn them at 1 AM. Go build something — with a Postgres table.