What Is a Kafka Topic? (And Whether You Actually Need One)

The Night I Almost Put Kafka Behind a URL Shortener

In 2023 I was building Clickly, my little Bitly clone. If you’ve read my accidental database degree, you already know how that went: a URL shortener has maybe three moving parts, and I found a way to spend three weeks benchmarking databases I would never use.

Somewhere in that spiral, at about 1 AM, I had the thought every over-engineer eventually has. Every click is an event. Events should go on a stream. I should put Kafka in front of my analytics.

I opened a dozen tabs. I read about brokers and partitions and consumer groups. I sketched an architecture on the back of an electricity bill. (If you want that whole picture — brokers, partitions, replication, consumer groups — I laid it out in Kafka architecture.) I felt like a real engineer building a real system. And then I did the math on what running Apache Kafka would cost a guy with zero users, and I closed all the tabs and went to bed.

That night saved me a couple hundred dollars a month and a lot of pain. So let me do for you what past-me needed someone to do: explain exactly what a Kafka topic is in plain English, and then talk you out of using it — for now.

---

What a Kafka Topic Actually Is (Plain English)

Forget the diagrams for a second. A Kafka topic is a named log. That’s the whole idea.

Picture a notebook where you only ever write on the next blank line, and you’re never allowed to erase or reorder anything. Every new entry goes at the bottom. Once it’s written, it’s frozen. The notebook has a name — clicks, signups, orders — and that name is the topic.

That’s it. A topic is a category of messages, stored as an ordered, append-only list. The programs that write to it are called producers. The programs that read from it are consumers. Kafka sits in the middle holding the notebook, letting many producers write and many consumers read at once.

The part that trips people up: a Kafka topic is not a queue. In a normal queue, you take a message out and it’s gone. In Kafka, reading a message doesn’t remove it. Ten different consumers can read the same orders topic completely independently — your billing service, your analytics, your email sender — and none of them affects the others. The log just sits there, and each reader keeps its own bookmark. That “everyone reads the same durable log” design is the whole reason Kafka exists, and it’s why it was built at LinkedIn to move activity data around a company at scale before Confluent was spun out to commercialize it.

---

Partitions, Offsets, and the Log That Never Forgets

If a topic were a single notebook, one writer could hog it and everything would crawl. So Kafka splits each topic into partitions — think of one topic as several notebooks that together make up clicks.

Partitions are how Kafka goes fast

Each partition is its own independent append-only log, and Kafka spreads them across different machines (brokers). More partitions means more writers and readers working in parallel, which is where Kafka’s famous throughput comes from — it can push toward a million messages a second when it’s tuned, versus a traditional broker sitting in the thousands-per-second range.

There’s a catch that matters: ordering is only guaranteed inside a single partition, not across the whole topic. So how do you keep all events for one user in order? You give the message a key — say, the user’s ID — and Kafka always routes the same key to the same partition. Same user, same notebook, guaranteed order. Different users can be scattered across partitions and processed in parallel. That one trick — key-based routing — is the thing to actually understand about topics.

Offsets are just bookmarks

Every message in a partition gets a number called an offset — 0, 1, 2, 3, counting up forever. A consumer doesn’t ask Kafka “what’s new?” It says “give me everything from offset 4,102 onward.” Because each consumer tracks its own offset, you can rewind. Reprocess yesterday. Replay the last month into a brand-new service you just built. The log remembers, and that replay ability is genuinely useful — it’s also exactly the feature you don’t need on day one.

---

Topics vs. Queues: Why Kafka Isn’t RabbitMQ

This is where most “do I need Kafka?” confusion lives, so let me be blunt about it.

RabbitMQ is a message broker. Its job is to hand a task to a worker, watch the worker finish it, and then forget the task ever existed. One job, done once, gone. It’s push-based and it’s great at “run this thing in the background.” (I put the two head-to-head in RabbitMQ vs Kafka if you’re actually choosing between them.)

Kafka is an event streaming platform. Its job is to keep a durable log that many services read from at their own pace, possibly replaying history. It’s pull-based and built for “broadcast what happened so anyone can react, now or later.”

Most broke founders reaching for Kafka actually want the RabbitMQ thing — “send this email in the background,” “process this upload.” They’ve just heard “Kafka” more often because it’s the name that shows up in FAANG engineering blogs. Picking Kafka for a job-queue problem is like buying a shipping container to mail a birthday card.

---

So… Do You Actually Need Kafka? (Almost Certainly Not Yet)

Here’s the hard truth, and it’s the same one from the database rabbit hole: the tool is real, powerful, and almost never what a bootstrapper needs on day one.

Look at what Kafka costs the moment you leave “localhost demo” behind. A self-hosted production cluster on cloud VMs runs somewhere around $850–$1,500 a month just for the infrastructure — before you count the human hours to babysit it. Managed options soften the ops pain but not the concept: Confluent Cloud’s serverless Basic tier bills roughly $0.11 per GB in, $0.11 per GB out, and about $0.10 per GB-month stored, and a genuinely moderate workload lands around $200/month. Amazon MSK is in the same neighborhood once it’s real.

Now remember who you are. You’re the person from the survival guide who felt physically ill paying $90/month with $0 MRR. Kafka is not a $5 Railway box you forget about. It’s a system with a personality, and it wants attention.

You don’t have a data-streaming problem. You have a “nobody uses my product yet” problem. Kafka fixes neither.

The signal that you don’t need Kafka: you can’t clearly name three different services that all need to independently read the same stream of events and occasionally replay history. If you’re squinting to invent a third consumer, you’re rationalizing. Been there, on the back of that electricity bill.

---

What to Use Instead When You’re Broke

Almost every “I need Kafka” instinct maps onto something you already have running. Here’s the honest ladder, cheapest first.

Just use Postgres. If you’re already running PostgreSQL (and you should be — it’s the only database answer you need for 99% of SaaS), you have a queue hiding in it. A plain table plus SELECT ... FOR UPDATE SKIP LOCKED gives you an atomic, race-free job queue where every worker grabs a different row. No new service, no new bill, and your jobs live in the same transaction as your data — so nothing gets half-done in a crash. Libraries like pg-boss wrap this up for Node so you don’t hand-roll it. Real teams have ripped out RabbitMQ and replaced it with exactly this, and their stack got simpler. This handles far more load than you think — comfortably into tens of thousands of jobs before you’d feel it.

Reach for Redis when Postgres feels heavy. If you want a dedicated, blazing-fast queue for background work, Redis with a library like BullMQ is the classic move. You can run it for a couple dollars, or use a serverless option like Upstash so you pay for what you touch. This covers “send the welcome email,” “resize the image,” “generate the PDF” — the actual jobs you have.

Use RabbitMQ when you truly have many workers and complex routing. If you outgrow a database table and genuinely need push-based delivery to a fleet of workers with fancy routing rules, RabbitMQ is the grown-up, boring, correct choice — and still an order of magnitude simpler to reason about than Kafka.

Nine times out of ten you stop at the first rung. I built the whole of Clickly’s tracking on a Postgres table and it was fine. The Kafka cluster I designed at 1 AM never wrote a single message — right around the same time I was also convincing myself I needed to run it all on a Kubernetes cluster. Same disease, twice in one weekend.

---

When Kafka Finally Earns Its Keep

I’m not anti-Kafka. It’s brilliant at the thing it’s for. You’ll know you’ve arrived when you can honestly say:

• Multiple independent services need to react to the same firehose of events — analytics, fraud, notifications, a data warehouse — each at its own pace.
• You genuinely need to replay history: rebuild a service’s state by re-reading the last month of events.
• You’re moving real volume, the kind where “just use a table” starts groaning — think hundreds of thousands to millions of events flowing through, continuously.

That’s a company with revenue and traffic, not a solo founder chasing their first ten users. And even then, if you want Kafka’s model without the JVM-and-cluster-babysitting operational tax, look at Redpanda — it speaks the Kafka protocol but ships as a single binary with no separate coordination service to run, so your existing Kafka code works while your ops burden drops. That’s the version of “adding Kafka” I’d actually respect from a small team.

But that’s a later problem. A good one. The problem you earn by having so many users and events that a log becomes the sanest way to move data around. Get there first.

---

The Bottom Line

A Kafka topic is a named, append-only log — split into partitions for speed, numbered with offsets so readers can bookmark and replay, keyed so related events stay in order. It’s not a queue; it’s a stream that many services can read forever. That’s genuinely worth understanding, because knowing what a tool is means you can decide, on purpose, not to use it.

And right now, you almost certainly shouldn’t. Not because Kafka is bad — because it’s a war machine for a battle you haven’t reached. Your Postgres table is enough. Your $5 Redis is enough. The engineering energy you’d pour into brokers and partitions and consumer groups is energy stolen from the only thing that actually decides whether you make it: getting someone to use the thing you built.

Learn what a Kafka topic is. Sketch it on an electricity bill if you want. Then close the tabs, go to bed, and ship with the boring stack that costs you a dollar.

This is the Broken Engineer Guide. I over-engineer everything, fail at business, and share the scars so you don’t have to earn them at 1 AM. Go build something — with a Postgres table.