Deep Dive: Building a Distributed Message Broker from Scratch

Table of contents

Why I Built a Distributed Message Broker from Scratch

Understanding how Apache Kafka works is one thing; actually building it is another. To truly grasp memory-mapped logs, sequential storage, offset management, and consensus, I decided to build a “Mini-Kafka” in Java from scratch.

This write-up breaks down the engineering design decisions I made, how my implementation maps to Kafka’s core components, and the lessons learned in low-level concurrency and systems storage.

1. The Storage Engine: The Heart of the Log

Original Kafka:

Kafka’s performance secret isn’t magic; it’s Sequential I/O. Instead of using complex B-trees (like MySQL) or Hash Maps (like Redis), Kafka uses a Distributed Partitioned Append-Only Log.

My Implementation: LogSegment.java & PartitionLog.java

• Immutability: Once a KafkaRecord is written, it is never modified. This avoids lock contention and makes caching highly effective.
• Segmentation: In the original Kafka, logs are split into “segments” to allow for easy cleanup of old data. I modeled this with LogSegment, where records are stored as JSON lines.
• Storage Layout:

  kafka-data/
  └── order-topic/
      ├── partition-0/
      │   └── 0000000000.log  <-- Persistent storage
      └── partition-1/
          └── 0000000000.log

• Sequential Writes: By using StandardOpenOption.APPEND, my implementation ensures the OS writes data to the end of the file, leveraging high-bandwidth sequential disk speeds.

2. Horizontal Scaling: Topics & Partitions

Original Kafka:

A Topic is a logical stream of data, but a Partition is the physical unit of scalability. Multiple partitions allow multiple brokers to share the load.

My Implementation: KafkaBroker.java

• Routing: I implemented a ConcurrentHashMap of topics, where each topic maps to multiple PartitionLog objects.
• Parallelism: Each partition operates independently with its own file locks. This mirrors Kafka’s ability to scale writes linearly with the number of partitions.
• Recovery: On startup, my broker scans the dataDir to reconstruct the topic/partition metadata, ensuring durability across restarts.

3. The Consumer Model: Offsets & Pull-based Protocol

Original Kafka:

Unlike RabbitMQ (which pushes data and tracks consumer state), Kafka consumers Pull data and track their own Offsets. This makes Kafka “stateless” from the broker’s perspective.

My Implementation: MiniKafkaConsumer.java

• Stateful Polling: My consumer maintains a local partitionOffsets map. When poll() is called, it asks the broker for data “starting from X”.
• Replayability: Because the broker doesn’t delete data after it’s read, a consumer can simple “reset its offset” to 0 and re-read the entire history of the topic—a cornerstone feature of Kafka.
• Fan-out: Multiple consumers can read from the same partition at different speeds because the offset is a simple pointer in the log.

4. Producer Intelligence: Partitioning Strategies

Original Kafka:

The producer decides which partition a message goes to, not the broker.

My Implementation: MiniKafkaProducer.java

• Key-Hashing: I implemented semantic partitioning. If a message has a key (e.g., user_id), the producer ensures all messages for that user go to the same partition (key.hashCode() % numPartitions). This preserves message ordering for that specific key.
• Round-Robin: For load balancing, if no key is provided, the producer rotates through available partitions.
• Metadata Caching: To avoid hitting the network on every send, the producer fetches and caches the partition count via a DESCRIBE_TOPIC command.

5. The Wire Protocol: Custom TCP

Original Kafka:

Kafka uses a optimized binary protocol over TCP to reduce the overhead of headers (like HTTP).

My Implementation: KafkaServer.java & KafkaCommand.java

• Command Dispatcher: I used a custom text-based protocol (COMMAND ARG1 ARG2...) handled by a multi-threaded ServerSocket.
• Enum-driven Logic: Using a KafkaCommand enum makes the server easily extensible. Adding a new feature like DELETE_TOPIC is just a new case in the switch block.
• JSON Serialization: While real Kafka uses a binary format, I used Jackson JSON serialization to keep the data human-readable for learning and debugging.

6. Metadata Consensus: KRaft & Raft

Original Kafka:

Legacy Kafka used Zookeeper. Kafka 3.x+ uses KRaft—an internal Raft-based metadata log.

My Implementation: RaftNode.java

This was the most complex part to replicate. I built a simplified Raft Consensus Engine:

• Leader Election: When the broker starts, it enters a FOLLOWER state. If it doesn’t hear a heartbeat, it becomes a CANDIDATE, increments its term, and votes for itself.
• Log Replication: Every metadata change (like “Create Topic”) is treated as a RaftRecord. It is only “committed” once the leader confirms it is stored safely.
• The Result: This ensures that even in a multi-node cluster, all brokers have a single, consistent “Source of Truth” for what topics exist.

Core Engineering Takeaways

Building this broker highlighted several core systems engineering challenges:

1. Thread Safety under Load: Managing concurrent client readers and writers on the same log files required careful locking and thread-safe collections.
2. Crash-Safe Persistence: Making sure that write failures or sudden crashes don’t corrupt the offset state or log segment files.
3. Consensus Protocols: Coding the state transitions (Follower, Candidate, Leader) of the Raft election algorithm and keeping term counts synchronized.
4. Network Optimization: Reducing round-trips by implementing metadata caching on the producer side.

Why Kafka Wins

Building this broker made it clear why Kafka is so dominant. Its simplicity—the append-only log—is its core strength. By eliminating complex indexing on the broker and pushing offset state tracking entirely to the consumer, Kafka achieves a level of write throughput and simplicity that traditional brokers can’t match.

Project source code and implementation documentation can be found in this repository.