Building a Mini-Kafka from Scratch in Java: Step-by-Step Learning Notes

Table of contents

Designing a Distributed Commit Log

When developers transition from traditional message queues (like RabbitMQ or ActiveMQ) to Apache Kafka, they often struggle with a fundamental paradigm shift: Queue vs. Commit Log.

In a traditional queue, messages are transient. The broker tracks consumption state, and as soon as a consumer acknowledges a message, it is deleted from the server. This makes multi-consumer replay or back-dating consumption pointers incredibly expensive or impossible.

Kafka flips this model on its head. It is a distributed, append-only commit log. The broker is essentially a high-performance disk-writer that stores events sequentially. The consumers maintain their own read offsets (pointers) and pull data at their own pace. Because the data is persistent, multiple consumers can read the same partition from different offsets, and messages can be replayed infinitely within the retention window.

To understand how this operates under the hood, we built a fully functional Mini-Kafka in Java. In this post, we break down the implementation step-by-step, analyzing why each component was designed the way it was.

The System Architecture

Here is the high-level layout of our lightweight message broker:

                           +------------------------+
                           |  Producer Application  |
                           +-----------+------------+
                                       | 
                                       | 1. Sends: PRODUCE topic partition key value
                                       v
                               +---------------+
                               |  Mini-Kafka   | (Socket listener on 9092)
                               |    Broker     | (ThreadPool worker handles connection)
                               +-------+-------+
                                       |
                                       | 2. Appends to disk sequentially
                                       v
                              +-----------------+
                              | Storage Engine  | -> Writes to data/topics/<topic>/partition-<id>/
                              +-----------------+
                                       ^
                                       | 3. Pulls: FETCH topic partition offset
                                       |
                           +-----------+------------+
                           |  Consumer Application  | (Maintains local read pointers)
                           +------------------------+

Our implementation is divided into 6 distinct steps, mirroring Kafka’s architectural pillars:

1. The Data Model (Record): Serialization and formatting.
2. The Storage Node (StorageEngine): Partitioning, log writing, thread safety, and crash recovery.
3. The Messaging Protocol (Protocol): Custom text-based command parser.
4. The TCP Server (Broker): Concurrency via thread pooling.
5. The Publisher Client (Producer): Client-side metadata discovery and semantic hashing.
6. The Poller Client (Consumer): Pull-based loop and local offset pointers.

Step 1: The Core Data Model (Record.java)

Before writing data to disk or transferring it over the wire, we need a standard representation for a message. In Record.java, we define a Record:

public class Record implements Serializable {
    private long offset;
    private long timestamp;
    private String key;
    private String value;
}

Why this design?

• Offset & Timestamp: Every record must be uniquely identifiable inside its partition. The offset acts as a logical sequence number (like an array index), while the timestamp allows time-based indexing and retention policies.
• Key-Value Split: Separating the key from the payload (value) is critical. The key is used by the producer to route the message to the correct partition, while the value remains an opaque byte payload (or string in our case) for the application.
• Text-Based Custom Serialization: Instead of importing heavy binary serialization libraries (like Protobuf or Avro) or verbose JSON engines, we serialize the record as a comma-separated line:

  String escapedKey = key == null ? "null" : key.replace(",", "%2C");
  String escapedValue = value == null ? "null" : value.replace(",", "%2C");
  return String.format("%d,%d,%s,%s", offset, timestamp, escapedKey, escapedValue);

  Why escape commas? Commas are our field delimiters. If a payload contains a comma, it would break our simple splitter on parse. By URL-encoding commas to %2C during serialization and decoding them back on parsing, we achieve bulletproof serialization with zero dependencies and high performance.

Step 2: Append-Only Partition Persistence (StorageEngine.java)

Kafka’s performance is largely credited to sequential I/O. Modern operating systems optimize sequential disk access heavily via read-ahead page caches, making linear writes almost as fast as memory access.

In StorageEngine.java, we simulate this behavior by writing messages to append-only log files structured around partition directories:

data/
└── topics/
    └── orders/
        ├── partition-0/
        │   └── 0000000000.log  <-- Append-only log file
        ├── partition-1/
        │   └── 0000000000.log
        └── partition-2/
            └── 0000000000.log

Why this design?

• Directory isolation per partition: By keeping partitions in separate folders, we isolate disk operations. Writing to partition-0 does not lock or affect partition-1. This matches Kafka’s physical storage layout perfectly.
• ReentrantReadWriteLock per Partition: Our broker handles multiple concurrent connections. If multiple clients produce and fetch messages simultaneously, we must prevent race conditions without crippling performance. Instead of globally synchronizing the append and fetch methods, we use a map of locks:

  private final ConcurrentHashMap<String, ReentrantReadWriteLock> partitionLocks = new ConcurrentHashMap<>();

  For each topic-partition, we obtain a dedicated ReentrantReadWriteLock.
  • Write locks are held when appending to the log, ensuring that only one writer appends to a physical file at a time.
  • Read locks are held when consumers fetch from the log. Multiple consumers can read from the same log file concurrently without blocking each other.
• Startup Recovery: If the broker crashes, we must recover the next active offset. When a write occurs on a partition for the first time after boot, the storage engine scans the log file to locate the maximum offset:

  private long calculateNextOffset(Path partitionDir) {
      // Reads the log file, parses records, and returns (maxOffset + 1)
  }

  This guarantees that restarts do not overwrite existing data, preserving durability.

Step 3: Custom TCP Protocol (Protocol.java)

Communication between Kafka clients and brokers occurs over raw TCP sockets.

In Protocol.java, we implement a raw text-based protocol. The command parser processes simple string messages from the wire:

• PRODUCE <topic> <partition> <key> <value> -> Response: OK <offset>\n
• FETCH <topic> <partition> <offset> -> Response: MESSAGES <count>\n[record1]\n[record2]...\n
• METADATA <topic> -> Response: METADATA <topic> PARTITIONS <count>\n

Why this design?

• Raw TCP over HTTP: HTTP carries heavy overhead (headers, status codes, cookies, chunked transfer formatting) and enforces a request-response model that can be inefficient for streaming. Raw TCP connections are persistent, lightweight, and let us control the packet format directly.
• Text-Based Protocol: While real Kafka uses a highly optimized binary protocol (for minimizing packet sizes), we opted for a text-based ASCII format. This makes debugging simple: you can open a terminal and interact with your broker using telnet localhost 9092 or nc localhost 9092 directly.

Step 4: Parallel Connection Handling (Broker.java)

The central component is the TCP server defined in Broker.java.

public class Broker {
    private static final int PORT = 9092;
    private final ExecutorService clientThreadPool;

    public Broker(String dataDir) {
        this.clientThreadPool = Executors.newCachedThreadPool();
    }
}

Why this design?

• Cached Thread Pool: Instead of blocking the main thread or creating a thread per connection without limits, we manage connections using a CachedThreadPool. Executors.newCachedThreadPool() is ideal because it dynamically scales to handle incoming clients and tears down idle worker threads after 60 seconds of inactivity. This prevents memory leaks while supporting spikes in consumer/producer counts.
• Stateless Broker Routing: The broker does not know or care who the consumer is. When it receives a request, it delegates to Protocol.parseRequest(line), executes the operation against the StorageEngine, and writes the response. By keeping the broker stateless, we maximize scalability.

Step 5: Client-Side Partition Routing (Producer.java)

A major design decision in Apache Kafka is Dumb Broker, Smart Client. In legacy brokers, the server is responsible for routing messages, clustering, and consumer tracking. In Kafka, the client does the heavy lifting.

In Producer.java, the producer initiates connection and queries partition info before sending any event payloads:

// 1. Fetch metadata to discover topic partitions
String metadataRequest = "METADATA " + topic + "\n";
out.write(metadataRequest.getBytes(StandardCharsets.UTF_8));
out.flush();
String metadataResponse = in.readLine();
int numPartitions = parsePartitionCount(metadataResponse);

Why this design?

• Semantic Routing (Hashing): Once the producer knows the partition count, it routes messages dynamically. If a key is provided, we guarantee that all messages with the same key go to the same physical partition:

  int partition = Math.abs(key.hashCode()) % numPartitions;

  In distributed systems, this is how ordering guarantees are made. Since single-partition logs are appended sequentially, routing all orders for user_101 to partition-0 ensures they are read in the exact order they were created.
• Broker CPU Offloading: By calculating the destination partition client-side, we offload routing logic from the broker. The broker does not need to parse metadata or compute hash functions on every write; it simply accepts the destination partition and appends it directly to disk.

Step 6: Pull-Based Consuming & Client Offsets (Consumer.java)

In Consumer.java, we implement the consumer client. The consumer runs a polling loop, querying the broker at regular intervals:

while (true) {
    for (int partition = 0; partition < numPartitions; partition++) {
        long currentOffset = partitionOffsets.get(partition);
        String request = String.format("FETCH %s %d %d\n", topic, partition, currentOffset);
        out.write(request.getBytes(StandardCharsets.UTF_8));
        out.flush();
        // Read response & parse messages ...
        partitionOffsets.put(partition, record.getOffset() + 1);
    }
    Thread.sleep(POLL_INTERVAL_MS);
}

Why this design?

• Why Pull over Push? In push-based brokers, the broker pushes messages as fast as possible. If a consumer is slow or performing complex computations, its buffer overflows, causing crashes or packet loss. In a pull-based model, the consumer requests messages only when it is ready. If a consumer falls behind, messages safely queue up on the broker’s disk, and the consumer catches up when resources free up.
• Stateless Offsets: The consumer maintains its own read pointers (partitionOffsets map). The broker does not track which client has read what. This gives us message replayability. If the consumer crashes or needs to re-process yesterday’s data due to a bug, it simply updates its local offset map back to 0 and restarts. The broker will happily serve the logs from the beginning.

Summary of Learnings

By recreating Kafka’s architecture in Java, we uncovered the fundamental reasons behind its design:

Building systems from scratch is the ultimate way to demystify complex enterprise tools. With just six classes and Java’s raw socket API, we replicated the core data flows of one of the world’s most robust data systems.

The full source code of this project can be found in the mini-kafka directory of this workspace.