The Distributed Commit Log

At its heart, Apache Kafka is not a traditional message queue. It is a Distributed Commit Log.

In a traditional queue (like RabbitMQ), when a message is consumed, it is deleted. In Kafka, messages are appended to a log and stay there until a retention policy expires them (e.g., 7 days).

Consumers track their own progress using an Offset (a bookmark). This allows different consumers to read the same stream at different speeds and replay history if needed.

1. Topics: The Logical Stream

A Topic is a category or feed name to which records are published.

• Logical: To the user, it looks like a single stream of data (e.g., user-clicks, payment-processed).
• Physical: Under the hood, a topic is split into Partitions.

2. Partitions: The Unit of Scalability

This is the single most important concept in Kafka.

If a topic were stored on a single machine, it would be limited by that machine’s I/O and storage. To solve this, Kafka breaks a topic into Partitions (P0, P1, P2…).

• Distributed: Partitions are spread across different brokers in the cluster.
• Parallelism: You can have as many consumers processing a topic as you have partitions.
• Ordering: Guaranteed ONLY within a partition. There is no global ordering across the entire topic.

[!IMPORTANT] Why Partitions? Partitions allow Kafka to scale horizontally. If you need to handle 10 GB/sec of data, you simply add more partitions and more brokers.

Analogy: The Supermarket Checkout

Imagine a supermarket with a single checkout lane. No matter how fast the cashier works, there is a physical limit to how many customers can be served per minute.

• One Lane (No Partitions): Slow. Customers queue up.
• 10 Lanes (10 Partitions): Fast. 10 cashiers work in parallel.
• Consumer Group: The team of cashiers.
• Partitioning Key: How you decide which lane to join (e.g., “10 items or less” lane, or “A-M” surnames).

3. Hardware Reality: The Physics of Sequential I/O

Why is Kafka so fast? Why does it use a log instead of a B-Tree like a relational database?

The answer lies in the physics of hard drives (and even SSDs).

• Random Access: Jumping around the disk requires the drive head to physically move (seek time). This is slow (milliseconds).
• Sequential Access: Writing to the end of a file (appending) is incredibly fast because the drive head doesn’t need to move.

[!NOTE] The Numbers:

• Random I/O: ~100 IOPS (Input/Output Operations Per Second) on HDD.
• Sequential I/O: ~100MB/sec to 500MB/sec.
• Hardware Reality: Modern hard drives (and SSDs) are heavily optimized for sequential access. By strictly appending to the end of partition logs, Kafka leverages the OS Page Cache and avoids expensive disk head seeks. This turns disk writes into O(1) operations, achieving speeds comparable to network throughput. It also prevents cache line thrashing and false sharing since data is written linearly in memory before being flushed.

4. Offsets: The Immutable Sequence

Each message within a partition is assigned a unique integer ID called an Offset.

• Immutable: Once written, a message never changes.
• Monotonic: Offsets strictly increase (0, 1, 2, 3…).
• Local: Offset 5 in Partition 0 is completely different from Offset 5 in Partition 1.

5. Message Keys & Partitioning

How does Kafka decide which partition a message goes to?

Strategy A: Round Robin (No Key)

If you send a message with key=null, Kafka distributes it in a round-robin fashion (or using the Sticky Partitioner for efficiency) to balance the load across all partitions.

• Pros: Maximize load balancing.
• Cons: No ordering guarantee for related messages.

Strategy B: Semantic Partitioning (With Key)

If you provide a key (e.g., user_id=101), Kafka guarantees that all messages with the same key go to the same partition.

• Mechanism: Partition = Hash(Key) % NumPartitions
• Benefit: Strict ordering for that specific user. User 101’s “Login” will always appear before their “Logout”.

[!WARNING] The Hot Partition Problem: If one key is extremely popular (e.g., “Justin Bieber” in a twitter stream), that specific partition will be overwhelmed while others are idle. Choose your keys carefully!

6. Consumer Groups

A Consumer Group is a set of consumers acting as a single logical subscriber.

• Load Balancing: Kafka automatically assigns partitions to consumers in the group.
• Rule of 1: A partition is consumed by exactly one consumer within a group.
• Scalability Limit: You cannot have more active consumers than partitions. If you have 10 partitions and 11 consumers, 1 consumer will sit idle.

Fan-Out Architecture

Multiple different consumer groups can read from the same topic independently.

• Group A (Fraud Service): Reads payments to detect fraud.
• Group B (Analytics): Reads payments to build dashboards.
• Both groups see every message, but manage their own offsets.

7. Interactive: The Partitioning Simulator

Visualise how keys determine message placement. Note how null keys cycle round-robin, while specific keys stick to specific partitions.

Message Key Key: null (Round Robin) Key: "user_123" (Hash → P0) Key: "user_456" (Hash → P1) Key: "user_789" (Hash → P2)

Producer

🏭

P0

P1

P2

System Ready. Select a key to begin.

8. Code: Controlling Partitioning

Here is how you control partition assignment using the Producer API.

Java

// Java: Sending a message with a Key (Strict Ordering)
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class ProducerExample {
  public static void main(String[] args) {
  Properties props = new Properties();
  props.put("bootstrap.servers", "localhost:9092");
  props.put("key.serializer", StringSerializer.class.getName());
  props.put("value.serializer", StringSerializer.class.getName());

  // Use try-with-resources to ensure the producer is closed
  try (Producer<String, String> producer = new KafkaProducer<>(props)) {
    String key = "user_123"; // Messages with this key always go to the same partition
    String value = "login_event";

    // Send with Key
    producer.send(new ProducerRecord<>("my-topic", key, value));

    // Send without Key (Round Robin / Sticky)
    producer.send(new ProducerRecord<>("my-topic", "some_random_event"));
  }
  }
}

Go

// Go: Using segmentio/kafka-go
package main

import (
  "context"
  "log"
  "github.com/segmentio/kafka-go"
)

func main() {
  w := &kafka.Writer{
  Addr:     kafka.TCP("localhost:9092"),
  Topic:    "my-topic",
  Balancer: &kafka.Hash{}, // Guarantees Key -> Partition mapping
  }
  // Ensure writer is closed
  defer w.Close()

  // Message with Key
  err := w.WriteMessages(context.Background(),
  kafka.Message{
    Key:   []byte("user_123"),
    Value: []byte("login_event"),
  },
  )
  if err != nil {
  log.Fatal("failed to write messages:", err)
  }

  // Message without Key (Note: Hash balancer sends all nil keys to same partition)
  err = w.WriteMessages(context.Background(),
  kafka.Message{
    Value: []byte("some_random_event"),
  },
  )
  if err != nil {
  log.Fatal("failed to write messages:", err)
  }
}

9. Real-World Async & Ordering

If you are new to Kafka, the most important paradigm shift is moving from Synchronous (Request/Response) to Asynchronous (Event-Driven) architectures.

• The Old Way (Synchronous): Service A calls Service B. Service A must wait for Service B to finish. If B is slow, A is slow. If B is down, A fails.
• The Kafka Way (Asynchronous): Service A writes an event to a Kafka Topic (e.g., user_registered) and immediately returns a success to the user. Service B, C, and D consume that event at their own pace. Service A doesn’t care if Service B is currently down; the event is safely stored in the Kafka log until B wakes up.

Why Partitioning Matters for Scale

A single Topic is a logical concept. Physically, a single server can only handle so much traffic. By dividing a Topic into multiple Partitions, Kafka allows you to distribute the data across dozens of servers (Brokers). This means you can have dozens of consumers reading in parallel.

The Ordering Trade-off

Because data is split across multiple partitions, Kafka cannot guarantee global ordering. If you write Event 1 to Partition 0, and Event 2 to Partition 1, a consumer reading from both partitions might read Event 2 before Event 1.

The Solution: If order is critical (e.g., in financial transactions where “Deposit $10” must happen before “Withdraw $10”), you must use a Partitioning Key (like account_id). Kafka guarantees that all events with the same key will always go to the same partition, thus guaranteeing strict sequential processing for that specific entity.