The Principles of event streaming: Partitions, Consumers, and Scale Limits
Apache Kafka is the standard event streaming platform used to ingest and process high-volume message traffic in real time. Inside Kafka, a message queue is organized into Topics, which are physically partitioned across a cluster of broker nodes. Sizing this architecture requires evaluating both write throughput limits (MB/s) and consumer processing capacities.
Partitions are the unit of scale in Kafka. They allow writes and reads to occur in parallel. When a message is written to a topic, the producer determines which partition receives it (typically by hashing a key, like user ID, to distribute messages). The physical ingress write rate is: I_(ingress) (MB/s) = (N_(messages) × S_(message))/1024 where N_(messages) is the message rate per second, and S_(message) is the average message size in kilobytes (KB).
To optimize downstream data storage and database processing limits, you can check database limits using the database sharding capacity planner or track cluster connection requirements using the load balancer concurrency planner. Sizing partitions correctly prevents messaging bottlenecks.
Let's calculate the partition requirements for a stream ingesting 25,000 messages/second with an average message size of 4 KB, under a 3x replication factor. The physical write ingress is: I_(ingress) = (25000 × 4)/1024 = 97.66 MB/s. At a 3x replication factor, the disk write traffic across the cluster is: D_(write) = (97.66 × 3600 × 3)/1024 = 1030 GB/hr (1.03 TB/hr). Sizing disk volume retention is critical to prevent brokers from running out of storage.
To determine partitions based on write throughput, we apply the partition write limit: P_(write) = ⌈ I_(ingress)/(L_(partition-write)) ⌉ where L_(partition-write) is the maximum safe write throughput of a single partition (typically capped at 10 MB/s to prevent disk I/O saturation). For our stream, we need at least: P_(write) = ⌈ 97.66 / 10 ⌉ = 10 partitions to handle the incoming write traffic safely.