The Principles of Routing: Active Connections and HTTP Keep-Alive
A load balancer (or reverse proxy like NGINX, HAProxy, or AWS ALB) acts as the entry point for all incoming application traffic, distributing requests across a pool of backend servers. Sizing this infrastructure component requires evaluating both throughput limits (RPS/Gbps) and concurrency limits (active TCP connections). If a load balancer runs out of socket descriptors or memory, it will refuse new connections, triggering gateway timeout errors.
The primary driver of concurrency load is HTTP Keep-Alive. Keep-Alive is an option that allows a client to reuse a single TCP connection to send multiple HTTP requests, avoiding the latency overhead of establishing a new connection for every file. The number of concurrent active connections is: C_(active) = R_(ingress) × T_(keep-alive) where R_(ingress) is the incoming requests per second (RPS), and T_(keep-alive) is the keep-alive duration in seconds.
To optimize downstream application capacity, you can trace container requirements using the Kubernetes HPA replica simulator or plan socket buffers with the TCP throughput latency calculator. Properly configuring load balancer bounds is critical to cluster safety.
Let's calculate the connection capacity for a load balancer handling 10,000 RPS at peak, with a standard 15-second HTTP keep-alive timeout. Applying the concurrency formula: C_(active) = 10,000 × 15 = 150,000 active TCP sessions. Sizing the load balancer server requires allocating socket memory buffers: Memory Footprint = (C_(active) × S_(socket-buffer))/1024 where the socket buffer is typically 16 KB. Sizing yields: $150,000 × 16 / 1024 = 2,343.75 MB (2.34 GB)$$ of RAM just to hold socket metadata.
If the server lacks sufficient RAM to allocate these socket buffers, the kernel's network stack will drop incoming packets, causing clients to experience connection timeouts. Furthermore, the operating system limits the number of file descriptors a process can open (controlled by the `ulimit -n` setting on Linux). Sizing this limit to exceed your peak active connections (e.g., setting it to 200,000) is a prerequisite for high-concurrency load balancing.