Load Balancing
Load balancing is the process of distributing incoming network traffic across multiple servers to ensure no single server is overwhelmed, improving availability, throughput, and response times.
What It Really Means
Imagine a single web server handling all your traffic. At 100 requests per second it is fine. At 10,000, it buckles. At 100,000, it crashes. You cannot keep buying bigger servers forever — there is always a ceiling. The alternative is horizontal scaling: run many servers and spread traffic across them.
A load balancer sits between clients and your server fleet. It receives every incoming request and decides which backend server should handle it. If a server goes down, the load balancer stops sending traffic to it. If you add new servers, the load balancer starts including them. Clients never know how many servers are behind the scenes — they talk to one address, and the load balancer does the rest.
Load balancing operates at different layers of the network stack, uses various algorithms to make routing decisions, and is often the first line of defense for both scalability and reliability. Every large-scale system — from Google Search to Netflix streaming — depends on multiple layers of load balancing.
How It Works in Practice
Layer 4 vs Layer 7 Load Balancing
Layer 4 (Transport Layer) load balancers route based on IP address and TCP port. They do not inspect the HTTP request content. This makes them extremely fast — they can handle millions of connections per second.
Google's Maglev is a Layer 4 load balancer that uses consistent hashing to distribute packets across backend servers. It runs on commodity hardware and handles Google's entire frontend traffic.
Layer 7 (Application Layer) load balancers inspect HTTP headers, URLs, cookies, and request bodies. They can make smarter routing decisions: send API traffic to one server pool, static assets to another, and WebSocket connections to a third.
Netflix uses Layer 7 load balancing with Zuul to route requests based on URL patterns, user segments, and canary deployment rules. A request to /api/profiles goes to the profile service fleet, while /api/recommendations goes to a completely different set of servers.
Load Balancing Algorithms
Round Robin: Requests go to servers in sequential order (1, 2, 3, 1, 2, 3...). Simple and fair when all servers have equal capacity and all requests have equal cost.
Weighted Round Robin: Servers with more capacity get more traffic. A 16-core server gets twice the traffic of an 8-core server.
Least Connections: Route to the server with the fewest active connections. Works well when request processing times vary significantly.
Consistent Hashing: Hash the request (e.g., by user ID) to deterministically route to the same server. Used when you want session affinity or cache locality without sticky sessions.
Random with Two Choices (Power of Two): Pick two random servers and route to the one with fewer connections. Surprisingly effective — provides near-optimal load distribution with minimal coordination.
Health Checks
A load balancer must know which servers are healthy. It periodically sends health check probes:
- TCP check: Can I open a connection to port 8080? (basic liveness)
- HTTP check: Does GET /health return 200? (application-level health)
- Deep health check: Does the server respond within 100ms and return valid data? (thorough but expensive)
Amazon ALB performs health checks every 30 seconds by default. If a target fails 2 consecutive checks, it is marked unhealthy and removed from rotation. After 3 consecutive successes, it is added back.