Load Balancing Strategies
How a load balancer spreads traffic, the main algorithms and when each fails, layer 4 versus layer 7, health checks, and why consistent hashing moves 19% of keys instead of 80% when you add a server.
A load balancer sits in front of several servers, spreads requests across them and stops using servers that fail health checks. Common strategies are round robin, least connections and hashing on a client or key. Interviewers expect consistent hashing: when a server is added or removed, only about 1/N of keys move, not nearly all.
A load balancer sits in front of several servers and spreads incoming requests across them, so no single server is overloaded and one failure does not take the service down. Common strategies are round robin, least connections, and hashing on the client or key. Consistent hashing keeps most keys in place when servers are added or removed.
- A layer 4 load balancer forwards TCP or UDP traffic by address and port, while a layer 7 load balancer parses HTTP and can route on paths, headers or cookies.
- In a simulation of 1,000 requests where 10% cost 50 units, round robin across 4 servers left loads from 985 to 1,769, while least-loaded kept all four within 44 of each other.
- Mapping 10,000 keys with hash(key) % N moved about 80% of keys when going from 4 to 5 servers, so nearly every server cache went cold.
- With consistent hashing and 100 virtual nodes per server, going from 4 to 5 servers moved 18.8% of 10,000 keys, close to the ideal 1/5, and all moved keys went to the new server.
- Sticky sessions pin a user to one server with a cookie or IP hash, while keeping session state in a shared store such as Redis lets any server handle any request.
01What a load balancer does
One server has a limit, so you run several identical ones and put a load balancer in front. Clients talk to the balancer's address. It picks a backend for each request or connection and forwards it.
Three jobs come out of that. It spreads load so no server is overwhelmed. It runs health checks, and stops sending traffic to servers that fail. And it hides the fleet, so you can add, remove and upgrade servers without clients noticing.
In practice you meet nginx, HAProxy, Envoy, and the cloud versions such as AWS's Application and Network Load Balancers.
02Layer 4 versus layer 7
A layer 4 balancer works at the TCP/UDP level. It sees addresses and ports, not the content, so it is fast and simple and works for any protocol. It cannot tell one URL from another.
A layer 7 balancer understands HTTP. It can route /api to one pool and /images to another, route on a header or cookie, rewrite URLs, terminate TLS, and cache or compress. It costs more CPU per request because it has to parse the traffic.
A common design is an L4 balancer in front for raw speed and availability, with L7 balancers behind it for smart routing.
03Round robin and weighted round robin
Round robin sends request 1 to server A, 2 to B, 3 to C, then back to A. It needs no state and is a fine default when requests cost about the same and servers are alike.
Weighted round robin gives a server with twice the capacity twice the turns. It also helps during a gradual rollout, where a new version gets a small weight first.
Its weakness is that it ignores how busy each server actually is. When request costs vary a lot, round robin can pile slow requests on one unlucky backend.
04Least connections
Least connections tracks open connections per server and picks the lowest. If one server is stuck on slow requests, it accumulates connections and gets fewer new ones.
To see the difference I simulated 1,000 requests where 90% cost 1 unit of work and 10% cost 50 (a few heavy requests, like a report), spread over 4 servers.
import random
random.seed(1)
costs = [random.choice([1]*9 + [50]) for _ in range(1000)]
def round_robin(costs, n):
load = [0] * n
for i, c in enumerate(costs):
load[i % n] += c
return load
def least_loaded(costs, n):
load = [0] * n
for c in costs:
load[load.index(min(load))] += c
return load
print(round_robin(costs, 4)) # [985, 1524, 1769, 1524]
print(least_loaded(costs, 4)) # [1434, 1435, 1478, 1455]Round robin left the busiest server with 1,769 units against 985 for the quietest, a spread of 784. Least-loaded kept all four within 44 of each other. (This simulation picks by total work, which a real balancer only approximates with connection counts. Variants such as least response time use measured latency.)
05Hashing, and why consistent hashing exists
Sometimes you want the same client or key to reach the same server every time, for example so that a server's local cache stays warm. Hash the key and pick hash(key) % N.
The problem shows up when N changes. I mapped 10,000 keys onto 4 servers with modulo, then added a fifth server. About 80% of the keys changed servers, so nearly every cache was suddenly cold.
Consistent hashing fixes this. Place servers at points on a circle (a ring), and send each key to the next server clockwise from its hash. Adding a server only takes over the arc just before it:
import hashlib, bisect
def h(s): return int(hashlib.md5(s.encode()).hexdigest(), 16)
class Ring:
def __init__(self, nodes, vnodes=100):
self.ring = sorted((h(f"{n}#{i}"), n) for n in nodes for i in range(vnodes))
self.points = [p for p, _ in self.ring]
def get(self, key):
i = bisect.bisect(self.points, h(key)) % len(self.points)
return self.ring[i][1]With the same 10,000 keys, going from 4 to 5 servers moved 18.8% of them, close to the ideal 1/5, and every moved key went to the new server. The vnodes setting places each server at 100 points on the ring. With only 1 point per server the split was badly uneven (135, 283, 3951 and 5631 keys for four servers), and with 100 points the five servers each held between about 1,780 and 2,290 keys.
06Health checks, stickiness and the balancer itself
Active health checks have the balancer probe each server (for example GET /health every few seconds) and take it out of rotation after a few failures. Passive checks watch real traffic for errors and timeouts. A good health endpoint checks that the app can really do work, such as reaching its database, but not so deeply that one slow dependency makes every server look dead at once.
Sticky sessions pin a user to one server with a cookie or IP hash, because their session lives in that server's memory. It works but makes the balance uneven and loses sessions when that server dies. A better design keeps session state in a shared store such as Redis, so any server can serve anyone.
The balancer itself can fail. Run at least two, with a floating IP or DNS that points to both. Cloud load balancers are already redundant across zones. For worldwide traffic, DNS-based geo routing sends users to the nearest region.
07Common interview questions
Q: Layer 4 versus layer 7 load balancing?
A: L4 forwards TCP or UDP by address and port, which is fast and protocol-agnostic. L7 parses HTTP and can route on paths, headers or cookies, and can terminate TLS.
Q: Which algorithm for requests of very different cost?
A: Least connections or least response time. Round robin ignores the load.
Q: Why consistent hashing?
A: When a server is added or removed, only about 1/N of the keys move, instead of almost all of them with hash mod N.
Q: What are virtual nodes?
A: Many points per server on the ring, which evens out the distribution and spreads a failed server's load across several others.
Q: How do you avoid the load balancer being a single point of failure?
A: Run a redundant pair (active-passive or active-active) with a floating IP or DNS, or use a managed balancer spread across zones.
Q: What are sticky sessions and the alternative?
A: Routing a user to the same server using a cookie or IP hash. The alternative is shared session storage so any server can handle any request.