1. The problem: why load balance?

Assume Instagram's API sees ~600,000 requests per second at peak (an assumption — see Capacity estimation), and one application server can comfortably handle ~1,000 requests per second. You need at least 600 servers just for the API tier — more with headroom. That raises immediate questions:

• Which server should each request go to? The app only knows one hostname. Something must distribute requests evenly, so no server is idle while another melts.
• What if a server dies? At this fleet size, hardware fails every day. Requests must stop going to dead or sick servers automatically, within seconds.
• How do we deploy new code? Restarting servers must not drop users' in-flight requests.
• How do we grow? Adding servers during the evening peak (and removing them at night) must be invisible to clients.

A load balancer answers all four. It is the component that makes horizontal scaling (more machines rather than a bigger machine — see Core principles) work in practice. It shows up in the Instagram diagram right after DNS, and — less visibly — between almost every pair of tiers inside the data center.

2. What a load balancer does

3. Layer 4 vs Layer 7 load balancing

"L4" and "L7" refer to the OSI layers the load balancer looks at (see Networking basics).

3.1 L4 (transport-level) load balancers

An L4 LB sees only the IP and TCP/UDP headers — source/destination IP and port. It picks a backend when a connection starts (usually by hashing the 5-tuple) and then forwards every packet of that connection to the same backend. It never decrypts or parses the payload, so it can't see URLs or cookies, but it is extremely fast (millions of packets per second per machine), works for any protocol (HTTP, gRPC, MQTT, databases, QUIC) and adds microseconds of latency.

Ways an L4 LB forwards packets:

• NAT mode: rewrite the destination IP to the backend's IP; responses must flow back through the LB to be rewritten again.
• Encapsulation / tunnelling (IP-in-IP, GUE): wrap the original packet in a new IP header addressed to the backend. The backend unwraps it and sees the original client IP.
• Direct Server Return (DSR): the backend replies directly to the client, bypassing the LB. Since responses (images, feed JSON) are much larger than requests, the LB only handles the small inbound half — a huge capacity win. Meta's Katran uses IP-in-IP encapsulation with DSR.

3.2 L7 (application-level) load balancers

An L7 LB terminates the client's connection (including TLS), parses the HTTP request, and then opens or reuses a separate connection to a backend. Because it understands requests, it can:

• Route by content: /api/v1/feed/* → Feed service, /api/v1/media/upload → Upload service, header X-Canary: 1 → canary pool.
• Balance per request, not per connection — essential with HTTP/2, where one client connection carries hundreds of requests.
• Retry idempotent requests on another backend, enforce timeouts, add/strip headers, compress, cache, rate limit, and log every request with status codes and latency.

The price: more CPU and memory per request (TLS crypto, HTTP parsing), more latency (sub-millisecond to a few ms), and the LB must understand the protocol.

4. Load-balancing algorithms

The algorithm decides which healthy backend gets the next connection or request. There's no universally best one — it depends on whether requests are uniform, whether servers differ in size, whether state lives on specific servers, and how many independent LB instances are making decisions at the same time.

4.1 Round robin

Send requests to servers in turn: 1, 2, 3, 1, 2, 3… Simple, stateless and perfectly fair if all servers are equal and all requests cost the same. It ignores load: if request #4 is a slow 2-second search query and #7 is a 5 ms "like", server 1 can pile up slow work while round robin keeps feeding it.

4.2 Weighted round robin

Give each server a weight proportional to its capacity. With A (weight 5, a big new machine), B (3) and C (2), out of every 10 requests A gets 5, B 3 and C 2. A good implementation (e.g. NGINX's "smooth" weighted round robin) interleaves them rather than sending A five in a row:

weights: A=5, B=3, C=2   (total 10)
each step: add every server's weight to its running score, pick the highest, subtract the total from the winner
sequence for 10 requests:  A B C A A B A C B A    → A×5, B×3, C×2, nicely spread

Weights are also used for slow start (a freshly started server with a cold cache gets weight 1, ramping to full over a minute) and canaries (new version gets 1% of traffic).

4.3 Least connections

Send the new connection to the server with the fewest active connections. If A has 12, B has 7 and C has 9, B gets it. This adapts automatically to slow requests and uneven work: a server stuck on slow requests accumulates connections and receives fewer new ones. Weighted least connections divides by weight (A: 12/5 = 2.4, B: 7/3 ≈ 2.3, C: 9/2 = 4.5 → B). It's the natural choice for long-lived connections (WebSockets, DM gateways — section 13), where round robin leads to imbalance over time.

4.4 Least response time (latency-aware)

Prefer the server with the lowest recent latency, typically an exponentially weighted moving average (EWMA) of response times, often combined with outstanding requests: score = ewma_latency × (active_requests + 1). If A averages 40 ms with 3 in flight (score 160) and B averages 25 ms with 5 in flight (score 150), B wins. This automatically avoids servers that are slow because of a noisy neighbour, garbage-collection pauses or a bad disk. Risk: latency measurements are noisy, and a server that fails fast (returning errors in 1 ms) looks "fastest" — so combine with error-aware outlier ejection.

4.5 IP hash (source hashing)

server = hash(client_ip) mod N. The same client IP always lands on the same server — a cheap form of stickiness without cookies, usable even at L4. Problems: thousands of users behind one carrier NAT all land on one server (hotspot); a phone changing networks changes IP and therefore server; and when N changes, almost everyone moves:

N = 4 → 5 servers with mod-N hashing:
a key stays put only if hash mod 4 == hash mod 5  → about 1 in 5 keys
→ ~80% of clients are remapped (and lose any server-local state or cache)

4.6 Consistent hashing

Consistent hashing places servers and keys on a hash ring; each key goes to the next server clockwise. Adding or removing one of N servers moves only ~1/N of the keys. It's used when affinity matters: routing the same user/key to the same cache server (keeping its local cache hot), or keeping the same connection on the same backend across many L4 LB instances. Full details, including virtual nodes, are in Consistent hashing.

 hash space: 0 ──────────────────────────────────────────────────────────────── 2^32 (wraps to 0)
                  S1          k1      k2        S2             k3        S3          k4
                  ●           ○       ○         ●              ○         ●           ○
 rule: each key goes to the first server to its RIGHT (wrapping around the end)
       k1, k2 → S2      k3 → S3      k4 → S1 (wraps)
 remove S2           → only k1, k2 move (to S3); k3 and k4 are untouched
 add S4 between k1,k2 → only k1 moves (to S4); everything else stays put
 (with mod-N hashing, the same change would reshuffle almost every key)

Google's Maglev and Meta's Katran use a Maglev-style consistent hashing table so that many independent L4 LB machines, each seeing a different subset of packets (because routers spread packets across them), all pick the same backend for a given connection — and so that adding or removing a backend disturbs as few existing connections as possible.

4.7 Power of two choices (P2C)

Least connections needs an accurate, global view of load. But a large system has many LB instances (or client-side balancers in every service), each with its own partial, slightly stale view. If all of them send to "the least-loaded server", they all pick the same server at the same moment and overload it — herd behaviour. Pure random choice avoids herding but produces noticeable imbalance.

Power of two choices splits the difference: pick two servers at random, send the request to the less loaded of the two.

function pick(servers):
    a = random_choice(servers)
    b = random_choice(servers)          # distinct from a
    return a if load(a) <= load(b) else b   # load = active requests, or EWMA latency × active

The maths is striking: with n servers, the busiest server under random assignment carries roughly log n / log log n more than average; with two random choices it drops to roughly log log n — an exponential improvement from checking just one more server. And because each LB samples different random pairs, they don't herd. Envoy's default "least request" balancer, Finagle (Twitter) and NGINX's random two least_conn all use P2C. It's an excellent default for service-to-service traffic with many independent balancers.

5. Health checks and outlier ejection

A load balancer is only as good as its knowledge of which backends are healthy. Sending 1/600th of traffic to a dead server means 0.17% of requests fail — at 600K QPS, that's 1,000 errors per second.

5.1 Active health checks

The LB periodically probes each backend: a TCP connect (L4), or an HTTP request such as GET /healthz expecting 200 (L7). Typical configuration:

interval: 5s            # probe every 5 seconds
timeout: 2s             # no answer within 2s = failure
unhealthy_threshold: 3  # 3 consecutive failures → mark DOWN   (~15 s to detect)
healthy_threshold: 2    # 2 consecutive successes → mark UP again
path: /healthz          # cheap endpoint: "process is up and ready to serve"

• Liveness vs readiness: liveness = "the process isn't hung" (restart it if not); readiness = "I can take traffic right now" (false while warming caches, during shutdown, or when overloaded). LBs should use readiness.
• Shallow vs deep checks: a deep check that also tests the database seems thorough, but if the shared database hiccups, every backend fails its check at once and the LB ejects the entire pool — turning a partial problem into a total outage. Keep LB health checks mostly about the server itself; handle dependency failures with timeouts, circuit breakers and graceful degradation (see Resilience patterns).
• Check traffic cost: 50 LB instances × 600 backends × one check per 5 s = 6,000 checks/s. Large systems centralise or share health information rather than having every LB probe every backend.

5.2 Passive health checks

The LB watches real traffic: connection failures, timeouts, 5xx responses, resets. Passive checks detect problems instantly on the request path and catch failures that a trivial /healthz misses (e.g. the server returns 200 for health checks but 500 for real requests because a config file is corrupt).

5.3 Outlier ejection

Outlier detection uses passive signals to temporarily remove misbehaving backends. Envoy-style rules, for example:

• Consecutive errors: 5 consecutive 5xx/connection failures → eject for 30 s; the ejection time grows with each repeat offence (30 s, 60 s, 90 s…).
• Success-rate outliers: eject backends whose success rate is far below the pool mean (e.g. more than ~2 standard deviations below).
• Latency outliers: eject hosts with p99 latency far above their peers.
• Max ejection percentage (e.g. 10–30%): never eject more than this fraction of the pool, so a systemic problem (bad deploy, shared dependency) doesn't empty the pool.

6. Session persistence (sticky sessions)

Sticky sessions (session affinity) mean the LB sends all requests from the same user to the same backend. Implementations: an LB-issued cookie naming the backend (SERVERID=app-17), consistent hashing on a user ID or session cookie, or IP hash at L4.

Why would you want it? Because the server holds something about the user in its own memory — a server-side session, a shopping cart, a partially uploaded file, a warm per-user cache. Why it's a problem:

• Uneven load: a few heavy users (or a big corporate NAT) pin one server hot while others idle; the LB can't rebalance.
• Fragile state: when that server crashes or is deployed, its users lose their sessions (logged out, upload lost).
• Harder scaling and draining: new servers receive only new users; draining an old server takes as long as its longest session.
• Hidden coupling: correctness now depends on routing, which breaks on network changes (IP hash) or across regions.

The standard fix — and the reason the Instagram diagram labels the application tier "stateless" — is to move state out of the servers:

• Authentication state lives in a signed token (JWT) or a session ID looked up in a shared store such as Redis (see Authentication).
• Durable data lives in databases and caches (Caching), uploads go to object storage via resumable sessions keyed by an upload ID.
• Any server can then handle any request, so the LB is free to use the best algorithm, eject servers, and autoscale. See Microservices & stateless services.

Legitimate uses of affinity remain: long-lived connections (a WebSocket is inherently tied to one server for its lifetime), cache locality (consistent hashing users to cache nodes improves hit ratio — "soft" affinity where any node still works if the preferred one is down), and per-connection protocol state at L4 (all packets of a TCP or QUIC connection must reach the same backend).

7. TLS termination

Where should HTTPS be decrypted? Three options:

Big consumer platforms terminate client TLS (and QUIC) at the edge L7 proxies, as close to the user as possible, because handshakes cost round trips (see Networking basics). The edge then re-encrypts traffic over long-lived connections across the backbone to the data center. Using SNI, one LB fleet can serve certificates for many hostnames. TLS termination is a serious CPU cost at scale — modern CPUs with AES instructions and session resumption keep it manageable — and the private keys on these machines are among the most sensitive secrets in the company (see Security).

8. Connection draining and deploys

Deploying new code means restarting every server, typically in batches (a rolling deploy). Simply killing a process would abort its in-flight requests — at 1,000 requests/second per server with 100 ms average latency, ~100 requests are in progress at any moment. Connection draining (AWS calls it "deregistration delay") removes a server gracefully:

1. Mark draining: the deploy system tells the LB (or the server starts failing its readiness check) that this backend should get no new connections or requests.
2. Finish in-flight work: existing requests complete normally. For HTTP/2 and HTTP/3, the server sends a GOAWAY frame so clients open new streams on another connection; HTTP/1.1 responses carry Connection: close.
3. Wait with a deadline: e.g. up to 30 s for API servers. Anything still open after the deadline is closed (clients retry idempotent requests).
4. Stop, upgrade, start: deploy the new version and wait for it to warm up (load config, open DB pools, fill caches).
5. Re-enable: readiness passes → the LB adds it back, ideally with slow start (gradually increasing weight) so a cold server isn't flooded.

Draining long-lived connections (WebSockets, DM gateways, MQTT) is harder because they may last hours: the server instead tells clients to reconnect (a "please reconnect" control message or close frame) gradually and with random jitter, so thousands of clients don't reconnect in the same second. Section 13 covers this.

9. Avoiding the load balancer as a single point of failure

We added an LB so that no single server failure takes the site down — but now the LB is a single box everything passes through. If it fails, everything fails. Three common patterns, often combined:

9.1 Active–passive pair with a floating IP (VRRP)

Two LB machines share a virtual (floating) IP. The active one owns it; the standby monitors the active using VRRP (Virtual Router Redundancy Protocol — implemented by tools like keepalived) heartbeats. If heartbeats stop, the standby claims the IP (announcing it with a gratuitous ARP so the network switches send traffic to it) within a second or a few. Simple and classic for small/medium deployments. Downsides: half the hardware sits idle; capacity is capped at one machine; connection state is lost on failover unless synchronised; and a network partition can cause split brain (both think they're active).

9.2 Active–active: multiple LBs behind DNS

Run several independent LBs (or LB clusters), each with its own IP, and return all of them from DNS (multi-value or round-robin records, with health checks). All capacity is used. Failover depends on DNS TTLs and on clients trying another IP — so it's best for coarse failover between sites, not for instant recovery of one box.

9.3 Active–active: anycast and ECMP

Many LB machines announce the same VIP to the upstream routers via BGP. Routers use ECMP (Equal-Cost Multi-Path) to spread flows across all of them by hashing each packet's 5-tuple, so a given connection's packets consistently reach the same LB machine. If an LB dies, its BGP session drops and routers stop sending to it within seconds; because all LBs use the same consistent-hashing table to pick backends, the connections that move to another LB machine still reach the same backend. The same VIP can also be announced from many PoPs worldwide (anycast), so users reach the nearest site. This is how Katran, Google's Maglev and cloud NLBs scale out "horizontally" with no single LB box in the path.

10. Global vs local load balancing, and LB tiers

10.1 GSLB vs local load balancing

GSLB is covered in depth in DNS and Multi-region. A global platform also needs capacity-aware global balancing: if the Mumbai PoP is full during an evening peak, some Indian users should spill over to Singapore rather than queue — the global map is recomputed continuously from load data.

10.2 Tiers: edge L4 → L7 → service

At large scale, load balancing isn't one box but a pipeline of tiers, each doing what it's best at:

• Tier 0 — GSLB (DNS + anycast): get the user to a good site.
• Tier 1 — L4: a stateless, very fast packet-level layer that spreads connections across many L7 machines using consistent hashing. It makes the L7 fleet horizontally scalable and absorbs SYN floods cheaply.
• Tier 2 — L7: terminates TLS/QUIC, parses HTTP, applies routing rules, retries and limits, and forwards to the right backend cluster over pooled connections.
• Tier 3 — service-level: the API gateway routes to services; inside the mesh, each service calls others through client-side load balancing (a library or sidecar like Envoy that picks an instance from service discovery, e.g. with P2C) — no central LB hop at all. See Microservices.

11. Hardware vs software load balancers

The industry trend at hyperscale is clear: software load balancers on commodity servers. Google publicly described Maglev (2016), and Meta open-sourced Katran (2018), an L4 load balancer built on the Linux kernel's XDP and eBPF that processes packets at the network driver level — before the normal kernel networking stack — so a single commodity server can forward millions of packets per second. At L7, Meta uses its open-source Proxygen HTTP framework as its edge and data-center reverse proxy, with HTTP/2 and HTTP/3 support. For almost everyone else, a managed cloud LB or HAProxy/NGINX/Envoy is the right answer.

12. Autoscaling integration

Load balancers and autoscalers work as a team: the autoscaler decides how many servers exist; the LB decides which one gets each request. The integration points:

1. Metrics: the LB reports requests per second per target, latency and errors; together with CPU, these drive scaling policies, e.g. "keep ~700 RPS per instance" (target tracking).
2. Scale out: new instances boot and register in the LB's target group (or service discovery). They join only after passing readiness checks, then receive gradually increasing traffic (slow start) while caches and JIT warm up.
3. Scale in: the instance is first deregistered and drained (section 8), then terminated — never the other way round.
4. Headroom: scaling takes minutes (boot, warm-up), so keep spare capacity for sudden spikes and use predictive/scheduled scaling for known daily peaks.

Assume evening peak 600K RPS, target 700 RPS/instance (70% of the 1,000 RPS max for headroom):
  needed = 600,000 / 700 ≈ 858 instances
Night trough 150K RPS → 150,000 / 700 ≈ 215 instances  → ~640 instances released for ~10 hours/day
If a region fails and its traffic shifts here (+50%), you need ~1,290 instances —
  either pre-provisioned headroom or the ability to scale out within minutes.

13. Load balancing long-lived WebSocket connections

Most load-balancing intuition assumes short requests. Real-time features — Instagram's Direct messaging, typing indicators, live notifications — use connections that stay open for minutes or hours (WebSocket, MQTT). That changes the problem:

• The unit of load is the connection, and it's sticky by nature: once a WebSocket is established, every message on it goes to the same server. You balance only at connection time.
• Use least connections (or P2C on connection count/memory), not round robin. Each connection costs memory (buffers, TLS state, maybe 10–50 KB) and heartbeat traffic, so capacity is "max connections per server" — assume ~500K–1M mostly idle connections per well-tuned gateway server.
• Imbalance after events: when a server restarts or new servers are added, existing connections stay where they are. New servers sit nearly empty while old ones are full. Fixes: least-connections routing for new connections; and gentle rebalancing by asking a fraction of clients on overloaded servers to reconnect.
• Draining takes long: you can't wait hours for connections to end during a deploy. The server sends a "reconnect" message and closes connections gradually with jitter — e.g. spreading 500K reconnects over 5–10 minutes — so clients don't stampede the LB and auth service (a thundering herd). Clients reconnect with exponential backoff and jitter, and resume from their last acknowledged message ID.
• Timeouts and heartbeats: LBs and NATs drop idle connections (e.g. a default idle timeout of 60 s on many cloud LBs). Configure long idle timeouts for WebSocket listeners and send application heartbeats more often than the shortest timeout on the path.
• Routing messages to the right server: since a user's connection lives on one specific gateway server, the system keeps a presence/session registry (user → gateway server) so other services can deliver messages to that server. This is covered in Direct messaging.
• Protocol support: an L7 LB must support the HTTP Upgrade handshake (or HTTP/2 extended CONNECT); an L4 LB just forwards the TCP connection. For QUIC, L4 LBs route by connection ID (not the 5-tuple) so a connection survives the client's IP change — Katran supports this.

 Before deploy (least-connections):      During gradual drain of GW-2:          After
 GW-1 ███████████ 400K                   GW-1 ████████████████ 600K             GW-1 ████████ 300K
 GW-2 ███████████ 400K   send "reconnect"GW-2 ████ 120K (draining, jittered)    GW-2 ████████ 300K (new version)
 GW-3 ███████████ 400K   to 2% per 10 s  GW-3 ███████████████ 480K              GW-3 ████████ 300K
                                                                                 GW-4 ████████ 300K (autoscaled)

14. How Instagram's request passes through the LB tiers

Follow a single "load my feed" request from a phone in Mumbai:

1. GSLB: Meta's authoritative DNS maps the user's network to a nearby edge PoP (Mumbai) based on latency, capacity and health, returning that PoP's VIP. See DNS.
2. Routers + ECMP: the VIP is announced via BGP by many Katran machines in the PoP; the PoP's routers hash each packet's flow to one of them.
3. Edge L4 (Katran): looks up the connection (or QUIC connection ID) in its consistent-hash table and encapsulates the packet (IP-in-IP) to one of many Proxygen L7 machines. Responses go straight back to the client (DSR), so Katran only handles the small inbound half of traffic.
4. Edge L7 (Proxygen): terminates QUIC/TLS a few milliseconds from the user, parses the HTTP request, and forwards it over a long-lived, pre-warmed connection across Meta's backbone to the best data-center region for this request.
5. Data-center L4 → L7: the same two-tier pattern again inside the region: an L4 tier spreads connections over an L7 proxy fleet, which routes by host/path to the right backend cluster with health checks, outlier ejection and retries.
6. API gateway → service: the gateway authenticates the token, applies rate limits and routes to the Feed service; inside the service mesh, each hop (Feed → Graph, Feed → Redis) uses client-side load balancing. Media URLs in the response send the app to the CDN, a separate path with its own L4/L7 load balancing.

Rough numbers

Assume 600K API RPS at peak worldwide, spread over ~30 edge PoPs → ~20K RPS per PoP on average
  (big PoPs like Mumbai much more, maybe 60–100K RPS)
L4 tier: a Katran-style XDP box can forward millions of packets/s → a handful per PoP suffices
L7 tier: assume ~5–10K TLS/QUIC RPS per proxy server → 10–20 proxies for a 100K RPS PoP (+ headroom)
Per-region app tier: 600K RPS / 3 regions = 200K RPS ÷ 700 RPS per instance ≈ 290 instances per region
If a 1-in-600 dead server isn't ejected: 0.17% errors ≈ 1,000 failed requests per second globally

15. Trade-offs & common pitfalls

Common pitfalls

• A single LB with no failover — you've just moved the single point of failure.
• Deep health checks on shared dependencies that eject the entire pool when the database blips; no panic threshold or max-ejection limit.
• Round robin for long-lived connections → badly unbalanced WebSocket servers after deploys.
• L4 balancing of HTTP/2 or gRPC: one long connection carries all of a client's requests, so per-connection balancing pins huge load to one backend. Balance per request at L7 or client-side.
• Sticky sessions as a crutch for server-local state, causing hotspots and lost sessions on restarts.
• No connection draining → errors on every deploy; or draining without jitter → reconnect stampedes.
• Retrying everything at the LB: retries of non-idempotent requests duplicate actions, and retries during overload multiply load (use retry budgets).
• Idle timeouts mismatched between client, LB and server (e.g. LB closes at 60 s while the server keeps the connection), causing random resets.
• Losing the client IP behind proxies, breaking rate limiting and abuse detection (pass X-Forwarded-For / PROXY protocol, and trust it only from your own LBs).
• Mod-N hashing for affinity — adding one server reshuffles nearly every key; use consistent hashing.

16. Real-world technologies

17. Interview corner

18. Check your understanding

Related lessons

• DNS — global server load balancing, GeoDNS and anycast
• Networking basics — L4 vs L7, TCP, TLS and QUIC under the hood
• API gateway — the L7 tier that adds auth, rate limiting and routing
• Consistent hashing — the ring and virtual nodes behind affinity routing
• Microservices — stateless services, service discovery and client-side load balancing
• Direct messaging — long-lived connections and session registries
• Resilience patterns — retries, timeouts, circuit breakers and load shedding
• Multi-region — failing over between regions
• Core principles — horizontal scaling and single points of failure