1. The problem: why do systems need resilience patterns?

Opening the Instagram home feed looks like one request, but behind the API Gateway the News Feed Service fans out to many others: the feed cache (Redis), the ranking service, the post metadata store, the like & comment counters, the user service for profile pictures, the Stories tray, ads, and more. Each of those talks to its own databases and caches.

Now do the arithmetic. Suppose each of 10 dependencies is independently available 99.9% of the time. If the feed needs all of them:

P(feed works) = 0.999^10 ≈ 0.990  → 99.0% — about 7 hours of failure per month
With 30 dependencies: 0.999^30 ≈ 0.970  → 97% — the app is broken ~22 hours/month

Adding dependencies multiplies failure probability. At scale, with tens of thousands of servers, hardware failure is not an exception — it is a constant background rate. Disks die daily, a top-of-rack switch fails weekly, a deploy introduces a bug, a garbage-collection pause freezes a JVM for 2 seconds. The goal of resilience engineering is to make the system degrade gracefully: a failure in the like counter should mean "like counts are missing for a minute", not "Instagram is down". (See availability and reliability for the definitions of nines.)

2. Kinds of failure

A key lesson from industry incident reports: slow is worse than dead. A dead dependency returns an error in milliseconds; you handle it and move on. A slow one silently consumes your resources while you wait.

3. Anatomy of a cascading failure

A cascading failure is when the failure of one component causes the failure of the components that depend on it, and so on, until a large part of the system is down. Walk through a realistic one:

1. Origin. A celebrity's post goes viral; one Redis shard holding its like counter hits 100% CPU. Its latency rises from 1 ms to 300 ms.
2. Slowness propagates. The Counter Service calls that shard for every feed that contains the post. Its own latency rises to 3 s. It's not erroring — just slow.
3. Resource exhaustion. The Feed Service has a pool of, say, 200 worker threads per instance. Normally a feed request takes 100 ms, so each instance handles 2,000 req/s. Now every request waits 3 s on the counter call: capacity drops to 200 / 3 s ≈ 67 req/s — a 30× collapse. Requests queue; queue latency grows; memory grows.
4. Healthy dependencies starve. Requests that don't even touch the slow counter can't get a thread. Ranking and User Service sit idle while the feed is "down".
5. Retries amplify. The gateway times out and returns 5xx; apps retry; the gateway may retry too. Load triples exactly when capacity has collapsed.
6. Death spiral. Overloaded Feed instances fail health checks and get removed, concentrating traffic on the remaining ones, which then fail too. Autoscaling adds instances that immediately receive the backlog and also fall over.

Every pattern in the rest of this lesson breaks one link in this chain: timeouts limit how long you wait (step 3), bulkheads stop one dependency consuming all threads (step 4), circuit breakers stop calling a sick dependency, retry budgets stop amplification (step 5), load shedding keeps an overloaded service doing useful work (step 6), and fallbacks let the feed render without counts.

4. Timeouts

A timeout is the maximum time a caller will wait for a response before giving up. Without one, a single hung dependency can hold a thread forever. Many HTTP clients and database drivers default to no timeout or to very long ones (60 s+) — one of the most common root causes of cascading failures.

4.1 Kinds of timeout

• Connect timeout: how long to wait to establish a TCP/TLS connection. Should be short (e.g. 100–500 ms inside a data center) — if a host doesn't accept connections quickly, it's likely dead.
• Request (read) timeout: how long to wait for the response once connected.
• Overall deadline: the total time budget for an operation including retries.
• Idle timeout: close pooled connections unused for a while (so a silently broken connection isn't reused).

4.2 Choosing values

Too long and you get the cascade above; too short and you fail requests that would have succeeded (and trigger retries). A practical method:

1. Measure the dependency's latency distribution (see percentiles). Suppose the Counter Service has p50 = 5 ms, p99 = 40 ms, p99.9 = 90 ms.
2. Set the timeout a bit above a high percentile — e.g. ~100–150 ms. You accept that ~0.1% of calls time out (and maybe get one retry), in exchange for never waiting seconds.
3. Check it fits in the caller's own budget. If the feed endpoint has a 500 ms SLO and calls 3 dependencies sequentially, their timeouts plus retries must sum to less than that.
4. Make timeouts configurable at runtime, and alert on timeout rates.

4.3 Deadline propagation

Per-hop timeouts alone waste work. Imagine the app gives up after 2 s, but deeper services each have a 1.5 s timeout and work sequentially — they keep computing an answer nobody is waiting for. Deadline propagation fixes this: the first service sets an absolute deadline ("must finish by 10:00:00.500") and passes the remaining time downstream in a header or RPC metadata. gRPC does this natively (grpc-timeout).

Gateway  deadline = now + 800 ms
  └─► Feed Service           receives "remaining 790 ms"
        ├─► Feed cache (Redis)    gets min(own 50 ms, remaining) → 50 ms
        ├─► Ranking               remaining now 720 ms → its own cap 300 ms
        │     └─► Feature store   remaining 280 ms → cap 40 ms
        └─► Counter Service       remaining 380 ms → cap 100 ms
If remaining ≤ 0 at any hop → don't even start; return DEADLINE_EXCEEDED immediately.

def call_with_deadline(stub, request, ctx, own_timeout_s):
    remaining = ctx.deadline - time.monotonic()
    if remaining <= 0.005:                       # less than 5 ms left: pointless to try
        raise DeadlineExceeded()
    timeout = min(own_timeout_s, remaining)
    return stub.Call(request, timeout=timeout)   # gRPC forwards the deadline downstream

5. Retries with exponential backoff and jitter

Many failures are transient: a packet was dropped, a pod was restarting, a leader election took 2 seconds. Retrying often succeeds. But retries done naively are one of the main causes of outages, so do them carefully.

5.1 What to retry

5.2 Exponential backoff + jitter

Backoff = wait between attempts, growing exponentially (100 ms, 200 ms, 400 ms…) so a struggling dependency gets breathing room. Jitter = randomise the wait. Without jitter, 10,000 clients that failed at the same instant all retry at exactly +100 ms, then +200 ms — synchronised waves that knock the recovering service over again (a thundering herd). "Full jitter" picks a random wait between 0 and the exponential cap, spreading retries evenly.

import random, time

def call_with_retries(op, max_attempts=3, base=0.1, cap=2.0, deadline=None):
    for attempt in range(max_attempts):
        try:
            return op()
        except RetryableError as e:
            if attempt == max_attempts - 1:
                raise
            if not retry_budget.try_acquire():          # see 5.4 — global brake on retries
                raise
            backoff = min(cap, base * (2 ** attempt))    # 0.1, 0.2, 0.4, ... capped
            sleep_s = random.uniform(0, backoff)         # FULL JITTER
            if e.retry_after:                            # server told us when to come back
                sleep_s = max(sleep_s, e.retry_after)
            if deadline and time.monotonic() + sleep_s >= deadline:
                raise                                    # no time left: fail fast
            time.sleep(sleep_s)

No jitter (10k clients):     ████          ████              ████        ← synchronized spikes
                             t+0.1         t+0.2             t+0.4
Full jitter (10k clients):   ▂▃▂▃▂▂▃▂▂▃▂▂▃▂▃▂▂▃▂▂▃▂▂▃▂▂▃▂▂▃▂▂▃▂▂          ← smooth, survivable

5.3 Retry storms

Retries multiply across layers. Suppose the app, the gateway, the Feed Service and the Counter Service client each make up to 3 attempts. If the bottom dependency is failing, one user action becomes:

attempts at the bottom = 3 (app) × 3 (gateway) × 3 (feed) × 3 (counter client) = 81

An 81× load multiplier, applied precisely when the dependency is already failing — a retry storm. It can keep a service down long after the original cause is gone, because recovery capacity is consumed by retries of requests whose users have already given up.

5.4 Retry budgets and other brakes

• Retry at one layer only — usually the layer closest to the failure that knows whether the operation is safe, or the edge (the app). Inner layers fail fast and propagate the error.
• Retry budget: a client may only add retries up to, e.g., 10% of its normal request volume. Implemented as a token bucket: each successful request deposits 0.1 tokens, each retry spends 1. When the dependency is healthy, occasional retries are free; when it's failing broadly, the budget runs dry and retries stop automatically, capping amplification at 1.1×.
• Respect Retry-After and 429/503 from overloaded servers.
• Don't retry on overload signals from a server that is explicitly shedding load.
• Combine with a circuit breaker (section 7) so a clearly dead dependency isn't retried at all.

6. Idempotency: making retries safe

An operation is idempotent if doing it twice has the same effect as doing it once. GET, PUT profile.bio = "hi" and DELETE like(user=7, post=42) are naturally idempotent. POST /comments or INCR like_count are not: a retried comment after a timeout (where the first attempt actually succeeded but the response was lost) creates a duplicate comment.

The standard fix is an idempotency key: the client generates a unique ID (UUID) per logical action and sends it with every attempt; the server records processed keys and returns the stored result for repeats.

POST /v1/media/42/comments
Idempotency-Key: 6f1c2b9e-0d4a-4c11-9a77-3b2e5f8a9d10
{"text": "Beautiful shot!"}

def create_comment(user_id, post_id, text, idem_key):
    # atomic "insert if absent": only the first attempt wins
    existing = idem_store.get(f"{user_id}:{idem_key}")
    if existing:
        return existing                                  # replay the original response
    comment = comments.insert(post_id, user_id, text)
    idem_store.set(f"{user_id}:{idem_key}", comment, ttl=24*3600)
    return comment

(In production the check-and-insert must be atomic — e.g. a unique constraint on (user_id, idem_key) or SET NX in Redis — to handle two concurrent attempts.) Other idempotency techniques used across Instagram-style systems:

• Natural keys: a like is a row keyed by (user_id, post_id); inserting it twice is a no-op. Counters are then derived, not blindly incremented.
• Client-generated IDs for uploads: the app creates an upload ID before uploading, so a retried upload resumes rather than creating a second post. See also ID generation.
• Consumer de-duplication: Kafka consumers (fan-out, notifications) track processed event IDs because delivery is at-least-once — you don't want two push notifications for one like. See Message Queues.

7. Circuit breaker

If a dependency is clearly broken, continuing to call it wastes time (each call waits for a timeout), wastes resources, and adds load to something trying to recover. A circuit breaker wraps calls to a dependency and tracks failures; when failures exceed a threshold it "trips" and subsequent calls fail immediately (or go straight to a fallback) for a cool-down period, then cautiously tests whether the dependency has recovered.

class CircuitBreaker:
    def __init__(self, failure_threshold=0.5, min_calls=20, open_seconds=10, probes=3):
        self.state, self.window = "CLOSED", RollingWindow(seconds=10)
        self.failure_threshold, self.min_calls = failure_threshold, min_calls
        self.open_seconds, self.probes, self.opened_at, self.probe_ok = open_seconds, probes, 0, 0

    def call(self, fn, fallback):
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at < self.open_seconds:
                return fallback()                          # fail fast: no waiting, no load
            self.state, self.probe_ok = "HALF_OPEN", 0     # cool-down over: test the water
        try:
            result = fn()                                  # fn has its own timeout!
        except Exception:
            self.on_failure()
            return fallback()
        self.on_success()
        return result

    def on_failure(self):
        if self.state == "HALF_OPEN":
            self.trip()                                    # still broken → back to OPEN
            return
        self.window.record(ok=False)
        if self.window.count() >= self.min_calls and self.window.failure_rate() >= self.failure_threshold:
            self.trip()

    def on_success(self):
        if self.state == "HALF_OPEN":
            self.probe_ok += 1
            if self.probe_ok >= self.probes:
                self.state = "CLOSED"; self.window.reset()
        else:
            self.window.record(ok=True)

    def trip(self):
        self.state, self.opened_at = "OPEN", time.monotonic()

Design notes:

• Count timeouts and 5xx as failures, but not 4xx (a 404 means the dependency is healthy and answered correctly).
• Use a minimum call volume (here 20) so two failures out of three calls at 3 a.m. don't trip it.
• Scope matters. One breaker per dependency per host or shard is often better than one per whole service: if 1 of 64 Redis shards is sick, you want to stop calling that shard only.
• Limit probes in half-open (only a few requests, not a flood).
• Always pair with a fallback and with metrics/alerts on state changes — an open breaker is a symptom someone should know about.

8. Bulkheads

A bulkhead isolates resources so that one failing part can only exhaust its own compartment. In the cascade above, the Feed Service shared one pool of 200 threads across all dependencies, so the slow Counter Service starved everything. With bulkheads:

                 Feed Service instance (200 threads total)
 ┌───────────────────┬───────────────────┬───────────────────┬──────────────────┐
 │ Ranking pool: 80  │ Feed cache: 50    │ Counters: 30      │ User svc: 40     │
 │ ✓ healthy         │ ✓ healthy         │ ✗ all 30 stuck    │ ✓ healthy        │
 └───────────────────┴───────────────────┴───────────────────┴──────────────────┘
 Counter calls beyond 30 concurrent are rejected instantly → fallback "hide counts".
 The other 170 threads keep serving feeds.

Bulkheads exist at many levels:

• Per-dependency thread pools / semaphores / connection pools inside a service (as above).
• Separate service deployments per traffic class: e.g. a separate fleet for internal batch traffic vs user-facing traffic, or for the web vs mobile API, so a scraper storm on one doesn't affect the other.
• Per-tenant / per-customer limits so one abusive client can't use all capacity (see Rate Limiting).
• Cells: whole independent copies of the stack serving a subset of users, the ultimate bulkhead — see Multi-Region & cell-based architecture.

9. Load shedding and admission control

Every server has a maximum throughput. When offered load exceeds it, queues grow without bound; every request waits longer; eventually every request exceeds its client's timeout and the server is doing 100% work for 0% useful output ("congestion collapse"). Load shedding deliberately rejects some requests early (cheaply, with HTTP 503/429) so that the rest are served fast.

9.1 Little's law and concurrency limits

Little's law: L = λ × W — the average number of requests in the system equals arrival rate times time in system. If a Feed instance can process 2,000 req/s at 50 ms each, it naturally has ~100 requests in flight. If in-flight climbs to 400, requests must be queueing (W ≈ 200 ms) — a clear overload signal. Admission control caps in-flight requests (or queue length / queue wait time) and rejects new arrivals above the cap. Adaptive limiters (inspired by TCP congestion control, e.g. Netflix's concurrency-limits library) adjust the cap automatically from observed latency.

9.2 Smarter shedding

• Shed early and cheaply: reject at the load balancer or gateway before doing expensive work (auth, DB reads).
• Priority shedding: drop the least important traffic first. For Instagram: background prefetch of the next feed page and analytics log uploads before user-initiated feed loads; feed loads before… nothing — sending a DM or posting is top priority.
• Queue-time based: if a request has already waited longer than its deadline, drop it — the client has given up. Facebook's publicly described "Fail at Scale" practices include controlled delay (shorten queue timeouts when the queue hasn't been empty recently) and adaptive LIFO (under overload, serve the newest requests first, because the oldest are likely already abandoned).
• Client cooperation: on 429/503 with Retry-After, apps back off instead of hammering.

10. Backpressure

Backpressure is how a slow consumer tells a fast producer "slow down", instead of silently buffering until memory runs out. Load shedding drops work; backpressure pushes the waiting back upstream to where it can be handled.

• Bounded queues: every in-memory queue has a maximum size. When full, the producer blocks or gets an error. An unbounded queue just converts overload into an out-of-memory crash later.
• Pull-based consumption: Kafka consumers pull at their own pace, so a slow fan-out worker builds up consumer lag (stored durably on disk) rather than being overwhelmed. Lag is then the signal to autoscale workers.
• Flow control in protocols: TCP windows, HTTP/2 and gRPC stream flow control, reactive streams (request(n)).
• Rate limits and 429s upstream: the rate limiter at the gateway is backpressure at the system boundary.

11. Fallbacks and graceful degradation

A fallback is what you return when the real answer isn't available in time. Graceful degradation is the product-level design that makes those fallbacks acceptable: the app keeps doing its most important job with reduced quality, rather than failing completely. The key design exercise is to classify every dependency of a request as critical (no answer without it) or optional (nice to have), and to decide in advance what the optional ones degrade to.

Rules for good fallbacks:

• Fallbacks must be cheaper and more reliable than the primary path — a fallback that calls another fragile service just moves the failure.
• Exercise fallbacks regularly (chaos tests). A fallback path that runs only during real incidents is likely broken when you need it.
• Make degradation visible in metrics and, if needed, to the user ("Couldn't refresh feed. Showing recent posts.").
• Never "fall back" in a way that violates correctness or security — e.g. if the auth/permission check fails, deny (fail closed), don't allow. Showing a private account's posts because the privacy service timed out is a far worse outcome than an error.
• Feature flags / kill switches let operators turn off expensive non-critical features (e.g. disable video autoplay prefetch) during an incident to free capacity.

12. Health checks: liveness vs readiness

Orchestrators (Kubernetes, or Meta's internal equivalents) and load balancers decide what to do with an instance based on health checks. There are two different questions, and mixing them up causes outages:

• A startup probe gives slow-starting processes (loading an ML model of several GB) time before liveness checks begin.
• Graceful shutdown: on deploy, mark unready first, let the LB drain in-flight requests, then exit.
• Load balancers often add passive health checks (outlier detection): eject a backend that returns many 5xx or is much slower than its peers — this catches gray failures that active /healthz pings miss. Cap ejection at a percentage (e.g. never eject more than 30% of backends) so a systemic issue doesn't empty the pool. See Load Balancing.

13. Hedged requests: fighting tail latency

When a request fans out to many servers, the slowest one determines the response time. If each of 100 shards answers in 10 ms 99% of the time but takes 1 s 1% of the time (GC pause, noisy neighbour, disk hiccup):

P(at least one of 100 shards is slow) = 1 − 0.99^100 ≈ 63%
→ the "rare" 1% tail becomes the COMMON case for the fan-out request.

A hedged request sends the request to one replica, and if no answer arrives within, say, the p95 latency, sends a second copy to another replica; use whichever answers first and cancel the other. Because only ~5% of requests trigger a hedge, extra load is ~5%, but the tail shrinks dramatically. (This technique was popularised by Google's "The Tail at Scale" paper.)

t=0ms     send to replica A ─────────────────────────────── (A stuck in GC… answers at 900 ms)
t=12ms    no answer by p95 → send to replica B ──► B answers at t=20 ms ✔ use it, cancel A

• Only hedge idempotent reads (e.g. Search shard queries, feed-cache reads, feature-store lookups).
• Hedge after a delay, not immediately — immediate duplication doubles load.
• Disable hedging under overload (it's a form of retry, so subject it to the retry budget).

14. Chaos engineering

You can't know a system is resilient until you've seen it survive failure. Chaos engineering is the practice of deliberately injecting failures in a controlled way to find weaknesses before real incidents do.

1. Define steady state: measurable normal behaviour, e.g. feed load success rate > 99.9%, p99 < 800 ms.
2. Hypothesise: "If one Redis feed-cache shard dies, feed success rate stays > 99.9% because we fall back to the chronological path."
3. Inject the failure: kill instances, add 500 ms latency to a dependency, drop packets between two services, fill a disk, fail a whole availability zone.
4. Limit blast radius: start in staging, then 1% of production traffic or one cell; have an abort button.
5. Observe and fix: if the hypothesis fails, you've found a bug cheaply. Automate the experiment so it runs continuously.

Famous examples: Netflix's Chaos Monkey randomly terminates production instances, and Chaos Kong simulates losing a whole AWS region. Meta has publicly described a program of large-scale disaster-readiness drills (called Storm in public talks) that take entire data centers offline to verify that services — including Instagram — fail over correctly; see Multi-Region. Smaller-scale versions are game days: a team schedules an afternoon to break its own service and practise the response.

15. Patterns at a glance

16. How Instagram uses resilience patterns

Follow one feed load through the main diagram with resilience in mind:

1. The app requests the feed with a timeout of a few seconds. If it fails, it shows the feed cached on the device and retries with jittered backoff. Uploads and likes performed offline are kept in a local queue with client-generated idempotency keys and replayed when connectivity returns (see Clients).
2. The load balancer and API gateway enforce rate limits, eject outlier backends, and shed low-priority traffic (prefetch, logging) first under overload. They attach a deadline to the request.
3. The Feed Service calls its dependencies in parallel through per-dependency bulkheads and circuit breakers. Critical: the timeline from the feed cache (with a fallback to a slower rebuild from the post store). Optional: ranking (fallback: chronological), counters (fallback: hide), stories tray (fallback: hide), ads (fallback: none).
4. Stores are replicated; reads can go to replicas, and latency-sensitive fan-out reads can be hedged.
5. Async paths (fan-out, notifications, media processing) run through Kafka with backpressure via consumer lag, idempotent consumers and dead-letter queues for poison messages.

Rough numbers (assumptions)

Feed Service instance: 200 threads, normal latency 100 ms → capacity ≈ 2,000 req/s
If one dependency becomes 3 s slow and there is NO timeout / bulkhead:
    capacity → 200 / 3 s ≈ 67 req/s   (≈ 30× collapse)
With a 100 ms timeout + 30-thread bulkhead on that dependency:
    worst case each request waits 100 ms extra → capacity ≈ 200 / 0.2 s = 1,000 req/s
    and with an OPEN breaker (fail fast) → back to ≈ 2,000 req/s, counts hidden

17. Trade-offs & common pitfalls

Common pitfalls

• No timeout or library defaults (often infinite or 60 s+).
• Retrying at every layer → multiplicative retry storms; retrying non-idempotent operations → duplicates.
• Backoff without jitter → synchronised retry waves.
• Timeouts longer than the caller's timeout → wasted work after the caller gave up (use deadline propagation).
• Readiness/liveness checks that test shared dependencies → whole-fleet restarts or empty LB pools.
• Unbounded queues → memory blow-up and latency far past any deadline.
• Fallbacks that are never exercised, or that fail open on security checks.
• Autoscaling as the only defence — scaling takes minutes; overload kills in seconds. And a scaled-out caller can overwhelm a dependency that didn't scale.

18. Real-world technologies

19. Interview corner

20. Check your understanding

Related lessons

• Scalability, Availability, Reliability & Performance — the definitions of nines and availability math
• Load Balancing — health checks and outlier ejection
• Rate Limiting — admission control at the edge
• Message Queues & Kafka — backpressure, at-least-once delivery, idempotent consumers
• Microservices — why many dependencies make resilience patterns essential
• News Feed — the request whose fallbacks we designed
• Multi-Region & Disaster Recovery — cells, failover and data-center drills
• Monitoring, Logging & Tracing — seeing timeouts, breaker states and tail latency