1. The problem: why do we need these principles?

Instagram's requirements (see Requirements) say: serve 500M people a day, at ~350K feed requests per second at peak, with p99 latency under 300 ms, 99.99% availability, and never lose a photo. Now imagine the simplest possible implementation: one server running the app and a database, with photos on its disk. Every one of those requirements fails:

• Scale: one server handles maybe a few thousand requests per second — two orders of magnitude short.
• Availability: when that server reboots for a kernel patch, Instagram is down for everyone.
• Durability: when its disk dies, every photo ever posted is gone.
• Performance: users in Mumbai wait 300 ms just for packets to reach Virginia and back, before any work is done.

The principles in this lesson are the vocabulary for describing how a design fixes each of these — and what it costs. They come up in every system design interview, usually in step 6 ("what are the bottlenecks and failure modes?" — see How to Approach).

2. Scalability

Scalability is a system's ability to handle increased load — more users, more data, more requests — by adding resources, while keeping performance acceptable. A scalable system's capacity grows roughly in proportion to the resources you add. "Load" must be described with load parameters: for Instagram, requests per second per endpoint, read:write ratio, follower-count distribution, data volume.

2.1 Vertical vs horizontal scaling

Real systems do both: scale up a database primary to the largest sensible machine (it buys time and simplicity), and scale out everything else. Instagram's early story, as told in public talks, followed this path: a few bigger servers first, then many servers and sharded databases as growth demanded.

2.2 Stateless services: the key to scaling out

A service is stateless when it keeps no client-specific data in its own memory or disk between requests: any instance can handle any request. All state lives in shared external stores — databases, caches, object storage. Then:

• The load balancer can send each request to any healthy instance (no "sticky sessions").
• Adding capacity = starting more instances; autoscaling becomes trivial.
• An instance crashing loses nothing except in-flight requests, which clients retry.
• Deploys are rolling restarts with zero downtime.

# ❌ Stateful: session kept in the server's memory — breaks when the LB picks another server
sessions = {}                                   # lives in this process only
def login(user):  sessions[token] = user
def feed(token):  user = sessions[token]        # KeyError on any other instance

# ✅ Stateless: identity carried in a signed token, data in shared stores
def feed(request):
    user_id = verify_jwt(request.headers["Authorization"])   # no server memory needed
    post_ids = redis.zrevrange(f"timeline:{user_id}", 0, 19)
    return hydrate(post_ids)                                  # from cache / Cassandra

In the Instagram diagram every application service (Upload, Feed, Like & Comment, Follow, Search, User/Auth) is stateless. The one notable exception is the Direct Messaging WebSocket tier, which must hold long-lived connections; it keeps only the connection itself in memory and stores everything else (messages, presence) externally, with a routing table saying which gateway holds which user's connection. See Microservices & Stateless Services.

2.3 The scale cube: three ways to scale

The scale cube (from the book The Art of Scalability) names three independent dimensions of scaling:

2.4 Scaling the data tier

Stateless compute is the easy part; data is where scaling gets hard, because it has state by definition.

• Read-heavy? Add caches (absorb most reads) and read replicas (X-axis for reads). Instagram's ~100:1 content read ratio makes this the first lever.
• Too much data or too many writes for one machine? Shard (Z-axis), ideally with consistent hashing or many logical shards so you can rebalance.
• Different access patterns? Use different stores for different data (polyglot persistence): PostgreSQL for users, Cassandra for posts, Redis for timelines, Elasticsearch for search, S3 for media (Databases).
• Expensive work on the request path? Move it off via a queue (Kafka) and process asynchronously — this scales the work independently of user-facing latency.

3. Availability

Availability is the proportion of time — or, better at scale, the proportion of requests — for which the system is working correctly:

availability (time-based)    = uptime / (uptime + downtime)
availability (request-based) = successful requests / total valid requests
availability (from failures) = MTBF / (MTBF + MTTR)
   MTBF = mean time between failures   MTTR = mean time to recovery (detect + fix)

The last formula reveals the two levers: fail less often (raise MTBF) or recover faster (cut MTTR). At scale, recovering faster is usually far cheaper. Example: a component fails once a month (MTBF = 720 h). If a human must notice and fix it (MTTR = 1 h), availability = 720/721 = 99.86%. If automatic failover recovers in 1 minute, availability = 720/(720 + 0.0167) = 99.998% — the same unreliable hardware, 60× less downtime.

3.1 The nines

Each extra nine is 10× less downtime, and typically costs far more than 10× the engineering effort: three nines can be reached with good practices in one region; five nines generally requires multiple regions, automated everything and extremely careful change management.

3.2 Availability in series vs in parallel

A request usually passes through several components. How do their availabilities combine? Assume failures are independent (an important caveat — see 3.3).

In series (the request needs all of them): multiply.

A_series = A1 × A2 × … × An

In parallel (redundant copies; the request needs at least one): the system fails only if all copies fail.

A_parallel = 1 − (1 − A1) × (1 − A2) × … × (1 − An)
two copies of 99%:   1 − 0.01 × 0.01   = 1 − 0.0001  = 99.99%
three copies of 99%: 1 − 0.01^3        = 99.9999%

Let's verify the numbers. Series: 0.9999³ = 0.99970; × 0.995 = 0.99470; × 0.999 = 0.99370 → 99.37%, i.e. 0.63% × 8,766 h ≈ 55 hours/year. With three feed instances and a Redis replica: 0.99970 × 0.9999999 × 0.999999 ≈ 99.97% (~2.6 h/year). The remaining downtime now comes from the three single-instance edge components — so we make those redundant too (multiple DNS providers/anycast, LB pairs, many gateway instances). That is precisely why every layer in the Instagram diagram is drawn as a single box but deployed as many.

3.3 The independence trap: correlated failures

The parallel formula assumes copies fail independently. In reality, failures are often correlated: all three replicas run the same buggy release; all sit in one data centre that loses power; all read the same bad config push; all depend on the same DNS or auth service. Three "independent" 99% replicas that share a deploy pipeline are not 99.9999% available. Defences: spread replicas across failure domains (racks, availability zones, regions), deploy gradually (canary one zone first), keep dependencies diverse where it matters, and design so that a bad change can be rolled back automatically.

4. Redundancy and single points of failure

A single point of failure (SPOF) is any component whose failure takes down the whole system (or a whole feature). Redundancy — having spare capacity or duplicate components — removes SPOFs.

4.1 Hunting SPOFs in a naive Instagram

4.2 How much redundancy?

• N+1: enough capacity to lose any one unit (server, zone) and still serve peak load. With 3 zones, each zone must handle 50% of peak → you provision 150% of peak.
• N+2: survive one planned outage (maintenance) plus one unplanned failure at the same time.
• Data redundancy: replication factor 3 is the common default for databases and logs; object stores use replication or erasure coding (e.g. 10 data + 4 parity fragments) for durability at lower overhead (Object Storage).

5. Failover: active-passive vs active-active

Failover is switching traffic from a failed component to a healthy one. It needs three things: detection (health checks, heartbeats), a decision (who's the new primary — ideally via consensus to avoid two primaries) and redirection (update the LB, DNS or client routing).

 ACTIVE-PASSIVE                                ACTIVE-ACTIVE
 ──────────────                                ─────────────
   clients                                       clients
      │                                         ┌───┴───┐
      ▼                                         ▼       ▼
 ┌─────────┐  replicate  ┌─────────┐       ┌───────┐ ┌───────┐
 │ ACTIVE  │ ──────────> │ PASSIVE │       │ node A│ │ node B│   both serve traffic;
 │ serves  │             │ standby │       │ 50%   │ │ 50%   │   replicate to each other
 └─────────┘             └─────────┘       └───────┘ └───────┘
   ✗ fails → standby promoted (seconds–minutes)   ✗ A fails → B takes 100% immediately

Standby flavours: hot (running and in sync, takes over in seconds), warm (running but behind or under-provisioned, minutes), cold (restored from backups when needed, hours). The choice trades cost against RTO (recovery time objective: how long you may be down) and RPO (recovery point objective: how much recent data you may lose).

6. Reliability vs availability vs durability vs fault tolerance

These words are often used interchangeably, but interviewers notice when you distinguish them precisely.

Some illuminating contrasts:

• Available but unreliable: the feed always loads, but sometimes shows posts from blocked users. Up, but wrong.
• Durable but unavailable: during a regional outage your photos are safe on disk but can't be viewed for an hour.
• Available but not durable: a cache that always answers but loses everything on restart — fine for a cache, unacceptable for a database of record.
• Fault tolerance is a means, availability is an end: you build fault tolerance (redundancy, failover) in order to achieve availability and durability targets.

A key design idea is graceful degradation: when part of the system fails, reduce functionality instead of failing entirely. If the ranking service is down, serve the timeline in chronological order; if like counts can't be fetched, hide them; if the CDN can't reach the origin, serve a slightly stale cached copy. Users experience "a bit worse" instead of "broken".

7. Performance

7.1 Latency vs throughput

• Latency: time to complete one operation (ms). "How long did my feed take to load?"
• Throughput: operations completed per unit time (requests/s, MB/s). "How many feeds can we serve per second?"
• Bandwidth: the maximum possible throughput of a channel (e.g. a 10 Gbps link).

Analogy: a motorway. Latency is how long one car takes to drive from A to B; throughput is how many cars pass per hour. Adding lanes raises throughput without making any car faster; raising the speed limit lowers latency. They interact through queueing: as throughput approaches capacity, queues form and latency rises sharply (section 7.5). Some techniques trade one for the other — batching (e.g. Kafka producers batching messages, or fan-out workers writing timelines in bulk) raises throughput but adds a little latency to each item.

7.2 Percentiles: p50, p95, p99

Latency is a distribution, not a number. The pN (N-th percentile) is the value below which N% of measurements fall.

20 feed requests, sorted latencies (ms):
 42 45 47 48 50 51 53 55 56 58 60 62 65 70 75 80 95 120 250 1,900

mean = 3,282 / 20 ≈ 164 ms     ← dragged up by one 1.9 s outlier; describes nobody
p50  (median) ≈ 59 ms          ← the typical experience
p95           ≈ 250 ms         ← 1 in 20 requests is at least this slow
p99           ≈ 1,900 ms       ← the tail (with only 20 samples, p99 ≈ the max)

• p50 shows typical performance; p95/p99 show the tail, which is what frequent users actually feel — someone loading 20 feed pages a day hits the p95 roughly once a day.
• Never average percentiles across servers or time windows (the average of p99s is not the p99). Aggregate the underlying histograms instead (Observability).
• Tail causes include garbage-collection pauses, cache misses, a slow disk, a noisy neighbour, retries, lock contention, and queueing at high utilisation.

7.3 Tail latency amplification under fan-out

Here is a subtle but crucial effect. When one user request fans out to many backend calls in parallel and must wait for all of them, the request is as slow as the slowest call. If each backend is "slow" (above its p99) 1% of the time, independently:

P(user request is slow) = 1 − P(all N calls are fast) = 1 − 0.99^N

Instagram example: rendering one feed page hydrates ~20 posts, each requiring metadata, counters, author info and "did I like it?" lookups spread over many cache shards — easily 20–100 parallel calls. With N = 20, 18% of feed requests experience some backend's p99. Flip it around: to keep the page p99 at a target, each backend must meet that target at the p99.95 level, because 0.99^1/20 ≈ 0.9995.

Mitigations (from Google's well-known paper The Tail at Scale and common practice):

• Reduce fan-out: batch lookups (one multi-get per shard instead of one call per item), denormalise data so fewer services are touched.
• Hedged requests: if a call hasn't returned by its p95, send a duplicate to another replica and take whichever answers first (costs a few % extra load).
• Tight timeouts + partial results: render the feed without one post's like count rather than wait (graceful degradation).
• Reduce variance at the source: avoid GC pauses, isolate noisy neighbours, keep utilisation moderate, use load-aware balancing.
• Pre-compute: serve a pre-built timeline instead of assembling it from many sources per request — exactly what the Feed Cache does.

7.4 Little's law

Little's law is a simple, universally applicable relationship for any stable system (a server, a queue, a whole service):

L = λ × W
L = average number of requests in the system (concurrency / in-flight)
λ = average arrival rate (throughput, requests/s)
W = average time each request spends in the system (latency, s)

Uses:

• Sizing concurrency: the feed tier at 350K req/s peak with 100 ms average latency has 350,000 × 0.1 = 35,000 requests in flight. If each server can work on 64 requests concurrently, you need at least 35,000 / 64 ≈ 550 servers — just for concurrency.
• Connection pools: a service doing 20,000 DB queries/s at 5 ms each needs ~20,000 × 0.005 = 100 connections on average (plus headroom). Too small a pool → requests queue; too large → the DB drowns in connections.
• Latency spikes eat capacity: if a dependency slows from 100 ms to 1 s, concurrency at the same throughput rises 10× — threads and connections run out and the slowdown cascades. This is why timeouts and circuit breakers are essential.
• Queues: a Kafka consumer group with 2M messages of lag processing 50K messages/s means new events wait ~40 s (W = L / λ) — directly the "post visible to followers" delay.

7.5 Utilisation and queueing: why you need headroom

Queueing theory gives a sobering rule: as a server's utilisation ρ (busy fraction) approaches 100%, waiting time explodes. For the simplest model (M/M/1 — random arrivals, one server), average time in system is:

W = S / (1 − ρ)        S = service time when idle
ρ = 50%  → W = 2 S
ρ = 70%  → W ≈ 3.3 S
ρ = 80%  → W = 5 S
ρ = 90%  → W = 10 S
ρ = 95%  → W = 20 S
ρ = 99%  → W = 100 S

Real systems differ in detail, but the shape — a "hockey stick" — is universal. That's why capacity plans target ~60–70% peak utilisation, not 100%: the last 30% of "unused" capacity is what keeps the p99 sane and absorbs spikes and failovers (Estimation).

7.6 Bottleneck analysis

A system's throughput is limited by its bottleneck — the most constrained resource on the path. Speeding up anything else doesn't help. A systematic approach:

1. Draw the request path (e.g. feed: LB → gateway → feed service → Redis → post cache → Cassandra on miss → ranking).
2. For each stage, compare demand to capacity: requests/s vs per-node limit × nodes; bytes/s vs NIC/disk bandwidth; memory vs working set.
3. Check resources with the USE method: for each resource (CPU, memory, disk, network, connection pools, locks) look at Utilisation, Saturation (queue length) and Errors.
4. Find the latency budget hogs with distributed tracing: which span dominates the p99? (Observability)
5. Fix the top bottleneck, then re-measure — the bottleneck moves. After caching the feed, ranking may become the slowest stage; after optimising ranking, the network to Redis may be.

Common bottlenecks in Instagram-like systems: a hot shard (a celebrity's posts or counters), the primary database for writes, cache-miss storms after a cache restart, fan-out workers lagging behind during a spike, and cross-region round trips. Amdahl's law is the reminder for parallelism: if 10% of the work is inherently serial, no amount of parallel workers can make it more than 10× faster.

8. Maintainability and operability

Most of a system's cost over its lifetime is not building it but running and changing it. Three facets (as framed in Martin Kleppmann's Designing Data-Intensive Applications):

In the Instagram diagram, Kafka is a big maintainability win: when the company wants a new feature that reacts to posts (say, a "creator insights" dashboard), a team adds a new consumer — the Upload Service doesn't change. Splitting services by function (Y-axis) lets hundreds of engineers deploy independently. Public Instagram talks have long emphasised operational simplicity — "do the simple thing first" and use proven technology — which let a tiny early team run a fast-growing service.

9. Cost

Every property above costs money: redundancy doubles or triples hardware, extra nines need extra regions, low latency needs RAM and edge locations. Good designers think in unit economics — cost per DAU, per upload, per GB served — and know the main levers:

10. How Instagram uses these principles: why each box exists

Read the Instagram diagram as a set of answers to the questions in this lesson. Every box buys at least one property:

11. Trade-offs & common pitfalls

Common pitfalls

• "We'll scale vertically forever." Works until it suddenly doesn't — and the migration happens under pressure.
• Hidden state in "stateless" services: local file uploads, in-memory sessions, local caches assumed to be authoritative.
• Redundancy in the same failure domain: two replicas on the same rack, or a "backup" in the same region.
• Assuming independent failures: correlated failures (bad deploys, config pushes) are the most common cause of big outages.
• Untested failover: a standby that has never been promoted probably won't work when needed. Practise it (game days, chaos testing).
• Reporting averages: the mean hides the tail; use p99 and p99.9.
• Running hot: 90% utilisation looks efficient but makes latency explode and leaves no room for failover.
• Ignoring fan-out: a page touching 100 services will be slow at the tail unless you mitigate.
• Every extra nine everywhere: over-engineering low-value features wastes money; match SLOs to user impact.

12. Real-world technologies

13. Interview corner

14. Check your understanding

Related lessons

• CAP, PACELC & Consistency — what availability costs in consistency during partitions
• Replication — leader/follower, failover, replication lag
• Sharding — Z-axis scaling in depth
• Load Balancing — X-axis scaling and health checks
• Resilience Patterns — timeouts, retries, circuit breakers, load shedding
• Multi-Region & Disaster Recovery — RTO/RPO and regional failover
• Observability — measuring percentiles and finding bottlenecks
• Back-of-the-Envelope Estimation — the numbers behind headroom and server counts