1. The problem: why estimate at all?

Should Instagram's feed be served from the database or from a cache? Will one PostgreSQL server hold all the posts? Do we need a CDN? You cannot answer any of these by intuition alone. "A lot of users" might mean 1,000 requests per second (one server is fine) or 1,000,000 (you need thousands of servers and a completely different design). Estimation turns vague scale into numbers you can reason with — in two to five minutes, with mental arithmetic.

Estimates are useful precisely because they are rough. Nobody cares whether storage is 580 TB/day or 610 TB/day; what matters is that it's "hundreds of terabytes per day", which immediately means "not in a relational database, use object storage", and "over 200 petabytes per year", which means "storage cost and tiering are first-class concerns". Getting the exponent right is the whole game; the leading digit is a bonus.

Real engineers do this constantly: capacity planning before a launch, checking whether a proposed design can possibly meet its latency target, sanity-checking a cloud bill. Interviewers use it to see whether you can connect requirements to architecture with evidence.

2. The toolkit: numbers worth memorising

2.1 Powers of two (and ten)

Computers count in powers of two; humans in powers of ten. The happy coincidence 2¹⁰ = 1,024 ≈ 10³ lets you switch freely between them with ~2–10% error — negligible for estimation.

2.2 Handy conversions

2.3 Latency numbers every programmer should know

This famous table (popularised by Jeff Dean and Peter Norvig) gives the cost of basic operations. Exact values vary by hardware generation — modern NVMe SSDs and memory bandwidth have improved several of them — but the ratios are what matter: memory is ~1,000× faster than SSD random reads, and the same data centre is ~300× closer than another continent.

A few practical system-level numbers to add to the classic list (orders of magnitude, assume rather than quote):

• Redis/Memcached GET inside a data centre: ~0.2–1 ms (mostly network). One node: ~100K+ simple ops/s.
• Indexed lookup in a well-tuned SQL database (hot in buffer cache): ~1–5 ms; a single primary handles roughly 5K–20K writes/s depending on hardware and row size.
• Cassandra-style node: ~10K–50K writes/s; comfortable data per node ~1–2 TB (to keep compaction and repair manageable).
• TCP + TLS 1.3 handshake to a server 100 ms away: ~2 round trips ≈ 200 ms before the first byte — which is why connection reuse and edge termination matter (Networking).
• A modern server NIC: 10–100 Gbps. A Kafka broker: ~100 MB/s of sustained produce traffic per broker is a conservative planning number.

 HUMAN-SCALE VERSION: multiply every latency by 2 billion (so L1 = 1 second)

   L1 cache reference ............ 0.5 ns  →  1 second        (a heartbeat)
   Main memory reference ......... 100 ns  →  3.3 minutes     (make a coffee)
   SSD random read 4 KB .......... 150 µs  →  3.5 days        (a long weekend)
   Same-datacenter round trip .... 500 µs  →  11.6 days       (a holiday)
   HDD seek ...................... 10 ms   →  7.6 months      (a pregnancy, almost)
   California ⇄ Netherlands ...... 150 ms  →  9.5 years       (primary school)

 → Every network hop or disk access is an eternity to a CPU.
 → That is why a feed request that does 20 sequential DB queries is slow,
   and why we pre-compute timelines and keep them in RAM.

3. The estimation method

1. State assumptions explicitly and round them to friendly numbers (500M not 487M). Write them on the board.
2. Convert per-day counts into per-second rates, separately for reads and writes. Multiply by a peak factor (2–3×).
3. Multiply counts by sizes to get storage (per day, per year, over the retention period) and bandwidth (in and out). Remember replication and multiple renditions.
4. Divide by per-machine capacity to get server, shard and cache-node counts. Add headroom (target ~60–70% utilisation) and redundancy (survive losing a zone).
5. Sanity-check and, most importantly, say what the number means for the design.

4. Full worked example: Instagram

We now estimate Instagram end-to-end. All inputs are assumptions for design purposes (consistent with the Requirements lesson), not official statistics. Follow along with a pen — the point is the method.

4.1 Assumptions

4.2 Requests per second (QPS)

Rule: QPS = per-day count ÷ 86,400. Using "1B/day ≈ 12K/s":

What it means:

• Post creation (~1.2K/s) is tiny. The upload service is not the scaling challenge; what happens after a post is created is.
• At the API level, reads:writes ≈ 3:1 — because likes are frequent writes. But at the content level: 10B feed pages × 10 posts = 100B post impressions/day vs 100M posts created → ~1,000:1. The feed read path dominates and must be served from memory.
• ~1.1M requests/s at peak means thousands of stateless app servers behind load balancers.
• Likes at ~174K/s peak are a serious write path: write-optimised store plus asynchronous counters (Likes & Comments).

4.3 Fan-out write amplification

With push-based fan-out (News Feed), every post is inserted into each follower's timeline:

timeline inserts/day = 100M posts × 200 followers (avg) = 2 × 10^10 = 20B/day
average  = 20B / 86,400 ≈ 231K inserts/s
peak     ≈ 700K inserts/s

Optimisation: only fan out to followers active in the last ~30 days.
If ~25% of edges point to recently active users → ~5B/day ≈ 58K/s average.
Celebrities (say >1M followers) are excluded and pulled at read time —
one 500M-follower post would otherwise be 500M inserts on its own.

So each 1 post/s of writes becomes ~200 timeline writes/s — the "small" write rate is amplified 200× inside the system.

4.4 Media storage

Photos:  85M/day × (1 MB original + 0.5 MB renditions) = 85M × 1.5 MB  = 127.5 TB/day
Videos:  15M/day × (15 MB original + 15 MB renditions) = 15M × 30 MB   = 450   TB/day
                                                     Total posts media ≈ 580 TB/day (logical)

Per year:   580 TB × 365  ≈ 212 PB/year   (logical, one copy)
Replicated: × 3 copies    ≈ 635 PB/year   (classic 3× replication)
   — or with erasure coding (~1.5× overhead, as large blob stores use) ≈ 318 PB/year raw

5 years, flat volume:      212 PB × 5   ≈ 1.06 EB logical  → ~1.6 EB (EC) … ~3.2 EB (3×)
5 years, +20% growth/yr:   212 PB × (1 + 1.2 + 1.44 + 1.73 + 2.07) = 212 × 7.44
                                       ≈ 1.6 EB logical    → ~2.4 EB (EC) … ~4.7 EB (3×)

Stories: 200M × (1.5 MB + 0.5 MB renditions) ≈ 400 TB/day, but expire after 24 h
         → a rolling ~400 TB hot footprint (× replication), unless archived.

What it means: hundreds of PB per year is far beyond any database — media must live in object storage, with only keys in the metadata DB. Video is ~15% of posts but ~78% of bytes, so video codec efficiency and rendition choices dominate storage cost. Old media is rarely viewed → move it to cheaper "cold" tiers; this one decision changes the storage bill more than anything else.

4.5 Metadata storage

What it means: metadata is ~650× smaller than media but still far too big for one machine (a single database server comfortably holds ~1–2 TB of hot data). Over 5 years the posts+likes+comments set reaches ~0.9 PB logical → it must be sharded across hundreds of nodes (see section 4.10). Notice that likes outweigh posts: small rows × huge counts add up.

4.6 Bandwidth: ingress and egress

INGRESS (uploads)
  posts:   85M × 1 MB + 15M × 15 MB = 85 TB + 225 TB  = 310 TB/day
  stories: 200M × 1.5 MB                              = 300 TB/day
  total ≈ 610 TB/day  →  610 × 10^12 B / 86,400 s ≈ 7 GB/s ≈ 56 Gbps avg  (~170 Gbps peak)

EGRESS (media views)
  images: 500M DAU × 100 × 200 KB  = 10 PB/day
  videos: 500M DAU × 30  × 5 MB    = 75 PB/day
  total ≈ 85 PB/day  →  85 × 10^15 / 86,400 ≈ 984 GB/s ≈ 7.9 Tbps avg  (~24 Tbps peak)

API (JSON) egress
  10B feed pages × ~20 KB = 200 TB/day ≈ 2.3 GB/s ≈ 19 Gbps   (~0.2% of media egress)

Egress : ingress ≈ 85 PB : 0.61 PB ≈ 140 : 1

What it means: ~8 Tbps average (~24 Tbps peak) of media cannot be served from a handful of data centres — it needs a globally distributed CDN. API traffic is a rounding error by comparison, which is why the API path and the media path are designed separately.

4.7 CDN vs origin traffic

The CDN absorbs most egress; only cache misses reach object storage (the origin):

How many edge servers? If one edge cache server can serve ~40 Gbps of real traffic, peak ~24 Tbps needs at least ~600 servers — and in practice many more, because traffic must be served from hundreds of locations near users, each sized for its local peak plus failover. That's why at this scale companies either buy from the largest CDN providers or, like Meta, build their own edge network.

4.8 Cache memory sizing (the 80/20 rule)

The Pareto (80/20) rule says ~20% of items receive ~80% of requests. So caching the hottest ~20% of the data gives a high hit ratio at a fraction of the memory. There are two ways to size a cache; the second is more reliable.

(a) The classic interview formula — and its trap:

cache ≈ 20% × (read requests/day × response size)
      = 0.2 × 10B feed pages × 20 KB = 40 TB      ← overestimate!

This counts the same popular post millions of times. Caches store distinct objects, so size by distinct hot objects instead.

(b) Bottom-up, per data structure:

Nodes by memory:     9 TB / ~50 GB usable per node (on a 64 GB box) ≈ 180 primaries
Nodes by throughput: feed hydration ≈ 350K pages/s × ~20 lookups = 7M ops/s (peak)
                     + fan-out inserts ~0.7M/s  ≈ 8M ops/s ÷ ~100K ops/s per node ≈ 80 nodes
→ memory is the binding constraint: ~180 primaries, ×2 with one replica each ≈ 360 cache nodes

What it means: the entire hot working set fits in a few hundred RAM-heavy machines — which is why serving the feed from Redis is feasible at all. Note the timelines dominate; capping them (e.g. 500 entries, older items fetched on demand) is a key cost lever.

4.9 Kafka event throughput

Every domain action publishes an event to Kafka; clients also emit analytics events (impressions, scrolls, video watch time) that feed ranking models.

DOMAIN EVENTS (post, story, like, comment, follow, DM events)
  ≈ total writes ≈ 91K events/s avg (~275K/s peak) × ~1 KB  ≈ 91 MB/s avg (~275 MB/s peak)

ANALYTICS / CLIENT EVENTS
  500M DAU × ~500 events/day = 250B events/day ≈ 2.9M events/s × ~500 B ≈ 1.45 GB/s avg
  peak ≈ 4.3 GB/s

TOTAL produce ≈ 1.5 GB/s avg, ~4.6 GB/s peak
With replication factor 3, brokers write ≈ 4.6 GB/s avg, ~14 GB/s peak to disk.

Brokers:   ~14 GB/s ÷ ~100 MB/s per broker (conservative) ≈ 140 brokers  → ~150–200 with headroom,
           typically split into separate clusters (critical domain events vs bulk analytics)
Retention: domain events 7 days:  91 MB/s × 604,800 s ≈ 55 TB  × 3 ≈ 165 TB
           analytics 3 days:      1.45 GB/s × 259,200 s ≈ 376 TB × 3 ≈ 1.1 PB
Partitions: domain topics ≈ 275 MB/s ÷ ~10 MB/s per consumer partition ≈ 28 minimum;
           choose e.g. 128–256 so hundreds of fan-out workers can consume in parallel.

What it means: the "important" event stream is modest (~100 MB/s); analytics is ~15× bigger. Separating them prevents a flood of impression events from delaying fan-out or notifications. Partition count is driven by consumer parallelism, not just throughput.

4.10 Number of application servers

Peak API load                     ≈ 1.1M requests/s
Assume one app server handles     ≈ 1,000 req/s   (I/O-bound handlers calling caches/DBs;
                                                   could be 200/s in Python, 5,000/s in Go/Java)
Raw servers                       = 1.1M / 1,000             ≈ 1,100
Target 70% CPU utilisation        = 1,100 / 0.7              ≈ 1,570
Survive loss of 1 of 3 zones      = × 3/2                    ≈ 2,360  → plan ~2,400 servers

DM WebSocket gateways: assume 50M concurrent connections at peak
                       ÷ ~100K connections per server        ≈ 500 servers (+ headroom)

The per-server number is the softest assumption — say so, and show how the answer scales with it: if a server only handles 250 req/s, you need ~4× as many (~9,500). That sensitivity is itself a design insight — efficient runtimes and fewer downstream calls per request save thousands of machines.

4.11 Number of database shards

Shard count is driven by whichever constraint is tightest: storage, write throughput, or read throughput. Compute all three and take the max.

What it means: for Instagram-like data, storage usually drives the node count of the write-heavy stores, while RAM drives the cache tier and QPS drives the app tier. Knowing which constraint binds tells you which optimisation matters (compression and TTLs for storage-bound stores; more replicas for read-bound ones). Instagram's publicly described approach of many logical shards mapped onto fewer physical PostgreSQL servers is exactly the "plan for growth" trick in the first row — see Sharding.

4.12 Summary table

5. Sanity-check techniques

Estimates go wrong by factors of 1,000 through a single slipped unit. Before you rely on a number, check it with at least one of these:

1. Per-user check. Divide the total by the number of users. 85 PB/day ÷ 500M DAU = 170 MB per user per day — about 30 short videos plus 100 photos. Plausible for a heavy media app on mobile. If you'd got 170 GB per user per day, something is off by 1,000×.
2. Per-machine check. Does any single box need to do something impossible? "One database handling 350K reads/s" or "one server pushing 8 Tbps" → wrong design or wrong number.
3. Two methods. Compute top-down (DAU × actions) and bottom-up (requests per screen × screens per session × sessions). If they disagree by more than ~3×, find out why.
4. Ratios. Does the read:write ratio match the product's nature? Does egress:ingress (~140:1) fit "each post viewed hundreds of times"?
5. Known anchors. Compare with physical limits you know: a 100 Gbps NIC, a ~2 TB comfortable DB node, ~100K ops/s per Redis node, 86,400 seconds per day.
6. Units on every line. Bits vs bytes (×8), per day vs per second (×86,400), KB vs MB (×1,000). Write the unit next to every number.
7. Order-of-magnitude rounding. If rounding an input up or down by 2× changes your design decision, the decision is sensitive — say so and pick the safe side.

6. Cheat sheet

TIME                          RATES                          SIZE
1 day   ≈ 86,400 s ≈ 10^5 s   1M/day  ≈ 12/s                 char 1 B · int 4 B · long/ID 8 B
1 month ≈ 2.5 × 10^6 s        1B/day  ≈ 12K/s                UUID 16 B · row 100 B–1 KB
1 year  ≈ 3 × 10^7 s          peak    ≈ 2–3 × average        photo ~200 KB–2 MB · 1 min video ~10–30 MB

BANDWIDTH                     STORAGE                        CAPACITY (rough, per machine)
1 Gbps = 125 MB/s             2^10 ≈ 10^3 (KB), 2^20 (MB)    app server:  ~200–5,000 req/s
1 TB/day ≈ 12 MB/s ≈ 93 Mbps  2^30 (GB), 2^40 (TB)           Redis node:  ~100K ops/s, ~50 GB usable
1 PB/day ≈ 12 GB/s ≈ 93 Gbps  2^50 (PB), 2^60 (EB)           SQL primary: ~5–20K writes/s, ~1–2 TB
                              2^31 ≈ 2.1B (int32 limit!)     Cassandra:   ~10–50K writes/s, ~2 TB
LATENCY                                                      Kafka broker: ~100 MB/s
RAM 100 ns · SSD 0.1 ms · same-DC RTT 0.5 ms                 WebSocket gw: ~100K connections
HDD seek 10 ms · cross-continent RTT 150 ms

FORMULAS
QPS          = daily count ÷ 86,400          storage   = count/day × size × replicas × days
bandwidth    = QPS × payload size            servers   = peak QPS ÷ (per-server QPS × utilisation)
cache        = distinct hot objects × size   in-flight = QPS × latency  (Little's law)

7. How Instagram uses estimation

Each box in the Instagram diagram can be justified by a number from section 4:

8. Trade-offs & common pitfalls

Common mistakes

• Bits vs bytes. 1 Gbps is 125 MB/s, not 1 GB/s — an 8× error.
• Forgetting peak. Sizing for the average means falling over every evening.
• Forgetting replication and renditions. Raw storage is 1.5–3× logical; each photo exists in 4–8 sizes/formats; videos in a whole bitrate ladder.
• Forgetting write amplification. 1 post = ~200 timeline writes, plus index updates, plus replicas.
• Mixing up DAU and MAU, or using MAU to size daily traffic.
• Over-precision. "1,157.407 QPS" wastes time and signals the wrong priorities. Say "~1.2K/s".
• Computing numbers that drive nothing. Every estimate should end with "…therefore we need X".
• The naive 80/20 cache formula applied to request volume rather than distinct objects (overestimates by orders of magnitude).
• Ignoring the skew. The average user has 200 followers; the top account has hundreds of millions. Hot keys, not averages, break systems.
• Not saying "assume". Present inputs as assumptions, not as facts about a real company.

9. Real-world technologies

10. Interview corner

11. Check your understanding

Related lessons

• Requirements — where the scale assumptions come from
• Core Principles — Little's law, utilisation, percentiles and availability maths
• CDN — how the 85 PB/day of egress is actually served
• Object Storage — replication vs erasure coding, storage tiers
• Caching — eviction, hit ratios and cache cluster design
• Sharding — turning a shard count into a partitioning scheme
• Message Queues & Kafka — partitions, consumer groups and retention
• News Feed — fan-out write amplification in practice