1. The problem: why do we need a CDN?

Instagram is a media-first, read-heavy product. Every time someone scrolls their feed, the app downloads dozens of images and video segments. Suppose all of that media lived only in one data center in Virginia, USA:

• Latency is physics. Light in fibre travels ~200,000 km/s. A round trip Mumbai ⇄ Virginia (~13,000 km each way) takes at least ~130 ms — and a single HTTPS image fetch needs several round trips (TCP handshake, TLS handshake, request). Realistically a user in India would wait 400–800 ms per image. Scrolling would feel broken.
• Bandwidth is enormous. With ~2 billion monthly users each viewing hundreds of images a day, media egress is measured in terabits per second. No single data center uplink can handle that, and transit bandwidth across oceans is expensive.
• The origin would melt. Object storage (S3-like) is built for durability, not for serving the same viral photo 50 million times in an hour.

2. How a CDN works

2.1 The building blocks

2.2 How does the user reach the nearest edge?

Two main techniques route the user to a nearby PoP:

1. DNS-based routing (GeoDNS). When the app resolves scontent.cdninstagram.com, the CDN's authoritative DNS server looks at where the request came from and returns the IP of a nearby, healthy PoP. Different users get different IPs for the same name.
2. Anycast. Many PoPs announce the same IP address via BGP. Internet routers naturally deliver packets to the topologically closest one. Cloudflare relies heavily on this; it also absorbs DDoS traffic by spreading it across all PoPs.

2.3 Cache hit vs cache miss

1. The app requests https://cdn.example.com/p/9f3a…/1080.jpg. The edge computes the cache key (usually host + path, plus selected query params/headers).
2. Cache HIT: the edge has the file and it's still fresh (within its TTL) → it returns it immediately. ⚡ ~10–30 ms total.
3. Cache MISS: the edge asks the origin shield; if the shield also misses, the shield fetches from the origin (object storage).
4. The response flows back, and each layer stores a copy according to the Cache-Control header, so the next user nearby gets a hit.

 User            Edge (Mumbai)           Shield (Asia)            Origin (S3)
  |  GET img.jpg      |                        |                        |
  |-----------------> |  lookup cache key      |                        |
  |                   |--- MISS -------------> |                        |
  |                   |                        |--- MISS ------------> |
  |                   |                        | <------ 200 + bytes -- |
  |                   | <------ 200 + bytes -- |  (store copy)          |
  | <-- 200 + bytes - |  (store copy)          |                        |
  |                                                                     |
  |  GET img.jpg  (another user in Mumbai, 2 s later)                   |
  |-----------------> |  HIT ✔ → served from RAM/SSD in ~10 ms          |

The single most important CDN metric is the cache hit ratio (CHR):

cache hit ratio = cache hits / (cache hits + cache misses)

Going from 90% to 99% CHR sounds small, but it means origin traffic drops from 10% to 1% of requests — a 10× reduction in origin load.

3. Pull CDN vs Push CDN

4. Controlling what gets cached (and for how long)

4.1 Cache-Control headers & TTL

The origin tells caches how to behave through HTTP headers:

Cache-Control: public, max-age=31536000, immutable   # cache for 1 year, never revalidate
Cache-Control: private, max-age=0, no-store           # never cache (e.g. personalised API responses)
Cache-Control: public, max-age=60, s-maxage=300       # browser: 60 s, shared caches (CDN): 300 s
ETag: "9f3a1c"                                        # version fingerprint for revalidation

• TTL (time to live) — how long a cached copy is considered fresh.
• Revalidation — after the TTL expires, the cache can ask the origin If-None-Match: "9f3a1c"; the origin replies 304 Not Modified (no body) if unchanged — cheap.
• stale-while-revalidate lets the edge serve a slightly stale copy while it refreshes in the background, hiding origin latency.

4.2 Cache keys

The cache key decides whether two requests are "the same file". If you accidentally include a random query param (e.g. a tracking id) in the key, every request becomes a miss and your hit ratio collapses. Good practice: normalise the key — include only what changes the response (path, size, format), and strip everything else.

4.3 Invalidation: the hard part

"There are only two hard things in computer science: cache invalidation and naming things." What if a user edits or deletes a photo that's cached in 300 PoPs?

5. Securing content on a CDN

• Signed URLs / tokens: the API server generates a URL containing an expiry time and an HMAC signature, e.g. …/1080.jpg?exp=1730000000&sig=ab12…. The edge verifies the signature without contacting the origin. Leaked links stop working after expiry. Instagram media URLs carry exactly this kind of signature (oe=, oh= params).
• TLS at the edge: the edge terminates HTTPS close to the user (faster handshakes) and re-encrypts to the origin.
• DDoS absorption & WAF: CDNs have enormous aggregate capacity and filter malicious traffic before it reaches you. See Security.
• Origin protection: only CDN IPs (or authenticated requests) may reach the origin bucket.

6. CDNs for video: adaptive streaming

Reels and Stories videos aren't served as one giant file. The media pipeline cuts each video into small 2–6 second segments at several bitrates (240p…1080p), plus a manifest (HLS .m3u8 / DASH .mpd). The player downloads segments one by one from the CDN and switches quality based on bandwidth. Segments are just small static files — perfect for CDN caching — and the first few segments of popular videos are almost always hot in the edge cache, giving near-instant playback.

Two HTTP features make this efficient:

• Range requests (Range: bytes=0-1048575) let a player fetch just part of a file, which the CDN can serve from a partially cached object. The edge often fetches large files from the origin in fixed-size chunks (e.g. 1–2 MB), so a user seeking to minute 3 doesn't force the whole file to be downloaded first.
• Byte hit ratio vs request hit ratio. Video segments are big, thumbnails are tiny. A CDN can have a 95% request hit ratio but only a 80% byte hit ratio if big videos miss more often. Bandwidth cost follows the byte hit ratio, so watch both.

7. Inside an edge server

An edge server is essentially a very fast HTTP reverse proxy with a huge cache. Understanding its internals explains most CDN behaviour.

7.1 The cache hierarchy inside one PoP

• Key ownership within a PoP. A PoP may have hundreds of servers. If each cached every object independently, the PoP would store 100 copies of the same photo and the effective cache size would shrink 100×. Instead, servers use consistent hashing on the cache key so each object has one "owner" server. Very hot objects are deliberately replicated to many servers to spread load.
• Eviction. When the disk is full, something must go. Plain LRU (least recently used) is easily polluted by "one-hit wonders" — objects requested once and never again, which are extremely common for user-generated content. CDNs therefore use smarter policies: segmented LRU, LFU-style admission, or "only cache on the second request" (admission filters, often implemented with a Bloom filter — see Caching). Meta has published research showing that smarter admission and eviction noticeably raise the hit ratio of its photo caches.
• Request collapsing (coalescing). If 1,000 requests for the same uncached object arrive within 50 ms, the edge sends one request upstream and makes the other 999 wait for that response. This is the main defence against the thundering herd described in Caching.

7.2 Serving stale content when things break

Two headers let edges keep serving when the origin is slow or down:

Cache-Control: max-age=600, stale-while-revalidate=60, stale-if-error=86400

• stale-while-revalidate=60: for 60 s after expiry, serve the stale copy immediately and refresh it in the background.
• stale-if-error=86400: if the origin returns 5xx or times out, keep serving the stale copy for up to a day.

This makes the CDN a resilience layer as well as a performance layer — a pattern covered in Resilience Patterns as graceful degradation.

8. Beyond static files: dynamic acceleration & edge compute

CDNs started as static-file caches but now do much more:

9. Operating a CDN: metrics, cost and multi-CDN

9.1 What to monitor

These feed the dashboards and SLOs described in Observability. Real-user monitoring from the app (how long images actually took to appear) is the ultimate truth.

9.2 The cost model

Commercial CDNs mostly charge per GB delivered (cheaper at higher volume and in North America/Europe, pricier in some regions), plus per-request fees and extras such as edge compute. Two levers dominate the bill: bytes delivered (shrink them with better formats like WebP/AVIF and the right rendition per screen) and origin egress (cut it with a higher byte hit ratio and an origin shield). A 30% smaller image format saves 30% of the CDN bill and makes the app faster — one of the rare wins on every axis.

9.3 Multi-CDN

Large companies that don't run their own CDN often use two or more providers at once:

• Availability: if one CDN has a global outage (it happens, e.g. the June 2021 Fastly incident that briefly took down many large websites), DNS steers traffic to the other.
• Performance: real-user measurements decide, per country or ISP, which CDN is fastest.
• Cost leverage: traffic can be shifted toward the cheaper provider.

The price is complexity: consistent configuration, signed-URL schemes and purges across providers, and a lower hit ratio per CDN since traffic is split.

10. How Instagram uses the CDN

1. The Feed service returns JSON containing post metadata and signed CDN URLs — never the bytes themselves. The API path and the media path are separate.
2. The app picks the right rendition for the device (e.g. 640px for a small phone, 1080px for a large one, WebP/AVIF if supported) — these renditions were generated by the media processing workers.
3. The app fetches media directly from the CDN. Popular content is served from edge RAM/SSD; long-tail content triggers an origin pull from object storage.
4. The app prefetches the next few posts' images while you scroll, so they're already on the device when you get there (see Clients).

Rough numbers

Assume 500M daily active users view ~100 images/day at ~200 KB average:

500M × 100 × 200 KB = 10 PB/day of media egress
10 PB / 86,400 s ≈ 116 GB/s ≈ 0.93 Tbps average (peak ~2–3×)
With a 95% hit ratio → origin serves only ~5.8 GB/s; the CDN absorbs the rest.

See Back-of-the-Envelope Estimation for how to do this math yourself.

11. Trade-offs & common pitfalls

Common pitfalls

• Caching personalised responses (e.g. a user's feed JSON) under a shared key → users see each other's data. Always mark them private / no-store.
• Cache-busting by accident — random query params in cache keys destroy the hit ratio.
• Thundering herd on the origin — a viral new post misses in 300 PoPs simultaneously. Fix with an origin shield and request collapsing (the edge sends one origin request and makes the others wait for it).
• Overwriting files in place instead of using new versioned URLs.

12. Real-world technologies

13. Interview corner

14. Check your understanding

Related lessons

• DNS — how users get routed to the nearest PoP
• Object Storage — the CDN's origin
• Media Processing — where the cached renditions come from
• Caching — the same ideas (TTL, hit ratio, invalidation) applied inside the data center
• Security — signed URLs, DDoS protection, WAF