1. The problem: why is the feed so hard?

When you open Instagram, the first thing you see is your feed: a scrolling list of photos, carousels and videos from accounts you follow (plus suggested posts and ads). It looks simple. Under the hood it is one of the hardest problems in the whole system, for three reasons:

• It is a many-to-many problem. You follow hundreds of accounts; each of them is followed by hundreds, thousands or hundreds of millions of people. Your feed is a personalised merge of hundreds of other people's activity, and nobody else in the world has exactly the same feed.
• It is extremely read-heavy and latency-sensitive. Opening the app must show posts in well under a second. Every feed refresh, every scroll to the next page, from hundreds of millions of daily users, hits this path. Reads outnumber posts by roughly 100:1 or more.
• The order matters. With hundreds of new posts available since your last visit and time to look at maybe 20–50, deciding which posts go at the top is a machine-learning problem in its own right.

The naive solution — "run a SQL query: select posts where author is in (people I follow) order by time desc limit 20" — works for a startup with 10,000 users. At Instagram scale the "people I follow" list spans hundreds of database shards, the query runs hundreds of thousands of times per second, and the result still has to be ranked. This lesson walks through how to make it fast, cheap and robust.

2. Requirements and back-of-the-envelope numbers

Always start a feed design by pinning down scope (see Requirements).

2.1 Functional requirements

• A user sees posts (photos, videos, carousels, Reels) from accounts they follow, newest-first or ranked by predicted interest.
• Infinite scroll: the feed is paginated; scrolling loads the next page without duplicates or gaps.
• Each post is shown with author name and avatar, media URLs, caption, like/comment counts and whether I already liked it.
• Freshness: a new post should appear in followers' feeds within seconds; the app shows a "New posts" indicator.
• Ads and recommended posts are blended into the feed.
• Unfollowing, blocking, deleting a post or switching an account to private must be respected (the post must disappear).

2.2 Non-functional requirements

• Low latency: feed API p99 under ~200–300 ms on the server side (the rest of the budget goes to network and media loading).
• High availability: the feed is the product. It's better to show a slightly stale or less well-ranked feed than an error (we prefer availability over consistency — see CAP & consistency).
• Eventual consistency is fine: if your friend's post shows up 5 seconds late, nobody notices. If it's missing for an hour, they do.
• Scalability: handle skewed follower distributions — most accounts have hundreds of followers, a few have hundreds of millions.

2.3 Rough numbers (assumptions, not official statistics)

Let's do the math (method from Back-of-the-Envelope Estimation):

Assume 500M daily active users (DAU)
Each DAU loads ~20 feed pages/day (several sessions × a few pages)
  → 500M × 20 = 10B feed requests/day
  → 10B / 86,400 s ≈ 115,000 feed requests/s average, ~250-350k/s at peak

Assume 100M new feed posts/day
  → 100M / 86,400 ≈ 1,150 posts/s average (~3k/s peak)

Assume a post's author has, on average, ~300 followers (weighted by posting volume)
  → 1,150 posts/s × 300 = ~345,000 timeline inserts/s if we push to everyone
  → if we skip followers inactive for 30+ days (say half), ~170,000 inserts/s

Read:write ratio for the feed ≈ 115,000 / 1,150 ≈ 100 : 1

Two observations drive the entire design. First, reads dwarf writes, so we should do work at write time to make reads cheap. Second, the average hides a brutal tail: one celebrity post can generate more fan-out work than a million ordinary posts. Keep both in mind.

3. The two fundamental approaches

The key term is fan-out: delivering one item to many recipients (one post → many followers' feeds). The big design decision is when fan-out happens.

3.1 Fan-out-on-read (the pull model)

In the pull model nothing special happens when a post is created: the post is written once to the post store (in Instagram's diagram, Cassandra sharded by user_id, so each author's posts sit together, sorted by time). All the work happens when somebody reads their feed:

1. Ask the Follow/Graph Service: "whom does viewer V follow?" → a list of, say, k = 500 account IDs.
2. For each followee, fetch their most recent posts (e.g. the last 10–20 post IDs with timestamps). Each followee's list is already sorted newest-first.
3. Merge the k sorted lists into one sorted list and take the top L (say 20–500 for ranking).

Step 3 is the classic k-way merge (the same algorithm as merging sorted runs in external sorting or LSM-tree compaction, see Indexing). You keep a min/max-heap holding the head of each list; repeatedly pop the newest item and push the next item from the same list:

import heapq

def merge_feeds(lists, limit):
    """lists: {author_id: [post, post, ...]} each sorted newest-first."""
    heap = []                                   # max-heap via negated timestamps
    for author, posts in lists.items():
        if posts:
            heapq.heappush(heap, (-posts[0].ts, author, 0))
    out = []
    while heap and len(out) < limit:
        neg_ts, author, i = heapq.heappop(heap)  # newest remaining post overall
        out.append(lists[author][i])
        if i + 1 < len(lists[author]):
            nxt = lists[author][i + 1]
            heapq.heappush(heap, (-nxt.ts, author, i + 1))
    return out
# Cost: O(k) to build the heap + O(L log k) to emit L posts.

The CPU cost of the merge is trivial (k = 500, L = 50 → a few thousand comparisons). The real cost is I/O and fan-in:

That last row is the killer. If each shard call has a 1% chance of exceeding its p99 latency of 20 ms, then a request that waits for 100 independent calls exceeds 20 ms with probability 1 − 0.99100 ≈ 63%. Fan-in turns every backend's rare slow response into the feed's common slow response — the "tail at scale" problem. You can mitigate it with hedged requests and timeouts that drop slow shards, but you are fighting physics on every single read.

Where pull shines: writes are O(1); there is no wasted work for inactive users; unfollow and delete are trivially correct (the next read simply doesn't see them); and the feed is always perfectly fresh. Facebook's original News Feed infrastructure ("Multifeed", described in public talks) was essentially pull-based: per-user activity is stored in leaf servers and aggregator servers query and merge them at read time — Facebook could afford it with huge in-memory leaf tiers.

3.2 Fan-out-on-write (the push model)

The push model flips the cost. Every user has a precomputed, materialised timeline — a list of post IDs that should appear in their feed. When someone posts, the post is fanned out to every follower's timeline:

1. The Upload Service stores the post and publishes a post.created event to Kafka, then immediately returns success to the author.
2. Fan-out workers consume the event, page through the author's follower list from the graph store (e.g. 1,000 follower IDs per page), and for each follower insert the post ID into timeline:{follower_id} in the Redis feed cache.
3. When a follower opens the app, the Feed Service reads the top of their own timeline — one key, one shard, one round trip.

Reads become trivially cheap, which is exactly what a 100:1 read-heavy system wants. The price is write amplification: one logical write (a post) turns into N physical writes (N = number of followers). Let's put numbers on it.

3.3 Write amplification: 1,000 vs 100,000,000 followers

If a celebrity posts 5 times a day, that's 500M inserts — more than the entire rest of a mid-sized platform. Worse, Kafka partitions are processed in order, so ordinary posts queued behind a celebrity's fan-out are delayed: your best friend's post shows up two minutes late because a pop star posted first. This is called the celebrity problem or hot-key / "Justin Bieber" problem, and it is why pure push does not work at Instagram scale.

3.4 Side-by-side comparison

4. The hybrid model: push for most, pull for celebrities

The observation that saves us: the two models fail in opposite places. Push fails for authors with huge follower counts; pull fails for viewers who follow many accounts. But a typical viewer follows only a handful of celebrities. So:

• Normal accounts (below a follower threshold — say 1M; the exact number is a tunable assumption) are pushed: their posts are fanned out into followers' timelines.
• Celebrity accounts (above the threshold) are not fanned out. Their posts are stored once, plus kept in a small, very hot "recent posts by author" cache.
• At read time, the Feed Service reads the viewer's precomputed timeline and pulls the recent posts of the few celebrities the viewer follows, then merges them — a k-way merge where k is ~5–20 instead of 500.

4.1 The hybrid read, step by step

def candidate_ids(viewer_id, since_cursor):
    # ① precomputed part: one Redis call
    pushed = redis.zrevrangebyscore(f"timeline:{viewer_id}", "+inf", since_cursor, limit=500)

    # ② pulled part: the few celebrities this viewer follows (cached list, ~0-30 IDs)
    celebs = graph.followed_celebrities(viewer_id)
    pulled = recent_posts_cache.multi_get(celebs)       # each list small & very hot

    # ③ k-way merge with small k, then hand to ranking
    return merge_feeds({"pushed": pushed, **pulled}, limit=500)

Why is ② cheap even though a celebrity's recent-posts list is read by tens of millions of viewers? Because it is the same small object for everyone, so it caches beautifully — replicated across many cache servers or even held in each Feed Service instance's local memory for a few seconds. A hot read key is easy to scale with replicas (see Caching: hot keys); a hot write fan-out is not.

4.2 Choosing and applying the threshold

• Threshold choice is an economic trade-off: lower thresholds move more work to read time (slower reads), higher thresholds create bigger write storms. In practice it's tuned from metrics — fan-out lag, Redis write load, read latency. Some systems use a gradient instead of a hard line: medium accounts get pushed only to their most active followers.
• Crossing the threshold: when an account grows past it, stop fanning out new posts; existing pushed entries stay and expire naturally. If it drops below, start pushing again. Hysteresis (e.g. switch to pull at 1.1M, back to push below 0.9M) avoids flapping.
• Celebrity-to-celebrity edge cases are fine: the merge doesn't care who follows whom.

4.3 Skipping inactive users

The second big optimisation: don't push to people who aren't going to look. A large fraction of any account's followers haven't opened the app in weeks. Fan-out workers check a cheap "last active" signal (a bitmap or a small key per user, updated on login) and skip anyone inactive for, say, 7–30 days. Their timelines are not maintained; if they come back, the timeline is rebuilt on demand by pulling (section 10).

Effects: if half of all follower edges point at inactive accounts, this alone halves fan-out writes and timeline memory. A further refinement is prioritised fan-out: push first to followers who are online right now or very active, then to the rest, so the people most likely to see the post get it fastest.

5. The timeline cache: data structure and commands

In Instagram's architecture diagram, timelines live in the Feed Cache (Redis). Each user's timeline is a single key holding post IDs only — never full post objects. This keeps entries tiny, and means an edited caption or an updated like count never needs to be propagated to millions of timeline copies: the ID is resolved to fresh data at read time (hydration, section 7).

5.1 Sorted set vs list

5.2 Example commands

# Fan-out worker: add post 3170538912345678 (created at 1730000000123 ms) to follower 42's timeline
ZADD    timeline:42  1730000000123  3170538912345678
# Cap the timeline at the newest 500 entries (remove ranks 0..-501 = everything older)
ZREMRANGEBYRANK timeline:42  0  -501
# Keep the key alive while the user is active (14 days)
EXPIRE  timeline:42  1209600

# Feed read, first page: newest 50 post IDs with their scores
ZREVRANGEBYSCORE timeline:42  +inf  -inf  WITHSCORES  LIMIT 0 50
# Next page: strictly older than the last score we returned (the "(" means exclusive)
ZREVRANGEBYSCORE timeline:42  (1729999000456  -inf  WITHSCORES  LIMIT 0 50

# Author deleted post 3170538912345678 → best-effort cleanup (read-time filter is the real guarantee)
ZREM    timeline:42  3170538912345678

Workers send these three write commands together in a pipeline (or a small Lua script so add + trim + expire are atomic), and batch many followers that live on the same Redis shard into one round trip. Timelines are distributed across a Redis cluster by hashing the key (consistent hashing / hash slots), so different users' timelines land on different shards and the load spreads evenly.

5.3 How big is the cache?

Assume 500M users with maintained timelines (active in the last ~2 weeks)
Cap = 500 entries per timeline, ~64 bytes per ZSET entry
  → 500 × 64 B = 32 KB per user
  → 500M × 32 KB = 16 TB of RAM
  → ~250 Redis nodes with 64 GB usable each, ×2-3 for replicas

Same data as packed 8-byte IDs + 4-byte timestamps:
  → 500 × 12 B = 6 KB per user → 3 TB total

16 TB of RAM is expensive but entirely normal for a company of this size, and the encoding optimisation shows how much headroom exists. Two levers keep it bounded: the cap (nobody scrolls 500 posts deep; older content can be fetched by pulling) and the TTL (inactive users' timelines evaporate and are rebuilt if they return).

6. The write path, end to end

Here is the full lifecycle of a post from "Share" to "it's in your followers' timelines":

 Author app    API GW    Upload Svc    Post store    Kafka      Fan-out worker    Graph svc     Redis timelines
     |  POST /posts |          |            |           |               |               |                |
     |------------->|--------->|            |           |               |               |                |
     |              |          |-- write -->|           |               |               |                |
     |              |          |  (post row, media refs)|               |               |                |
     |              |          |-- publish post.created(post_id, author_id, ts) --->   |                |
     |  201 Created |<---------|            |           |               |               |                |
     |<-------------|          |            |           |-- consume --->|               |                |
     |  (author sees post instantly via optimistic UI)  |               |               |                |
     |              |          |            |           |               |-- follower_count(author) ----> |
     |              |          |            |           |               |   > threshold? stop: celebrity |
     |              |          |            |           |               |   (pulled at read time)        |
     |              |          |            |           |               |-- followers(author, page 1) ->|
     |              |          |            |           |               |<-- 1,000 follower IDs --------|
     |              |          |            |           |               |  drop inactive followers       |
     |              |          |            |           |               |  group the rest by Redis shard |
     |              |          |            |           |               |-- pipeline per shard: ZADD +   |
     |              |          |            |           |               |   ZREMRANGEBYRANK + EXPIRE --->|
     |              |          |            |           |               |  ... next page until done ...  |
     |              |          |            |           |               |-- commit Kafka offset          |

Key properties of this design:

• The author never waits for fan-out. The API returns as soon as the post is durably stored and the event published. Fan-out is asynchronous — the essence of event-driven decoupling.
• At-least-once is safe. If a worker crashes mid-fan-out, Kafka redelivers the event and the worker redoes the pages. Because ZADD of an existing member just updates its score, replays are idempotent.
• Large fan-outs are split. A 500k-follower post is broken into many "fan-out jobs" (one per page of followers) that different workers process in parallel, so one big account doesn't block a partition.
• Separate queues by size. Many systems route big fan-outs to a different topic/worker pool so ordinary posts are never stuck behind them (a bulkhead, see Resilience patterns).
• The same event feeds other consumers: the Search Indexer, the Notification Service, analytics — each with its own consumer group.

6.1 Unfollow, block, delete and privacy changes

Timelines are a cache of a derived view, so they will occasionally contain IDs that should no longer be shown. Two layers keep the feed correct:

1. Best-effort cleanup: on unfollow, an async job removes that author's IDs from the viewer's timeline; on delete, a job can ZREM the post from followers' timelines (or simply skip it — see next point).
2. Read-time filtering (the real guarantee): during hydration the Feed Service drops any post that is deleted, from an author the viewer no longer follows or has blocked, or from a now-private account. Because this check runs on every read, correctness does not depend on cleanup jobs finishing.

7. The read path: from post IDs to a rendered feed

Getting candidate IDs is only the first step. A feed request goes through four stages: candidates → hydration → ranking → pagination/blending.

7.1 Hydration: turning IDs into posts

Hydration means filling in a list of bare IDs with everything the app needs to draw each post. For a page of 20 posts the Feed Service needs, per post:

The rules that make hydration fast:

• Batch everything. Never do 20 sequential lookups ("N+1 query problem"). Each data type is one parallel multi-get; all types are fetched concurrently, so hydration latency ≈ the slowest single batch, typically a few ms from cache.
• Cache aggressively. Post metadata is immutable-ish and shared by everyone who sees the post — perfect for a look-aside cache (Memcache/Redis; Facebook's public Memcache and TAO papers describe exactly this pattern for social data). Hit ratios above 95% are normal.
• Over-fetch slightly. Visibility filtering can drop items, so fetch ~25 to return 20 instead of making a second round trip.
• Tolerate partial failure. If the counter service times out, render the post without counts rather than failing the page; if post metadata is missing, drop that post.

Hydration happens after ranking for the final page, but ranking also needs some of this data (as features), which is why ranking systems keep their own feature store optimised for bulk reads.

7.2 The read-path sequence

 App        API GW     Feed Svc      Redis        Graph/celeb    Ranking svc    Hydration caches   Ads svc
  | GET /feed  |           |           |            cache        (+features)    (post,user,counts)    |
  |----------->|---------->|           |              |              |                |              |
  |            |           |-- ZREVRANGEBYSCORE timeline:V (500 IDs) ->|            |                |   |
  |            |           |-- followed_celebrities(V) + their recent posts -------->|              |   |
  |            |           |   (both calls in parallel; ~2-5 ms)       |              |                |   |
  |            |           |   merge + drop already-seen IDs           |              |                |   |
  |            |           |-- score(V, ~500 candidates) ------------------------------->|            |   |
  |            |           |-- request ad candidates (in parallel) ---------------------------------------->|
  |            |           |<-- ranked list (scores) ----------------------------------|              |   |
  |            |           |   re-rank: diversity, integrity, freshness                |              |   |
  |            |           |   blend ads into slots; cut first page of ~12             |              |   |
  |            |           |-- multi-get posts, authors, counts, viewer_liked ---------------------->|   |
  |            |           |<-- hydrated objects ---------------------------------------------------|   |
  |            |           |   visibility filter, sign CDN URLs, build next cursor     |              |   |
  |<-- 200 JSON (posts + next_cursor) ----|            |              |              |                |   |
  |  app downloads media from the CDN, logs impressions (seen posts) asynchronously    |              |   |

A realistic server-side latency budget for this path (assumptions): timeline + celebrity fetch 5 ms, ranking 50–100 ms (the biggest chunk), hydration 10–20 ms, serialisation 5 ms — about 100–150 ms total at p50 and under ~300 ms at p99.

8. Ranking: from chronological to ML-ranked

For its first six years Instagram's feed was reverse-chronological: newest posts first. In 2016 Instagram announced it would move to a ranked feed, because users were missing a large share of posts from people they cared about — with hundreds of posts arriving between visits, "newest first" means "whatever happened to be posted in the last hour first". (Later, Instagram also added "You're all caught up" markers and, in 2022, optional "Following" and "Favorites" views that are chronological again — a good reminder that ranking is a product choice, not just a technical one.)

8.1 Stage ① — candidate generation

Collect everything that could be shown: the pushed timeline, pulled celebrity posts, and (in modern Instagram) recommended posts from accounts you don't follow, produced by separate retrieval systems similar to Explore's. Already-seen posts are removed here. The output is a few hundred to a few thousand post IDs with minimal metadata.

8.2 Stage ② — feature lookup

A feature is a number the model can use. Instagram has publicly described the main signal families it uses (paraphrased):

Features come from a low-latency feature store (a key-value store of precomputed aggregates, refreshed by batch and streaming jobs that consume the Kafka event stream) plus real-time counters (likes in the last 10 minutes). The Analytics & ML pipeline computes them and trains the models offline.

8.3 Stage ③ — ML scoring

Modern feed rankers are multi-task models: a neural network predicts several probabilities at once for (viewer, post):

P(like)  P(comment)  P(share)  P(save)  P(dwell > 5 s)  P(profile tap)
P(hide)  P(report)  P(“see fewer”)                          ← negative events

score = w1·P(like) + w2·P(comment) + w3·P(share) + w4·P(save) + w5·P(dwell)
        − w6·P(hide) − w7·P(report)

The weights encode product priorities (e.g. comments and shares indicate more meaningful interaction than a quick like). Because the heavy model costs maybe a millisecond of GPU/CPU time per candidate, it only runs on the few hundred survivors of the first-pass model. Scoring is a batched call to a model-serving fleet; the Feed Service sends the viewer's features once plus a matrix of candidate features.

8.4 Stage ④ — re-ranking for diversity, integrity and freshness

Sorting purely by score produces bad feeds: five posts in a row from the same prolific friend, all videos, or one viral topic everywhere. A rules + lightweight-model pass adjusts the list:

• Diversity: no more than N consecutive posts from the same author; mix formats (photo, carousel, video); spread topics. Often implemented as a greedy pass that penalises an item's score based on similarity to items already placed.
• Integrity: demote content classifiers flag as borderline, misleading, spammy or low quality; remove anything violating policy that slipped through.
• Freshness: boost recent posts so the feed doesn't feel stale; guarantee that very recent posts from close friends appear near the top.
• Business rules: "You're all caught up" marker after all followed posts from the last ~2 days have been seen, then suggested posts.

8.5 Stage ⑤ — ad insertion

Ads are not stored in timelines. They're chosen per request by a separate ads system, called in parallel with ranking so it adds no latency:

1. The ads service retrieves eligible ads for the viewer (targeting, budget, frequency caps).
2. It runs an auction: roughly bid × predicted action rate + ad quality decides which ads win.
3. The Feed Service places winning ads into slots with spacing rules — e.g. no ad in the first position, at least several organic posts between ads, ad load adjusted per user. The positions themselves can be optimised by a model that weighs ad revenue against the predicted drop in engagement.
4. Impressions and clicks are logged as events for billing and model training.

9. Pagination, deduplication and freshness

9.1 Why offset pagination breaks

The naive API is GET /feed?page=2 (offset = 20). But the feed changes between requests: if three new posts arrive while you look at page 1, page 2 starts three items "earlier" and you see three duplicates. Deletions cause the opposite — skipped posts. Offsets are also expensive for databases (they must skip rows).

9.2 Cursor pagination

A cursor is an opaque token that marks where you stopped, not how many you've seen (see API design). For a chronological feed the cursor encodes the last (timestamp, post_id) returned:

GET /v1/feed?limit=12
→ { "items": [...12 posts...], "next_cursor": "eyJ0cyI6MTcyOTk5OTAwMDQ1NiwiaWQiOjMxNz..." }

GET /v1/feed?limit=12&cursor=eyJ0cyI6MTcyOTk5OTAwMDQ1NiwiaWQiOjMxNz...
→ server decodes {ts: 1729999000456, id: 317...} and reads strictly older entries:
  ZREVRANGEBYSCORE timeline:V (1729999000456 -inf LIMIT 0 12

New posts arriving at the head never shift what's "older than the cursor", so there are no duplicates or gaps. Base64-encoding (and ideally signing) the cursor keeps clients from depending on its internals.

9.3 Pagination of a ranked feed

Ranked feeds are trickier: there's no single sort key, and re-running the ranker for page 2 could reorder everything (new likes change scores). The common solution is a feed session:

1. On the first request (app open or pull-to-refresh), rank the full candidate set once and store the ordered list of ~200–500 IDs in a short-lived cache entry feedsession:{session_id} (TTL ~30 minutes).
2. The cursor encodes {session_id, position}. Page 2 simply reads positions 12–23 from the stored ranking and hydrates them — no re-ranking, stable order, very cheap.
3. When the session's list runs out, the server ranks the next batch of older candidates, excluding everything already served.

9.4 Dedup of seen posts

Users should not see the same post again and again across sessions. The app reports impressions (a post was on screen for, say, > 50% of its area for a moment) in batches. The server keeps a per-user seen set: a capped set of recently seen post IDs or a compact Bloom filter (a probabilistic set: no false negatives, a small rate of false positives — occasionally hiding an unseen post, which is acceptable). Candidate generation removes seen IDs; the ranker may also demote rather than remove, because re-showing a friend's post you scrolled past quickly can be fine. Within a single response, a simple in-memory hash set guarantees the same ID is never returned twice (e.g. if a post arrived via both push and pull).

9.5 Freshness and the "New posts" indicator

Freshness has two parts:

• Fan-out lag — time from "Share" to the ID being in followers' timelines. Monitor it as an SLO (e.g. p99 under ~10 s for normal accounts, an assumption). Kafka consumer lag is the leading indicator.
• Telling the user — while you're scrolling, the app should not shove new posts in at the top (content jumping under your thumb is awful). Instead it periodically calls a cheap endpoint, or receives a hint over an existing real-time connection (like the one used for DMs), saying "N new posts newer than your head cursor". It shows a "New posts" pill; tapping it or pulling to refresh starts a new feed session.

# cheap "anything new?" check: count entries newer than the client's head timestamp
ZCOUNT timeline:42 (1730000000123 +inf      → 3   (show "New posts" pill)

A related trick: on app open, the client renders its locally cached last feed immediately (see Clients), then swaps in fresh results — the app feels instant even if the network is slow.

10. Rebuilding a timeline on cache miss

A user's timeline key may be missing because they were inactive (TTL expired, or fan-out skipped them), because they're brand new, or because a Redis node was lost. The Feed Service then falls back to pull for that one user:

def get_timeline(viewer_id):
    ids = redis.zrevrange(f"timeline:{viewer_id}", 0, 499, withscores=True)
    if ids:
        return ids
    # Cache miss → rebuild. Single-flight lock so 5 concurrent requests from the
    # same user (app open + prefetch) don't all rebuild at once.
    with lock(f"rebuild:{viewer_id}", ttl_s=5) as got_lock:
        if not got_lock:
            return wait_for_or_fallback(viewer_id)
        followees = graph.following(viewer_id)                 # e.g. 500 IDs
        recent = post_index.recent_posts_multi(followees, per_author=10, max_age_days=7)
        ids = merge_feeds(recent, limit=500)                   # k-way merge (section 3.1)
        redis.pipeline() \
             .zadd(f"timeline:{viewer_id}", {pid: ts for pid, ts in ids}) \
             .expire(f"timeline:{viewer_id}", 14 * 86400).execute()
        mark_active(viewer_id)        # fan-out workers will push to this user again
        return ids

• The rebuild is the expensive pull read (hundreds of lookups), but it happens once per returning user, not on every read.
• To keep the first screen fast, the service can return a quick partial result (e.g. merge the 50 most-interacted-with followees first) and complete the rebuild asynchronously.
• Mass rebuilds are dangerous. If a Redis shard holding 2 million timelines dies and has no replica, 2 million users trigger rebuilds — a thundering herd on the post store. Protect with replicas (promote a follower instead of rebuilding), rate-limited rebuild queues, and serving degraded (chronological, cached) feeds while rebuilding.
• There's a small race: a post fanned out between "rebuild read" and "rebuild write" could be missed. Writing with ZADD (a merge, not an overwrite) and marking the user active before reading closes most of the gap; the read-time celebrity merge and next refresh cover the rest.

11. Failure modes and graceful degradation

The feed must almost never show an error screen. Design every dependency as optional where possible (see Resilience patterns):

12. How Instagram uses fan-out and ranking

Putting it together with the main architecture diagram:

1. Upload Service writes the post metadata to the Post store (Cassandra, sharded by user_id) and publishes post.created to Kafka.
2. Feed Fan-out Workers consume it, read the author's followers from the Social Graph store, skip celebrities and inactive followers, and write the post ID into each follower's timeline in the Feed Cache (Redis).
3. The News Feed Service reads the viewer's timeline, pulls in followed celebrities' recent posts, removes seen items, calls the ranking models (trained by the Analytics & ML pipeline), blends ads, hydrates the top items and returns JSON with signed CDN URLs.
4. The app downloads media from the CDN and logs impressions, which flow back through Kafka into the seen-set and the training data.

13. Trade-offs & common pitfalls

Common pitfalls

• Storing full post objects in timelines — multiplies memory by 100× and makes every edit or like-count change a fan-out. Store IDs; hydrate at read time.
• Pure push with no celebrity handling — one viral account stalls the whole fan-out pipeline.
• Offset pagination — duplicates and gaps as the feed changes. Use cursors / feed sessions.
• N+1 hydration — fetching author and counts one post at a time. Batch and parallelise.
• Trusting timelines for correctness — always re-check visibility (block, private, delete) at read time.
• Using 64-bit IDs as Redis scores — precision loss; use ms timestamps.
• Unbounded timelines — no cap/TTL means memory grows forever for users who never scroll that far.
• Hard dependency on the ranker — no fallback means an ML outage becomes a full product outage.
• Mass rebuild stampede after losing a cache node — replicate, rate-limit rebuilds, serve degraded feeds.

14. Real-world technologies

15. Interview corner

16. Check your understanding

Related lessons

• Social Graph & Follow Service — where follower lists for fan-out and followee lists for pull come from
• Caching — Redis data structures, hot keys, stampedes and TTLs used by timelines and hydration
• Message Queues & Kafka — the event bus that drives asynchronous fan-out
• Analytics & ML Ranking — how the ranking models and features are trained and served
• Unique ID Generation — time-sortable post IDs that make timelines and cursors work
• Likes, Comments & Counters — the counts and viewer-liked flags fetched during hydration
• API Design — cursor pagination and response shapes for the feed endpoint
• Resilience patterns — timeouts, circuit breakers and fallbacks on the feed read path
• End-to-End Request Flows — the feed load traced through the whole architecture