TL;DR
- Every Instagram feature is a chain of components. This lesson follows nine real user actions hop by hop, from the phone to storage and back, naming every box on the architecture diagram.
- Almost every flow has two parts. A short synchronous part, where the user waits (target: a few hundred ms), and a longer asynchronous part that runs after the response through Kafka (fan-out, transcoding, indexing, notifications, analytics).
- The API path (small JSON through Load Balancer → API Gateway → services) and the media path (large bytes going straight to Object Storage on upload and coming from the CDN on download) are always kept separate.
- Reads are fast because the work was done ahead of time. Feeds are pre-computed by fan-out, renditions are pre-generated, and search is pre-indexed. Writes are fast because the expensive work is deferred to background workers.
- Each flow below has an ASCII sequence diagram, numbered steps, a latency budget, a failure analysis and links to the lesson behind each concept. The lesson ends with a sync-vs-async summary table.
1. The problem: why trace requests end to end?
The other lessons teach one building block at a time: DNS, load balancing, caching, sharding, Kafka and so on. Real systems are never one block, though. When you double-tap a photo, at least six components take part in under a tenth of a second, and a few more keep working for seconds afterwards. Until you can trace that path, you don't really understand the system. Interviewers know this, so "walk me through what happens when…" is one of the most common system design questions.
Tracing a request end to end teaches you three things no single lesson can:
- Where the time goes. A 200 ms budget gets split between network round trips, the gateway, cache lookups, database queries and ranking. If you know the budget, you know what to optimise.
- What the user waits for and what they don't. The design trick behind Instagram's snappy feel is to make the user wait only for the minimum: "your post is saved". Everything else (resizing, fan-out to 500 followers, search indexing, push notifications) happens asynchronously.
- What breaks, and what the user sees when it does. Every hop can fail. A good design decides, for each hop, whether to retry, fall back, degrade or fail loudly.
🧠 Analogy
Think of sending a parcel. At the counter (the synchronous part) the clerk weighs it, takes payment and hands you a tracking number. You leave after two minutes. Sorting, trucks, planes and the final delivery van (the asynchronous part) take two days, and you don't stand at the counter waiting for them. If the clerk made you wait until the parcel arrived, the queue would reach around the block. Instagram's "201 Created" is the tracking number. Kafka is the sorting centre. The notification is the "delivered" text message.
2. The cast: every component on the diagram
All nine flows use the same architecture. Here it is again, with the synchronous request path drawn solid and the asynchronous event path dashed.
Solid arrows: the synchronous API path the user waits on. Dashed purple: events flowing through Kafka to background consumers, which write their results back into the data layer (index, timelines, renditions). Dashed orange: media served by the CDN, origin-pulled from Object Storage.
2.1 Conventions used in every flow
- DNS and the TLS connection are set up once. Flow 1 shows DNS resolution and the TLS handshake in full. After that the app reuses one long-lived HTTP/2 or HTTP/3 connection, so later flows skip those hops.
- "LB + API Gateway" stands for two hops. The Load Balancer spreads connections over healthy gateway nodes. The API Gateway terminates TLS, validates the JWT (authentication), applies rate limits and routes to the right service. They are merged into one column to keep the diagrams readable.
- Service-to-service calls inside the data center are gRPC or Thrift over persistent connections, with deadlines (see API design and microservices). Every call carries a trace ID so the whole flow shows up as one distributed trace.
- "Post Meta (Cassandra)" is the diagram's Post/Comment/Story Metadata (Cassandra) store, partitioned by
user_idorpost_id(see sharding). - Arrow styles in the ASCII diagrams:
----->synchronous call,.....>response,~~~~>asynchronous event via Kafka,--local work inside one component. - All numbers are assumptions for a typical user on a good 4G/5G connection with ~40 ms round-trip time (RTT) to the nearest region. They are not official Instagram measurements.
2.2 The two golden rules behind every flow
⚡ Do the minimum synchronously
The synchronous path writes the source of truth (one row, one edge, one message), publishes an event, and returns. Everything that can be eventually consistent is pushed to Kafka consumers.
📦 Bytes never go through services
Uploads go straight to Object Storage with a pre-signed URL. Downloads come from the CDN. The API only ever carries IDs, metadata and URLs, so app servers stay small, stateless and cheap.
🧮 Precompute for reads
Instagram is read-heavy (assume ~100 reads per write). Feeds, story trays, renditions, counters and search indexes are built at write time so that reads are a few key lookups.
🔁 Make every step retry-safe
Networks fail and Kafka redelivers. Every write uses an idempotency key or a natural primary key, so doing it twice has the same effect as doing it once (see resilience patterns).
3. Flow 1 — App open & login
The user installs the app, types a username and password and taps Log in. This flow sets up everything the other flows reuse: the IP address of the nearest region, an encrypted connection, and the tokens that prove who the user is.
Clients DNS Load API User/Auth User DB Kafka
Balancer Gateway Service (PostgreSQL)
| | | | | | |
| 1 resolve i.instagram.com | | | |
|------->| | | | | |
| 2 IP of nearest region (GeoDNS) | | |
|<.......| | | | | |
| 3 TLS 1.3 + POST /v1/auth/login | | |
|----------------->| | | | |
| | | 4 forward to healthy node | |
| | |--------->| | | |
| | | |-- 5 rate-limit by IP + username |
| | | | 6 VerifyCredentials (gRPC) |
| | | |--------->| | |
| | | | | 7 SELECT user by username
| | | | |----------->| |
| | | | | 8 user_id, password_hash
| | | | |<...........| |
| | | | |-- 9 argon2/bcrypt verify, risk check
| | | | | 10 INSERT session + refresh token
| | | | |----------->| |
| | | | | 11 user.logged_in |
| | | | |~~~~~~~~~~~~~~~~~~~~~~~>|
| | | | 12 access JWT + refresh token |
| | | |<.........| | |
| 13 200 OK {tokens} | | | |
|<............................| | | |
== app now fires feed, stories-tray and push-token calls in parallel ==|
| | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients → DNS: the app resolves
i.instagram.com. The phone's OS cache or the ISP's recursive resolver usually already has the answer. If not, the query reaches the authoritative DNS, which uses GeoDNS to pick a nearby, healthy region. - DNS → Clients: returns the virtual IP (VIP) of that region's Load Balancer, with a short TTL (e.g. 60 s) so traffic can be moved quickly during an outage (see multi-region).
- Clients → Load Balancer: the app opens a TCP connection, performs a TLS 1.3 handshake (1 RTT, or 0 RTT on resumption; see networking basics) and sends
POST /v1/auth/login {username, password, device_id}. - Load Balancer → API Gateway: an L4 load balancer forwards the connection to a healthy gateway node, chosen by consistent hashing or least-connections. Health checks keep failed nodes out of rotation.
- API Gateway: terminates TLS. Login is one of the few unauthenticated routes, so instead of checking a token the gateway applies a strict rate limit per IP and per username (e.g. 5 attempts/min) to block credential stuffing.
- API Gateway → User/Auth Service: forwards the request as an internal gRPC call with a deadline (e.g. 1 s).
- User/Auth Service → User DB (PostgreSQL): looks up the account. Usernames map to a
user_idthrough a lookup table. Theuser_idtells the service which shard holds the row, and reads can go to a read replica. - User DB → User/Auth Service: returns
user_id, the salted password hash, the account status and the 2FA settings. - User/Auth Service (local): verifies the password with a deliberately slow hash (argon2 or bcrypt, ~50–100 ms of CPU) and runs risk checks (new device? impossible travel?). Risky logins get a 2FA or e-mail challenge instead of tokens. See authentication and security.
- User/Auth Service → User DB: writes a session record: the device and a hash of a new refresh token. This write goes to the shard's primary.
- User/Auth Service → Kafka: publishes
user.logged_inasynchronously. Analytics & ML Training uses it for fraud models, and the Notification Service can send "new login from Chrome on Windows" alerts. - User/Auth Service → API Gateway: returns a short-lived signed access token (a JWT, valid ~15–60 min) and a long-lived opaque refresh token (valid for weeks).
- API Gateway → Clients:
200 OK. The app stores the tokens in the iOS Keychain or Android Keystore. It then fires three calls in parallel over the same connection: the home feed (Flow 3), the stories tray (Flow 6) andPOST /v1/devices {push_token}, which registers the APNs/FCM token used in Flow 9.
3.1 Every later app open (already logged in)
Login happens rarely. A normal cold start is much cheaper. The app paints the last feed it cached on disk immediately (see clients), reuses its tokens, and refreshes in the background. The gateway validates the JWT locally: it checks the signature with the Auth service's public key (cached), the expiry and the audience. No call to the User/Auth Service is needed, which takes the Auth service off the hot path of every request. If the access token has expired, the app calls POST /v1/auth/refresh once. The Auth service checks the refresh token against the User DB (where it can also be revoked) and issues a new JWT.
3.2 Latency budget (first login)
| Hop | Typical time | Notes |
|---|---|---|
| DNS lookup | 0–50 ms | 0 if cached on the device; one recursive lookup otherwise |
| TCP + TLS 1.3 handshake | ~80 ms | 2 RTTs at 40 ms (1 with QUIC/HTTP/3, ~0 when resumed) |
| Request upload + LB + gateway | ~25 ms | Half an RTT plus ~3 ms of routing and rate-limit checks |
| User DB lookup | ~3 ms | Indexed primary-key read on a replica |
| Password hash verify | ~70 ms | Deliberately slow to make offline cracking expensive |
| Session write + token signing | ~6 ms | Primary write plus a signature |
| Response download | ~20 ms | Half an RTT |
| Total | ~200–250 ms | Acceptable for a once-in-months action. Every later request skips ~180 ms of it. |
3.3 What happens on failure
- DNS outage or slow resolver: the app keeps using cached IPs. Many large apps also ship fallback IPs or use multiple resolvers. A GeoDNS failover moves users to another region within roughly a TTL.
- A gateway node dies: LB health checks drop it within seconds. Clients on that node reconnect, and the login is retried safely because nothing was committed.
- User/Auth Service is down: new logins fail, but every existing session keeps working because JWTs are validated at the gateway without calling Auth. That containment is the main reason for using signed tokens. Refreshes fail too, so a long Auth outage slowly logs people out as access tokens expire.
- User DB primary fails: credential reads still succeed from replicas, but session writes fail until a replica is promoted (seconds to tens of seconds; see replication). The gateway returns a retryable 503 and the app retries with exponential backoff plus jitter.
- Brute-force attack: rate limits return
429 Too Many Requests. Repeated failures trigger CAPTCHA or account lockout.
4. Flow 2 — Uploading a photo post
This is the most important write in Instagram, and it touches the most components. The design goal is to make the user wait only until the photo is durably stored and the post row exists. Everything else (renditions, moderation, delivery into 500 followers' feeds, hashtag indexing, notifications) happens in the background. The flow has three phases.
The upload's critical path is the three green and yellow boxes. Everything purple runs after the user has been told "posted", in parallel, through Kafka consumers.
4.1 Phase A — the synchronous upload
Clients LB + Upload Object Post Meta Kafka
API Gateway Service Storage (Cassandra)
| | | | | |
|-- 1 resize, compress, hash, idempotency key | |
| 2 POST /v1/media/uploads | | |
|---------->| | | | |
| | 3 auth OK, route | | |
| |---------->| | | |
| | |-- 4 new media_id + pre-signed PUT URL
| | | 5 upload session = PENDING |
| | |-------------------->| |
| 6 {media_id, upload_url} | | |
|<......................| | | |
| 7 PUT bytes directly (resumable)| | |
|-------------------------------->| | |
| 8 200 + ETag | | | |
|<................................| | |
| 9 POST /v1/posts {media_id, caption} | |
|---------->| | | | |
| | 10 route (Idempotency-Key) | |
| |---------->| | | |
| | | 11 HEAD: object exists? |
| | |-------->| | |
| | | 12 INSERT post (PROCESSING) |
| | |-------------------->| |
| | | 13 post.created | |
| | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>|
| 14 201 Created {post_id} | | |
|<......................| | | |
| | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients (local): the app applies filters, resizes the image to at most 1080 px wide and compresses it (a 12 MP camera photo of ~4 MB becomes ~300–600 KB). It also computes a content hash and generates an idempotency key (a random UUID) for this post. Doing this on the phone saves upload time and server CPU (see clients).
- Clients → LB + API Gateway:
POST /v1/media/uploads {type: "image/jpeg", size, sha256}with the JWT. - API Gateway → Upload Service: the JWT is valid and the upload rate limit is fine (e.g. 100 posts/day), so the request is routed.
- Upload Service (local): generates a globally unique, time-sortable
media_id(see ID generation) and signs a pre-signed PUT URL forraw/{media_id}in Object Storage, valid for ~15 minutes and only for that key, content type and size. See media upload. - Upload Service → Post/Comment/Story Metadata (Cassandra): records an upload session with
status = PENDING, so abandoned uploads can be found and cleaned up later. - Upload Service → Clients: returns
{media_id, upload_url}. This hop is ~60 ms end to end. - Clients → Object Storage: the app
PUTs the bytes directly to Object Storage, bypassing the Load Balancer, API Gateway and Upload Service entirely. Large files (Reels) use multipart or resumable chunks of ~5–8 MB, so a dropped connection resumes from the last chunk instead of starting over. - Object Storage → Clients:
200 OKwith an ETag (checksum). The object is now durably stored, replicated or erasure-coded across devices and zones. - Clients → LB + API Gateway:
POST /v1/posts {media_id, caption, location, user_tags}with the headerIdempotency-Key: 7c9e…. - API Gateway → Upload Service: routed. If the Upload Service has already seen this key (because the client retried after a timeout), it returns the original response instead of creating a second post. See API design.
- Upload Service → Object Storage: a
HEAD raw/{media_id}confirms the object exists and its size and hash match, so a client cannot create a post pointing at nothing. - Upload Service → Post/Comment/Story Metadata (Cassandra): inserts the post (
post_id, user_id, media_key, caption, status = PROCESSING, created_at) intoposts_by_idandposts_by_user. The post count on the profile is incremented in Feed Cache (Redis) counters. - Upload Service → Kafka: publishes
post.created {post_id, user_id, media_key, created_at}, keyed byuser_idso that all of one user's events land in order on one partition (see Kafka). To guarantee the event is never lost when the DB write succeeds, use the transactional outbox pattern: write the event to an outbox table in the same write, and a relay publishes it. - Upload Service → Clients:
201 Created {post_id}. The app already shows the post on the user's profile from the local file. The user is done.
4.2 Phase B — asynchronous media processing
Kafka Media Processing Object Post Meta CDN
Workers Storage (Cassandra)
| | | | |
| 15 consume post.created | | |
|~~~~~~~~~~~~>| | | |
| | 16 GET original | |
| |------------->| | |
| |-- 17 renditions, WebP/AVIF, moderation
| | 18 PUT renditions | |
| |------------->| | |
| | 19 status=READY + keys | |
| |------------------------->| |
| 20 post.ready | | |
|<~~~~~~~~~~~~| | | |
| | 21 (optional) pre-warm hot PoPs |
| |------------------------------------>|
| | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Kafka → Media Processing Workers: a worker in the consumer group takes
post.createdfrom its partition. - Media Processing Workers → Object Storage: downloads the original from
raw/. - Media Processing Workers (local): generates renditions (e.g. 150 px thumbnail, 320, 640 and 1080 px, in JPEG plus WebP/AVIF), a tiny blur placeholder, strips EXIF location metadata for privacy, and runs moderation classifiers (nudity, violence, spam) plus perceptual hashing against known-bad content. Videos are transcoded into HLS/DASH segments at several bitrates. See media processing.
- Media Processing Workers → Object Storage: writes each rendition under a content-versioned key such as
p/{media_id}/{hash}_1080.webp, so the CDN can cache them forever. - Media Processing Workers → Post/Comment/Story Metadata (Cassandra): updates the post to
status = READYwith the rendition keys, width, height and blur hash. If moderation rejected the content, the status becomesREMOVEDand the flow stops here. - Media Processing Workers → Kafka: publishes
post.ready. Followers are only told about posts whose media can actually be displayed. - Media Processing Workers → CDN (optional): for accounts with huge followings, the worker pre-warms the new renditions in major PoPs to avoid a thundering herd on the origin when millions of people open the post at once.
4.3 Phase C — asynchronous distribution
Kafka Fan-out Social Graph Feed Cache Search Search Index Notification Analytics
Workers store (Redis) Indexer (ES) Service & ML
| | | | | | | |
| 22 consume post.ready | | | | |
|~~~~~~~~>| | | | | | |
| | 23 page through followers | | | |
| |---------->| | | | | |
| | 24 ZADD post_id to each timeline | | | |
| |------------------------>| | | | |
| 25 consume post.ready | | | | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | |
| | | | | 26 index caption, hashtags |
| | | | |---------->| | |
| 27 consume (tags, mentions) | | | | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| |
| 28 events to warehouse | | | | |
|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>|
| | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Kafka → Feed Fan-out Workers: consume
post.ready. - Feed Fan-out Workers → Social Graph store: page through the author's followers (e.g. 1,000 per page). If the author is a celebrity (say more than ~1M followers) the workers skip push fan-out entirely; their posts are pulled at read time in Flow 3. See news feed and social graph.
- Feed Fan-out Workers → Feed Cache (Redis): for each follower,
ZADD timeline:{follower_id} {created_at} {post_id}and trim to the newest ~500 entries. Pipelining batches thousands of writes per round trip. Inactive followers (no app open in 30 days) can be skipped; their feed is rebuilt on demand. - Kafka → Search Indexer: a separate consumer group reads the same event independently. That is the power of a log: one event, many readers.
- Search Indexer → Search Index (Elasticsearch): indexes the caption, hashtags and location so the post shows up under
#sunsetwithin a second or two. See search. - Kafka → Notification Service: sends notifications to users tagged or mentioned in the caption, and to followers who turned on "post notifications" for this author (Flow 9).
- Kafka → Analytics & ML Training: the event lands in the data warehouse and feature stores. It's used for metrics, abuse detection and training the feed and Explore ranking models. See analytics & ML.
4.4 Latency budget
| Segment | Assumed time | Who waits? |
|---|---|---|
| Init upload (RTT + ID + URL signing + PENDING row) | ~60 ms | User (behind a progress bar) |
| PUT 500 KB at 5 Mbps uplink | ~0.8 s | User (progress bar; the app can start this while the caption is typed) |
| Create post (RTT + HEAD + insert + publish) | ~60 ms | User |
| Image renditions + moderation | 2–5 s (minutes for long video) | Nobody; post visible to followers after this |
| Fan-out to 500 followers | < 100 ms of Redis writes; ~1 s including queueing | Nobody |
| Fan-out to 200k followers | seconds to tens of seconds | Nobody (this is why celebrities use pull instead) |
💡 Hiding the upload entirely
A widely used trick: start uploading the media as soon as the user picks the photo, while they're still choosing a filter and typing the caption. By the time they tap Share, only the ~60 ms "create post" call is left. If they abandon, an Object Storage lifecycle rule deletes the orphaned
raw/ object after 24 h.4.5 What happens on failure
- The network drops mid-upload: resumable/multipart upload continues from the last acknowledged chunk. If the pre-signed URL expires, the app asks for a new one with the same
media_id. - The client times out on "create post" and retries: the idempotency key makes the retry return the same
post_id, so there is no duplicate post. - Kafka is unavailable: with the outbox pattern the post row and the pending event are committed together, and a relay publishes it when Kafka recovers. Followers see the post late but never miss it. Without an outbox you get "posted but never appeared in feeds", a classic dual-write bug.
- A media worker crashes halfway: it never committed its Kafka offset, so another worker gets the same event (at-least-once delivery). Renditions are written to deterministic keys, so redoing the work simply overwrites identical files. The operation is idempotent.
- A poison message (e.g. a corrupt file that crashes the decoder) is retried a few times and then moved to a dead-letter queue; the post is marked
FAILEDand the user sees "Couldn't post, try again". - Fan-out lags behind (consumer lag during a spike): posting still succeeds. Followers' feeds are a little stale, autoscaling adds workers, and Flow 3's read path can pull very recent posts directly to fill the gap.
5. Flow 3 — Loading the home feed
This is the most frequent request in the system (assume ~5 billion feed loads a day, around 60k requests per second on average and 150k at peak). The goal is a p99 under ~300 ms server time. That is only possible because most of the work, deciding which posts are candidates, was done at write time in Flow 2.
Clients LB + News Feed Feed Cache Follow/ Post Meta CDN
API Gateway Service (Redis) Graph (Cassandra)
| | | | | | |
|-- 1 render cached feed from disk | | | |
| 2 GET /v1/feed?cursor=... | | | |
|---------->| | | | | |
| | 3 JWT ok, route | | | |
| |----------->| | | | |
| | | 4 ZREVRANGE timeline:{uid} | |
| | |---------->| | | |
| | | 5 followed celebrities? | |
| | |---------------------->| | |
| | | 6 recent posts of celebs (pull) | |
| | |---------------------------------->| |
| | |-- 7 merge, filter, rank ~500 -> 20| |
| | | 8 MGET post objects + counters | |
| | |---------->| | | |
| | | 9 fill cache misses | | |
| | |---------------------------------->| |
| 10 20 posts + signed CDN URLs + cursor | | |
|<.......................| | | | |
| 11 GET images (prefetch next) | | | |
|---------------------------------------------------------------------->|
| 12 bytes (hit) / origin pull (miss)| | | |
|<......................................................................|
| 13 impression batch -> Kafka | | | |
|~~~~~~~~~~>| | | | | |
| | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients (local): the app immediately renders the feed it saved on disk last time, so the user sees content in under 100 ms even on a bad network. It then asks the server for fresh content (stale-while-revalidate; see clients).
- Clients → LB + API Gateway:
GET /v1/feed?cursor=<opaque>&limit=20with the JWT. The cursor encodes where the previous page ended (cursor pagination, never offsets). - API Gateway → News Feed Service: the JWT signature and expiry are verified locally, the user is under the rate limit, and the request is routed to any News Feed Service instance. The service is stateless (see microservices).
- News Feed Service → Feed Cache (Redis):
ZREVRANGE timeline:{uid} 0 499returns up to 500 post IDs that the fan-out workers pushed into this user's timeline. It's a single in-memory read of ~1 ms. See caching. - News Feed Service → Follow/Graph: "which celebrities does this user follow?" These are the accounts excluded from push fan-out. The answer is small, and cached per user. In parallel with step 4.
- News Feed Service → Post/Comment/Story Metadata (Cassandra): fetches the last ~48 h of posts from each followed celebrity (
posts_by_useris partitioned by author and clustered by time, so each fetch is one partition read; celebrities' recent posts are also heavily cached). This is the pull half of the hybrid model. See news feed. - News Feed Service (local): merges the pushed and pulled candidates, removes posts that are deleted, blocked, muted or already seen, then ranks. A cheap first-stage model trims ~500 candidates to ~100, then a heavier model scores those using features like your affinity with the author, the post's early engagement and media type. It also inserts a few suggested posts. See analytics & ML ranking.
- News Feed Service → Feed Cache (Redis): hydrates the top 20. A multi-get fetches cached post objects (author, caption, rendition keys) and the like/comment counters.
- News Feed Service → Post/Comment/Story Metadata (Cassandra): fills any cache misses and writes them back to the cache (cache-aside).
- News Feed Service → Clients: JSON with 20 posts, each containing signed CDN URLs for several rendition sizes (never the bytes), plus
next_cursor. Typical size ~30–60 KB, gzip/brotli compressed. - Clients → CDN: the app picks the rendition for its screen and format support and fetches images in parallel. It prefetches the next few posts' media and the next page while the user scrolls.
- CDN → Clients: served from edge RAM/SSD on a hit (~10–30 ms). On a miss the edge goes through the origin shield to Object Storage. See CDN.
- Clients → Kafka (via the gateway): the app batches impression and view-time events and sends them every few seconds. They flow into Analytics & ML Training, which records what was seen (so it isn't shown again) and produces training labels for the ranking model.
Ranking is the largest single server-side cost; the network and CDN dominate what the user feels. The timeline read and the celebrity pull run in parallel, so the slower of the two counts.
5.1 Latency budget
| Stage | p50 (assumed) | p99 target | How it's kept low |
|---|---|---|---|
| Network to region and back | 40 ms | 150 ms | Nearest region via GeoDNS, warm HTTP/2 or QUIC connection |
| Gateway (auth, rate limit, route) | 3 ms | 10 ms | Local JWT validation, in-memory token buckets |
| Candidate fetch (timeline + celebrity pull, parallel) | 18 ms | 50 ms | Precomputed timeline in Redis; celebrity posts cached |
| Ranking | 50 ms | 120 ms | Two-stage ranking, features prefetched, hard deadline with fallback |
| Hydration (post objects + counters) | 12 ms | 40 ms | Batched multi-gets; cache-aside with high hit ratio |
| Server total | ~85 ms | < 300 ms | Every downstream call has a deadline shorter than the parent's |
| First images from CDN | 30 ms | 200 ms | Edge hits, right-sized renditions, prefetching |
5.2 What happens on failure
- Timeline missing from Redis (new user, evicted inactive user, or a lost cache node): fall back to fan-out-on-read. Fetch the user's followees from the Social Graph store, read each one's recent posts from Cassandra, merge, then write the result back into Redis. This is slower (hundreds of ms) but correct, and it only happens once.
- The ranking service is slow or down: after its deadline (say 80 ms) the News Feed Service degrades gracefully to a simple reverse-chronological order of the candidates. A slightly worse feed beats an error screen. See resilience patterns.
- Some Cassandra replicas are slow: hedged requests or a lower consistency level (e.g.
ONE) keep reads fast. Posts that can't be hydrated in time are dropped from this page rather than blocking it. - A cache node holding a celebrity's post gets hot: replicate hot keys across nodes or keep a small in-process cache in each News Feed Service instance (see caching on hot keys).
- The whole feed backend is unavailable: the app keeps showing its cached feed with a "Couldn't refresh feed" banner and retries with backoff. A circuit breaker at the gateway stops retries from overwhelming a recovering service.
- CDN edge miss storm or PoP failure: the origin shield collapses requests. Anycast or DNS moves users to the next PoP. Images load more slowly but still load.
6. Flow 4 — Liking a post
Likes are the highest-volume write in Instagram (assume ~5 billion a day, ~60k/s on average, with heavy spikes on viral posts). Each one is tiny, but together they create two classic problems: hot keys (millions of people liking the same celebrity post) and counting at scale. The user, meanwhile, expects the heart to turn red instantly.
Clients LB + Like & Post Meta Feed Cache Kafka Notification Analytics
API Gateway Comment (Cassandra) (Redis) Service & ML
| | | | | | | |
|-- 1 heart turns red, count+1 (optimistic) | | | |
| 2 PUT /v1/posts/{id}/like | | | | |
|---------->| | | | | | |
| | 3 route | | | | | |
| |---------->| | | | | |
| | | 4 upsert likes_by_post | | | |
| | |---------->| | | | |
| | | 5 was it new? | | | |
| | |<..........| | | | |
| | | 6 INCR like_count:{post}:{shard} | | |
| | |----------------------->| | | |
| | | 7 like.created | | | |
| | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | |
| 8 200 OK | | | | | | |
|<......................| | | | | |
| | | | | | 9 aggregate, push |
| | | | | |~~~~~~~~~~>| |
| | | | | | 10 ranking signal |
| | | | | |~~~~~~~~~~~~~~~~~~~~~~~~>|
| | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients (local): the heart turns red and the count goes up by one immediately. This is optimistic UI: the app assumes success and reconciles later (see clients).
- Clients → LB + API Gateway:
PUT /v1/posts/{post_id}/like.PUTrather thanPOSTbecause liking is naturally idempotent: "make it so that I like this post". - API Gateway → Like & Comment: authenticated and rate limited (bots that like 1,000 posts a minute get throttled), then routed.
- Like & Comment → Post/Comment/Story Metadata (Cassandra): upserts a row into
likes_by_post(partition keypost_id, clustering keyuser_id), plus a mirror row inlikes_by_userfor "posts you've liked". The primary key(post_id, user_id)means a repeated like overwrites the same row, so the write is idempotent. See likes & comments. - Post/Comment/Story Metadata → Like & Comment: the service needs to know whether this like is new before touching the counter. It can do a cheap read-before-write, or a set-add in Redis that returns 1 for new and 0 for existing.
- Like & Comment → Feed Cache (Redis): if new,
INCR like_count:{post_id}:{shard}, whereshardis random in 0..N−1 for hot posts. Splitting one hot counter into N sub-counters spreads the writes across N keys and nodes; reads sum them. A periodic job flushes counts durably into Cassandra. - Like & Comment → Kafka: publishes
like.created {post_id, liker_id, owner_id}keyed bypost_id. - Like & Comment → Clients:
200 OK. Total server time ~10–20 ms. - Kafka → Notification Service: aggregates likes rather than sending one push per like. You get "alice and 23 others liked your photo", sent at most once every few minutes per post, then delivered through Flow 9.
- Kafka → Analytics & ML Training: likes are among the strongest signals for feed and Explore ranking, and early like velocity is a ranking feature for the post itself.
6.1 Latency budget
| Stage | Assumed time |
|---|---|
| UI feedback | 0 ms (optimistic) |
| Network RTT | 40 ms |
| Gateway | 3 ms |
| Cassandra upsert (quorum write across replicas in one region) | 3–8 ms |
| Redis INCR + Kafka publish (async, acks from the leader) | 2–5 ms |
| Request complete | ~55 ms, invisible to the user |
| Post owner gets a push | seconds to minutes (aggregation window) |
6.2 What happens on failure
- The request fails or times out: the app retries in the background with exponential backoff. Because the operation is idempotent, a retry after a "lost" success does no harm. If it finally gives up, the heart quietly turns back to grey.
- Double-tap, like, unlike, like in one second: the client debounces and sends only the final state. The server applies requests in order per user, and each request carries a client timestamp so a stale "unlike" can't overwrite a newer "like".
- Redis counter node lost: counts are approximate anyway. They can be rebuilt from the durable Cassandra totals plus a replay of recent
like.createdevents. Users may briefly see a count that's slightly off, which is an acceptable eventual-consistency trade (see consistency models). - Viral post hot partition: the
likes_by_postpartition for one post can grow to tens of millions of rows. Bucket it as(post_id, bucket)so no single Cassandra partition grows unbounded (see sharding on hot spots). - Notification Service lagging: likes still succeed. Notifications arrive late, which nobody notices.
7. Flow 5 — Following a user
Alice taps Follow on Bob's profile. The source of truth is a single relationship, but it must be stored twice, once on each user's side, because Instagram needs both "whom does Alice follow?" (to build Alice's feed) and "who follows Bob?" (to fan out Bob's posts). Those two lists live on different shards.
Clients LB + Follow/ User DB Social Graph Feed Cache Kafka Fan-out Notification
API Gateway Graph (PostgreSQL) store (Redis) Workers Service
| | | | | | | | |
|-- 1 button shows Following (optimistic) | | | | |
| 2 POST /v1/users/{bob}/follow | | | | | |
|---------->| | | | | | | |
| | 3 rate limit, route | | | | | |
| |---------->| | | | | | |
| | | 4 bob private? blocked? | | | | |
| | |---------->| | | | | |
| | | 5 following(alice -> bob)| | | | |
| | |------------------------->| | | | |
| | | 6 followers(bob <- alice)| | | | |
| | |------------------------->| | | | |
| | | 7 INCR follower counts | | | | |
| | |--------------------------------------->| | | |
| | | 8 follow.created | | | | |
| | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | |
| 9 200 {state: following} | | | | | |
|<......................| | | | | | |
| | | | | | | 10 consume |
| | | | | | |~~~~~~~~>| |
| | | | | | 11 backfill bob's posts |
| | | | | |<-------------------| |
| | | | | | | 12 'alice followed you'
| | | | | | |~~~~~~~~~~~~~~~~~~~~>|
| | | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients (local): the button changes to Following optimistically, or Requested if the app already knows the account is private.
- Clients → LB + API Gateway:
POST /v1/users/{bob_id}/follow. - API Gateway → Follow/Graph: authenticated and rate limited. Follow limits (e.g. 200 per hour) are a key anti-spam measure. The gateway then routes the request.
- Follow/Graph → User DB (PostgreSQL): is Bob's account private? Has Bob blocked Alice? These settings are read through a cache. A private account creates a
follow_requestinstead of an edge, and the flow ends with{state: requested}. - Follow/Graph → Social Graph store: writes the forward edge
following(alice_id → bob_id, created_at)on Alice's shard. - Follow/Graph → Social Graph store: writes the reverse edge
followers(bob_id ← alice_id, created_at)on Bob's shard. These are two writes on two shards with no distributed transaction. See the failure section for how they're kept consistent, and social graph for the data model. - Follow/Graph → Feed Cache (Redis): increments Alice's
following_countand Bob'sfollower_count. - Follow/Graph → Kafka: publishes
follow.created {follower: alice, followee: bob}. - Follow/Graph → Clients:
200 {state: "following"}. - Kafka → Feed Fan-out Workers: consume the follow event.
- Feed Fan-out Workers → Feed Cache (Redis): backfill. Bob's last ~10–20 posts are inserted into Alice's timeline, so his content appears the next time she pulls to refresh, instead of only after his next post. Celebrities are skipped because they're pulled at read time anyway.
- Kafka → Notification Service: "alice started following you" to Bob (Flow 9). Kafka → Analytics & ML Training also consumes the event: new edges feed "Suggested for you" and affinity features.
Unfollow is the mirror image: delete both edges, decrement counters, publish follow.deleted. Rather than deleting Bob's posts from Alice's Redis timeline eagerly, the News Feed Service filters them out at read time because Bob is no longer in her following set, and they age out of the timeline naturally.
7.1 What happens on failure
- Forward edge written, reverse edge write fails: the graph is now inconsistent: Alice "follows" Bob, but Bob's follower list doesn't include her, so she won't receive his posts. Fixes, from simplest to most robust: (a) retry the second write with backoff; (b) write only the forward edge synchronously and let a Kafka consumer of
follow.createdwrite the reverse edge idempotently, which gives at-least-once delivery and eventual consistency; (c) run a periodic repair job that compares the two indexes. A saga mindset ("every step has a retry or a compensation") is the right frame. - Double tap or retries: edge writes are upserts keyed on
(follower, followee), so they're idempotent. Counters are only incremented when the edge was actually new. - Backfill lags: Alice doesn't see Bob's old posts for a few seconds. That's harmless.
- Follow/unfollow spam bots: rate limits at the gateway and abuse models in Analytics & ML Training throttle or block them (see security).
8. Flow 6 — Posting and viewing a story
Stories look like posts, but they behave differently in three ways. They expire after 24 hours, so every storage layer needs a TTL. They're consumed as a tray of accounts at the top of the feed rather than as individual items. And they're mostly video, so playback must start instantly. See stories & TTL data.
8.1 Posting a story
Clients LB + Stories Object Story Meta Kafka Media Proc. Fan-out Feed Cache
API Gateway Service Storage (Cassandra) Workers Workers (Redis)
| | | | | | | | |
| 1 POST /v1/stories/uploads | | | | | |
|---------->| | | | | | | |
| | 2 route | | | | | | |
| |---------->| | | | | | |
| 3 pre-signed PUT URL | | | | | | |
|<......................| | | | | | |
| 4 PUT photo/video | | | | | | |
|-------------------------------->| | | | | |
| 5 POST /v1/stories (via LB+GW) | | | | | |
|---------------------->| | | | | | |
| | | 6 INSERT ... USING TTL 86400 | | | |
| | |-------------------->| | | | |
| | | 7 story.created | | | | |
| | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | |
| 8 201 (your ring updates) | | | | | |
|<......................| | | | | | |
| | | | | | 9 transcode HLS, thumbnails |
| | | | | |~~~~~~~~~~>| | |
| | | | 10 PUT segments | | | |
| | | |<---------------------------------| | |
| | | | | | 11 consume| | |
| | | | | |~~~~~~~~~~~~~~~~~~~~~~>| |
| | | | | | | | 12 ZADD author to trays
| | | | | | | |--------->|
| | | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients → LB + API Gateway:
POST /v1/stories/uploadsto request an upload slot. - API Gateway → Stories Service: authenticated, routed.
- Stories Service → Clients: a new
story_idand a pre-signed PUT URL, exactly as in Flow 2 (see media upload). - Clients → Object Storage: the photo or the 15-second video clip is uploaded directly. The object goes under a
stories/prefix that has a lifecycle rule to delete objects after ~24–48 h unless they were archived. - Clients → Stories Service (via LB + API Gateway):
POST /v1/stories {story_id, stickers, close_friends_only}. - Stories Service → Post/Comment/Story Metadata (Cassandra):
INSERT INTO stories_by_user (...) USING TTL 86400. Cassandra deletes the row automatically after 24 h. No cron job is needed, though tombstone compaction must be tuned (see LSM trees). - Stories Service → Kafka:
story.created. - Stories Service → Clients:
201. Your own avatar gets the coloured ring immediately. - Kafka → Media Processing Workers: transcode the video into short HLS segments at 2–4 bitrates and produce a thumbnail. Photos get renditions as in Flow 2.
- Media Processing Workers → Object Storage: write the segments and manifests, then mark the story as ready.
- Kafka → Feed Fan-out Workers: consume
story.createdonce the media is ready. - Feed Fan-out Workers → Feed Cache (Redis): for each (non-celebrity) author,
ZADD tray:{follower_id} {ts} {author_id}with a key TTL just over 24 h. Note that the tray stores authors, not individual stories: posting 10 stories updates the tray once.
8.2 Viewing the stories tray
Clients LB + Stories Feed Cache Follow/ Story Meta CDN Kafka
API Gateway Service (Redis) Graph (Cassandra)
| | | | | | | |
| 13 GET /v1/stories/tray | | | | |
|---------->| | | | | | |
| | 14 route | | | | | |
| |---------->| | | | | |
| | | 15 tray:{uid} (pushed authors) | | |
| | |--------->| | | | |
| | | 16 followed celebs with live story | |
| | |--------------------->| | | |
| | | 17 live items per author | | |
| | |--------------------------------->| | |
| | |-- 18 drop expired, rank by closeness | |
| 19 tray + signed CDN URLs | | | | |
|<......................| | | | | |
| 20 prefetch first segments | | | | |
|-------------------------------------------------------------------->| |
| 21 POST /v1/stories/seen (batched) | | | |
|---------------------->| | | | | |
| | | 22 story.seen | | | |
| | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>|
| | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients → LB + API Gateway:
GET /v1/stories/tray, fired in parallel with the feed request on app open. - API Gateway → Stories Service: routed.
- Stories Service → Feed Cache (Redis): reads
tray:{uid}, the authors pushed by fan-out. - Stories Service → Follow/Graph: followed celebrities whose stories weren't pushed (pull side of the hybrid), filtered by a cached "has active story" flag.
- Stories Service → Post/Comment/Story Metadata (Cassandra): live story items for the first ~10 authors (rows past their TTL simply don't come back), plus the viewer's seen state.
- Stories Service (local): drops expired or fully-seen authors, then ranks the tray with unseen first, ordered by closeness (how often you interact with them).
- Stories Service → Clients: the ordered tray with signed CDN URLs for the first items.
- Clients → CDN: the app prefetches the first segment of the first few stories, so tapping an avatar starts playback in under ~100 ms.
- Clients → Stories Service: as the user watches, seen events are batched (e.g. every 5 s) into
POST /v1/stories/seen. - Stories Service → Kafka:
story.seen. Consumers append the viewer to the author's "seen by" list, which is also stored with a TTL, and feed Analytics & ML Training.
8.3 Latency budget & failure handling
| Stage | Assumed time | If it fails… |
|---|---|---|
| Tray request (server) | ~40 ms | Show the tray cached on the device. The tray is non-critical, so the feed still loads. |
| First segment from CDN (prefetched) | ~0–30 ms | Miss → origin pull (~150 ms). Player shows the thumbnail meanwhile. |
| Adaptive bitrate switches | per segment (~2 s) | Bad network → player drops to a lower bitrate rather than stalling. |
| Seen events | async, batched | Lost batch → the story may show as unseen again. Retried with idempotent (viewer, story) keys. |
| Expiry | exactly 24 h | TTLs in Cassandra and Redis plus lifecycle rules in Object Storage. The API also filters created_at < now − 24h as a safety net, so a late deletion is never visible. |
9. Flow 7 — Search with typeahead
The user types s, se, sel… and suggestions (accounts, hashtags, places) must appear as they type. Every keystroke is potentially a request, so typeahead traffic is many times larger than "real" search traffic, and the latency target is aggressive: under ~100 ms end to end, or the suggestions lag behind the fingers. See search & inverted indexes.
Clients LB + Search Feed Cache Follow/ Search Index CDN
API Gateway Service (Redis) Graph (Elasticsearch)
| | | | | | |
|-- 1 debounce ~100 ms, show local recents | | |
| 2 GET /v1/search/typeahead?q=sel | | | |
|---------->| | | | | |
| | 3 rate limit, route | | | |
| |---------->| | | | |
| | | 4 prefix cache 'sel'?| | |
| | |--------->| | | |
| | | 5 miss | | | |
| | |<.........| | | |
| | | 6 completion / edge-ngram query | |
| | |----------------------------------->| |
| | | 7 my followees & affinities | |
| | |--------------------->| | |
| | |-- 8 blend + personal boost, top 10 | |
| | | 9 SET prefix cache (TTL) | |
| | |--------->| | | |
| 10 results + avatar URLs | | | |
|<......................| | | | |
| 11 GET avatars | | | | |
|------------------------------------------------------------------------>|
| | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Clients (local): the app shows recent searches and a client-side cache of earlier results instantly, and debounces: it waits ~100 ms after the last keystroke before sending, so typing "selena" quickly sends one or two requests instead of six.
- Clients → LB + API Gateway:
GET /v1/search/typeahead?q=sel&seq=17. Theseqlets the app discard responses that arrive out of order. - API Gateway → Search Service: rate limited per user, since typeahead is an easy target for scraping, then routed.
- Search Service → Feed Cache (Redis): checks a prefix result cache for
sel. Short, popular prefixes have a very high hit ratio because millions of people type the same first letters. - Feed Cache → Search Service: miss in this example.
- Search Service → Search Index (Elasticsearch): a prefix query against fields indexed with edge n-grams or a completion suggester (a trie-like structure in memory), across the users, hashtags and places indexes. Results are pre-sorted by global popularity (follower count, hashtag volume).
- Search Service → Follow/Graph: fetches the user's followees and top interaction partners (cached) for personalisation, in parallel with step 6.
- Search Service (local): blends the results and boosts accounts you follow or talk to. "sel" should show your friend selin_k above a global celebrity if you message her daily. Keeps the top ~10.
- Search Service → Feed Cache (Redis): stores the global (non-personalised) part under the prefix key with a short TTL (minutes). Personalisation is re-applied on every request.
- Search Service → Clients: the results, with usernames, verified badges and avatar CDN URLs.
- Clients → CDN: avatars, tiny 150 px renditions that are almost always edge hits.
How results get into the index is the asynchronous half. When a user signs up, changes their username or posts with a new hashtag, an event flows like this:
User/Auth Serviceusername changed
→user.updated
Kafka
→consume
Search Indexerbuild document
→bulk index
Search IndexElasticsearch, ~1 s refresh
Elasticsearch makes new documents searchable after its refresh interval (~1 s by default), so search is near-real-time and eventually consistent with the source-of-truth databases.
9.1 Latency budget
| Stage | Assumed time |
|---|---|
| Debounce (client, not counted as server time) | ~100 ms after last keystroke |
| Network RTT | 40 ms |
| Gateway | 3 ms |
| Prefix cache hit / Elasticsearch on miss | 1 ms / 15–40 ms |
| Personalisation (parallel) + blend | 5 ms |
| Total per keystroke | ~50 ms on hit, ~90 ms on miss |
9.2 What happens on failure
- Elasticsearch slow or a shard down: a query fans out to all shards, so it's as slow as the slowest one. The Search Service sets a tight timeout and returns partial results from the shards that answered. The user rarely notices a slightly incomplete suggestion list.
- The whole index is unavailable: serve from the prefix cache and the client's recent searches only. Full search shows "Try again later". Search is non-critical, so the rest of the app is unaffected (a bulkhead).
- Indexer lag: a renamed account is unfindable by its new name for a few seconds or minutes. Acceptable eventual consistency, and recovery is automatic as the consumer catches up.
- Out-of-order responses: the response for "se" arrives after the one for "sel". The client compares
seqand ignores stale responses.
10. Flow 8 — Sending a direct message
DMs are different from everything above. They need real-time push from server to client, strict ordering within a conversation, exactly-once display despite retries, and delivery receipts. The connection model therefore changes from request/response HTTP to a long-lived WebSocket (or MQTT, which public talks describe Meta using for its messaging apps). See direct messaging and API design.
10.1 Recipient online
Alice LB + Direct Messages Redis Bob
(Clients) API Gateway Messaging Store (presence) (Clients)
| | | | | |
| 1 WebSocket upgrade (JWT) | | |
|----------->| | | | |
| | 2 pinned to a connection server | |
| |----------->| | | |
| | | 3 SET online:alice = host-3 |
| | |--------------------->| |
| 4 send {client_msg_id, thread, text} | |
|------------------------>| | | |
| | |-- 5 authz + assign per-thread seq |
| | | 6 INSERT (thread_id, seq, ...) |
| | |--------->| | |
| 7 ack seq=42 (sent) | | | |
|<........................| | | |
| | | 8 where is bob? | |
| | |--------------------->| |
| | | 9 host-17| | |
| | |<.....................| |
| | | 10 push over bob's WebSocket |
| | |---------------------------------->|
| | | 11 delivered ack | |
| | |<..................................|
| 12 delivered receipt | | | |
|<........................| | | |
| | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Alice's Clients → LB + API Gateway: when the app opens it upgrades an HTTPS connection to a WebSocket, authenticating once with the JWT. The connection then stays open for the whole session, with heartbeats every ~30 s.
- LB → Direct Messaging: the connection is pinned to one connection server (a "chat gateway" host inside the Direct Messaging tier). The connection is stateful, so this is one of the few places where sticky routing matters.
- Direct Messaging → Redis (presence):
SET online:alice host-3 EX 60, refreshed by heartbeats. This presence and connection registry is a Redis cluster like the Feed Cache, and tells the system which server holds a user's socket. - Alice → Direct Messaging: sends
{client_msg_id: uuid, thread_id, text: "hey!"}over the socket. Theclient_msg_idis the idempotency key for retries. - Direct Messaging (local): checks Alice is a member of the thread and not blocked, deduplicates on
client_msg_id, and assigns the next per-thread sequence number (e.g. 42). Ordering per conversation is what matters; global ordering is unnecessary and expensive. - Direct Messaging → Messages Store:
INSERTinto a table partitioned bythread_idand clustered byseq. A conversation is one partition, read sequentially, which suits Cassandra or HBase. - Direct Messaging → Alice: ack with
seq = 42. Alice's app now shows the message as sent (one tick). The message is durable before it's acknowledged. - Direct Messaging → Redis (presence): where is Bob connected?
- Redis → Direct Messaging:
host-17. - Direct Messaging → Bob's Clients: host-3 forwards the message to host-17 (via internal RPC or a pub/sub channel), which pushes it down Bob's open WebSocket.
- Bob's Clients → Direct Messaging: a delivered ack for seq 42. The app also records that it has everything up to seq 42 for this thread.
- Direct Messaging → Alice: a delivered receipt. When Bob opens the thread, a read receipt follows the same path.
10.2 Recipient offline
Alice Direct Messages Redis Kafka Notification Push Providers Bob
(Clients) Messaging Store (presence) Service APNs/FCM (Clients)
| | | | | | | |
| 1 send message (WebSocket) | | | | |
|---------->| | | | | | |
| | 2 INSERT (thread_id, seq) | | | |
| |--------->| | | | | |
| 3 sent | | | | | | |
|<..........| | | | | | |
| | 4 where is bob? | | | | |
| |--------------------->| | | | |
| | 5 no live connection | | | | |
| |<.....................| | | | |
| | 6 message.created (offline) | | | |
| |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | |
| | | | | 7 consume | | |
| | | | |~~~~~~~~~~>| | |
| | | | | | 8 push 'Alice: hey!' |
| | | | | |-------------->| |
| | | | | | | 9 OS shows banner
| | | | | | |------------->|
| | | | | | | |-- 10 Bob taps
| | 11 reconnect + sync(after seq N)| | | |
| |<---------------------------------------------------------------------------|
| | 12 read thread where seq > N | | | |
| |--------->| | | | | |
| | 13 missed messages | | | | |
| |...........................................................................>|
| 14 delivered receipt | | | | | |
|<..........| | | | | | |
| | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Alice's Clients → Direct Messaging: sends over her WebSocket, exactly as before.
- Direct Messaging → Messages Store: persists with the next sequence number. The store is the mailbox, so nothing depends on Bob being online.
- Direct Messaging → Alice: sent ack.
- Direct Messaging → Redis (presence): where is Bob?
- Redis → Direct Messaging: no key, because Bob's presence entry expired when his heartbeats stopped.
- Direct Messaging → Kafka:
message.created {thread_id, seq, recipient: bob, offline: true}. - Kafka → Notification Service: consumes it, checks Bob's preferences (thread muted?), and collapses bursts ("Alice sent 5 messages").
- Notification Service → Push Providers APNs/FCM: sends a push with a short preview (or just "New message" if previews are disabled). Details in Flow 9.
- Push Providers APNs/FCM → Bob's Clients: the phone OS displays the banner even though the app isn't running.
- Bob's Clients (local): Bob taps the notification.
- Bob's Clients → Direct Messaging: the app opens a WebSocket and sends
sync {thread_id, after_seq: N}for each thread, where N is the last seq it has stored locally. This is the key idea: the client pulls the gap, so delivery stays correct even if pushes are lost or arrive out of order. - Direct Messaging → Messages Store:
SELECT ... WHERE thread_id = ? AND seq > N, a single-partition range scan. - Direct Messaging → Bob's Clients: the missed messages, in order.
- Direct Messaging → Alice: delivered receipts flow back to Alice, if she's online.
10.3 Latency budget & failure handling
| Stage | Assumed time | Failure behaviour |
|---|---|---|
| Alice → server → durable write → ack | ~50 ms (RTT + ~5 ms write) | No ack within ~5 s → client resends the same client_msg_id; the server dedupes. Unsent messages show a red "!" with retry. |
| Server → Bob (online) | ~25 ms extra (cross-host hop + half RTT) | Bob's socket died silently → no delivered ack → the message stays in the store and is fetched on reconnect via sync. |
| End-to-end online delivery | ~100 ms | Connection server crashes → all its clients reconnect (with jittered backoff to avoid a thundering herd) to other hosts and resync. |
| Offline push | ~1–3 s (APNs/FCM are best-effort) | Push lost → no data lost; Bob still gets everything on the next app open. |
⚠️ Push is a hint, not a transport
A common interview mistake is to treat APNs/FCM as the delivery channel. Push providers may delay, collapse or drop notifications. The Messages Store plus sync-from-last-seq is the real delivery guarantee; push only wakes the user up.
11. Flow 9 — Notification delivery
Flows 2, 4, 5 and 8 all end with "→ Notification Service". This flow follows a notification from the event to a banner on the lock screen. Notifications sound trivial, but at Instagram scale (assume billions per day) the hard parts are not annoying people (aggregation, rate caps, preferences), handling dead device tokens, and coping with third-party providers you don't control. See notifications.
Kafka Notification User DB Feed Cache Post Meta Push Providers Clients Analytics
Service (PostgreSQL) (Redis) (activity) APNs/FCM & ML
| | | | | | | |
| 1 like/comment/follow/message events | | | | |
|~~~~~~~~~~>| | | | | | |
| |-- 2 drop self-actions, blocked, muted | | | |
| | 3 prefs + device tokens | | | | |
| |------------->| | | | | |
| | 4 dedupe/aggregate, rate cap | | | |
| |--------------------------->| | | | |
| | 5 append to activity feed | | | | |
| |---------------------------------------->| | | |
| | 6 send (HTTP/2, token auth)| | | | |
| |------------------------------------------------------->| | |
| | 7 200 | 410 gone | 429 | | | | |
| |<.......................................................| | |
| | | | | | 8 deliver to device |
| | | | | |------------>| |
| | | | | | |-- 9 tap -> deep link
| | | | | | | 10 opened events
| | | | | | |~~~~~~~~~>|
| | | | | | | |
legend: ----> synchronous call ....> response ~~~~> asynchronous event (Kafka) -- local work
- Kafka → Notification Service: consumes
like.created,comment.created,follow.created,post.ready(for mentions and post notifications) andmessage.created. Each event type can have its own consumer pool and priority. DMs are urgent; "someone you may know joined" is not. - Notification Service (local): drops events that shouldn't notify anyone: liking your own post, actors the recipient has blocked or muted, or recipients who disabled that category.
- Notification Service → User DB (PostgreSQL): loads the recipient's notification preferences, language and device tokens (one per installed device, registered in Flow 1). These are read through a cache.
- Notification Service → Feed Cache (Redis): dedupe and aggregate. A key like
notif:{owner}:{post}:likescollects likers over a window of a few minutes, so 50 likes become one push ("alice and 49 others…"). A per-user counter enforces a rate cap (e.g. at most N pushes per hour). - Notification Service → Post/Comment/Story Metadata (activity): appends an entry to the recipient's in-app Activity feed (the heart tab). This in-app record is written even when no push is sent.
- Notification Service → Push Providers APNs/FCM: renders the localised text and sends over persistent HTTP/2 connections with token-based auth: APNs for iOS, FCM for Android. Bursts are smoothed with an internal queue.
- Push Providers → Notification Service: the response code matters.
200means accepted.410 Unregisteredor an invalid token means the app was uninstalled, so the token is deleted from the User DB.429or5xxmeans retry with exponential backoff and jitter, and after N attempts move the message to a dead-letter queue. - Push Providers APNs/FCM → Clients: the OS displays the notification. Delivery is best-effort; a phone that's off receives only the latest notification per collapse key when it comes back.
- Clients (local): the user taps, and a deep link (
instagram://media?id=…) opens the right screen, triggering Flow 3-style reads for that post. - Clients → Analytics & ML Training: delivered, opened and dismissed events flow back via Kafka. Models learn which notifications people value, and send fewer of the kind people ignore.
11.1 What happens on failure
- Notification Service is down: events accumulate in Kafka (retention of days), and nothing is lost. On recovery, stale notifications (e.g. likes older than an hour) are skipped rather than sent late; the Activity feed is still backfilled.
- APNs/FCM outage: a circuit breaker stops hammering the provider, messages wait in a retry queue, and the in-app Activity feed still shows everything.
- Duplicate events (at-least-once Kafka): a dedupe key per (recipient, event_id) in Redis with a TTL prevents double pushes.
- Notification storm (a celebrity's post tagging 1M users, or a bug): per-user rate caps and global rate limiting on the sender protect both users and provider quotas.
12. Summary: synchronous vs asynchronous in every flow
Put the nine flows side by side and the pattern is obvious. The synchronous part writes one source-of-truth record and returns; the asynchronous part does the expensive, fan-out-heavy work.
| Flow | Synchronous (the user waits) | Asynchronous (after the response) | Consistency the user sees |
|---|---|---|---|
| 1. App open & login | DNS, TLS, gateway rate limit, password check, session write, tokens | user.logged_in → fraud analytics, "new login" alert | Strong (you're either logged in or not) |
| 2. Upload photo | Pre-signed URL, direct PUT to Object Storage, post row insert, publish event | Renditions, moderation, fan-out to timelines, search indexing, notifications, analytics | Author: read-your-writes. Followers: eventual (seconds). |
| 3. Home feed | Timeline read, celebrity pull, ranking, hydration, CDN image fetch | Impression/view events → ranking training and seen-state | Eventual (may lag new posts by seconds) |
| 4. Like | Upsert like row, counter increment, publish | Aggregated notification, ranking signals, counter flush to durable store | Your like: immediate. Counts: approximate/eventual. |
| 5. Follow | Privacy/block check, edge write(s), counters, publish | Reverse edge (if async), timeline backfill, notification, suggestions | Edge: immediate for you. Feed backfill: eventual. |
| 6. Story post / view | Post: URL, upload, TTL insert. View: tray read, item fetch, CDN prefetch. | Transcoding to HLS, tray fan-out, seen-state, viewer lists, expiry | Eventual for followers. Expiry enforced at read time. |
| 7. Search typeahead | Prefix cache / Elasticsearch query, personalisation, avatars from CDN | Indexing of new users, renames and hashtags via Search Indexer | Near-real-time (~1 s+ index lag) |
| 8. Direct message | Durable write with per-thread seq, ack to sender, push over recipient's WebSocket if online | Offline: Kafka → Notification Service → APNs/FCM. Receipts. | Ordered per thread. At-least-once, deduped to exactly-once display. |
| 9. Notifications | (none on the user's request path) | Filter, preferences, aggregate, rate-cap, activity write, push, token cleanup | Best-effort push. Activity feed is the source of truth. |
13. How Instagram uses these patterns
📸 In Instagram
Public engineering talks and blog posts describe how Instagram's request flows evolved. It started as a Python/Django application tier talking to sharded PostgreSQL, with IDs generated inside PostgreSQL using a 64-bit time + shard + sequence scheme (see ID generation). Later it added Cassandra for high-write data such as feeds, activity and direct messages, Memcache/Redis caches, and async task queues for fan-out and media work. After joining Facebook, it moved onto Meta's data centers, edge network and shared infrastructure. The exact internals are private, but the shape described in this lesson matches what has been made public: a thin synchronous path, direct-to-storage media, a CDN for bytes, precomputed feeds with special handling for celebrity accounts, and heavy use of asynchronous pipelines for everything else.
Three cross-cutting patterns run through all nine flows:
- Event-carried state. Every write publishes a domain event (
post.created,like.created,follow.created…). New features (a new notification type, a new ML signal, a new index) are added as new consumers without touching the write path. That is the main architectural payoff of Kafka. - Optimistic clients plus idempotent servers. The app assumes success (heart turns red, post appears, "Following" shows) and retries in the background. That is only safe because every server operation is idempotent: idempotency keys for posts, natural primary keys for likes and follows,
client_msg_idfor DMs. - Graceful degradation per feature. Each service has a fallback: feed → chronological or cached, search → recents, tray → cached, push → Activity feed. A failure in one feature doesn't take down the app. These are bulkheads in practice.
📸 Rough end-to-end numbers (assumptions)
With 500M daily active users: ~60k feed requests/s on average (~150k peak), ~60k likes/s, ~1.2k photo uploads/s, ~100k typeahead requests/s at peak, tens of millions of concurrent WebSocket connections for DMs, and media egress around 1 Tbps from the CDN. Notice that the async side multiplies these numbers: one upload by an account with 1,000 followers creates 1,000 timeline writes, so fan-out workers perform over a million Redis writes per second. See capacity estimation.
14. Trade-offs & common pitfalls
✅ Benefits of the "thin sync, fat async" shape
- Low, predictable user-facing latency regardless of follower counts or video length
- Failures in background work don't fail the user's request
- Natural load levelling: Kafka absorbs spikes and workers drain at their own pace
- Easy extensibility: add consumers without changing producers
- Media path scales independently through Object Storage and the CDN
⚠️ Costs
- Eventual consistency everywhere: followers, counts, search and notifications lag
- Harder debugging: one user action becomes ~10 asynchronous hops (needs tracing)
- Duplicate delivery means every consumer must be idempotent
- Dual-write risk between DB and Kafka (needs an outbox or CDC)
- Operational weight: Kafka, worker fleets, DLQs, lag monitoring
Common pitfalls when describing flows
- Sending media through the API servers. It ties up app servers on slow uploads and wastes bandwidth twice. Always use pre-signed URLs and the CDN.
- Doing fan-out synchronously in the post request. A user with 2M followers would wait minutes. Fan-out belongs in workers, and celebrities belong on the pull path.
- Forgetting the failure path. For each hop, say what happens if it's slow or down: retry, fallback, degrade, or fail.
- Treating push notifications as reliable delivery. They're hints. Durable stores plus client sync are the guarantee.
- Ignoring idempotency. Mobile networks guarantee retries, so without idempotency keys you get duplicate posts and double-counted likes.
- Publishing the event before (or without) committing the DB write. Consumers then act on data that doesn't exist. Commit first; use an outbox.
- Calling the Auth service on every request. Validate signed tokens at the gateway instead.
- Unbounded timeouts. Each hop needs a deadline smaller than its caller's, or one slow dependency cascades into a site-wide outage.
15. Real-world technologies
| Concern in the flows | Technologies | When to use |
|---|---|---|
| Edge routing | Route 53, NS1, Cloudflare DNS; Anycast | GeoDNS for region choice; Anycast for edge/CDN IPs |
| Load balancer / gateway | Envoy, NGINX, HAProxy, AWS ALB/NLB, Kong, Meta's Katran (L4) and Proxygen (L7) | L4 for raw connection spreading; L7 for auth, routing and rate limiting |
| Service calls | gRPC, Thrift, Finagle | Low-latency internal RPC with deadlines and retries |
| Event bus | Apache Kafka, Pulsar, AWS Kinesis, Google Pub/Sub | Durable, replayable, multi-consumer events |
| Timelines, counters, presence | Redis Cluster, Memcached, Meta's TAO (graph cache) | Sub-ms reads of precomputed data |
| Metadata & messages | Cassandra, ScyllaDB, HBase, DynamoDB | High write throughput, partition-per-entity access |
| Media pipeline | FFmpeg, libvips/ImageMagick, AWS Elemental, custom GPU encoders | Transcoding, resizing, thumbnailing at scale |
| Search | Elasticsearch, OpenSearch, Solr, Meta's Unicorn | Inverted-index full-text and prefix search |
| Real-time messaging | WebSockets, MQTT, Socket.IO; Erlang/Elixir, Netty | Millions of long-lived connections |
| Push | APNs, FCM, OneSignal, AWS SNS | Reaching phones whose app isn't running |
| Tracing the flow | OpenTelemetry, Jaeger, Zipkin, Datadog APM | Seeing every hop of one request with timings |
16. Interview corner
Walk me through what happens, end to end, when a user posts a photo.
The client resizes and compresses the photo, then asks the Upload Service (through the LB and API Gateway, which authenticate and rate-limit) for an upload slot. The service generates a time-sortable media ID and a pre-signed URL. The client PUTs the bytes directly to Object Storage, then calls POST /posts with an idempotency key. The Upload Service verifies the object exists, writes the post row to Cassandra (status PROCESSING), publishes post.created to Kafka (via an outbox), and returns 201. The user is done after ~100 ms of server work plus upload time. Asynchronously, media workers create renditions and run moderation, mark the post READY and emit post.ready. Fan-out workers then push the post ID into each follower's Redis timeline (skipping celebrities, who are pulled at read time), the Search Indexer indexes hashtags, the Notification Service handles mentions, and analytics records the event. Followers fetch the images from the CDN, which origin-pulls from Object Storage.
Why does the client upload directly to object storage instead of through the Upload Service?
Bytes through app servers would cost bandwidth twice, hold an app-server thread or connection for the full (possibly minutes-long) upload, and make the stateless tier scale with media volume instead of request count. A pre-signed URL keeps authorisation in our hands (it's scoped to one key, size and content type, and it expires) while Object Storage handles durability, multipart and resumable transfer at practically unlimited scale. The API path stays small and fast.
A user follows 2,000 accounts, including 15 celebrities. How is their feed served within 200 ms?
Most of the work happened at write time. Normal accounts' posts were already pushed into the user's Redis timeline by fan-out workers, so one sorted-set read returns ~500 candidate IDs. Celebrities were excluded from push fan-out, so in parallel the service reads the recent posts of those 15 accounts. These are 15 partition reads, heavily cached because millions of users need the same data. It merges, filters and ranks (two-stage, with a deadline and a chronological fallback), hydrates the top 20 from the post-object cache, and returns JSON with CDN URLs. The server-side p50 is ~85 ms; images then come from the nearest CDN edge.
How do you make sure a like isn't counted twice when the client retries?
Store likes keyed by (post_id, user_id), so a repeat is an upsert of the same row, and only increment the counter when the like is new: check the insert result, or use a Redis set-add that returns whether the member was new. The client uses PUT semantics ("I like this"), and the server orders a user's like/unlike operations by client timestamp. Kafka consumers downstream (notifications, analytics) dedupe on an event ID. Counters on hot posts are sharded and periodically reconciled against the durable count, because display counts may be approximate but must converge.
How does DM delivery differ when the recipient is offline?
In both cases the message is first durably written to the Messages Store with a per-thread sequence number and acked to the sender. If the presence registry shows a live connection, the server pushes it over the recipient's WebSocket. If not, it publishes an event that the Notification Service turns into an APNs/FCM push. Push is only a wake-up hint. When the recipient opens the app, it reconnects and asks for "all messages in thread X after seq N", which pulls the gap from the store. This makes delivery correct even if pushes are lost or duplicated, and the client dedupes by client_msg_id/seq.
What happens if Kafka is down when a post is created?
With a naive dual write ("write DB, then publish"), you'd get posts that exist but are never fanned out, indexed or processed. Instead use the transactional outbox: the post row and an outbox record are committed together, and a relay (or CDC from the commit log) publishes outbox records when Kafka is reachable, retrying until acknowledged. The user's request succeeds. Followers see the post late, but they never miss it. Consumers must be idempotent because the relay may publish twice.
Which parts of these flows need strong consistency and which can be eventual?
Strong (or at least read-your-writes): authentication and sessions, the author seeing their own new post, the per-thread order of DMs, privacy and block checks (showing a blocked user's content is a real harm), and payments if any. Eventual: follower feeds, like/view counters, search index, notifications, stories tray, analytics. The rule of thumb is to ask "what's the cost of the user seeing stale data for a few seconds?" For most social features it's negligible, and accepting it buys latency and availability (CAP/PACELC).
How would you find out why feed loads became slow last night?
Start from the SLO dashboard (p99 latency per endpoint and region) to locate when and where. Then use distributed traces: each feed request has spans for the gateway, timeline read, celebrity pull, ranking and hydration, so you can see which span grew. Correlate with deploys, cache hit ratio, Redis and Cassandra latency, ranking model version, and Kafka consumer lag (a lagging fan-out pushes more work onto the read path). Logs with the trace ID give details for individual slow requests. See observability.
17. Check your understanding
In Flow 2, which steps must succeed before the user sees "posted"? Which can fail without the user's request failing?
Must succeed: the upload slot (pre-signed URL), the PUT to Object Storage, the post row insert and the durable recording of the event (outbox). Can fail without failing the request, because they're retried asynchronously: renditions, moderation, fan-out, search indexing, notifications and analytics. If moderation later rejects the post, it's removed after the fact.
An account with 50,000 followers posts. Fan-out workers write to Redis with pipelines of 1,000 commands, ~2 ms per pipeline. Roughly how long does push fan-out take for one worker, and why is that acceptable?
50,000 / 1,000 = 50 pipelines × 2 ms ≈ 100 ms, plus paging through the follower list. It runs asynchronously, so nobody waits on it, and followers see the post within about a second. For a 50M-follower account it would be ~100 s of work per post and 50M writes, which is why such accounts use pull instead.
Why does the stories tray store author IDs rather than story IDs?
The tray shows one avatar per author. Storing authors means an author posting 10 stories updates each follower's tray once instead of 10 times (10× less fan-out). Individual story items are fetched from the per-author partition only for the authors actually displayed, and expired items disappear automatically via TTL.
The ranking service starts timing out. What does the user see, and why?
A feed in roughly reverse-chronological order (slightly less relevant), not an error. The News Feed Service calls ranking with a deadline and degrades gracefully to a simple ordering of the candidates it already has. Combined with a circuit breaker, this protects the user experience and gives the ranking service room to recover.
Typeahead: a user types "selena" in 600 ms. Without debouncing, how many requests are sent, and with a 100 ms debounce, roughly how many?
Without debouncing, 6 (one per keystroke). With a 100 ms debounce and keystrokes ~100 ms apart, typically 1–3, usually just the final "selena" plus perhaps an intermediate prefix. Across hundreds of millions of users that cuts typeahead QPS by 2–5×, and the prefix cache absorbs most of the rest.
Bob's phone was off for 6 hours and APNs dropped several notifications. Does Bob lose any DMs? Explain.
No. Messages were durably stored in the Messages Store before being acked to the sender. On reconnect, Bob's app syncs each thread from its last known sequence number and receives everything it missed, in order. Push notifications are only a hint to wake the user; they're not the delivery mechanism.
Which component(s) would you check first if new posts appear on the author's profile but not in followers' feeds?
The asynchronous path between the write and the timelines. Is post.created being published (outbox relay healthy)? Are media workers marking posts READY and emitting post.ready? What is the Kafka consumer lag for the Feed Fan-out Workers, and are any messages piling up in the dead-letter queue? Are the Redis timeline writes succeeding? The author's profile reads Cassandra directly, which is why it looks fine.
🎯 Key takeaways
- Every flow is thin synchronous + fat asynchronous: write the source of truth, publish an event, return. Kafka consumers do the rest.
- API path and media path are separate. Uploads go client → Object Storage via pre-signed URLs; downloads go CDN → client.
- Reads are cheap because writes did the work: precomputed timelines (hybrid push/pull), renditions, counters and indexes.
- Optimistic clients require idempotent servers: idempotency keys, natural primary keys and dedupe in consumers.
- Know each flow's latency budget and each hop's failure behaviour (retry, fallback, degrade). That is what interviewers probe.
- Push notifications are hints. Durable stores plus client sync are the delivery guarantee.
Related lessons
- Full interview walkthrough: put these flows into a 45-minute "Design Instagram" answer
- Media upload and media processing: Flow 2 in depth
- News feed: fan-out on write vs read, ranking (Flow 3)
- Likes, comments & counters: sharded counters and hot keys (Flow 4)
- Social graph: edges in both directions (Flow 5)
- Stories: TTL data and the tray (Flow 6)
- Search: inverted indexes and typeahead (Flow 7)
- Direct messaging: WebSockets, ordering, receipts (Flow 8)
- Notifications: APNs/FCM, aggregation, preferences (Flow 9)
- Kafka: the backbone of every asynchronous half
- Resilience patterns: timeouts, retries, circuit breakers and degradation used in every failure section
- Glossary: quick definitions of every term used here