1. The problem: why is messaging different from everything else?

Almost everything else in Instagram is client-initiated: the app asks ("give me my feed", "like this post") and the server answers. Direct messages break that model. When your friend sends you "are you coming?", nobody on your side asked for it — the server has to reach your phone on its own, within a fraction of a second, while you're looking at a chat screen.

That single difference creates a cascade of hard problems:

• Server push: plain HTTP can't push. We need a long-lived connection from every online device — that's tens or hundreds of millions of open connections at once.
• Routing: the message arrives at one server; the recipient is connected to a different one, among hundreds. Who knows where they are?
• Reliability: phones drop off networks constantly (elevators, tunnels, app backgrounded). A message must never be lost, never be shown twice, and must reach the user eventually even if they're offline for a week.
• Ordering: if Alice sends "Do you want pizza?" then "Or sushi?", Bob must see them in that order — on every one of his devices, and in group chats everyone must see the same order.
• State signals: delivered/seen receipts, "online now", "typing…" — all of which are high-frequency and must be cheap.

2. Requirements and numbers

2.1 Functional requirements

• 1:1 and group conversations (groups up to a few hundred members — an assumption).
• Real-time delivery to online recipients; offline delivery when they return, plus push notifications.
• Ordering: all participants see a conversation's messages in the same order.
• Receipts: sent, delivered and read ("Seen").
• Presence ("Active now", "Active 5m ago") and typing indicators.
• Media: photos, videos, voice notes, shared posts/Reels.
• Multi-device sync: phone, tablet and web show the same history and read state.
• History: scroll back through old messages; inbox list sorted by last activity.
• Optional: end-to-end encryption (E2EE).

2.2 Non-functional requirements

• Low latency: end-to-end delivery to an online recipient in ~100–500 ms.
• Durability: once the sender sees "sent", the message must never be lost.
• Exactly-once appearance: no visible duplicates, even though the network forces retries.
• High availability: chat is expected to "always work"; a gateway crash should cost a reconnect, not data.
• Consistency: per-conversation ordering is required; global ordering across conversations is not.

2.3 Back-of-the-envelope (assumptions)

Assume 300M daily DM users, 40 messages sent per user per day
  → 12B messages/day → 12B / 86,400 ≈ 140,000 msgs/s average, ~400,000/s peak
  → deliveries are higher: a message to a 10-person group = 9 deliveries

Storage: ~300 bytes per message row (text + IDs + metadata; media stored separately in S3)
  → 12B × 300 B ≈ 3.6 TB/day ≈ 1.3 PB/year, ×3 replicas ≈ 4 PB/year

Concurrent connections at peak: assume ~150M online devices
  Each gateway server holds ~500k connections (tuned event-loop server)
  → 150M / 500k = 300 gateway servers (+ headroom for failures → ~400-500)

Heartbeats: 150M connections pinging every 30 s → 5M tiny frames/s
  (this is why heartbeats are handled at the gateway and never touch a database)

Notice the two very different workloads: a connection-holding tier (huge numbers of mostly idle sockets, low CPU) and a message-processing tier (hundreds of thousands of writes per second). Separating them is the first architectural decision.

3. Real-time transport: how can the server reach the phone?

HTTP is request/response: the client speaks first, the server answers, done. There are four classic ways to get "server push" anyway, plus MQTT.

 SHORT POLLING                         LONG POLLING
 client          server                client          server
   |-- any new? ----->|                  |-- any new? ----->|
   |<--- no ---------|                  |                  |  (server HOLDS the request
   |   ...wait 5 s... |                  |                  |   until data or ~30 s timeout)
   |-- any new? ----->|                  |<--- msg #1 ------|
   |<--- no ---------|                  |-- any new? ----->|  (immediately re-asks)
   |-- any new? ----->|                  |                  |
   |<--- msg #1 -----|  up to 5 s late   |<--- msg #2 ------|

 SERVER-SENT EVENTS (SSE)              WEBSOCKET
 client          server                client          server
   |-- GET /events -->|                  |-- HTTP Upgrade ->|
   |<== 200, stream open, never ends    |<-- 101 Switching-|
   |<--- event: msg #1                  |<=== full-duplex frames ===>|
   |<--- event: msg #2                  |--- send msg ---->|
   |  (one direction only: server→client) |<--- msg #1 ------|
   |  client sends via separate POSTs     |<--- typing… -----|

3.1 The options compared

• Short polling at Instagram scale is absurd: 150M devices polling every 5 s = 30M requests/s, almost all answering "nothing new", and still up to 5 s of latency.
• Long polling was the classic pre-WebSocket solution (early Facebook chat used it). It works through any proxy, but each delivered message costs a new HTTP request with full headers, and every hop must tolerate requests hanging for 30+ seconds.
• SSE is great when only the server talks (live scores, a notification stream) and it auto-reconnects with a Last-Event-ID, but sending still needs separate HTTP requests.
• WebSocket starts as an HTTP request with Upgrade: websocket; after a 101 Switching Protocols response the same TCP (+TLS) connection carries small bidirectional frames. This is the standard for browser chat and what our architecture diagram shows. See API design for how it fits next to REST and gRPC.
• MQTT is a publish/subscribe protocol designed for unreliable, low-bandwidth networks. Facebook publicly wrote (2011) that Messenger used MQTT to keep a persistent connection with low battery and bandwidth cost; Meta's mobile apps have continued to use MQTT-based connections for real-time features. The concepts below are identical whichever protocol carries the bytes.

4. Gateway servers: holding millions of connections

A gateway (also called connection server, edge server or "front-end") is a server whose main job is to keep sockets open and shuttle frames between devices and the backend. It contains no business logic, which keeps it simple, stable and rarely redeployed.

4.1 How one server holds 500,000+ connections

• Event-driven I/O: instead of one thread per connection (500k threads would need hundreds of GB of stack memory and drown in context switches), the server uses an event loop on top of epoll (Linux) / kqueue: a few threads are told by the kernel "these 30 sockets have data", handle them, and go back to waiting. Netty (Java), Go's netpoller, Node.js, Erlang/BEAM all work this way. WhatsApp publicly reported holding over 2 million connections on a single Erlang server back in 2012.
• Memory per connection is the real limit: kernel socket buffers + TLS session state + application state ≈ 10–50 KB each. 500k × 30 KB ≈ 15 GB of RAM — fine for a modern server.
• OS tuning: raise the file-descriptor limit (each socket is a file descriptor), tune TCP buffer sizes and keep-alive. A server listening on one port can hold millions of connections (each is identified by the client's IP and port, so the server's ports don't run out).
• TLS termination at the gateway (or a TLS-terminating L4 proxy in front of it), using session resumption so reconnects are cheap (see Networking basics).

4.2 Load balancing long-lived connections

A load balancer spreads new connections across gateways (least-connections works well). But connections live for hours, so the usual "add servers and traffic rebalances" doesn't happen by itself:

• Deploys and draining: to restart a gateway, stop routing new connections to it, then close existing ones gradually (e.g. over 10 minutes) with a "please reconnect" frame so clients move to other gateways.
• Reconnect storms: if a gateway with 500k connections crashes, 500k clients reconnect at once — and each reconnect triggers a TLS handshake, an auth check and a sync. Clients must use exponential backoff with random jitter (e.g. retry after random 0–2 s, then 0–4 s…), and the backend must rate-limit sync work (see Resilience patterns).
• Heartbeats (ping/pong every ~30–60 s) keep NATs and mobile carrier middleboxes from silently dropping idle connections and let the gateway detect dead clients.

5. The session registry: who is connected where?

When Alice's message for Bob arrives, the Chat Service must know: is Bob online, on which devices, and which gateway holds each device's connection? That mapping is the session registry, typically a Redis cluster (fast, in-memory, with TTLs).

# Bob's phone connects to gateway 47 (connection id c-88123)
HSET   sess:bob  phone-7f3a  "gw47|c-88123|1730000000"
EXPIRE sess:bob  90                    # refreshed by every heartbeat; auto-expires if gateway dies

# Bob's laptop connects to gateway 12
HSET   sess:bob  web-21cc    "gw12|c-55102|1730000004"

# Chat Service routing a message to Bob
HGETALL sess:bob
→ phone-7f3a: gw47|c-88123|…   web-21cc: gw12|c-55102|…   → deliver to BOTH gateways

# Clean disconnect
HDEL   sess:bob  phone-7f3a

• Per device, not per user: each device has its own connection and its own sync cursor (multi-device).
• TTL as a safety net: if a gateway crashes without cleaning up, entries expire within ~90 s. Until then a route may point at a dead gateway — delivery fails, and the message is simply picked up on reconnect via sync. The registry is a hint, never the source of truth for delivery.
• Load: writes happen on connect/disconnect (and TTL refresh, which can be batched per gateway), reads once per message recipient. Both are cheap single-key operations, sharded across a Redis cluster by user ID.
• Gateway-to-gateway routing: once you know "gw47", how do you send to it? Either a direct RPC (Chat Service → gw47's internal address), or a pub/sub channel per gateway (Redis pub/sub or a lightweight broker topic gw47 that the gateway subscribes to). Direct RPC is simplest and lowest latency.

6. Sending a message: the full sequence

Here is the complete path of one message from Alice (connected to gateway 12) to Bob (connected to gateway 47), including every acknowledgement:

 Alice app      Gateway 12      Chat Service        Messages store    Session registry   Gateway 47     Bob app
    |                |                |                     |                 |               |              |
    | 1. SEND {conv:c9, client_msg_id:"a1b2", text:"pizza?"} (shown locally with a 🕓 clock)    |              |
    |--------------->|                |                     |                 |               |              |
    |                |-- 2. forward (+ authenticated user=alice) -->|          |               |              |
    |                |                | 3. check alice ∈ c9, not blocked, rate limit, size    |              |
    |                |                | 4. dedup: seen client_msg_id "a1b2" before? → no      |              |
    |                |                | 5. assign msg_id (time-ordered) + seq=42 for c9       |              |
    |                |                |-- 6. write row ---->|                 |               |              |
    |                |                |<-- ok (quorum) -----|                 |               |              |
    |                |<-- 7. ACK {client_msg_id:"a1b2", msg_id, seq:42} ---  |               |              |
    |<-- SENT ✓ -----|                |                     |                 |               |              |
    |                |                |-- 8. publish msg event to Kafka (inbox update, notif, analytics)     |
    |                |                |-- 9. HGETALL sess:bob --------------->|               |              |
    |                |                |<-- phone-7f3a @ gw47 -----------------|               |              |
    |                |                |-- 10. deliver(conn c-88123, msg) --------------------->|              |
    |                |                |                     |                 |               |-- 11. push ->|
    |                |                |                     |                 |               |<- 12. ACK ---|
    |                |                |<-- 13. delivered(bob, c9, seq 42) ---------------------|  (stores msg,|
    |                |                |   update delivered watermark          |               |   dedups by  |
    |                |<-- 14. receipt: delivered up to seq 42 --------------  |               |   msg_id)    |
    |<-- DELIVERED ✓✓|                |                     |                 |               |              |
    |                |                |                     |                 |               |  Bob opens chat
    |                |                |<-- 15. read(bob, c9, up to seq 42) ---------------------------------- |
    |<-- SEEN -------|<---------------|   update read watermark, relay receipt to alice's devices            |

Important details hidden in those steps:

• Persist before ack (step 6 → 7): Alice's "sent ✓" means the message is durably stored (written to a quorum of replicas). If the Chat Service crashes after step 6 but before step 7, Alice's app retries with the same client_msg_id and step 4 recognises the duplicate.
• Persist before deliver: Bob may receive it via real-time push or via sync later — both read from the same stored log, so nothing depends on the push succeeding.
• Delivery is decoupled from sending: steps 9–12 can fail (Bob's gateway just died) without affecting Alice. The message waits in storage; Bob's next sync fetches it.
• Sender's other devices: Alice's laptop must also show the message she sent from her phone — so the Chat Service delivers it to all of Alice's other devices too (multi-device sync is "deliver to everyone in the conversation, including the sender's other devices").
• Offline path: if step 9 finds no sessions for Bob, the Kafka event at step 8 lets the Notification Service send a push notification (step 9 may also explicitly trigger it).
• Media messages: the app first uploads the photo/voice note to object storage with a pre-signed URL, then sends a normal message containing the media key. Recipients download via the CDN with signed URLs. Chat servers never relay large binary payloads.

7. Storing messages

7.1 Access patterns decide the model

Chat storage is overwhelmingly write-heavy and read in one specific way: "the latest N messages of conversation X, then the N before that as I scroll up", plus "everything in X after sequence S" for sync. Messages are almost never updated (except edits, unsends and reactions). That's a perfect fit for a wide-column store (see Databases) built on LSM-trees, which turns random writes into sequential ones (see Indexing).

-- Cassandra-style schema (CQL)
CREATE TABLE messages (
  conversation_id  bigint,
  bucket           int,        -- e.g. month number: bounds partition size
  message_id       bigint,     -- 64-bit time-ordered ID (Snowflake/Instagram-style)
  seq              bigint,     -- per-conversation sequence number
  sender_id        bigint,
  client_msg_id    uuid,       -- sender-generated idempotency key
  type             text,       -- text | media | post_share | reaction | system
  body             blob,       -- text, or ciphertext if E2EE
  media_ref        text,
  PRIMARY KEY ((conversation_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

-- "Load the chat": newest 30 messages
SELECT * FROM messages WHERE conversation_id = 9 AND bucket = 658 LIMIT 30;
-- "Scroll up": 30 older than the oldest I have
SELECT * FROM messages WHERE conversation_id = 9 AND bucket = 658
  AND message_id < 7234011234567890 LIMIT 30;

• Partition key = conversation_id (plus a time bucket): all messages of a conversation live together on the same replica set, so reading a chat is a single-partition, sequential read. Different conversations spread evenly across the cluster (sharding by hash of the partition key — consistent hashing in Cassandra).
• Clustering key = message_id descending: within a partition rows are physically sorted, so "latest 30" is the first 30 rows — no sorting at query time. A time-ordered 64-bit ID (timestamp in the high bits, see ID generation) gives roughly chronological order and uniqueness without coordination.
• Buckets prevent unbounded partitions: a group chat active for five years could accumulate millions of messages; Cassandra partitions should stay well under ~100 MB. Bucketing by month means "scroll up" occasionally crosses into the previous bucket.

7.2 The inbox (conversation list)

The DM inbox shows your conversations sorted by latest activity with a preview and unread badge. That's a different access pattern ("all conversations for user U, newest first"), so it gets its own table, maintained asynchronously from the Kafka message events:

user_inbox: PRIMARY KEY (user_id, last_activity_ts DESC, conversation_id)
            → last message preview, unread count, muted flag
conversation_members: PRIMARY KEY (conversation_id, user_id) → role, joined_at, read/delivered watermarks

Because Cassandra can't "update the sort key in place", moving a conversation to the top means delete-old-row + insert-new-row; many systems instead keep the inbox in Redis sorted sets (score = last activity) backed by the durable table.

7.3 Why HBase / Cassandra?

8. Ordering: everyone sees the same sequence

8.1 Why timestamps alone fail

• Client clocks lie. A phone with a clock 3 minutes fast would place its messages "in the future". Never order by device time.
• Server clocks drift too. Two Chat Service instances receiving Alice's and Bob's simultaneous messages may disagree by a few milliseconds; time-ordered IDs from different machines aren't strictly ordered across machines (see ID generation).
• Concurrent sends in a group chat: if Carol and Dan send at the same moment, both orders are "correct" — what matters is that everyone sees the same one.

8.2 Per-conversation sequence numbers

The robust solution is a monotonic sequence number per conversation: 1, 2, 3, … assigned by the server at write time. It gives a total order within a conversation (which is all users can perceive) while different conversations proceed fully in parallel — no global ordering bottleneck.

Who assigns seq without races?

On the client side:

• The sender's message is shown immediately with a pending state at the bottom, then repositioned (if necessary) when the ACK returns its real seq.
• Recipients insert messages by seq. If they receive seq 45 while their last is 43, they know 44 is missing (perhaps still in flight, perhaps lost on a dying connection) and request the gap via sync.
• Sequence numbers double as sync cursors and receipt watermarks ("read up to 45") — one concept, three uses.

9. Delivery guarantees: at-least-once plus dedup

Networks offer no "exactly-once delivery". A send can fail in three indistinguishable ways: the request was lost, the server crashed after processing, or the ACK was lost. The client can't tell which — so it must retry, and retries create duplicates. The standard recipe (see delivery guarantees and idempotency keys):

1. The client generates a unique client_msg_id (a UUID) when the user hits send, and stores the message in a local outbox.
2. It sends and waits for an ACK; on timeout it retries with the same client_msg_id (with backoff), across reconnects and even app restarts.
3. The server keeps a dedup record (sender, client_msg_id) → (msg_id, seq) for a period (e.g. 24 h in Redis, or a unique index in the store). A duplicate send simply returns the original ACK — idempotent.
4. Server → recipient delivery is also at-least-once: the gateway pushes, waits for the device's ACK, and the backend relies on sync to fill anything un-acked. The recipient dedups by msg_id (a local unique index) before displaying.

// client-side outbox (simplified)
function send(convId, text) {
  const msg = { clientMsgId: uuidv4(), convId, text, state: "PENDING" };
  localDb.outbox.insert(msg);  ui.append(msg);
  trySend(msg);
}
function trySend(msg, attempt = 0) {
  socket.send({ type: "SEND", ...msg });
  timers.set(msg.clientMsgId, setTimeout(
    () => trySend(msg, attempt + 1),
    Math.min(30000, 1000 * 2 ** attempt) * (0.5 + Math.random())));  // backoff + jitter
}
socket.on("ACK", ({ clientMsgId, msgId, seq }) => {
  clearTimeout(timers.get(clientMsgId));
  localDb.outbox.markSent(clientMsgId, msgId, seq);    // ✓ sent
});

The combination — at-least-once transport + idempotent processing + dedup on display — gives users the experience of exactly-once, which is the only kind of exactly-once real distributed systems can offer.

10. Offline users, push and sync

Most of the time the recipient is not connected: the app is closed or the phone is asleep. The design must treat offline as the normal case.

1. Store: the message is already durable in the messages store (step 6 above). Nothing is "queued for Bob" separately — the conversation log is the queue.
2. Notify: the Notification Service, consuming the message event from Kafka, sends a push via APNs/FCM: "Alice: pizza?" (or a generic "New message" when content must stay private). Pushes for the same conversation are collapsed so 20 messages don't produce 20 banners.
3. Reconnect: when Bob opens the app, it establishes a connection, registers in the session registry, and immediately syncs.
4. Sync from a cursor: the device sends the last position it has fully processed; the server returns everything after it, in order, in pages.

Two common cursor designs:

GET /v1/sync?cursor=88214&limit=200
→ { "events": [
      {"useq": 88215, "type": "message", "conv": "c9", "seq": 42, "from": "alice", "text": "pizza?"},
      {"useq": 88216, "type": "read",    "conv": "c12", "user": "carol", "up_to_seq": 1007},
      {"useq": 88217, "type": "reaction","conv": "c9", "seq": 42, "emoji": "❤️", "from": "dan"} ],
    "next_cursor": 88217, "has_more": false }

This same mechanism solves multi-device sync: each device keeps its own cursor into the same per-user log, so the phone and laptop independently catch up to the same state. A device offline for months may get a "cursor too old" response and do a full resync of recent conversations instead.

11. Receipts: sent, delivered, seen

A naive design stores a status row per message per recipient and updates it twice. For a 100-member group that's 200 writes per message — far more than the message itself. The efficient design uses watermarks:

• For each (conversation, member) store last_delivered_seq and last_read_seq. Because sequence numbers are ordered, "read up to 45" implies 1–45 are all read. One small update covers any number of messages.
• Bob's device sends read(c9, up_to=45) when the chat is visible; the server updates the watermark only if it increases (monotonic — duplicates and out-of-order receipts are harmless) and relays it to the other members' devices as an event.
• The sender's UI computes each message's state by comparing its seq with the watermarks: in a group, "Seen by Carol, Dan + 3" = members whose last_read_seq ≥ seq.
• Receipts are coalesced: if Bob reads 30 messages, one receipt is sent, not 30. Delivered receipts can be batched per device every second or so.
• Users can disable read receipts; the server then simply doesn't relay them (it may still store the watermark to sync Bob's own devices' unread badges).

12. Presence and typing indicators

12.1 Presence via heartbeats

Presence = is the user online now, or when were they last active? The gateway already knows when a connection opens, closes or goes silent, so presence is derived from connection state plus heartbeats:

# on connect and on every heartbeat (~30 s), the gateway batches updates:
SET presence:bob  "online"  EX 70          # expires if 2 heartbeats are missed
SET last_active:bob 1730000123             # for "Active 5m ago"

# a viewer's inbox shows 20 conversations → one batched read:
MGET presence:alice presence:carol presence:dan ...

• Don't broadcast every change to everyone. If each of 150M users toggles online/offline ~10 times a day and has ~200 contacts, pushing every change would be 300B presence messages/day. Instead use interest-based delivery: fetch presence when you open the inbox or a chat (pull), and subscribe to live changes only for the conversations currently on screen.
• Debounce flapping: a phone switching from Wi-Fi to 4G disconnects for a second. Mark someone offline only after the TTL expires (e.g. ~60 s without heartbeat), and treat a quick reconnect as continuous presence.
• Approximate is fine: "Active now" being 30 s stale is invisible to users, so presence lives only in memory with TTLs — never in the durable store.
• Privacy: users can hide activity status; presence is only shown to people you follow or have chatted with.

12.2 Typing indicators

"Alice is typing…" is the most ephemeral signal of all:

• The client sends typing(c9) when the user starts typing and then at most once every ~3–5 s while they keep typing — never per keystroke.
• The Chat Service relays it straight to the online participants' gateways: no persistence, no Kafka, no retries, no push notifications, no sync. If it's lost, nobody cares.
• The receiver shows the indicator and auto-hides it after ~5–6 s without a refresh (so a sender who closes the app doesn't leave "typing…" forever).
• Under load, typing indicators are the first thing to shed (see graceful degradation).

13. Group conversations and fan-out

A message to a group of n members needs n − 1 deliveries (times the number of devices each). The question — just like in the news feed — is where to fan out.

Delivery itself always fans out: the Chat Service looks up the members (cached conversation_members), batch-reads their sessions from the registry (one pipelined call), groups the target devices by gateway, and sends one batched RPC per gateway containing all of its recipients. For a 250-member group with ~1.5 devices online each, that's ~370 device pushes but only a few dozen RPCs. Offline members get a (collapsed, possibly muted) push notification and pick the message up on sync.

• Cap group size (a few hundred members) so fan-out per message stays bounded; very large audiences are a different product (broadcast channels) with fan-out-on-read semantics.
• Ordering in groups comes from the same per-conversation seq — critical because concurrent senders are common.
• Membership changes are themselves messages in the log ("Dan added Erin") with a seq, so every device applies them in the same order relative to messages; a removed member stops receiving deliveries from that seq onward.

14. End-to-end encryption (E2EE) basics

With normal (transport) encryption, TLS protects messages on the wire, but the server decrypts and stores them — so the company (or an attacker who breaches it) could read them. With end-to-end encryption, only the participants' devices hold the keys; the server stores and forwards ciphertext it cannot read.

14.1 The Signal protocol idea

The Signal protocol, used by Signal and WhatsApp and adopted by Meta for Messenger and Instagram DMs, combines two pieces:

1. Key publication: each device generates a long-term identity key pair, a signed prekey and a batch of one-time prekeys, and uploads only the public halves to the server's key directory. Private keys never leave the device.
2. X3DH (initial key agreement): to start a chat with Bob even while he's offline, Alice fetches Bob's public "prekey bundle" and combines several Diffie-Hellman calculations (her identity and a fresh ephemeral key with his identity, signed prekey and a one-time prekey). Both sides can derive the same shared secret, but the server — which only saw public keys — cannot.
3. Double Ratchet (ongoing messages): every message is encrypted with a new key derived by "ratcheting" a chain forward, mixed with fresh Diffie-Hellman exchanges as replies flow. Stealing today's key doesn't decrypt past messages (forward secrecy), and the conversation heals after a compromise once new DH exchanges happen (post-compromise security).

 Bob's device                    Server (relay + key directory)                 Alice's device
 gen IK_B, SPK_B, OPK_B[1..100]
 ── upload PUBLIC keys ─────────► stores IK_B, SPK_B, OPK_B…
                                 ◄──────────────────── fetch Bob's bundle ───── (first message)
                                                         X3DH: DH(IK_A,SPK_B), DH(EK_A,IK_B),
                                                               DH(EK_A,SPK_B), DH(EK_A,OPK_B) → secret
                                 ◄─── ciphertext + EK_A(public) + OPK id ────── Double Ratchet: msg key #1
 ◄── relay ciphertext ──────────  (sees: sender, recipient, time, size —
 same DH math → same secret        NOT the content)
 decrypt; ratchet forward

14.2 What E2EE changes in the system design

• Multi-device: each device has its own keys, so the sender encrypts a message separately for every device of every participant (client-side fan-out). Groups use "sender keys": a sender distributes a group key once per member device (pairwise-encrypted), then encrypts each message once.
• Server features shrink: no server-side search of message text, no server-generated link previews, no content-based spam/abuse scanning (systems rely on metadata, user reports — which forward the reported plaintext — and client-side signals instead).
• History and new devices: a new device can't decrypt old messages from the server; history needs encrypted backups or device-to-device transfer.
• Metadata remains visible: who talks to whom and when is still needed for routing; minimising and protecting it is a separate effort (see Security & privacy).
• The storage, ordering, receipts and sync designs above are unchanged — they just carry opaque body bytes.

15. Scaling: the numbers end to end

Latency budget for an online-to-online message (assumptions): client → gateway ~30–80 ms (mobile network), gateway → Chat Service ~1 ms, persist with quorum ~5–10 ms, registry lookup ~1 ms, route to recipient gateway ~1–2 ms (same region), gateway → device ~30–80 ms. Total ≈ 70–180 ms; cross-region conversations add an inter-region hop (~50–150 ms).

Multi-region: users connect to the nearest region's gateways. A conversation needs one home for sequencing (e.g. the region of its creator or a consistent hash), and cross-region delivery goes through region-to-region links; the message store replicates across regions for disaster recovery (see Multi-region).

16. How Instagram uses real-time messaging

1. When the app is open, it keeps a persistent connection through the load balancer to a gateway (the diagram's "Direct Messaging (WebSocket)" service).
2. Sending a DM goes over that connection to the Direct Messaging service, which validates, sequences and persists it in the Messages Store (Cassandra/HBase), then routes it via the session registry to the recipient's gateway.
3. A Kafka event drives the inbox update, analytics, and the Notification Service → APNs/FCM push for offline recipients.
4. Photos, videos and voice notes go to Object Storage via pre-signed URLs and are served by the CDN; shared posts and Reels are sent as references (IDs), hydrated like feed posts.
5. On reconnect, the app syncs from its cursor, so the phone, tablet and web client converge on the same history and read state.

17. Trade-offs & common pitfalls

Common pitfalls

• Acking before persisting — a crash loses messages the sender believes were sent.
• Treating the push notification or the WebSocket push as the delivery guarantee — both are best-effort; the stored log + sync is the guarantee.
• Ordering by client timestamps — clock skew scrambles conversations. Use server-assigned per-conversation sequence numbers.
• Retrying without an idempotency key — duplicate messages on flaky networks.
• Per-message receipt rows — use monotonic watermarks.
• Broadcasting presence to all contacts — use pull + subscribe-while-visible.
• Business logic in gateways — every logic change then forces reconnecting millions of clients.
• Clients reconnecting without jitter — a gateway crash becomes a self-inflicted DDoS.
• Unbounded partitions for long-lived busy conversations — bucket by time.

18. Real-world technologies

19. Interview corner

20. Check your understanding

Related lessons

• API Design — WebSockets vs REST/gRPC, idempotency keys
• Networking Basics — TCP, TLS and keep-alive under persistent connections
• Load Balancing — L4 balancing and draining of long-lived connections
• Message Queues & Kafka — delivery guarantees and the events behind inbox and notifications
• Databases — why wide-column stores fit message history
• Unique ID Generation — time-ordered message IDs
• Notification System — pushes for offline recipients
• Security & Privacy — encryption in transit, at rest and end to end
• Resilience patterns — backoff with jitter, reconnect storms, graceful degradation