1. The problem: why go asynchronous?

When you tap "Share" on a new Instagram post, a surprising amount of work must happen:

• the photo must be resized into several renditions and a video transcoded (Media Processing),
• the post ID must be pushed into the timelines of every follower (Feed fan-out) — maybe 300 followers, maybe 300 million,
• hashtags and captions must be added to the search index,
• tagged users must get push notifications (Notifications),
• analytics and ML pipelines must record the event (Analytics & ML).

If the Upload/Post service called each of these synchronously, one after another, the user would wait for the sum of all their latencies (seconds, or minutes for a celebrity fan-out); if any one were down, posting would fail; and every new feature would require changing the Post service. Instead, the Post service writes the post, publishes a single "post created" event, and returns "posted!" in milliseconds. Everything else happens asynchronously.

1.1 What asynchronous messaging buys you

The costs: eventual consistency (a follower might see the post a few seconds later; search may lag), more moving parts, harder debugging across services, and new failure modes (duplicates, out-of-order messages, lag). Use async messaging for work that does not need to complete before you answer the user.

2. Core concepts and messaging models

• Message / event — a small record (bytes, usually JSON/Avro/Protobuf, typically < 1 MB, often ~1 KB) with optional key and headers. A command says "do X" (ResizeImage); an event states a fact "X happened" (PostCreated). Events are preferred for decoupling.
• Producer (publisher) writes messages; consumer (subscriber) reads them; the broker stores and delivers them.
• Acknowledgement (ack) — the consumer's signal that a message has been processed and need not be redelivered.

2.1 Point-to-point queues vs publish/subscribe

3. Traditional brokers vs log-based brokers

There are two families of messaging systems, and choosing between them is a frequent interview question.

• Traditional message brokers (RabbitMQ, ActiveMQ, Amazon SQS) are queues: the broker tracks each message's state, hands it to a consumer, waits for an ack and then deletes it. RabbitMQ routes messages through exchanges (direct, topic, fanout) into queues; SQS uses a visibility timeout — a received message is hidden for N seconds and reappears if not deleted in time.
• Log-based brokers (Apache Kafka, Amazon Kinesis, Apache Pulsar, Redpanda) are append-only logs: messages are written in order and retained for a period regardless of consumption. Each consumer group just remembers an offset (position) per partition and moves it forward.

For Instagram's event bus — high volume, many independent consumers, replay to rebuild indexes, analytics — Kafka is the natural choice, and that's what our architecture diagram shows. A task-queue system (Celery on RabbitMQ/Redis, or SQS) still fits well for discrete jobs like "send this email".

4. Kafka internals

4.1 Topics, partitions and offsets

A topic is a named stream of related events, e.g. post.created. A topic is split into partitions; each partition is an ordered, append-only log stored on disk. Every message in a partition gets a sequential offset (0, 1, 2, …), its permanent address within that partition.

 topic: post.created          (3 partitions; each is an independent ordered log)

 partition 0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |  ← producers append here (log end offset = 9)
                               ↑               ↑
                     group "search" at 4       group "fanout" at 8

 partition 1 | 0 | 1 | 2 | 3 | 4 | 5 |
 partition 2 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |

 • Order is guaranteed only WITHIN a partition (p0 offset 3 is before p0 offset 4).
 • Messages are NOT deleted when read; each consumer group keeps its own offset.
 • Old log segments are deleted by retention (e.g. 7 days) or compacted.

Why partitions? A single log can only be as fast as one disk and one machine. Partitions let a topic spread across many brokers (more write throughput) and let many consumers read in parallel (one consumer per partition per group). Kafka is fast because each partition is written and read sequentially (disks and the OS page cache love sequential I/O), messages are batched and compressed, and data is sent from page cache to socket with zero-copy (sendfile).

4.2 Brokers, leaders, followers and the ISR

A Kafka cluster is a set of servers called brokers. Each partition is replicated to replication factor (RF, typically 3) brokers. One replica is the leader — all produce and (by default) consume requests go to it — and the others are followers that continuously fetch from the leader to stay in sync.

• ISR (in-sync replicas) — the set of replicas that are fully caught up with the leader (within replica.lag.time.max.ms, default 30 s). A slow or dead follower is dropped from the ISR and rejoins after catching up.
• High watermark — the highest offset replicated to all ISR members. Consumers only see messages up to the high watermark, so they never read data that could be lost in a leader failover.
• Leader election — when a leader dies, the cluster controller promotes an ISR member. Modern Kafka manages this metadata with its built-in Raft-based quorum (KRaft), which replaced ZooKeeper (ZooKeeper support was removed in Kafka 4.0). unclean.leader.election.enable=false (default) forbids promoting an out-of-sync replica, trading availability for no data loss.

4.3 acks: how durable is a write?

Pair acks=all with min.insync.replicas=2 and RF = 3: a write succeeds only if at least 2 replicas have it; if the ISR shrinks to 1, producers get errors rather than writing data that only one machine holds. That's the standard "don't lose events" configuration for business events like post.created. For high-volume, loss-tolerant telemetry (impressions), acks=1 may be acceptable.

4.4 Retention and log compaction

• Time/size retention (retention.ms, retention.bytes): partitions are stored as segment files; whole old segments are deleted after, say, 7 days. Consumers can be down for days and still catch up; you can replay history within the window. Tiered storage (Kafka 3.6+) offloads old segments to object storage for long, cheap retention.
• Log compaction (cleanup.policy=compact): instead of deleting by age, Kafka keeps at least the latest message for every key, discarding older values of the same key in the background. A message with a null value (a tombstone) deletes the key. The topic becomes a durable changelog / snapshot of current state — e.g. user.profile keyed by user_id lets a new service bootstrap a full copy of all profiles by reading the topic from the start.

 Before compaction (key:value)       After compaction
 0  u1:{name:"ana"}                   
 1  u2:{name:"raj"}                   1  u2:{name:"raj"}
 2  u1:{name:"ana_k"}                 
 3  u3:{name:"li"}                    
 4  u1:{name:"ana.k"}                 4  u1:{name:"ana.k"}
 5  u3:null   (tombstone)             5  u3:null   (removed later)

5. Producers and partition keys

The producer decides which partition each message goes to:

• With a key: partition = hash(key) mod num_partitions (Kafka's default uses murmur2). Every message with the same key lands in the same partition — therefore all events for one key are ordered.
• Without a key: the default "sticky" partitioner fills a batch for one partition then switches, spreading load evenly (no ordering guarantees).

Choosing the key is a design decision:

Other producer knobs: linger.ms and batch.size (wait a few ms to send bigger batches → much higher throughput), compression.type (lz4/zstd), and enable.idempotence=true (default since 3.0): the broker de-duplicates producer retries using a producer ID + per-partition sequence numbers, so a network retry doesn't create a duplicate in the log.

from confluent_kafka import Producer
p = Producer({"bootstrap.servers": "kafka:9092", "acks": "all",
              "enable.idempotence": True, "linger.ms": 5, "compression.type": "zstd"})

event = PostCreated(post_id=4185136929178974185, author_id=31341, media_keys=[...])
p.produce("post.created", key=str(event.author_id), value=event.SerializeToString(),
          on_delivery=lambda err, msg: err and log.error("publish failed", err))
p.flush()

6. Consumer groups, partition assignment and rebalancing

A consumer group is a set of consumer processes that cooperatively read a topic. Kafka assigns each partition to exactly one consumer in the group, so the group processes every message once (queue semantics), while different groups each read everything (pub/sub semantics).

• Parallelism ≤ partitions. A group can't usefully have more active consumers than partitions. Plan partition counts for peak consumer parallelism, not just producer throughput.
• Offsets are committed by the consumer to an internal topic (__consumer_offsets). After a crash or reassignment, the new owner resumes from the last committed offset.
• Rebalancing happens when consumers join, leave, crash (miss heartbeats / exceed max.poll.interval.ms) or partitions change. The group coordinator reassigns partitions. Old "eager" rebalancing stopped the whole group ("stop-the-world"); the cooperative incremental protocol only moves the partitions that need to move, and static membership (group.instance.id) avoids rebalances on quick restarts during deploys. Kafka 4.0's new consumer group protocol moves assignment to the broker for faster, smoother rebalances.
• Assignment strategies: range, round-robin, sticky, cooperative-sticky (minimises movement — the usual choice).

from confluent_kafka import Consumer
c = Consumer({"bootstrap.servers": "kafka:9092", "group.id": "search-indexer",
              "enable.auto.commit": False, "auto.offset.reset": "earliest",
              "partition.assignment.strategy": "cooperative-sticky"})
c.subscribe(["post.created"])
while True:
    msg = c.poll(1.0)
    if msg is None or msg.error():
        continue
    index_post(PostCreated.FromString(msg.value()))   # must be idempotent!
    c.commit(message=msg, asynchronous=False)           # commit AFTER processing → at-least-once

7. Delivery semantics

Networks drop packets, consumers crash mid-processing, producers retry. What can a messaging system promise about how many times a message is processed?

 At-least-once duplicate:
  consumer: poll offset 41 → process (write to Elasticsearch ✔) → CRASH before commit
  restart:  last committed offset = 40 → poll offset 41 AGAIN → process again

Kafka's exactly-once semantics (EOS) combine the idempotent producer (no duplicate writes on retry) with transactions: a stream processor reads from input topics, writes to output topics and commits its consumer offsets atomically in one transaction; downstream consumers with isolation.level=read_committed never see aborted writes. This gives exactly-once for Kafka-to-Kafka pipelines (e.g. Kafka Streams, Flink). As soon as a side effect leaves Kafka — sending a push notification, writing to Redis — you're back to at-least-once, and the consumer must be idempotent.

7.1 Idempotent consumers

An operation is idempotent if doing it twice has the same effect as doing it once. Techniques:

• Natural idempotence: upserts keyed by ID (PUT doc 4185… into Elasticsearch), ZADD timeline post_id (a set ignores duplicates), "set status = PUBLISHED".
• Dedup table / processed-set: store each event_id you've handled (with TTL); skip if present. Do the check and the side effect in the same DB transaction when possible.
• Conditional writes / versions: apply an update only if event.version > stored.version — also fixes out-of-order delivery.
• Avoid non-idempotent increments: "likes += 1" on redelivery double-counts. Instead record the like as a row keyed by (post_id, user_id) and derive counts, or dedupe by event ID before incrementing.

def handle_like(evt):                       # at-least-once delivery, idempotent effect
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO likes (post_id, user_id, created_at) VALUES (%s, %s, %s) "
            "ON CONFLICT (post_id, user_id) DO NOTHING", (evt.post_id, evt.user_id, evt.ts)).rowcount
    if inserted:                            # only the first delivery changes the counter
        redis.hincrby(f"post:{evt.post_id}:counts", "likes", 1)
        notify_author_once(evt.event_id)    # notification service dedupes by event_id too

(There is still a tiny window between the DB commit and the Redis increment; counters are periodically reconciled from the likes table — see Likes & Counters.)

8. Dead-letter queues and poison messages

A poison message is one that always fails — malformed JSON, a schema the consumer can't parse, a reference to a deleted user, or one that triggers a bug. In a partitioned log this is particularly nasty: if the consumer keeps retrying, it blocks the whole partition (everything behind it waits). Standard handling:

1. Classify errors: transient (timeout, 503 from Elasticsearch) → retry; permanent (deserialization error, validation failure) → don't retry forever.
2. Retry in place a few times with exponential backoff and jitter (see Resilience Patterns).
3. Retry topics: if still failing, publish the message to post.created.retry.1m, then .retry.10m, consumed by a delayed consumer — the main partition keeps flowing.
4. Dead-letter queue (DLQ): after N attempts, park the message in post.created.dlq with error details in headers (exception, attempts, original partition/offset), commit the offset and move on.
5. Alert and redrive: DLQ depth is an alert. Engineers fix the bug, then replay the DLQ back to the main topic.

Trade-off: moving a message aside breaks per-key ordering (later events for that key get processed first). For order-critical streams, either pause the partition and page someone, or make consumers tolerant (versions/timestamps). SQS and RabbitMQ provide DLQs natively (SQS redrive policy with maxReceiveCount; RabbitMQ dead-letter exchanges); in Kafka it's an application pattern supported by frameworks (Spring Kafka, Kafka Connect).

9. Consumer lag and backpressure

Consumer lag = (log end offset) − (group's committed offset), per partition. It is the single most important health metric for an async pipeline: it tells you how far behind real time a consumer is.

Partition 7 of post.created:  log end offset = 1,250,000   committed (feed-fanout) = 1,190,000
lag = 60,000 messages.  Consume rate 2,000 msg/s, produce rate 1,500 msg/s
→ catching up at 500 msg/s → back to real time in 60,000 / 500 = 120 s
Lag in TIME (age of oldest unprocessed message) is what users feel: "followers see posts ~2 min late".

Lag grows when consumers are slower than producers — traffic spike, slow downstream (Elasticsearch GC pauses), a poison message, or a rebalance storm. Responses:

• Autoscale consumers on lag (e.g. KEDA in Kubernetes) — up to the partition count.
• Batch downstream writes (bulk index 500 docs at a time) and pipeline Redis commands.
• Prioritise: separate topics or consumer pools for urgent vs bulk work (e.g. DMs and normal users' fan-out vs celebrity fan-out, which goes through the pull path anyway).
• Shed or degrade: skip non-essential work under extreme lag (e.g. drop analytics enrichment, sample impressions).

Backpressure is how a system tells upstream "slow down". Kafka consumers pull, so they're never overwhelmed — the backlog simply accumulates in the log (that's the load-levelling buffer, bounded by retention: if lag exceeds retention, data is lost — alert well before!). Producers get backpressure when the broker can't keep up: the client's buffer (buffer.memory) fills and send() blocks or fails, which the service should turn into rate limiting or 503s rather than unbounded memory growth. Push-based brokers (RabbitMQ) use consumer prefetch limits for the same reason.

10. Publishing reliably: the transactional outbox and CDC

The Post service must do two things: write the post to its database and publish post.created to Kafka. These are two different systems with no shared transaction — the dual-write problem:

• DB commit succeeds, then the service crashes before publishing → the post exists but nobody fans it out or indexes it.
• Publish first, then the DB write fails → consumers process a post that doesn't exist.

Transactional outbox: in the same local DB transaction as the business write, insert the event into an outbox table. A separate relay reads the outbox and publishes to Kafka, marking rows as sent. Atomicity comes from the database; delivery is at-least-once (the relay might publish twice), so consumers stay idempotent.

  Post service                      Post DB (one transaction)                 Relay / CDC         Kafka
     | BEGIN                                                                     |                  |
     |-- INSERT INTO posts (...) ------------>|                                  |                  |
     |-- INSERT INTO outbox (event_id, topic, key, payload) -->|                 |                  |
     | COMMIT  (both rows or neither)                                            |                  |
     |                                         | -- commit log / WAL ------->   | (Debezium tails)  |
     |                                                                           |-- publish ------>|
     |                                                                           |   post.created   |

Change Data Capture (CDC) is the best relay: tools like Debezium (running on Kafka Connect) read the database's replication log — PostgreSQL's WAL via logical decoding, MySQL's binlog, Cassandra's commit log/CDC — and emit every committed change as a Kafka event, with no polling and no extra load on tables. Debezium has a built-in "outbox event router" that turns outbox-table inserts into clean domain events on the right topic with the right key. CDC is also how you keep caches, search indexes and data warehouses in sync with a database without the application having to remember to publish (see Caching for CDC-driven cache invalidation).

11. Event schemas and schema evolution

Events outlive the code that produced them: a topic may hold 7 days of messages written by three different versions of the Post service, read by a dozen consumers deployed on their own schedules. Changing an event's shape carelessly breaks consumers at 3 a.m.

• Use a schema'd binary format: Avro (schema stored separately, compact, popular in the Kafka ecosystem) or Protobuf (numbered fields, great codegen) — or JSON Schema if you must use JSON. Binary formats are 3–10× smaller than JSON and enforce types.
• Schema registry: a service (Confluent Schema Registry, Apicurio, AWS Glue) storing every version of each topic's schema. Producers register/lookup the schema and prefix each message with a small schema ID; consumers fetch the writer's schema by ID to decode. The registry rejects incompatible changes at deploy time.
• Compatibility modes: backward (new consumers can read old data — upgrade consumers first), forward (old consumers can read new data — upgrade producers first), full (both). Transitive variants check against all past versions.

// post_created.proto — evolving safely
message PostCreated {
  fixed64 post_id     = 1;
  fixed64 author_id   = 2;
  int64   created_ms  = 3;
  repeated string media_keys = 4;
  string  caption     = 5;
  // v2: added — old consumers ignore unknown field 6; new consumers see "" for old events
  string  location_id = 6;
  // NEVER reuse a removed field's number; mark it reserved instead
  reserved 7;  // was: string filter_name
}

Safe rules of thumb: add optional fields with defaults; never change a field's type or meaning; never reuse field numbers/names; to make big changes, publish a new event version on a new topic (post.created.v2) and run both during migration. Include standard envelope fields in every event: event_id (for dedupe), event_type, occurred_at, producer, schema_version, and a trace ID (see Observability).

12. How Instagram uses message queues

In our architecture diagram, producer services publish domain events to Kafka and five kinds of consumers react: the Search Indexer, Feed Fan-out Workers, Media Processing Workers, the Notification Service, and Analytics & ML Training.

Why author_id as the key for post events? It keeps a user's create/edit/delete events in order (a delete is never processed before its create). Post events are low-rate even for celebrities, so hot partitions aren't a concern. For likes, post_id keeps like/unlike for a post ordered; a viral post can make one partition hot, so the like consumer aggregates in memory and writes batched counter updates.

12.1 Sizing example

All inputs are assumptions for practice (see Back-of-the-Envelope Estimation):

Domain events per day (assume):
  likes 1B + comments 100M + posts 100M + stories 300M + follows 50M + misc 450M ≈ 2B events/day
  average size (Protobuf + envelope) ≈ 1 KB

Throughput:
  2B / 86,400 s ≈ 23K msg/s ≈ 23 MB/s average;  peak ×3 ≈ 70K msg/s ≈ 70 MB/s

Storage (7-day retention, replication factor 3):
  2 TB/day × 7 × 3 ≈ 42 TB  (plus ~30% headroom → ~55 TB)

Broker network/disk:
  writes: 70 MB/s × 3 replicas = 210 MB/s cluster-wide at peak
  reads:  ~5 consumer groups × 70 MB/s = 350 MB/s (mostly from page cache)
  → ~9 brokers with ~6 TB each gives comfortable headroom (a broker can sustain hundreds of MB/s)

Partitions (driven by consumer parallelism, not bytes):
  post.created peaks at only ~3.5K posts/s, but each post fans out to hundreds of timelines;
  assume one fan-out worker handles 50 posts/s → 70 workers → give the topic 96–128 partitions.
  like.created at ~35K/s peak, 2,000/s per consumer → ≥ 18 → choose 64 for headroom and hot-key spread.

Separate cluster for client.events: tens of billions of impressions/day → 100s of MB/s to GB/s.

13. Trade-offs & common pitfalls

Common pitfalls

• Non-idempotent consumers under at-least-once delivery → double notifications, inflated counters.
• Dual writes (DB then publish) without an outbox → lost or phantom events.
• Too few partitions → consumer parallelism capped; adding partitions later reshuffles keys.
• Hot partitions from skewed keys (a celebrity, a viral post) → one consumer lags while others idle.
• Poison messages blocking a partition forever — no retry limit or DLQ.
• Lag exceeding retention → unprocessed events deleted. Alert on lag in time, not just count.
• Huge messages (images in Kafka) — send references (object-storage keys), not blobs ("claim check" pattern).
• Using a queue for request/response where a synchronous call is simpler and the user needs the answer now.
• Breaking schema changes deployed without registry checks.
• Slow processing inside the poll loop exceeding max.poll.interval.ms → endless rebalances.

14. Real-world technologies

15. Interview corner

16. Check your understanding

Related lessons

• News Feed — the fan-out workers consuming post.created
• Media Processing — the worker pools behind media.uploaded
• Notification System — consuming likes, comments and follows to send pushes
• Search — keeping Elasticsearch in sync from events
• Analytics & ML — stream and batch processing of the event firehose
• Microservices — sync vs async communication and data ownership
• Resilience Patterns — retries, backoff and idempotency
• Replication — leader/follower and ISR ideas in general