1. The problem: why are notifications hard?

"Maya liked your photo." It seems trivial: when a like happens, send a push. But consider what Instagram has to get right:

• Volume: billions of likes, comments, follows and mentions per day. Turning every event into a push would mean hundreds of thousands of pushes per second — and furious users.
• Relevance and restraint: a post going viral gets 50,000 likes in an hour. Nobody wants 50,000 pushes. They want one: "maya and 49,999 others liked your post" — and maybe not at 3 a.m.
• Many channels, many rules: push vs in-app vs email vs SMS; per-category user settings; muted accounts; blocked users; OS-level permission; language and time zone.
• External dependencies: the last hop goes through Apple and Google, which have their own APIs, rate limits, error codes and failure modes — and they don't tell you whether the user actually saw anything.
• Correctness under retries: duplicates are very visible ("why did I get this twice?"), but so are losses for important notifications like login codes and DMs.
• Spikes: a celebrity going live can mean "notify 10 million followers now" — and then 10 million people open the app at the same moment.

2. Types of notifications and requirements

2.1 The four channels

A single event may produce output on several channels: a new comment creates an in-app activity item and (if allowed) a push. A login from a new device produces an in-app security alert, a push and an email.

2.2 Functional requirements

• Generate notifications from domain events: like, comment, reply, mention, tag, follow / follow request, DM, live, story mention, new post from a "post notifications on" account, security events.
• Respect per-user preferences per category and channel, muted accounts/threads, blocks, and OS permission.
• Aggregate similar events ("X and 20 others…").
• Render in the recipient's language with correct plurals, names and deep links.
• Enforce rate limits and quiet hours; prioritise urgent notifications.
• Maintain an in-app activity feed with unread counts.
• Track delivery and opens; support A/B experiments.

2.3 Non-functional requirements

• Latency: high-priority (DMs, login codes) within a couple of seconds; social notifications within seconds to minutes (aggregation deliberately delays some).
• Reliability: no loss of critical notifications; occasional loss of a low-priority "like" push is acceptable (it's still in the activity feed).
• No duplicates visible to users.
• Scalability to billions of events per day and multi-million spikes.
• Isolation: notification problems must never slow down posting, liking or the feed — the whole system is downstream of Kafka.

2.4 Back-of-the-envelope (assumptions)

Input events/day (assumptions): likes 4B, comments 500M, follows 300M,
  mentions/tags 200M, DMs 12B (handled mostly in-chat), lives/new-post subscriptions ~1B fan-out
Notification-worthy social events ≈ 5-6B/day → ~65,000/s average, ~200,000/s peak

After preferences, mutes, aggregation, caps:
  In-app activity writes ≈ 3B/day  (~35k/s; aggregated items are UPSERTS, not new rows)
  Push sends          ≈ 2-3B/day (~30k/s average, ~100k/s+ at peak and during spikes)
  Email               ≈ 100M/day; SMS ≈ 10-20M/day (mostly verification codes)

Push payload ≈ 1-2 KB → 3B × 1.5 KB ≈ 4.5 TB/day of outbound API traffic to APNs/FCM
Tracking events (created, sent, opened, dismissed…) ≈ 5× pushes ≈ 15B/day → data warehouse

The headline: aggregation and filtering cut the event stream roughly in half or more before anything is sent, and tracking data is several times larger than the notifications themselves.

3. Architecture overview

Each box is a stage that can be scaled independently, and every stage boundary is a queue or topic (see Message queues & Kafka), so a slow provider never backs up into the services that produce events. Let's walk through the stages.

4. The pipeline, stage by stage

4.1 Ingest: events and recipient resolution

Producer services don't call the Notification Service directly; they publish domain events they would publish anyway — the Like & Comment Service emits like.created, the Follow Service emits follow.created, and so on. The Notification Service is just another consumer group on those Kafka topics.

{ "event_id": "evt_01HF3K9...",          // unique, used for idempotency
  "type": "like.created",
  "actor_id": 4821,                       // maya
  "object": { "type": "post", "id": 3170538912345678, "owner_id": 17 },
  "ts": 1730000000123 }

The first job is recipient resolution — who should hear about this event?

The output is a stream of (recipient, notification candidate) pairs, partitioned by recipient_id so that all of one user's notifications are processed by the same worker — which makes per-user aggregation, rate limiting and ordering simple and lock-free.

4.2 Preferences and mute checks: filter early

The cheapest notification is the one you drop first. Before any expensive work, check:

• Category/channel settings: e.g. likes → push "From people I follow", comments → "From everyone", email digests off. Stored as a small document per user (user_id → settings), read through a cache — this lookup happens for every candidate, so it must be a sub-millisecond cache hit.
• Relationship filters: has the recipient muted or blocked the actor? Is the actor restricted? Is the actor's account flagged as spam? (Spammy accounts generate a lot of notifications; suppressing them protects users.)
• Object state: has the comment already been deleted? Has the like been undone? (Re-checked again at send time for delayed notifications.)
• "Pause all" and per-thread mutes.
• Channel availability: does the user have any registered device tokens with push permission? If not, only the in-app item is created.

Filtering happens per channel: a like might be suppressed for push but still recorded in the activity feed.

4.3 Aggregation and batching

Aggregation turns many similar events into one notification. The aggregation key identifies "the same kind of thing about the same object for the same person":

agg_key = (recipient_id, type, object_id)      e.g. (17, "like", post 3170538912345678)

A common strategy is "first one fast, then batch":

1. The first like on a post in a window → create the activity item and send a push quickly (users like immediate feedback on the first reaction).
2. Further likes within the window (say 15–60 minutes) → update the aggregate (count, latest actors) and the in-app item, but don't push yet.
3. When the window closes — or when the count crosses a milestone (10, 50, 100, 1,000…) — send one aggregated push: "maya, arjun and 48 others liked your post".
4. The aggregated push uses the same collapse ID as the earlier one, so on the device it replaces the previous banner instead of stacking a new one.

# Aggregation state in Redis (one hash per agg_key), processed by the recipient's worker
HINCRBY agg:17:like:3170538912345678  count 1
LPUSH   agg:17:like:3170538912345678:actors 4821
LTRIM   agg:17:like:3170538912345678:actors 0 2        # keep the 3 most recent actors for display
EXPIRE  agg:17:like:3170538912345678  3600

def on_like(recipient, post, actor):
    key = f"agg:{recipient}:like:{post}"
    count = redis.hincrby(key, "count", 1)
    activity_store.upsert(recipient, key, actor, count)          # in-app item, always fresh
    if count == 1:
        enqueue_push(recipient, key, priority="normal")          # first like: send soon
        schedule_flush(key, delay_s=window_for(recipient))       # delay queue / timer
    elif count in MILESTONES:                                    # 10, 50, 100, 1000 ...
        enqueue_push(recipient, key, priority="low")

def on_flush(key):                                               # window closed
    if redis.hget(key, "count") > redis.hget(key, "count_at_last_push"):
        enqueue_push(recipient_of(key), key, priority="low")     # one summary push

• Windows adapt: a user who gets 3 likes a day can get every like quickly; a creator getting 10,000 likes an hour gets milestone summaries. The window can be chosen per user (and tuned by experiments, section 10).
• Different types aggregate differently: likes and follows aggregate heavily; comments aggregate per post but show the comment text of the latest one; DMs and security alerts are never aggregated away.
• Retractions: if a like is undone before the push is sent, decrement the aggregate; if the count drops to zero, cancel the pending push and remove the activity item.

4.4 Template rendering and localization

Notifications are never stored as final text. Each is a structured record (type, actors, count, object, deep link) plus a template ID, rendered as late as possible — at send time for pushes, at read time for the activity feed — so that names, counts and language are current.

# template "like_post_agg" in ICU MessageFormat (handles plurals per language)
en: "{actor1}{count, plural,
        =1 { liked your post.}
        =2 { and {actor2} liked your post.}
        other { and {others, number} others liked your post.}}"
hi: "{actor1}{count, plural, =1 { ने आपकी पोस्ट पसंद की।} other { और {others, number} अन्य लोगों ने आपकी पोस्ट पसंद की।}}"

render(template="like_post_agg", locale="en-IN",
       args={actor1:"maya", actor2:"arjun", count:50, others:48})
→ "maya and 48 others liked your post."

• Locale and plural rules: languages have different plural categories (English has 2 forms, Arabic 6, Japanese 1). Never build sentences by string concatenation; use a localization library with per-language plural rules, number formatting, and right-to-left (RTL) support.
• Recipient's locale, not the actor's, from the user profile/device registry.
• Safe truncation: comment previews are cut on grapheme boundaries (don't split an emoji) and stay within provider limits (APNs and FCM payloads are capped at 4 KB).
• Deep links: the payload carries a URI such as instagram://media?id=3170538912345678&comment=… plus a notification_id for open tracking.
• Rich media: a thumbnail URL (signed CDN URL) that the app's notification extension downloads to show an image in the banner.
• Privacy-aware text: DM previews may be hidden ("New message") depending on settings or E2EE, where the server can't read the content.

4.5 Rate limits, fatigue and quiet hours

Even after aggregation, an active user can receive a lot. Over-notifying has a real cost: users turn off push or uninstall, and a disabled channel is very hard to win back. Guards:

• Per-user caps: e.g. at most N non-critical pushes per hour and M per day (a token bucket per user, stored in Redis). Excess notifications are dropped from push but remain in the in-app feed.
• Per-category caps: e.g. at most one "suggested account" push per day.
• Quiet hours: using the device's time zone, defer low-priority pushes during the night (e.g. 23:00–08:00) to a delay queue; at wake-up, send a single summary instead of the backlog. Critical notifications (login codes, security alerts, DMs per user settings) bypass quiet hours.
• Relevance models: large platforms use ML to predict, for each candidate, the probability of an open versus the probability the user disables notifications, and send only when the expected value is positive. The volume budget per user becomes a learned quantity rather than a fixed constant.
• Global and per-provider limits: protect APNs/FCM quotas and your own backend from self-inflicted spikes (section 7).

4.6 Priority queues

Not all notifications are equal, and they must not share a single queue. If a celebrity's post generates 5 million low-value pushes, a login code queued behind them would arrive 10 minutes late. So the pipeline uses separate queues (Kafka topics or queue partitions) per priority, each with dedicated worker capacity — a bulkhead (see Resilience patterns).

Workers pull from the high-priority queue first (strict priority) or use weighted scheduling so that low-priority work still progresses. A delay queue (e.g. a sorted set keyed by send-time, or a scheduler service) holds deferred items for quiet hours and aggregation windows.

4.7 Delivery workers and the push providers

Delivery workers take a rendered notification, look up the recipient's device tokens, call the right provider for each device, and report outcomes. You can't push to a phone directly — the OS vendors run the only connection that can wake a sleeping device:

# APNs request (HTTP/2) — one stream per notification, many streams per connection
POST /3/device/<device_token>
authorization: bearer <JWT signed with the app's .p8 key>
apns-topic: com.example.instagram
apns-push-type: alert
apns-priority: 10                       # 10 = immediate, 5 = power-friendly
apns-expiration: 1730003600             # drop if not deliverable by then
apns-collapse-id: like:3170538912345678 # newer push with same id replaces older
apns-id: 6f1c2d4e-...                   # our notification id (UUID)
{ "aps": { "alert": { "title": "Instagram", "body": "maya and 48 others liked your post." },
           "badge": 7, "sound": "default", "thread-id": "post-3170538912345678",
           "mutable-content": 1 },
  "deeplink": "instagram://media?id=3170538912345678", "nid": "n_8e1f..." }

# Responses the worker must handle
200                      → accepted by APNs (NOT the same as displayed!)
400 BadDeviceToken       → token malformed / wrong environment → delete token
410 Unregistered         → app uninstalled / token invalid      → delete token
429 TooManyRequests      → back off and retry
500 / 503                → provider trouble → retry with exponential backoff

• Connections: APNs uses HTTP/2, so one TLS connection multiplexes many concurrent requests; workers keep pools of long-lived connections (see Networking basics). FCM's HTTP v1 API is similar (OAuth2 access token, JSON body; UNREGISTERED error for dead tokens; collapse_key / android.ttl for collapsing and expiry).
• "Accepted" ≠ "delivered" ≠ "seen": providers only confirm they accepted the message. They store-and-forward a limited number per device while it's offline (APNs keeps only the most recent one per app; FCM keeps a bounded number), and may collapse or drop. Real delivery confirmation requires the app to report back (section 8).
• Silent / data pushes (no banner) can wake the app to sync content — e.g. pre-fetch the DM before the user opens it — but the OS throttles them heavily, so never rely on them.
• Email and SMS workers follow the same pattern with different providers (an email delivery API; an SMS aggregator), with extra concerns: email bounce/complaint handling and unsubscribe links; SMS cost control and fraud ("SMS pumping" attacks that trigger codes to premium numbers — rate-limit and verify).

5. The device token registry

A device token is an opaque string issued by APNs/FCM to one installation of the app on one device; it's the "address" for push. Users have several devices, tokens change, and apps get uninstalled — so tokens need a well-maintained registry.

device_tokens  (partition key: user_id)
  user_id      bigint
  device_id    text        -- stable per installation (generated by the app)
  platform     text        -- ios | android | web
  token        text        -- APNs / FCM token (unique index token → user for reassignment)
  environment  text        -- production | sandbox (APNs)
  app_version  text
  locale       text        -- e.g. "hi-IN"   (used for rendering)
  timezone     text        -- e.g. "Asia/Kolkata" (used for quiet hours)
  push_enabled boolean     -- OS permission as last reported by the app
  last_seen_at timestamp
  PRIMARY KEY (user_id, device_id)

5.1 Token lifecycle

1. Register: on every app start (and when the OS reports a new token), the app calls PUT /devices/{device_id} with its current token, permission state, locale and time zone. Upsert is idempotent.
2. Rotate: tokens can change after OS updates, app reinstalls or restores; the newest registration for a device_id wins.
3. Reassign: if user A logs out and user B logs in on the same phone, the same token now belongs to B. The unique token index lets registration remove it from A — otherwise A's notifications would appear on B's phone, a privacy bug.
4. Logout: delete the device's token row (and tell the app to stop showing cached notifications).
5. Prune on feedback: when APNs answers 410 Unregistered/BadDeviceToken or FCM answers UNREGISTERED, the delivery worker emits a token.invalid event and the registry deletes that token.
6. Prune stale tokens: a periodic job removes tokens not seen for, say, 60–90 days — they almost certainly belong to abandoned installs and waste send capacity.

Why pruning matters: without it, a growing fraction of every send goes to dead tokens. If 30% of tokens are dead, 30% of provider calls and worker capacity are wasted, and providers may view high invalid-token rates as abusive. Reads of the registry are a single-partition lookup per recipient (cached), so it scales simply by sharding on user_id.

6. Reliability: retries, dedup and ordering

6.1 Retries with backoff

Failures happen at every step: a worker crashes mid-batch, APNs returns 503, a network blip times out a request. The rules (see Resilience patterns):

• Retry transient errors only (timeouts, 429, 5xx). Never retry permanent errors (410, 400 BadDeviceToken, payload too large) — prune or drop and log.
• Exponential backoff with jitter: wait ~1 s, 2 s, 4 s, 8 s… with randomisation, so thousands of workers don't retry in lockstep against a recovering provider. Honour Retry-After headers.
• Bounded attempts and expiry: after N attempts or once the notification's expiry passes, give up and move it to a dead-letter queue for inspection. A "like" from 3 hours ago is not worth delivering late.
• Circuit breaker per provider: if FCM is failing broadly, stop hammering it, buffer high-priority items, shed low-priority ones.
• Fallback channels for critical messages: if a login-code push isn't confirmed quickly, offer SMS/email.

6.2 Deduplication with idempotency keys

Kafka consumers are at-least-once: after a crash, a worker reprocesses events whose offsets weren't committed, and a retry after a lost provider response may resend something that was actually delivered. Users must not get the same push twice. Layers of dedup:

1. Event-level: the Notification Service records processed event_ids (or makes processing naturally idempotent — e.g. adding actor 4821 to an aggregate set twice has no effect).
2. Notification-level idempotency key: each outgoing notification gets a deterministic ID, e.g. nid = hash(recipient, type, object_id, aggregation_window, version). Before sending, the worker does SET sent:{nid} 1 NX EX 86400 in Redis; if the key already exists, the send is skipped. Deterministic means a reprocessed event produces the same nid, so the check catches it.
3. Device-level: the same apns-collapse-id / FCM collapse_key makes a duplicate replace rather than stack; the app can also ignore an nid it has already displayed.

6.3 Ordering

Global ordering of notifications is neither possible (providers don't guarantee order) nor needed. What matters:

• Per-user processing order: partitioning by recipient_id keeps one user's events in order through the pipeline, which makes aggregation and caps consistent.
• Display order: the activity feed sorts by the item's last updated time, so an aggregate that gained a like moves to the top. The app sorts pushes in its notification center by timestamp, and collapse IDs ensure the newest version of an aggregate wins even if an older push arrives late.
• Causality checks: "maya commented" must not be sent if a later "comment deleted" event was already processed — hence re-validation of object state at send time, rather than trusting event order.

7. Celebrity events: millions of notifications at once

Two different celebrity problems hit the notification system, one on each side.

7.1 Outbound: a celebrity posts or goes live

Followers can turn on "post notifications" or "live notifications" for an account. For an account with 400M followers, assume 2% did → 8 million pushes for one event. Doing this naively has three failure modes: it floods the shared queues (delaying everyone's DMs), it can exceed provider throughput, and — most dangerously — it makes millions of people open the app in the same minute, all requesting the same post, the feed, the profile and the media: a self-inflicted thundering herd.

1. Maintain a subscriber list per account (users who enabled post/live notifications) instead of scanning all 400M followers and checking each one's settings.
2. Create a fan-out job on the bulk queue, never on the normal/high queues. The job pages through subscribers (e.g. 10,000 per page) and emits per-recipient tasks.
3. Render once per locale, not once per recipient: group recipients by locale and reuse the payload ("taylor just went live" in ~50 languages).
4. Throttle with a token bucket per job (e.g. 50,000 sends/s) and globally across concurrent jobs.
5. Stagger and prioritise: send first to the most engaged subscribers (most likely to open; for live video, to start the audience quickly), and spread the rest over several minutes; respect quiet hours per time zone.
6. Prepare the backend: pre-warm the CDN with the post's media and cache the post object before the first push goes out.
7. Re-check before each page: if the post was deleted or the live ended mid-fan-out, stop the job.

8M recipients at a throttle of 50,000/s  → 160 s to send everything
If 10% open within ~2 minutes of receiving:
   unstaggered → 800k app opens in ~2 min ≈ 6,700 opens/s extra, each = feed + post + media requests
   staggered over 10 min → ~1,300 opens/s extra — absorbed by normal headroom

FCM also offers topic messaging (devices subscribe to a topic; one API call fans out inside Google), which offloads fan-out for broadcast-style messages, at the cost of less per-user control (no per-user caps, personalisation or tracking granularity). APNs has no equivalent for regular alerts, so each device is a separate (cheap, multiplexed) HTTP/2 request.

7.2 Inbound: a celebrity receives a million likes

When a celebrity's post gets 1M likes in 10 minutes, all those events target one recipient. Because the pipeline partitions by recipient_id, they all land on one Kafka partition and one worker — a hot partition (see hot keys). Fixes:

• Pre-aggregate upstream: a stream processor counts likes per (post, minute) before the notification stage, so the Notification Service sees "+1,700 likes in the last minute" instead of 1,700 events.
• Different rules for very large accounts: no per-like processing at all; periodic summaries ("Your post has 1M likes") and activity items only from accounts they follow or interact with.
• Sharded counters for the aggregate itself (see Likes & counters).

8. Delivery and open tracking

You can't improve what you don't measure, and providers tell you little. The system emits a tracking event at each step, keyed by nid:

All events go to Kafka and then to the data warehouse and real-time dashboards (see Analytics and Observability). Key metrics: delivery rate (delivered/sent — exposes dead tokens and provider issues), open rate (opened/delivered, per type), end-to-end latency (event ts → provider accepted, per priority), suppression breakdown, and opt-out rate. These same events are training data for the relevance/fatigue models in section 4.5.

9. The in-app activity feed

The Activity tab is a per-user, reverse-chronological list of aggregated items with an unread badge — structurally a tiny news feed with only one writer per item (the notification pipeline) and one reader (the user).

-- Item details: one row per aggregation key; UPSERTed as new actors arrive
CREATE TABLE activity_items (
  user_id     bigint,
  agg_key     text,          -- "like:3170538912345678", "follow:2024-10-23", "comment:...:c991"
  type        text,
  actors      list<bigint>,  -- latest 3 actors for display
  actor_count int,
  object_ref  text,          -- post / comment / user being referenced
  created_at  timestamp,
  updated_at  timestamp,
  PRIMARY KEY (user_id, agg_key)
) WITH default_time_to_live = 7776000;      -- 90 days

-- Time index: append a row each time an item is updated; read newest-first
CREATE TABLE activity_timeline (
  user_id    bigint,
  updated_at timestamp,
  agg_key    text,
  PRIMARY KEY (user_id, updated_at, agg_key)
) WITH CLUSTERING ORDER BY (updated_at DESC) AND default_time_to_live = 7776000;

• Write: upsert the item; append a timeline row. A bumped aggregate leaves an older timeline row behind; the read path dedups by agg_key (keep the newest), and old rows expire via TTL. (Alternatively keep the recent window in a Redis sorted set with score = updated_at, where re-adding a member just moves it.)
• Read: page through activity_timeline with a cursor (updated_at < last), multi-get the items, hydrate actor names/avatars and post thumbnails, render templates in the viewer's locale, and group into "New / Today / This week / This month".
• Unread badge: store last_seen_activity_at per user; unread = number of items updated after it (capped, e.g. "99+"), maintained as a counter incremented on new items and reset when the tab is opened. The badge number in push payloads comes from the same counter.
• Retention: 30–90 days of activity is plenty; TTLs keep storage bounded. Rough size: 3B upserts/day of ~200 B ≈ 600 GB/day, but with TTL and aggregation the live dataset stays at tens of TB — a moderate Cassandra cluster partitioned by user_id.
• Visibility at read time: drop items whose actor is now blocked or whose object was deleted — same "derived data is not the source of truth" rule as the feed.

10. Experimentation

Almost every parameter in this pipeline is a product decision with uncertain effects: How long should the like-aggregation window be? Which copy gets more opens: "maya liked your post" or "maya liked your photo from Goa"? Should we push "suggested account" notifications at all? Notification systems are therefore built for A/B testing:

• Assignment: users are deterministically bucketed (hash(user_id, experiment) mod 100), so the same user always sees the same variant across stages.
• Hook points: preference defaults, aggregation windows, caps, ML thresholds, template text, send-time (immediately vs batched), channel choice. Each stage reads the user's variant from a config/experiment service.
• Metrics: open rate and sessions are the obvious success metrics, but the guardrails matter more — notification disable rate, uninstall rate, spam reports, long-term retention. A variant that lifts opens 5% but doubles opt-outs is a loss.
• Long-term holdouts: a small group receiving fewer notifications for months reveals fatigue effects short tests miss.
• Network effects: notifications cause actions that cause notifications for others (B gets notified, comments, which notifies C). Treating users as independent can mislead; cluster-based randomisation is sometimes used.

11. End-to-end sequence: a like becomes a push

 Maya app   Like Svc   Kafka      Notif Svc    Prefs    Aggregator   Activity   Renderer +   Priority   Delivery   Token     APNs     Raj's iPhone
    |  like   |          |        (consumer)   cache    (Redis)       store      rate limit   queue      worker     registry
    |-------->|-- write like, publish like.created ------->|            |            |            |          |          |         |           |
    |<-- 200 -|          |-- event (partition = recipient 17 = Raj) -> |            |            |          |          |         |           |
    |         |          |          |-- likes push? muted maya? ------->|            |            |          |          |         |           |
    |         |          |          |<-- allowed --|          |          |            |            |          |          |         |           |
    |         |          |          |-- add actor to agg:17:like:P ---->|            |            |          |          |         |           |
    |         |          |          |<-- count = 1 (first like) --------|            |            |          |          |         |           |
    |         |          |          |-- upsert in-app item --------------------------->|          |          |          |         |           |
    |         |          |          |-- render "maya liked your post." (Raj's locale) ----------->|          |          |         |           |
    |         |          |          |   check per-user cap, quiet hours (Raj's timezone) → OK     |          |          |         |           |
    |         |          |          |-- enqueue {nid, priority: normal} --------------------------->|          |          |         |           |
    |         |          |          |   schedule flush of agg:17:like:P in 30 min                  |-- pull -->|          |         |           |
    |         |          |          |          |              |             |            |          |  SETNX sent:nid → OK |         |           |
    |         |          |          |          |              |             |            |          |-- tokens(17) ------>|         |           |
    |         |          |          |          |              |             |            |          |<-- 2 devices --------|         |           |
    |         |          |          |          |              |             |            |          |-- POST /3/device/… (HTTP/2) ---->|        |
    |         |          |          |          |              |             |            |          |<-- 200 (device 1) / 410 (device 2) |        |
    |         |          |          |          |              |             |            |          |-- token.invalid(device 2) → prune  |        |
    |         |          |          |          |              |             |            |          |-- tracking: sent ──► Kafka         |--push->|
    |         |          |          |          |              |             |            |          |          |          |  Raj taps banner → app opens
    |         |          |          |          |              |             |            |          |          |          |  deep link → post; tracking: opened(nid)
    |   ... 30 min later, 48 more likes arrived: flush → ONE push "maya and 48 others liked your post." (same collapse id → replaces banner)

12. How Instagram uses the notification system

In the main architecture diagram, the Notification Service is a consumer of the Kafka event bus, alongside the Search Indexer, Feed Fan-out Workers, Media Processing Workers and Analytics. It sends through Push Providers (APNs/FCM) to phones. Concretely:

1. The Like & Comment, Follow/Graph, Upload, Stories and Direct Messaging services publish events; none of them know notifications exist — adding a new notification type means adding a consumer rule, not changing producers.
2. The Notification Service resolves recipients, checks preferences and relationships, aggregates, writes the in-app Activity item, renders in the recipient's language, applies caps and quiet hours, and queues by priority.
3. Delivery workers fetch tokens from the device registry and call APNs/FCM; invalid tokens are pruned; outcomes go back to Kafka.
4. When a user taps a push, the app deep-links to the post/comment/profile, loading data through the normal API path and media through the CDN.
5. Tracking data feeds Analytics & ML, which trains models deciding which notifications are worth sending.

13. Trade-offs & common pitfalls

Common pitfalls

• Sending pushes synchronously in the request path — a slow APNs call makes liking slow. Always go through a queue.
• One queue for everything — a bulk fan-out delays login codes. Separate by priority.
• No aggregation — 500 pushes for 500 likes; users disable notifications forever.
• Building text by concatenation — broken plurals and grammar in most languages; use ICU-style templates.
• Never pruning tokens — wasted capacity, misleading metrics, provider warnings.
• Not handling token reassignment on logout/login — notifications for one account appear on another person's phone.
• Treating "200 from APNs" as delivered — measure delivery and opens from the app.
• Retrying permanent errors or retrying without backoff/jitter — hammering a struggling provider.
• Sending stale notifications — no expiry or re-validation, so users get "live now" for an ended stream or a deleted comment.
• Celebrity fan-out without staggering — the notification system DDoSes your own backend through app opens.

14. Real-world technologies

15. Interview corner

16. Check your understanding

Related lessons

• Message Queues, Pub/Sub & Kafka — the event bus, partitions and delivery guarantees the pipeline depends on
• News Feed — the same fan-out trade-offs, and the in-app activity feed's bigger sibling
• Real-time Direct Messaging — pushes for offline DM recipients
• Rate Limiting — token buckets used for per-user caps and fan-out throttling
• Resilience patterns — retries, backoff, circuit breakers and bulkheads
• Likes, Comments & Counters — the events and hot-key counters behind most notifications
• Analytics & ML — tracking data, relevance/fatigue models and experiments
• Caching — Redis for preferences, aggregation state and idempotency keys