1. The problem: why does Instagram need data pipelines and ML?

Everything you have seen so far in this guide — uploads, feeds, likes, messages — is about serving users right now. But a product with billions of users also needs to learn from what those users do:

• "Is the product healthy?" How many people opened the app today? Are new users still around a month later?
• "Did our change help?" Does a bigger like button increase engagement or just annoy people? You need a controlled experiment on millions of users.
• "What should each person see?" Users follow hundreds of accounts that post far more than anyone can scroll, and Explore must pick a few dozen posts out of billions. Pure time order buries your best friend's wedding photo under 40 meme posts.

The live databases can't answer these. The User DB and the Cassandra post store fetch one user's rows in a millisecond; "average session length per country over 90 days" would scan every row for hours and slow the app for everyone. So Instagram builds a parallel world: an analytics and ML platform fed by a copy of everything that happens. In the main diagram this is the Analytics & ML Training consumer on Kafka, whose trained ranking models flow back into the News Feed Service and Explore.

2. Event logging: the raw material

2.1 What is an event?

An event is an immutable record that "something happened at time T". A database row holds current state ("post 42 has 1,203 likes"); events hold history ("user 7 liked post 42 at 10:03:11"). History can rebuild state, never the reverse — which is why analytics systems love events.

{
  "event_id":    "8b1f0c2e-6a1d-4c3e-9e7a-2f6b1d9c0a11",   // UUID → lets consumers de-duplicate
  "event_name":  "feed_media_impression",
  "event_time":  "2026-10-23T09:12:44.318Z",              // when it happened on the device
  "server_time": "2026-10-23T09:12:47.902Z",              // when the server received it
  "user_id":     184467440737,
  "device":      {"os": "android", "app_version": "312.0.0", "country": "IN"},
  "session_id":  "s-55f1…",
  "props":       {"media_id": 3192…, "author_id": 9921…, "position": 4,
                  "surface": "home_feed", "view_ms": 2350, "experiment_ids": [881, 1204]}
}

2.2 Client events vs service events

2.3 How client events travel

• Batch and compress on the device. One HTTP request per scroll event would drain batteries and flood the load balancers; apps buffer events in a local file and upload batches.
• Persist before sending. Failed uploads stay on disk and are retried — so the pipeline is at-least-once, and event_id lets downstream jobs drop duplicates.
• Two timestamps. Analyse by event_time, but device clocks can be wrong by hours; server_time lets you detect skew. Events can arrive days late (a phone offline on a flight) — see watermarks below.
• Schemas. Every event type has a declared schema (Avro/Protobuf/Thrift) in a registry; malformed events are quarantined so one buggy app version can't poison every table. Evolution rule: only add optional fields, never change a field's meaning.
• Sampling. Ultra-high-volume, low-value telemetry may be sampled at 1% (multiply counts by 100); business-critical events are never sampled.
• Privacy at the source. Log IDs, not content; never log passwords or message bodies; tag fields by sensitivity for deletion/retention (see Security & Privacy).

Rough volume (assumption): 500M daily users × ~1,000 events each (impressions are the bulk) = 500 billion events/day. At ~200 bytes each after compression that's ~100 TB/day of raw logs, or ~1.2 GB/s on average. This is why the transport is Kafka-style partitioned logs and the storage is cheap object storage, not a relational database.

3. OLTP vs OLAP and columnar storage

3.1 Two very different workloads

Running analytics on OLTP databases is a classic mistake: one analyst's full-table scan evicts the hot working set from the buffer cache and makes the live app slow. Keep them separate and copy data across (events, CDC, snapshots). See Databases and Indexing & Storage Engines for the OLTP side.

3.2 Row vs column layout

Imagine a impressions table with columns (user_id, media_id, country, view_ms, ts). A row store keeps each row's fields together on disk; a column store keeps each column's values together:

ROW-ORIENTED (good for "give me row 2")
  [u1, m9, IN, 2350, t1] [u2, m4, BR, 800, t2] [u3, m9, IN, 4100, t3] ...

COLUMN-ORIENTED (good for "average view_ms by country")
  user_id : u1   u2   u3   u4   ...
  media_id: m9   m4   m9   m7   ...
  country : IN   BR   IN   IN   ...   → dictionary + run-length: {IN:0,BR:1} 0 1 0 0 ...
  view_ms : 2350 800  4100 1200 ...   → delta / bit-packed integers
  ts      : t1   t2   t3   t4   ...

Why columns win for analytics:

1. Read only what you need. 2 of 5 columns here; real tables have 100+ columns, so savings are often 95%+.
2. Dramatic compression. Similar, repeated values compress 5–20× with dictionary, run-length and bit-packing encodings.
3. Skip whole blocks. Per-block min/max statistics let day = '2026-10-22' skip non-matching blocks.
4. Vectorised execution. SIMD over tight arrays of one type is far faster than walking mixed rows.

Worked example (assumption): 1 trillion impression rows × 120 columns × ~8 bytes ≈ 960 TB raw. A query needs 3 columns → 24 TB; with 8× compression → 3 TB; with partition pruning to one day of 90 retained → ~33 GB actually read. Spread across 1,000 workers that's 33 MB each — seconds, not days. The cost: writing or updating a single row is awkward (you'd touch 120 column files), which is why columnar stores are append-heavy and batch-loaded.

4. Data lake vs data warehouse (and the lakehouse)

The modern compromise is the lakehouse: Parquet on object storage plus a table format (Apache Iceberg, Delta Lake, Hudi) adding ACID commits, schema evolution, time travel and row-level deletes (essential for GDPR deletion), readable by Spark, Trino and Flink alike.

Partitioning is the key lake design choice: /impressions/ds=2026-10-22/hour=09/part-0001.parquet lets date-filtered queries read only matching directories. Avoid too many tiny files (the "small files problem"); compaction jobs merge them. A common layering is raw/bronze (as received) → clean/silver (deduplicated, validated, bots filtered) → aggregated/gold (daily metrics, feature tables, experiment results).

5. Batch processing

5.1 MapReduce intuition

Problem: count likes per post for yesterday from 20 TB of like events. One machine reading at 200 MB/s would take ~28 hours. Split the work across 1,000 machines and it takes under 2 minutes of reading — but you must combine partial counts correctly. MapReduce (Google, 2004; open-sourced as Hadoop) gave a simple recipe:

1. Map: each worker reads a chunk of input and emits (key, value) pairs — here (post_id, 1) per like.
2. Shuffle: the framework routes all pairs with the same key to the same reducer, by hash(key) mod R — the same idea as hash sharding. This network-heavy step is usually the bottleneck.
3. Reduce: each reducer sums the values for its keys and writes output files.

map(like_event):
    emit(like_event.post_id, 1)

combine(post_id, counts):          # optional local pre-aggregation on the mapper
    emit(post_id, sum(counts))     # sends (p42, 917) instead of 917 × (p42, 1)

reduce(post_id, counts):
    emit(post_id, sum(counts))

Workers are stateless and inputs immutable, so failed tasks are simply re-run elsewhere — free fault tolerance. The weakness is skew: a celebrity post with 30M likes sends 30M pairs to one reducer, the straggler for the whole job. Fixes: combiners, "salting" hot keys (p42#0…p42#9, then re-aggregate), broadcast joins.

5.2 The modern batch engines

A typical daily pipeline is a DAG run by a scheduler such as Airflow: wait for yesterday's partitions → clean → join dimensions → compute metrics, experiment results, training sets. Each step is idempotent (re-running ds=2026-10-22 overwrites that partition), making backfills after bug fixes safe.

6. Stream processing

Batch answers arrive hours later. Some questions can't wait: "Is this post going viral?" (pre-warm the CDN), "Has this account liked 500 posts in 2 minutes?" (spam), trending hashtags for Search, or fresh interest features for the next feed refresh. Stream processors consume Kafka topics continuously and update results incrementally.

6.1 Engines

• Apache Flink — distributed stream processor with large managed state (RocksDB), event-time windows, watermarks and exactly-once state via checkpoints. The usual choice for serious real-time pipelines.
• Kafka Streams — a Java library embedded in your service, state in local RocksDB backed by Kafka changelog topics. Simple when input and output are both Kafka.
• Spark Structured Streaming — micro-batches every few hundred ms to seconds with the Spark API.

6.2 Event time vs processing time

Event time = when it happened on the device; processing time = when the processor sees it. Counting likes per minute by processing time, a 5-minute consumer lag looks like likes dropping to zero then spiking — a lie. Group by event time — but how long do you wait for stragglers?

6.3 Windows

time ─────────────────────────────────────────────────────────────►
events:   x  x   x x     x        x  x x x            x     x

TUMBLING (fixed, non-overlapping, e.g. 1 min)
          |── w1 ──|── w2 ──|── w3 ──|── w4 ──|── w5 ──|

SLIDING / HOPPING (size 2 min, slide 1 min → overlapping)
          |──── w1 ────|
                   |──── w2 ────|
                            |──── w3 ────|

SESSION (gap-based: window closes after 30 s of inactivity per user)
          |─ s1 ──────|       |─ s2 ───────|        |─ s3 ──────|

• Tumbling: "likes per post per minute" — each event belongs to exactly one window.
• Sliding/hopping: "likes in the last 10 minutes, updated every minute" — smooth trending scores.
• Session: "how long was each app session?" — windows are defined by gaps in a user's activity.

6.4 Watermarks and late data

A watermark is the stream processor's declaration: "I believe I have now seen all events with event time ≤ W." It typically trails the maximum observed event time by an allowed lateness, e.g. W = max_event_time_seen − 2 minutes. When the watermark passes the end of a window, the window is closed and its result emitted.

 window [09:00, 09:01)        watermark = max event time − 2 min
 ─────────────────────────────────────────────────────────────────
 09:00:10 event (on time)          → added to window
 09:02:30 max seen → W = 09:00:30  → window still open
 09:03:05 max seen → W = 09:01:05  → W ≥ 09:01 → EMIT count, close window
 09:03:20 a 09:00:40 event arrives → LATE: drop, send to side output,
                                     or update result if "allowed lateness"

The trade-off is explicit: a larger watermark delay gives more complete results but higher latency; a smaller one gives fast but less complete results. Very late events (hours) are typically left to the batch layer, which recomputes exact numbers later.

6.5 Exactly-once (really "effectively once")

Kafka delivers at-least-once. Flink achieves exactly-once state by checkpointing operator state together with the matching Kafka offsets and rewinding both on failure. External outputs also need transactional sinks or idempotent writes (SET count[post][minute] = 917, not INCR) — see Resilience.

7. Lambda vs Kappa architectures

Most large companies run hybrids: streaming for fresh features and counters, batch for canonical daily metrics, training and backfills.

8. ETL vs ELT

Instagram-like systems mostly do ELT into the lake — but drop or pseudonymise unneeded personal fields at ingestion (a little "T" before the "L").

9. Product metrics: measuring health

9.1 Active users and stickiness

• DAU (daily active users): distinct users who did a precisely defined qualifying action on a day ("opened the app" vs "viewed a post" can differ by tens of millions).
• MAU (monthly active users): distinct users active at least once in the trailing 28/30 days.
• Stickiness = DAU / MAU. Assume DAU = 500M and MAU = 800M → 62.5%: the average monthly user shows up ~19 of 30 days. Social apps aim for 50%+.

Exact distinct counts come from daily batch jobs; real-time dashboards use HyperLogLog sketches (~12 KB each, ~1–2% error, mergeable across days).

9.2 Retention and cohorts

Retention asks "of the users who signed up (or were active) in period X, what fraction is still active N days later?" Group users into cohorts by signup week and track each cohort:

A flattening curve (D30 ≈ D60 ≈ D90) means the product has found users who stay. Cohorts prevent a classic trap: total DAU can grow (marketing brings new users) while every cohort retains worse than the last. Other common metrics: time spent, sessions per user, % of users who post (the supply side of a social network), and crash-free sessions (see Observability).

Teams usually pick a north-star metric that captures long-term value (e.g. "meaningful daily engagement") plus guardrail metrics that must not get worse (crash rate, reports of bad content, latency, unfollows). A change that increases time spent by showing more outrage-bait but also increases "hide"/"report" rates fails its guardrails.

10. A/B testing and experimentation

An A/B test (online controlled experiment) randomly splits users into a control group (current product, A) and one or more treatment groups (B). Because assignment is random, the only systematic difference between groups is the change — so a metric difference beyond random noise is caused by the change. This is how you answer "did the bigger like button help?" without guessing.

10.1 Assigning users

def assign(user_id, experiment):
    # deterministic: same user always lands in the same bucket, no DB lookup needed
    bucket = murmurhash(f"{experiment.salt}:{user_id}") % 1000      # 0..999
    if bucket < experiment.control_buckets:   return "control"      # e.g. 0..49   = 5%
    if bucket < experiment.treatment_buckets: return "treatment"    # e.g. 50..99  = 5%
    return "not_in_experiment"

• Randomisation unit: usually the user (consistent experience across sessions), not the request.
• Per-experiment salt: experiments shuffle users independently, so many can run on overlapping users ("layers").
• Exposure logging: log when the user actually encounters the change (the experiment_ids field above) and join metrics to exposures.

10.2 Reading the result: significance, power, sample size

Suppose control users comment on 10.0% of days and treatment users on 10.1%. Is +1% relative real, or noise? Statistics answers with a p-value: the probability of seeing a difference at least this big if the change truly did nothing. By convention p < 0.05 is "statistically significant". Power (often 80%) is the probability you detect a real effect of a given size.

A handy rule of thumb for 80% power at α = 0.05 is n per group ≈ 16 × σ² / δ², where σ² is the metric's variance and δ the absolute effect you want to detect. For a yes/no metric with p = 0.10, σ² = p(1−p) = 0.09:

Detect δ = 0.001 (10.0% → 10.1%, a 1% relative lift):
  n ≈ 16 × 0.09 / 0.001² = 1,440,000 users per group
Detect δ = 0.0001 (0.1% relative lift):
  n ≈ 144,000,000 users per group   ← 100× more, because n ∝ 1/δ²

That 1/δ² is why Instagram-scale companies can measure tiny improvements that a startup never could — and why even they need variance-reduction techniques like CUPED (using each user's pre-experiment behaviour to cancel out noise).

10.3 Experimentation pitfalls

• Peeking: stopping the moment p < 0.05 inflates false positives. Fix the duration up front (or use sequential tests).
• Multiple comparisons: test 40 metrics and ~2 will be "significant" by chance. Pre-register a primary metric.
• Novelty effects: users click a shiny new button for a week, then stop. Run 2+ weeks and check the trend.
• Sample ratio mismatch (SRM): configured 50/50, got 50.6/49.4 over millions of users → something (crash, logging bug) drops users from one arm; results are untrustworthy.
• Network effects: treatment users interact with control users (more comments sent → control receives more), shrinking the measured difference. Fix with cluster randomisation (friend-clusters or countries).
• Holdouts: keep ~1% of users without a quarter's launches to measure the cumulative long-term effect.

11. ML ranking: the multi-stage pipeline

For each feed request, the News Feed Service has a candidate pool of perhaps a few thousand unseen posts from accounts the user follows (plus, today, suggested posts). For Explore and Reels, the pool is effectively all public content — billions of items. Running a large neural network that takes ~1 ms of compute per item over billions of items for every request is impossible. The solution is a funnel: cheap methods shrink the pool, expensive methods order the survivors.

11.1 Stage 1 — Candidate generation (retrieval)

Goal: high recall cheaply — don't miss good items; precision comes later. Typically several sources run in parallel and their results are unioned:

• Graph-based: recent posts from accounts you follow (read from your precomputed timeline in the Redis feed cache, plus pulled celebrity posts — see fan-out); posts liked by accounts similar to ones you engage with.
• Embedding-based: nearest neighbours of your user embedding in item-embedding space (section 12).
• Heuristic: trending in your country, popular in topics you like, fresh posts needing exposure (cold start).

11.2 Stage 2 — Lightweight ranking

A small model (logistic regression, a small gradient-boosted tree, or a distilled neural net trained to mimic the heavy model) scores thousands of candidates using a handful of cheap, precomputed features and keeps the top few hundred. Its job is to be roughly right, very fast.

11.3 Stage 3 — Heavy ranking

A large deep neural network with thousands of features — often a multi-task model that predicts several probabilities at once for each (user, post) pair:

P(like), P(comment), P(share), P(save), P(tap profile), E[watch time], P(follow author),
P(hide / "not interested"), P(report)

These predictions are combined by a value model — a weighted formula that encodes what the product wants to optimise:

score = w1·P(like) + w2·P(comment) + w3·P(share) + w4·P(save)
      + w5·E[watch_time] − w6·P(hide) − w7·P(report)

The weights are a product decision tuned by A/B tests. Predicting components separately makes the system debuggable and lets you change goals (value comments and shares over passive likes) without retraining.

11.4 Stage 4 — Re-ranking for diversity, integrity and business rules

Sorting purely by score produces bad pages — five carousels from the same meme account in a row, or three near-duplicate posts about the same news. The final stage applies list-level logic:

• Diversity: no more than N consecutive items from one author or one topic; mix photos, carousels and videos; techniques include MMR (maximal marginal relevance) — penalise items too similar to those already chosen.
• Integrity: demote or remove content that classifiers flag as borderline (misinformation, spam, clickbait), content from accounts under enforcement, or items the user has already seen.
• Business rules: insert ads at allowed slots (with their own auction), boost "close friends" or new posts from accounts you frequently interact with, respect user controls (muted accounts, "not interested", sensitive-content settings).

11.5 Features: what the models look at

A feature is a numeric input to a model. Ranking quality comes far more from good features than from clever model architectures.

Freshness varies: "likes in the last 5 minutes" needs stream jobs; "90-day watch completion" can be daily batch. Feature computation is often the most expensive part of the system.

11.6 The feature store

A feature store computes, stores and serves features consistently for training and serving: an offline store (full timestamped history in the lake, for training sets) and an online store (latest value per user/post/user×author in a low-latency KV store such as Redis or Cassandra, read in a few ms). It prevents two notorious bugs:

1. Training–serving skew: training computes "likes in last 7 days" in Spark SQL, serving computes it in C++ slightly differently (time zones, dedup, defaults) — the model quietly underperforms. Define each feature once and materialise it to both stores.
2. Label leakage: a training example "did the user like post P shown at 10:00?" must use feature values as of 10:00. Using today's "likes on this author" includes the very like being predicted — brilliant offline, useless online. Feature stores do point-in-time joins.

11.7 Offline training vs online inference

The release flow: train → evaluate offline → model registry → shadow scoring (output not shown; compare distributions) → 1% A/B test → ramp up. Offline gains often don't translate online, so the A/B test is the final judge.

11.8 Latency budgets

The user expects the feed in a few hundred ms end-to-end, so ranking might get ~100–200 ms of server time. An illustrative budget (assumption):

When ranking is overloaded or down, the feed must still work: fall back to a cheaper model or to the chronological/cached timeline. That is covered in Resilience patterns (graceful degradation).

11.9 Feedback loops

Ranking models learn from data the ranker itself produced: items ranked high get more impressions, so more likes, so rank even higher — "we showed it a lot" looks like "people love it". Remedies:

• Position bias: position 1 gets more clicks just for being first. Log position as a training feature (fixed at serving time) or use inverse-propensity weighting.
• Exploration: reserve a few slots for items the model is uncertain about (epsilon-greedy, Thompson sampling) so it keeps learning about new content.
• Selection bias: you only have labels for items you showed; a small randomised traffic slice yields unbiased data.
• Goodhart's law: optimising raw clicks rewards clickbait. Balance with surveys ("worth your time?"), negative signals (hide, report) and integrity demotions.

12. Explore recommendations: embeddings, two towers, ANN

Feed ranks posts from accounts you chose. Explore (and Reels) must discover content from accounts you've never heard of — retrieval over billions of items. The core tool is the embedding.

12.1 Embeddings

An embedding is a vector (e.g. 128 floats) representing an entity so that similar entities get nearby vectors (by dot product or cosine). It's learned from behaviour: if users who engage with account A also engage with B, training pulls A and B together — word2vec's idea applied to "accounts appearing in the same sessions".

         ▲ dim 2                      (2-D toy; real embeddings have ~128+ dims)
         │     • latte art
         │   • coffee shop  • barista reels          ← "coffee" region
         │
         │                       • road bike
         │                    • cycling tours        ← "cycling" region
         │  ★ you
         │
         └──────────────────────────────────────► dim 1
  Your user vector ★ lies near coffee content → nearest neighbours = coffee posts

12.2 Two-tower models

Training uses positive pairs (user, item they engaged with) and negatives (random or other in-batch items); the loss pushes u·v up for positives and down for negatives. The key property: the towers don't interact until the dot product. That costs some accuracy versus a heavy ranker, but makes retrieval over billions of items feasible — item vectors are computed offline, and per request you compute one user vector and look up the highest dot products.

12.3 Approximate nearest-neighbour (ANN) search

Brute force is too slow: 1 billion items × 128 dims = 128 billion multiply-adds per request. ANN algorithms trade a little recall (finding ~95% of the true top-100) for 100–1000× speed:

• IVF (clustering): k-means splits vectors into, e.g., 65,536 cells; a query searches only the few dozen cells whose centroids are closest — ~0.1% of the data.
• Product quantisation (PQ): compresses each 512-byte vector to ~16–64 bytes of codebook IDs, so a billion vectors fit in tens of GB of RAM.
• HNSW: a multi-layer neighbour graph; a greedy walk from the top layer reaches the query's neighbourhood in ~log(N) hops. Excellent recall/latency, memory-hungry.

Libraries include FAISS (open-sourced by Meta), ScaNN and hnswlib. Indexes are rebuilt periodically, updated incrementally as new posts arrive, and sharded across machines — the query fans out to every shard and results are merged.

12.4 Cold start

Collaborative signals need history. Three cold-start cases:

12.5 Filter bubbles and integrity

A recommender that only exploits learned preferences narrows over time — three fitness reels lead to only fitness reels — a filter bubble. Worse, engagement optimisation can amplify sensational or harmful content. Mitigations:

• Diversity and exploration in re-ranking (topic caps, novelty bonuses).
• Recommendation eligibility: some content may stay on the platform but not be recommended to non-followers; Meta has publicly published recommendation guidelines of this kind.
• Integrity classifiers (spam, nudity, violence, misinformation) in the media pipeline and in ranking.
• User controls: "not interested", sensitive-content settings, a non-personalised/chronological view (the EU Digital Services Act requires a non-profiling option for very large platforms).
• Long-term metrics and surveys, not only short-term clicks.

13. How Instagram uses analytics & ML

1. Logging. The apps log impressions (which post was on screen, for how long), interactions and performance telemetry; services log domain events (the same post created / like / follow events that drive Kafka consumers in the main diagram) and ranking decisions.
2. Lake & warehouse. Events land in partitioned, columnar tables. Facebook created Hive and Presto for this workload and has publicly described Scribe (log transport) and Scuba (real-time analytics); Instagram, inside Meta's infrastructure since its migration (see Multi-Region), is described as using this shared platform.
3. Metrics & experiments. Daily pipelines compute DAU/MAU, retention and creator metrics; an experimentation platform evaluates every launch.
4. Ranking. Feed became ranked (not chronological) in 2016. Instagram's leadership has publicly explained that Feed, Stories, Explore and Reels each have their own ranking system predicting actions such as time spent, like, comment, share/save and profile tap, combined into a score with integrity rules applied. Chronological "Following" and "Favorites" views were added in 2022.
5. Explore. Instagram's engineering blog (2019) publicly described Explore as multi-stage: account embeddings learned from interaction sequences (ig2vec), candidate retrieval, then a few hundred candidates funnelled through a distilled lightweight model and heavier neural networks — the funnel of section 11.

Rough numbers (assumptions)

Feed requests:   500M DAU × ~20 feed loads/day ≈ 10B requests/day ≈ 115k/s avg (~300k/s peak)
Heavy scoring:   ~300 candidates per request → ~35M model evaluations/s on average
                 → this is why stages 1–2 exist, and why scoring runs batched on accelerators
Training data:   500B impression events/day; even a 1% sample = 5B labelled examples/day
Event storage:   ~100 TB/day compressed raw logs → ~36 PB/year before retention/aggregation

14. Trade-offs & common pitfalls

Common pitfalls

• Analytics on the production database — scans that evict the hot cache and slow the app. Use CDC/events into a separate store.
• Grouping by processing time instead of event time; or no watermark strategy, so late events vanish or windows never close.
• Not de-duplicating at-least-once events → inflated counts.
• Training–serving skew and label leakage — use a feature store with point-in-time joins.
• Trusting offline metrics — always confirm with an online A/B test.
• Forgetting deletion — user data must also leave the lake and feature stores (see Security & Privacy).

15. Real-world technologies

16. Interview corner

17. Check your understanding

Related lessons

• Message Queues, Pub/Sub & Kafka — the event bus that feeds every pipeline here
• News Feed: Fan-out & Ranking — where the ranking funnel is served
• Search & Inverted Indexes — retrieval and trending signals for search
• Databases: SQL vs NoSQL — the OLTP side of the OLTP/OLAP split
• Caching — Redis timelines and online feature serving
• Monitoring, Logging & Tracing — operational telemetry vs product analytics
• Security & Privacy — deletion and minimisation across analytical stores