1. The problem: why do we need observability?

It's 9 p.m. in Brazil, peak time. Some users report that the Instagram feed takes ten seconds to load. Somewhere among the load balancers, the API Gateway, the News Feed Service, ranking, Redis feed caches, Cassandra, the user DB, the CDN and thousands of servers in several regions, something is wrong. Questions you must answer in minutes:

• Is it real, and how many users are affected? Only Brazil? Only Android? Only one app version?
• Which component is slow — and is it the cause, or just a victim of something further down?
• Did anything change recently — a deploy, a config flag, a traffic spike from a viral post?

In a monolith on one server you'd SSH in and look at the logs. In a distributed system with hundreds of services, a single user request touches dozens of machines; no human can inspect them one by one. You need the system to emit telemetry that is collected, stored and queryable centrally. That is what this lesson is about.

2. Monitoring vs observability

3. The pillars of telemetry

The pillars are most powerful when linked: a latency spike on a dashboard → click through to example traces from that minute (via exemplars) → a slow span → the logs for that span (same trace_id) → the profile of the slow host.

4. Metrics in depth

4.1 Metric types

4.2 Labels and cardinality

Metrics carry labels (dimensions): http_requests_total{service="feed", endpoint="/v1/feed/timeline", status="200", region="eu"}. Each unique combination of label values is a separate time series stored by the database. Cardinality multiplies: 50 endpoints × 10 status codes × 5 regions × 2,000 hosts = 5M series. Never put unbounded values like user_id, post_id or raw URLs in metric labels — that would create billions of series and take down the metrics system. High-cardinality questions ("which user?") belong in logs and traces.

4.3 Pull vs push

Prometheus pulls ("scrapes") a /metrics endpoint on each target every 15–30 s; a target that disappears is immediately noticed (up == 0). Push systems (StatsD, OpenTelemetry OTLP, Datadog agents) have processes send metrics to a collector — better for short-lived jobs and for crossing network boundaries. Both are widely used; large fleets often combine local agents with a central, horizontally scalable TSDB.

# What a service exposes on /metrics (Prometheus text format)
# TYPE feed_requests_total counter
feed_requests_total{endpoint="timeline",status="200"} 18233412
feed_requests_total{endpoint="timeline",status="503"} 1204
# TYPE feed_request_duration_seconds histogram
feed_request_duration_seconds_bucket{endpoint="timeline",le="0.1"}  15011290
feed_request_duration_seconds_bucket{endpoint="timeline",le="0.25"} 17950003
feed_request_duration_seconds_bucket{endpoint="timeline",le="0.5"}  18190117
feed_request_duration_seconds_bucket{endpoint="timeline",le="+Inf"} 18234616
feed_request_duration_seconds_sum{endpoint="timeline"}   2735192.4
feed_request_duration_seconds_count{endpoint="timeline"} 18234616

5. What to measure: golden signals, RED and USE

5.1 The four golden signals

Google's SRE book proposes that if you can only measure four things about a user-facing system, measure these:

5.2 RED — for request-driven services

Rate (requests/s), Errors (failed requests/s), Duration (latency distribution). Every microservice — Upload, Feed, Search, Like & Comment — gets the same three graphs per endpoint, which makes any service's dashboard instantly readable by any engineer.

5.3 USE — for resources

For every resource (CPU, memory, disk, network interface, connection pool, thread pool): Utilisation (% of time busy / % capacity used), Saturation (extra work queued that can't be served — run-queue length, swap, dropped packets), Errors (disk errors, NIC errors, OOM kills). USE finds the bottleneck; RED tells you whether users are affected.

6. Percentiles and histograms: why averages lie

A percentile pN is the value below which N% of observations fall: p50 is the median; p99 means "99% of requests were faster than this". Averages hide what users actually experience:

100 feed requests: 98 take 50 ms, 2 take 5,000 ms (stuck on a slow shard)
  average = (98×50 + 2×5000) / 100 = 149 ms        ← "looks fine"
  p50     = 50 ms
  p99     = 5,000 ms                                ← 1 in 50 users waits 5 seconds

Bimodal cache: 80% hits at 5 ms, 20% misses at 200 ms
  average = 0.8×5 + 0.2×200 = 44 ms                 ← no request ever took 44 ms!

Why the tail matters even more at Instagram's scale:

• Volume: at 100,000 req/s, p99 describes 1,000 requests every second — real people.
• Heavy users hit the tail most: a user loading 50 feed pages a day will see the p99 experience at least once with probability 1 − 0.99⁵⁰ ≈ 40%.
• Fan-out amplifies tails: a request waiting on 100 shards sees at least one p99 response 63% of the time (see hedged requests).

6.1 Computing percentiles correctly

You can't store every latency of 100,000 req/s forever, and you cannot average percentiles: the p99 of host A averaged with the p99 of host B is not the fleet p99. The solution is the histogram: each host counts requests into fixed buckets; bucket counts can be summed across hosts and time; the fleet percentile is estimated from the merged buckets.

latency bucket   ≤10ms  ≤25ms  ≤50ms  ≤100ms  ≤250ms  ≤500ms  ≤1s   ≤2.5s  +Inf
host A counts     120    800   2400    600      60      15     4      1      0
host B counts      90    700   2600    500      80      20     8      2      0
sum (fleet)       210   1500   5000   1100     140      35    12      3      0   total 8000
cumulative        210   1710   6710   7810    7950    7985  7997   8000
p99 → rank 7920 falls in the ≤250ms bucket (7810 < 7920 ≤ 7950) → p99 ≈ 100–250 ms (interpolate ≈ 220 ms)

Accuracy depends on bucket boundaries — choose them around your SLO thresholds. Libraries such as HdrHistogram, DDSketch and Prometheus native histograms use exponential buckets to keep relative error small (~1–2%) at any scale.

7. SLIs, SLOs, SLAs and error budgets

7.1 Error budgets — worked example

An SLO of 99.9% means you are allowed 0.1% bad requests — that allowance is the error budget. Assume 10 billion feed requests per day:

28-day window: 10B × 28 = 280B requests
Error budget  = 0.1% × 280B = 280M bad requests per 28 days
In time terms = 0.001 × 28 days × 1,440 min = ~40 minutes of total outage per 28 days

Incident: a bad deploy makes 30% of feed requests fail for 20 minutes
  requests in 20 min = 10B / 1,440 × 20 ≈ 139M
  bad requests       = 30% × 139M ≈ 42M  → 15% of the 28-day budget burned

The budget turns reliability into an engineering decision instead of an argument:

• Budget remaining → ship features, run experiments, take risks.
• Budget exhausted → freeze risky launches, prioritise reliability work until it recovers.
• 100% is the wrong target: users can't tell 99.99% from 100% because their own phone network fails more often than that, and every extra nine costs roughly 10× more. Pick the SLO that matches what users notice.

7.2 Burn-rate alerting

The burn rate is how fast you're consuming budget relative to plan: burn rate 1 = you'll use exactly 100% of the budget by the end of the window; burn rate 14.4 sustained for 1 hour consumes 2% of a 30-day budget. A widely used scheme (from Google's SRE workbook) uses multiple windows:

The short confirmation window makes the alert stop quickly once the problem is fixed; the long window prevents paging on a brief blip.

8. Logging

8.1 Structured logging

Unstructured logs ("Upload failed for user 42 after 3 retries") are readable by humans but painful to search and aggregate. Structured logs are key-value records (usually JSON) that machines can index and filter:

{
  "ts": "2026-10-23T15:42:07.118Z",
  "level": "ERROR",
  "service": "upload-service",
  "host": "upload-7f9c-eu-3",
  "region": "eu",
  "version": "2026.10.23-1",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "user_id": 184467440737,
  "event": "s3_put_failed",
  "media_id": "3192…",
  "attempt": 3,
  "error": "TimeoutError: PUT exceeded 2000 ms",
  "duration_ms": 2004
}

• Correlation IDs: every log line carries the trace_id (and a request ID), so you can pull all logs for one request across every service.
• Levels: DEBUG (off in production or sampled), INFO (significant business events), WARN (unexpected but handled), ERROR (a request failed), FATAL.
• Log events, not noise: log state changes and failures with context; don't log every successful cache hit — that's what metrics are for.
• Never log secrets or sensitive content: passwords, tokens, message bodies, full card numbers. Redact or hash PII; access to logs is itself audited (see Security).

8.2 Log pipelines

The buffer (Kafka) matters: during an incident, error logs spike 10–100×, exactly when you need the log system most. Without a buffer, the log store falls over or the agents drop logs. Cost controls: sample repetitive INFO/DEBUG logs (keep 1%, keep 100% of errors), set per-service ingestion quotas, keep only a few days in the expensive full-text index and the rest in cheap object storage queried on demand. Rough scale (assumption): 20,000 servers × 500 lines/s × 500 bytes ≈ 5 GB/s ≈ 430 TB/day before sampling — logs can easily cost more than the service itself if unmanaged.

9. Distributed tracing

Metrics say the feed is slow; logs show errors in several services. But which service made this request slow? A distributed trace records the full journey of one request through every service as a tree of timed operations.

9.1 Traces, spans and IDs

• Trace: everything that happened for one end-to-end request, identified by a globally unique trace ID (128-bit).
• Span: one timed operation within the trace (an RPC call, a DB query, a cache lookup) with a span ID, a parent span ID, start time, duration, status and attributes (db.system=redis, shard=17, user.app_version=312.0).
• The parent/child links form a tree; drawn on a timeline it becomes a waterfall that shows which work was sequential, which was parallel, and where the time went.

9.2 An Instagram feed request, traced

trace_id 4bf92f3577b34da6a3ce929d0e0e4736   GET /v1/feed/timeline   user 1844…   total 412 ms
time (ms)        0        50       100      150      200      250      300      350      400
                 |--------|--------|--------|--------|--------|--------|--------|--------|
lb + gateway     [████████████████████████████████████████████████████████████████████████] 412
 ├ auth check    [██] 12
 └ feed-service     [█████████████████████████████████████████████████████████████████████] 395
    ├ redis: timeline  [█] 6                                  (precomputed post IDs)
    ├ counters svc       [████] 20          ┐
    ├ user svc           [███] 18           ├ run in parallel
    ├ stories tray       [████] 22          ┘
    ├ ranking svc        [██████████████████████████████████████████████] 250   ◄ slowest child
    │  ├ feature store     [████████████████████████████████] 180            ◄ ROOT CAUSE
    │  │   attrs: shard=17, keys=480, cache_hit=0.31 (normally 0.95)
    │  └ model inference                                    [████████] 55
    └ hydrate + sign CDN URLs                                            [█████] 30

Reading it: the gateway and auth are fine; the parallel calls are fast; the ranking call dominates, and within it the feature-store lookup takes 180 ms because its cache hit rate on shard 17 dropped from 95% to 31%. Without tracing, the ranking team, the feed team and the feature-store team would each see "slow" in their own metrics and argue about whose fault it is. With tracing, the critical path is obvious in seconds.

9.3 Context propagation

For spans from different services to join one trace, the trace context must travel with the request. The W3C Trace Context standard defines an HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └ flags (01 = sampled)
             │  │                                └ parent span ID (the caller's span)
             │  └ trace ID (same for the whole request)
             └ version

• Synchronous calls: tracing libraries (OpenTelemetry instrumentation for HTTP/gRPC clients and servers) read and inject the header automatically.
• Asynchronous hops: when the Upload Service publishes "post created" to Kafka, the context goes into the message headers; the fan-out worker that consumes it starts a span linked to the producer's span. Now you can trace "user posted → fan-out → push notification delivered" end to end.
• In-process: context must follow the request across thread pools and async callbacks — a common source of broken traces.
• Baggage: small key-values propagated alongside (e.g. experiment=881) so deep services can tag their telemetry. Keep it tiny; it travels on every hop.

9.4 Sampling

Recording every span of every request is too expensive (a feed request might produce 50+ spans; at 100k req/s that's 5M spans/s). So traces are sampled:

Practical systems combine them: head-sample a baseline, always keep errors and very slow requests, and allow forced tracing for specific debugging (e.g. an engineer's own account or a flagged request).

10. Alerting

10.1 Symptom-based vs cause-based

Alert on symptoms users feel ("feed error rate above SLO burn threshold", "DM delivery p99 > 5 s"), not on every possible cause ("CPU > 80% on host 1234", "one Cassandra node down"). Causes are many and often harmless (a host at 85% CPU serving fine; a replica down with others healthy); symptoms are few and always matter. Cause metrics belong on dashboards for diagnosis. Exceptions: causes that predict imminent user impact without a symptom yet — "disk will be full in 4 hours", "certificate expires in 7 days", "Kafka consumer lag growing for 30 min".

10.2 Pages vs tickets

10.3 Runbooks

Every alert should link to a runbook: what the alert means, the likely causes, the dashboards and queries to check, safe mitigation steps (roll back the last deploy, drain a region, enable degraded mode, flip a kill switch), and who to escalate to. A tired engineer at 4 a.m. should not have to reverse-engineer an alert.

# Prometheus alert rule: fast burn of the feed availability SLO (99.9%)
- alert: FeedSLOFastBurn
  expr: |
    (sum(rate(feed_requests_total{status=~"5.."}[1h])) / sum(rate(feed_requests_total[1h])))  > (14.4 * 0.001)
    and
    (sum(rate(feed_requests_total{status=~"5.."}[5m])) / sum(rate(feed_requests_total[5m]))) > (14.4 * 0.001)
  labels:
    severity: page
    team: feed
  annotations:
    summary: "Feed is burning its error budget 14x too fast"
    runbook: "wiki/runbooks/feed-slo-fast-burn"
    dashboard: "grafana/d/feed-overview"

10.4 Alert fatigue

When engineers receive dozens of alerts per shift, most of them unactionable or duplicates, they start ignoring them — and miss the real one. This is alert fatigue. Remedies:

• Every page must be actionable, urgent and real; delete or downgrade alerts that are routinely acknowledged and ignored.
• Group and deduplicate (one page for "Redis cluster degraded", not 64 pages for 64 shards); add inhibition rules (don't page for every service when the region is down).
• Require a minimum duration (for: 5m) to skip brief blips; use burn-rate alerts instead of raw thresholds.
• Review alert statistics in a weekly on-call handover; track pages per shift as a team health metric (a common target: no more than ~2 incidents per 12-hour shift).

11. Dashboards

Dashboards are for humans to understand state quickly. Good practice:

• Hierarchy: a top-level "Instagram health" board (SLOs per user journey: open app, load feed, post, send DM, view story) → per-service RED boards → per-resource USE boards → drill-down queries.
• Same layout for every service (RED at the top, dependencies' RED next, saturation below) so on-call engineers can read any team's board.
• Overlay change events (deploys, flag flips, failovers) as vertical markers.
• Show percentiles and rates, not averages and raw counters; compare to the same time last week to separate daily patterns from anomalies.
• Break down by key dimensions: region, app version, platform (iOS/Android/web), endpoint — most incidents are localised to one of these.
• Avoid "wall of 200 graphs" dashboards — nobody can read them under pressure.

12. On-call, incident management and postmortems

12.1 On-call

Each service team runs an on-call rotation: one primary (and a secondary backup) responsible for responding to pages, typically for a week at a time, often "follow-the-sun" across time zones so nobody is woken at night routinely. Healthy on-call has manageable page volume, good runbooks, the authority to roll back, and time after a rough shift to fix the root causes.

12.2 Incident management

1. Detect: alert fires or users report a problem.
2. Declare & triage: open an incident, assign a severity (SEV1 = major user-facing outage, e.g. feed down in a region; SEV3 = minor degradation).
3. Assign roles: Incident Commander (coordinates, decides — doesn't debug), Operations/Tech lead (hands on keyboard), Communications lead (status page, internal updates), Scribe (timeline).
4. Mitigate first, root-cause later: roll back the deploy, drain the region, turn off the feature flag, enable degraded mode. Restoring users comes before understanding.
5. Resolve & monitor: confirm metrics are back to normal; keep watching.
6. Postmortem within days.

12.3 Blameless postmortems

A postmortem is a written analysis of an incident. Blameless means it assumes everyone acted reasonably given the information and tools they had, and asks "how did the system allow this?" rather than "who messed up?". If people are punished for mistakes, they hide them, and the organisation stops learning. A typical template:

13. How Instagram does observability

• SLIs measured at the edge and in the app. Server-side metrics can't see failures that happen before the request arrives (DNS, TLS, mobile network) or after (image decode, rendering). So user-journey SLIs — "feed loaded with first image visible", "photo posted successfully", "DM delivered" — are computed from both load-balancer logs and client telemetry (see Clients).
• Per-service RED + dependency signals: the Feed Service dashboard shows its own RED, its dependencies' RED (Redis timeline, ranking, counters), Kafka lag for fan-out, and cache hit ratios.
• Every deploy is an event: Instagram's engineering talks describe continuous deployment of its large Python/Django service through canary stages, where automated checks compare error rates and latency of the canary against the rest of the fleet and halt the rollout on regressions. That is observability used as a release gate.
• Correlation via IDs: request and trace IDs flow from the gateway through services and Kafka messages, so an engineer can jump from a user report to the exact request path.

Rough numbers (assumptions)

Metrics: 2,000 services × 300 metric names × ~500 label combos ≈ 300M active series
         → requires a horizontally sharded TSDB, aggressive downsampling (10 s → 5 min after 2 weeks)
Traces:  100k feed req/s × 50 spans × 1% head sampling = 50k spans/s (+ all errors and slow traces)
Logs:    ~400 TB/day raw before sampling → keep ~1 week hot, sampled archive in object storage
Error budget for a 99.9% feed SLO at 10B requests/day ≈ 280M bad requests per 28 days

14. Trade-offs & common pitfalls

Common pitfalls

• Averages instead of percentiles; averaging per-host percentiles.
• Unbounded label values (user_id, URL) in metrics.
• Alerting on causes (CPU, single hosts) instead of user-facing symptoms → noise and fatigue.
• Alerts without runbooks or without an owner.
• Only server-side measurement — missing failures in DNS, the network, or the app.
• Broken trace propagation across async boundaries (Kafka, thread pools) → fragmented traces.
• The monitoring system depends on the system it monitors (same region, same Kafka): when production fails, observability fails with it. Run monitoring out-of-band and alert on "no data".
• Logging PII or secrets; keeping logs forever.
• Blame-oriented postmortems that make people hide mistakes.

15. Real-world technologies

16. Interview corner

17. Check your understanding

Related lessons

• Resilience patterns — timeouts, breakers and fallbacks that observability must make visible
• Scalability, Availability, Reliability & Performance — what the SLOs are measuring
• Analytics, Data Pipelines & ML — product analytics vs operational telemetry, shared event pipelines
• Message Queues & Kafka — log transport and consumer-lag monitoring
• Load Balancing — where request-level SLIs are measured
• Multi-Region & DR — detecting regional failures and replication lag
• Security & Privacy — audit logs and keeping PII out of telemetry