1. The problem: why run in more than one region?

Imagine every Instagram server — all the application services, the sharded PostgreSQL User DB, the Cassandra clusters, Redis caches, Kafka — lives in a single region in Virginia. Three things go wrong:

• Disasters take everything down. A region is a cluster of data centers within a few tens of km. Storms, floods, power-grid failures, fibre cuts, a bad network configuration push, or a software bug in a regional control plane can make the whole region unavailable. With one region, a regional outage is a global outage.
• Latency for distant users. The CDN serves photos from nearby edges, but API calls — loading the feed JSON, posting a comment, sending a DM — still travel to the origin. A user in Singapore is ~230 ms round-trip from Virginia; a request needing several round trips (TCP, TLS, request, and the app's sequential API calls) feels sluggish.
• Law and regulation. Some countries and regulations restrict where personal data may be stored or transferred (e.g. the EU's GDPR rules on transfers). A single-country deployment can make compliance very hard.

Multi-region architecture addresses all three, but it's one of the hardest things in system design because the speed of light makes cross-region coordination slow — and the CAP theorem says that during a partition between regions you must choose between consistency and availability.

2. Vocabulary: regions, zones, data centers, cells

Multi-AZ deployment (replicas in 2–3 zones of one region) protects against a data-center failure with almost no latency cost — synchronous replication is affordable at ~1 ms. Multi-region protects against region-wide events, but replication must usually be asynchronous. Most systems do both: multi-AZ inside each region, multi-region on top.

3. Active-passive vs active-active

4. RPO and RTO

Disaster-recovery requirements are written as two numbers:

• RPO — Recovery Point Objective: the maximum acceptable data loss, measured in time. "RPO = 5 minutes" means after a disaster you may lose at most the last 5 minutes of writes. It is determined by how often/how fast data is copied elsewhere (backup frequency or replication lag).
• RTO — Recovery Time Objective: the maximum acceptable downtime — how long until the service works again. It is determined by how quickly you can detect the failure, make a decision, and bring up or redirect to the recovery site.

         last good copy                disaster                    service restored
               │                          │                               │
  ─────────────●══════════════════════════✖═══════════════════════════════●──────► time
               │◄──── data lost (≤ RPO) ─►│◄────── downtime (≤ RTO) ─────►│

  Async replication with 2 s lag        →  RPO ≈ seconds
  Nightly backup at 02:00, crash 18:00  →  RPO = 16 hours
  Automated DNS failover in 3 min       →  RTO ≈ minutes (+ DNS TTL)
  Restore 50 TB from backup at 1 GB/s   →  RTO ≥ 14 hours just to copy data

Different data deserves different targets. For Instagram (assumptions): DMs and posts need RPO of seconds; the like counter can tolerate approximate recovery (rebuild from the likes table); the analytics lake can be hours behind; the feed cache needs RPO = ∞ — it can be rebuilt from source data and doesn't need backup at all.

5. Disaster-recovery strategies

Cloud providers popularised four standard DR strategies. They form a ladder: each step lowers RPO/RTO and raises cost.

Choose per system, based on the business cost of downtime and data loss. An internal reporting tool can live with backup & restore; the Instagram feed and DMs need active-active; the user DB might be active-passive per shard (one primary region per shard, replicas elsewhere) inside an overall active-active deployment.

6. Cross-region data replication and lag

Stateless services (the Feed, Upload and Search services) are easy to run in many regions — just deploy them. State is the hard part. Every stateful store needs a cross-region replication strategy (see Replication for the fundamentals).

6.1 Synchronous vs asynchronous

A common middle ground is to spread a consensus group across three nearby regions (e.g. US-East, US-Central, US-West): each write waits for 2 of 3, adding ~20–40 ms rather than 80+, while surviving the loss of any one region with RPO = 0. Most social-media data, though, uses asynchronous replication and accepts a small RPO — losing the last second of likes in a once-a-decade disaster is acceptable; adding 80 ms to every like is not.

6.2 Living with replication lag

Replication lag is how far behind a replica is — measured in seconds or in log position. It causes user-visible anomalies:

• Read-your-writes violation: you change your bio (written in the US region), refresh, and the EU replica still shows the old bio. Fix: route a user's reads to their home region for a few seconds after they write, or carry a "last write position" token and only read from replicas that have caught up.
• Monotonic reads violation: one refresh hits a fresh replica, the next hits a stale one — a comment appears then disappears. Fix: stick each user to one replica/region.
• Cache invalidation across regions: if the EU cache is invalidated before the EU DB replica receives the write, the next read re-caches the old value. Facebook's published memcache paper (NSDI 2013) describes driving invalidations from the database's own replication/commit stream in each region, so a remote cache is invalidated only once the remote replica has the new data.

Monitor lag continuously and alert on it — it is literally your current RPO. A replica 10 minutes behind means a failover now loses 10 minutes of data.

7. Write routing and conflict handling

7.1 Home region per user (partitioned writes)

The simplest way to be active-active without conflicts is to ensure each record has exactly one writable home at any time. Assign each user a home region (usually where they mostly connect from); all writes to data owned by that user go to that region's primary, and other regions hold read replicas.

 user 184467 (home = EU)          user 99021 (home = US)
 ─────────────────────────        ─────────────────────────
 EU region:  PRIMARY  ✎           EU region:  replica  👁
 US region:  replica  👁           US region:  PRIMARY  ✎

 A US-homed user travelling in Europe:
   reads  → EU region (local replicas, fast)
   writes → forwarded to US primary (+~80 ms, rare enough to be fine)

• The mapping user_id → home region lives in a small, highly replicated directory (or is encoded in the shard ID — Instagram's public ID scheme embeds a logical shard ID in every 64-bit ID, see ID generation, and a shard → region map can be kept separately).
• Users can be re-homed (move a shard's primary) when they move countries, or to rebalance load — a careful, online migration.
• Data that involves two users (a like from an EU user on a US user's post; a DM between them) needs a rule: e.g. the like is owned by the post's shard, the DM thread by the thread's shard. Writes go to the owner's region.

7.2 When multiple regions can write the same data

Some systems allow writes to the same record in multiple regions (multi-leader / leaderless), for availability during partitions. Then conflicts are inevitable and need a resolution strategy:

# PN-counter CRDT for a post's like count, replicated across regions
state = {"US": {"inc": 1040, "dec": 12}, "EU": {"inc": 388, "dec": 3}, "APAC": {"inc": 905, "dec": 9}}
value = sum(r["inc"] - r["dec"] for r in state.values())   # 2309
merge(a, b) = per region: max(a.inc, b.inc), max(a.dec, b.dec)   # commutative, idempotent

8. Global traffic management

How does a user's phone reach the right region, and how is traffic moved away from a failed one?

• GeoDNS / latency-based DNS: the authoritative DNS answers i.instagram.com with the IP of the closest healthy region (or edge), based on the resolver's location or measured latency. Failover = stop handing out the failed region's IPs. Caveat: resolvers and devices cache answers for the TTL (and some ignore it), so a full shift takes minutes; keep TTLs short (30–60 s) on failover-critical records.
• Anycast: many edge locations announce the same IP via BGP; packets go to the topologically nearest. Withdrawing a route shifts traffic within seconds, without waiting for DNS caches.
• Edge proxies + private backbone: the user's TLS connection terminates at a nearby edge PoP (fast handshake); the edge then forwards the request over the company's private backbone to a healthy region chosen by a global load balancer. The edge can change the backend region instantly, per request — the most flexible option, and the model used by large companies (Meta's edge PoPs serve this role).
• Weighted traffic control: the global load balancer sends e.g. 30/30/40% to three regions and can drain a region gradually (100% → 50% → 0%) for maintenance or drills.
• Health signals: per-region health is judged from many signals — synthetic probes, error rates, latency — not one ping, to avoid failing over on a false alarm.

9. Failover, failback and drills

9.1 A failover runbook

A runbook is a written, rehearsed procedure. A regional failover for an Instagram-like active-active system might be:

1. Detect & declare. Alerts fire (error rate, health probes). The incident commander confirms it's a regional problem (not a bad deploy everywhere — failing over won't fix that) and declares a failover. Automation can do this for clear-cut cases; many organisations keep a human decision for full-region failover to avoid flapping on false positives.
2. Check capacity. Confirm surviving regions have headroom; pre-emptively scale them up; shed low-priority load (background jobs, prefetch, batch analytics) to make room.
3. Drain traffic. Set the failed region's weight to 0 in the global load balancer / withdraw anycast routes / update GeoDNS.
4. Promote data. For shards whose primary was in the failed region, promote replicas in a surviving region to primary. Fence the old primaries (revoke their write permission/lease) so that if the failed region comes back unexpectedly it can't accept writes — otherwise you get split brain: two primaries diverging.
5. Re-point async systems. Kafka producers/consumers, cron jobs, and batch pipelines that ran in the failed region are started elsewhere.
6. Verify. Dashboards for error rates, latency, replication health; synthetic user journeys (log in, load feed, post, DM).
7. Communicate. Status page, internal updates.

9.2 Failback

Failback — returning to the original region — is often riskier than failover because it's done less urgently and with less practice. The recovered region's databases are stale and may contain un-replicated writes made just before the failure (the RPO window). Steps: rebuild the old primaries as replicas of the new primaries (don't just "turn them back on"); reconcile or discard the orphaned writes (log them for review); wait until lag is near zero; shift traffic back gradually (1% → 10% → 50% → 100%); move primaries back during a quiet period.

9.3 Drills

A failover plan that has never been executed will fail: credentials expire, scripts reference deleted hosts, a "stateless" service turns out to depend on a config store in the failed region, and surviving regions turn out to lack capacity. The only cure is regular drills in production:

• Start with draining a single service from one region, then a whole region at low-traffic times, then at peak.
• Measure actual RTO and RPO against the objectives; record every surprise and fix it.
• Meta has publicly described running large-scale disaster-recovery exercises (internally called Storm in public talks) that disconnect entire data centers or regions from the network to verify that products, including Instagram, keep working. Netflix's "Chaos Kong" exercises do the same for AWS regions. See Chaos engineering.

10. Cell-based architecture and blast radius

Blast radius is how many users a single failure can affect. A region failover limits blast radius to "everyone in that region" — still tens or hundreds of millions of people. And many outages aren't regional disasters at all: they're a bad deploy, a poison request, or a runaway client that hits every replica of a service at once. Cells shrink the blast radius further.

• What a cell contains: a complete set of services plus the data shards for its users. Cells don't call each other on the request path (or do so only through well-guarded interfaces).
• Cell router: a deliberately thin, simple, highly available layer mapping user_id → cell (often via the shard ID). It must be the most reliable part of the system because it's shared by all cells.
• Deploy cell by cell: new code goes to one canary cell first, then progressively to more. A bad release hurts 1–2% of users for minutes, not 100%.
• Fixed maximum cell size: growth adds new cells instead of growing cells, so every cell stays at a size that has been load-tested.
• Cross-cell data: social products are highly connected (your feed contains posts from users in every cell), so pure cells are easiest for per-user data (accounts, DMs, settings) and harder for global data (social graph, feed fan-out). Global services then get their own isolation (sharding, bulkheads).

Regions, AZs, cells and shards are nested failure domains. A well-designed system asks, for each: "if this unit fails, what fraction of users is affected, and what happens to them?"

11. Backups: replication is not a backup

Replication protects against losing a machine or region. It does not protect against bad writes: if a bug or an attacker deletes or corrupts data, replication faithfully copies the damage to every replica within milliseconds. You need backups — point-in-time copies you can go back to.

11.1 The 3-2-1 rule

11.2 Backup techniques

• Snapshots: periodic full copies (or copy-on-write volume snapshots) of each database shard.
• Point-in-time recovery (PITR): a base snapshot plus the continuous write-ahead log (PostgreSQL WAL archiving). Restore = load snapshot, replay the log up to just before the bad write (e.g. 14:31:59 before a 14:32 bad migration). RPO for corruption becomes seconds.
• Delayed replicas: a replica that intentionally applies changes 1 hour late — a quick way to recover from a recent accidental delete.
• Object storage: versioning plus cross-region replication for photos and videos; deletions create "delete markers" that can be undone within a retention window (balanced against privacy deletion obligations — see Security & Privacy).
• Encryption and access control: backups contain everything, so they must be encrypted and tightly access-controlled.

11.3 Restore testing

"Backups don't matter; restores do." Many organisations discover during a disaster that backups were incomplete, corrupt, missing a table, or would take days to restore. Practice:

• Automated restore tests: continuously restore random shards' backups into a sandbox, run integrity checks (row counts, checksums, application-level queries) and alert on failure.
• Measure restore time: a 40 TB shard restored at 500 MB/s takes ~22 hours — if your RTO is 4 hours, you need parallel restores, smaller shards, or a warm replica instead.
• Full-scale exercises periodically, including rebuilding derived stores (caches, search indexes) from restored sources.

12. Data residency and GDPR

Data residency means keeping certain data within a geographic or legal boundary. The EU's GDPR does not, by itself, require that all EU data be stored in the EU; it restricts transfers of personal data outside the European Economic Area unless there's a legal basis (an adequacy decision, standard contractual clauses with extra safeguards, etc.). In practice, legal uncertainty around EU→US transfers (the 2020 "Schrems II" ruling invalidated the EU-US Privacy Shield; Meta was publicly fined €1.2 billion by the Irish regulator in 2023 over Facebook's EU-US transfers before a new EU-US Data Privacy Framework was adopted) has pushed many companies to localise more data. Other countries (e.g. Russia, China, India for some data categories) have their own localisation rules.

Architectural techniques:

• Region pinning: use the home-region mechanism to keep a regulated user's primary data (and backups) in an in-boundary region; don't replicate those tables outside.
• Data classification: tag fields by sensitivity and residency requirement; replicate public content (posts visible worldwide must be served worldwide) differently from private data (DMs, contact lists, location history).
• Regional keys: encrypt data with keys held in a KMS inside the boundary, so copies elsewhere are unreadable without in-region access.
• Pseudonymised analytics: send only aggregated or pseudonymised data to global analytics pipelines (see Analytics).
• Deletion everywhere: GDPR's right to erasure applies to every regional copy, cache and backup — see Security & Privacy.

13. How Instagram does multi-region

Instagram's public history is a textbook progression up the DR ladder (dates and details as publicly reported; internal specifics are not public):

1. 2010–2012 — a single AWS region. The early Django + PostgreSQL + Redis + memcached stack ran on AWS in the US-East region. On 29 June 2012, severe storms caused a power failure in an AWS US-East data center; Instagram (along with Netflix and Pinterest) was widely reported to be down or degraded for many hours. With everything in one region and recovery depending on bringing that region back, the outage illustrated precisely the single-region risk in section 1, and it is often cited as a lesson in why critical services should not depend on a single data center or region.
2. 2013–2014 — migration into Facebook's data centers. After the 2012 acquisition, Instagram publicly described a large migration ("Instagration") from AWS into Facebook's own data centers, completed in 2014 — moving a live service with hundreds of millions of users, its sharded PostgreSQL and Cassandra data, without a user-visible outage. Being inside Facebook's infrastructure gave Instagram access to Meta's networks, edge PoPs, tooling and data-center fleet.
3. 2015 onwards — multiple data centers. Instagram's engineering blog publicly described expanding from one data center to several, and running across multiple regions: stateless Django application tiers deployed in each data center; Cassandra replicated across data centers with local-quorum reads/writes; PostgreSQL with a primary per shard in one location and replicas elsewhere, writes routed to the primary; and careful handling of cache invalidation across regions. This enabled serving traffic from multiple places and surviving the loss of a data center.
4. Today — part of Meta's global infrastructure. Meta publicly describes running many data-center regions, a global edge network, and regular disaster-recovery drills (Storm) that take whole data centers offline — Instagram participates in these like other Meta products.

Rough numbers (assumptions)

3 active regions, peak 1M API req/s total → ~333k req/s each
Survive loss of 1 region → each must handle 500k req/s → run at ≤ 67% of capacity
Write rate 50k writes/s, avg 1 KB → 50 MB/s replicated to each other region (~400 Mbps)
Async lag normally ~100–500 ms → RPO on sudden region loss ≈ 0.5 s × 50k = ~25k writes at risk
Restoring a 2 TB shard from backup at 400 MB/s ≈ 1.4 h → per-shard RTO if no replica

14. Trade-offs & common pitfalls

Common pitfalls

• Hidden single-region dependencies: an auth service, config store, secrets manager, DNS provider or CI system that exists only in one region — discovered during the disaster.
• Insufficient headroom: surviving regions overload and fail in a cascade right after failover.
• Treating replication as backup: a bad migration deletes data everywhere.
• Never-tested restores and runbooks.
• Split brain: promoting a new primary without fencing the old one.
• Long DNS TTLs that make traffic shifts take hours.
• Assuming cross-region synchronous writes are cheap — 80+ ms per write adds up fast for chatty code.
• Forgetting async systems (cron jobs, queue consumers, batch) in the failover plan — jobs run twice or never.

15. Real-world technologies

16. Interview corner

17. Check your understanding

Related lessons

• Replication — leader/follower, quorum and lag fundamentals
• CAP, PACELC & Consistency — the theory behind cross-region trade-offs
• Sharding — shards are the unit of placement and failover
• DNS and CDN — global traffic steering and edge serving
• Resilience patterns — chaos engineering, load shedding during failover
• Security & Privacy — GDPR rights, encryption keys and deletion across regions
• Monitoring, Logging & Tracing — detecting regional failures and replication lag