1. The problem: why split a system into services at all?

Picture Instagram in October 2010: a handful of engineers, one Django application, one PostgreSQL database. Every feature — sign-up, photo upload, the feed, likes, comments, follows — lived in one codebase and ran in one process type. That is a monolith, and it was exactly the right choice: one thing to build, test, deploy and debug.

Now fast-forward to a company with thousands of engineers and 2+ billion monthly users. New problems appear that have nothing to do with any single feature:

• Team coupling. Hundreds of engineers commit to one codebase daily; a bug in comment ranking can block the Stories team's deploy.
• Uneven scaling. The feed is read tens of thousands of times per second; "change password" a few times a second. A monolith scales everything together.
• Blast radius. A memory leak in the upload path can crash the same process that serves likes.
• Technology fit. A messaging gateway holding millions of WebSocket connections wants a different runtime than a CRUD API in Python.

Microservices are one answer: split the system along business boundaries into separately deployable services, each owned by one team. But — the most important sentence in this lesson — splitting trades code complexity for operational and distributed-systems complexity. Function calls become network calls that can time out; one database transaction becomes several databases that can disagree.

2. Monolith vs modular monolith vs microservices

2.1 Definitions

• Monolith — a single deployable unit (one Django project, one image) containing all features, usually over one shared database. The defining trait: built, deployed and scaled as one.
• Modular monolith — still one deployable, but divided into modules with explicit interfaces (feed.api, graph.api), enforced import rules and per-module table ownership. Calls stay in-process, yet a module could later be extracted.
• Microservices — many independently deployable services, each owning one bounded context (a business capability with its own language and data, e.g. "social graph") and its own datastore, communicating only via APIs (REST, gRPC) or events (Kafka).

2.2 Side-by-side comparison

2.3 Why a monolith scaled so far for Instagram

The key insight: "monolith" describes how code is packaged, not how it scales. A stateless monolith can be copied onto 10,000 servers behind a load balancer just as easily as a microservice. What it can't easily do is let 200 teams deploy independently — Instagram compensated with tooling (continuous deployment with canaries, code ownership, static analysis enforcing boundaries, feature flags). Meanwhile the data tier was split aggressively: sharded PostgreSQL, Cassandra, Memcached/Redis. Scale comes mostly from the data layer and async workers.

3. Benefits and costs of microservices

3.1 Benefits

3.2 Costs (the part people underestimate)

• Network unreliability. The fallacies of distributed computing — "the network is reliable", "latency is zero", "bandwidth is infinite", "topology doesn't change" — are all false. Every call needs a timeout, retry policy and fallback (see Resilience patterns).
• Availability multiplies down. If a request synchronously needs 10 services that are each 99.9% available, the request succeeds only 0.99910 ≈ 99.0% of the time — about 7 hours of failure per month instead of 43 minutes.
• Tail latency amplifies. If each of 10 parallel downstream calls is slow 1% of the time, the chance that at least one is slow is 1 − 0.9910 ≈ 9.6%. Your p99 becomes your p90.
• No more cross-feature transactions or joins. Consistency across services is eventual; you need sagas, outboxes and idempotent consumers (sections 8–9).
• Operational overhead and harder debugging. Every service needs CI/CD, dashboards, on-call and a versioned API; one request may touch 30 services, so you need distributed tracing.

4. Stateless services

4.1 What "stateless" means

A service is stateless when an instance keeps no data between requests that another instance would need. Every request carries (or can look up) everything required to process it: an auth token identifying the user, the IDs it operates on, and access to shared datastores. The instance's memory and local disk are scratch space only — losing the instance loses nothing important.

By contrast, a stateful server keeps something meaningful locally: a user's login session in memory, an uploaded file on local disk, a counter it increments, an open WebSocket connection, a partition of a database.

4.2 Why statelessness matters so much

1. Horizontal scaling is trivial. Need more capacity? Start 200 more identical instances and register them with the load balancer. No data needs to move.
2. Any instance can serve any request. The load balancer can use simple round-robin or least-connections — no sticky sessions needed. Load spreads evenly.
3. Failure is cheap. If an instance dies, the load balancer's health check removes it and in-flight requests are retried elsewhere. Nobody loses their login.
4. Deploys and autoscaling are painless. Rolling deploys replace instances one batch at a time; autoscalers add and remove instances at will (section 10).

4.3 Where the state goes instead

State doesn't disappear — it is pushed down into systems designed to hold it durably and share it:

4.4 Sticky sessions — the tempting shortcut

A load balancer can pin a user to one instance ("session affinity"). It lets stateful code limp along, but load becomes uneven and scale-in or deploys log people out. Use stickiness only for performance, never correctness. See Load balancing.

5. Service discovery: how does service A find service B?

Instances change IPs every minute under autoscaling. Service discovery maps a logical name ("graph-service") to the current set of healthy addresses.

5.1 The service registry

A service registry is a highly available database of "service name → instances (IP, port, health)". Instances get into it by:

• Self-registration: the instance registers on start-up and sends heartbeats; if they stop, the entry expires after a TTL (ZooKeeper's ephemeral znodes vanish when the session dies).
• Third-party registration: the platform does it — Kubernetes tracks which pods pass their readiness probes and updates each Service's endpoints.

Because everything depends on it, the registry runs as a 3- or 5-node consensus cluster: ZooKeeper (ZAB; ephemeral nodes + watches, also leader election), etcd (Raft; the brain of Kubernetes) or Consul (Raft; built-in health checks, DNS interface, multi-DC). Netflix Eureka deliberately chooses availability, serving possibly-stale lists during partitions. Clients cache the last known list so a registry outage doesn't instantly break traffic.

5.2 Client-side vs server-side discovery

CLIENT-SIDE DISCOVERY                         SERVER-SIDE DISCOVERY

 Feed svc                Registry              Feed svc
   |  (1) where is graph-svc?  |                 |  (1) GET http://graph-svc/...
   |-------------------------->|                 v
   |  (2) [10.0.1.5, 10.0.1.9, |            +-----------+   watches   +----------+
   |       10.0.2.3]           |            | LB / VIP  |<------------| Registry |
   |<--------------------------|            | (or K8s   |             +----------+
   |  (3) client picks one     |            |  Service) |
   |      (round robin / P2C)  |            +-----------+
   |---------> 10.0.1.9                           |  (2) forwards to a healthy instance
                                                  v
                                             10.0.1.9

A service mesh (section 11) is effectively a hybrid: the "client" is a local sidecar proxy, so the application calls localhost, while the sidecar does smart client-side balancing using endpoint lists pushed from a control plane.

5.3 DNS-based discovery

The simplest registry is DNS: graph-service.prod.internal resolves to healthy instances (A records; SRV records add ports and weights). Consul exposes a DNS interface; Kubernetes' CoreDNS resolves graph.default.svc.cluster.local to a Service VIP. It works in every language for free, but resolvers cache answers for the TTL (some ignore it), so removed instances keep getting traffic, and DNS carries no load information. Hence DNS usually finds a load balancer or VIP that does health-aware balancing.

6. How services talk: synchronous vs asynchronous

6.1 Synchronous request/response (REST, gRPC)

The caller sends a request and waits — needed when it can't continue without the result (the Feed service needs post metadata and counts to build its response).

• REST over HTTP/JSON — universal and easy to debug; verbose, informal schemas.
• gRPC — HTTP/2 + Protocol Buffers: compact binary, generated stubs, strict evolvable schemas, streaming, deadline propagation. The default for internal calls at scale (Meta's equivalent is Thrift).

See API design. Every sync call needs a timeout (ideally a shrinking deadline budget), bounded retries with backoff and jitter, and a circuit breaker — otherwise one slow service exhausts upstream thread pools (cascading failure).

6.2 Asynchronous messaging (events)

The producer publishes to a broker (Kafka, RabbitMQ, SQS) and does not wait. Commands/jobs ("resize this photo") go to a queue for one worker; domain events (PostCreated{post_id: 42, author_id: 7}) are facts about the past that any number of unknown consumers can react to.

That's the Instagram diagram: PostCreated goes to Kafka and fan-out workers, the search indexer, notifications and analytics consume it on their own schedule. If the indexer is down for 10 minutes, posts still publish; it catches up from the log. See Message queues & Kafka.

6.3 Choosing between them

7. Data ownership: database per service

7.1 The rule

Each service owns its data exclusively. Only the Graph service touches follow tables; only the Like service touches likes. Others go through the owner's API or consume its events. Each picks its own store: a TAO-like association store for Graph, Redis for Feed, PostgreSQL for Users, Elasticsearch for Search.

Why so strict? A shared database is a hidden, unversioned API: if Feed runs SELECT … FROM likes JOIN posts … directly, the Like team can never change schema, move to Cassandra or reshard without breaking Feed — and shared tables mean shared load.

7.2 But then how do I query across services?

1. API composition — the caller (or an API gateway / BFF) calls services in parallel and stitches results: feed rendering gets post IDs from the timeline, then batch-fetches metadata, profiles, counts and "viewer has liked" flags.
2. Replicated read models (CQRS) — subscribe to others' events and keep a denormalised copy: Search builds Elasticsearch from UserUpdated/PostCreated and never queries the User DB.
3. Denormalise into the event — PostCreated carries author_id, timestamp, media type and audience so consumers needn't call back.

8. Distributed transactions: 2PC vs sagas

In one database, "do A and B atomically" is BEGIN; …; COMMIT;. Across services with separate databases there is no shared transaction.

8.1 Two-phase commit (2PC)

A coordinator asks every participant to prepare (write durably, hold locks, promise to commit). If all vote "yes" it tells everyone to commit; otherwise everyone aborts.

 Coordinator            Post DB                 User DB
     |--- PREPARE -------->|                        |
     |--- PREPARE ------------------------------->  |
     |<-- YES (locks held) |                        |
     |<-- YES (locks held) ------------------------ |
     |  write "COMMIT" to coordinator log           |
     |--- COMMIT --------->|                        |
     |--- COMMIT ---------------------------------->|
     |<-- ACK ------------ |<-- ACK ----------------|

 If the coordinator crashes after PREPARE, participants are stuck holding
 locks ("in doubt") until it recovers — 2PC is a BLOCKING protocol.

Problems at scale: blocking (locks held across round trips; stuck if the coordinator dies), availability (every participant must be up), latency (two round trips plus forced log writes), and support (many NoSQL stores and brokers don't speak XA/2PC). It's used inside engineered databases (Spanner and CockroachDB run 2PC over Paxos/Raft groups so the coordinator can't vanish), rarely between microservices.

8.2 Sagas

A saga breaks a business transaction into a sequence of local transactions, each committed in one service's own database. If a later step fails, the saga runs compensating transactions that semantically undo the earlier steps (you can't roll back a committed transaction, but you can apply its opposite). Sagas give up isolation — other requests can see intermediate states — in exchange for availability and loose coupling. This is why posts have explicit states like PROCESSING.

8.3 Instagram example: publishing a post as a saga

Publishing touches several domains. Suppose these steps (simplified):

1. Post service: insert post row with status = PROCESSING (local transaction) → emit PostSubmitted.
2. Media processing: generate renditions in S3 → emit MediaReady (or MediaFailed).
3. Integrity / moderation: classify the media and caption → emit ContentApproved (or ContentRejected).
4. Post service: set status = PUBLISHED; User service increments the author's post_count → emit PostCreated.
5. Fan-out, search, notifications: consume PostCreated.

Compensations: if moderation rejects the post after it was published (e.g. a delayed review), the saga sets status = REMOVED, decrements post_count, emits PostRemoved so fan-out workers delete the ID from timelines and the indexer drops it, deletes renditions, and notifies the author. If media processing fails, the post goes to FAILED and the client is told to retry. Notice that compensations must be idempotent and always succeed eventually (retry until done).

8.4 Choreography vs orchestration

9. The transactional outbox pattern

9.1 The dual-write problem

The Post service must update its database and publish PostCreated — two different systems — so this naive code is broken:

db.execute("UPDATE posts SET status='PUBLISHED' WHERE id=%s", post_id)
db.commit()
kafka.publish("post-events", PostCreated(post_id))   # what if we crash right here?

• Crash after commit, before publish → the post is published but no follower ever sees it in their feed, and it's never indexed.
• Publish first, then the DB commit fails → consumers fan out a post that doesn't exist.

9.2 The fix: write the event in the same local transaction

Add an outbox table to the service's own database. In one local ACID transaction, update the business row and insert the event into the outbox. A separate relay process reads the outbox and publishes to Kafka, marking rows as sent.

BEGIN;
  UPDATE posts SET status = 'PUBLISHED' WHERE post_id = 42;
  INSERT INTO outbox (event_id, aggregate_id, type, payload, created_at)
  VALUES ('evt-9f1c', 42, 'PostCreated',
          '{"post_id":42,"author_id":7,"media":"photo","audience":"followers"}', now());
COMMIT;      -- both happen, or neither does

-- relay loop (separate process)
loop:
  rows = SELECT * FROM outbox WHERE sent_at IS NULL ORDER BY created_at LIMIT 500
  for r in rows: kafka.publish(topic="post-events", key=r.aggregate_id, value=r.payload)
  kafka.flush()                         # wait for broker acks
  UPDATE outbox SET sent_at = now() WHERE event_id IN (rows.ids)

• Atomicity of "state changed" and "event recorded" comes from the local transaction.
• The relay can crash between publishing and marking sent → at-least-once delivery, so consumers must be idempotent (dedupe on event_id, or naturally idempotent ops like a sorted-set add).
• Keying Kafka messages by post ID keeps a post's events ordered on one partition.
• CDC relays (Debezium) tail the replication log (PostgreSQL WAL, MySQL binlog) instead of polling.

The consumer-side mirror is the inbox / idempotent consumer: record processed event_ids in the same transaction as the state change.

10. Deploying and running many services

10.1 Containers

A container (Docker/OCI image) packages a service with its exact runtime, libraries and config into one immutable artifact. The same image runs on a laptop, in CI and in production — no more "works on my machine". Containers share the host OS kernel (unlike virtual machines), so they start in seconds and you can pack dozens onto one host. Each build produces a versioned image (e.g. feed-service:2026.10.23-ab12cd); deploying means "run this image version".

10.2 Kubernetes in five concepts

• Pod — one or more containers sharing a network namespace (e.g. app + sidecar proxy).
• Deployment — "run 40 replicas of feed-service:v123"; Kubernetes reconciles reality to this desired state, rescheduling pods from dead nodes.
• Service — a stable name and virtual IP in front of changing pods (server-side discovery).
• Probes — liveness (restart if wedged) and readiness (no traffic until warmed up).
• Requests/limits — CPU and memory per pod, for bin-packing and noisy-neighbour protection.

10.3 Autoscaling

Because services are stateless, capacity can follow demand. The Kubernetes Horizontal Pod Autoscaler (HPA) computes:

desiredReplicas = ceil( currentReplicas × currentMetric / targetMetric )

Example: 40 feed pods at 90% average CPU, target 60%
         → ceil(40 × 90 / 60) = 60 pods

Metrics can be CPU, memory, requests per second or queue lag (e.g. scale fan-out workers on Kafka consumer lag). A cluster autoscaler adds or removes whole machines when pods can't be scheduled or nodes sit idle. Practical details matter: scale up fast, scale down slowly (avoid flapping); keep headroom because new instances need time to start and warm caches; and pre-scale for predictable peaks (evenings, New Year's Eve, big sports finals) rather than relying purely on reactive scaling.

10.4 Release strategies

11. Service mesh and sidecars

With hundreds of services, every one of them needs the same plumbing: mutual TLS, retries, timeouts, circuit breaking, load balancing, metrics, tracing headers. Re-implementing these in every language's client library is painful. A service mesh moves this plumbing out of the application into a sidecar proxy (most commonly Envoy) that runs next to every instance — in the same Kubernetes pod — and intercepts all inbound and outbound traffic.

• Data plane — the sidecars carrying traffic: client-side load balancing, retry budgets, outlier ejection, uniform metrics for every hop.
• Control plane — distributes routes (5% to v2: a mesh-level canary), policy (only Feed may call Graph) and short-lived certificates for mTLS.
• Costs — an extra local proxy hop per side (sub-millisecond, but it adds up), CPU/memory per pod, and a complex system to operate; sidecar-less designs (Istio ambient, eBPF/Cilium) reduce this.

Meta built equivalent capabilities into shared RPC libraries (Thrift-based) long before off-the-shelf meshes existed; a mesh is the "buy" version of that "build".

12. Conway's law: architecture mirrors the org chart

"Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure." — Melvin Conway, 1968

If you have three teams building a compiler, you get a three-pass compiler. Applied to microservices: service boundaries follow team boundaries, whether you plan it or not. The practical lessons:

• Design teams and services together (the "inverse Conway manoeuvre"): want a Stories service? Create a Stories team that owns it end to end.
• One owner per service; a service owned by everyone is owned by no one.
• Small orgs shouldn't copy big-org architectures. Ten engineers running 40 services contradicts the org chart.

13. When to split — and when not to

13.1 Good reasons to extract a service

• Different scaling or runtime profile: feed reads at 100k+ QPS vs settings at 10 QPS; GPU transcoding; long-lived WebSocket connections.
• Different security requirements: auth and payments benefit from a small, tightly controlled surface.
• Team autonomy: a large team blocked by the shared release train.
• A clean bounded context with a narrow API and little shared data.

13.2 Bad reasons

• "Microservices are modern." Fashion is not a requirement.
• A small team, or domain boundaries still changing weekly — you'll cut in the wrong place.
• Messy code: a messy monolith split up becomes a messy distributed system. Modularise first.
• Entities constantly read and written together — you'll recreate joins over the network.

13.3 How to split safely: the strangler fig

1. Modularise in place: define the module's API and stop other modules touching its tables.
2. Route through a facade (gateway or in-code) and build the new service behind the same API; shadow traffic to compare responses.
3. Migrate data: backfill, keep in sync via CDC, then move ownership.
4. Shift traffic gradually (canary %), then delete the old code — the new service has "strangled" the old path like a strangler fig around a tree.

14. How Instagram uses microservices & stateless services

Putting it together, a realistic reading of the diagram:

• The web tier is stateless: thousands of identical Django servers behind load balancers; tokens for auth, data in sharded PostgreSQL/Cassandra/caches, media in object storage — so the fleet scales horizontally and redeploys many times a day.
• "Services" in the diagram are domains (Upload, Stories, Like & Comment, Feed), largely modules in the monolith with clear ownership of tables and events.
• Separate systems exist where runtime or scale differs: messaging connections, media pipelines, search, notification delivery, ML ranking, and Meta-wide graph and cache infrastructure.
• Async events glue domains together (PostCreated, StoryCreated, LikeCreated, FollowCreated) — choreography made reliable by outbox-style publishing and idempotent consumers.

15. Trade-offs & common pitfalls

Common pitfalls

• Distributed monolith — services that must deploy together or share a database.
• Chatty synchronous chains — A calls B calls C calls D; latencies add up and availability multiplies down. Prefer parallel fan-out, batching APIs and async events.
• Dual writes without an outbox — "write DB, then publish" silently loses events on crashes.
• Non-idempotent consumers — at-least-once delivery means duplicates; a duplicate LikeCreated must not double-count.
• Hidden state in "stateless" services — files on local disk, in-memory rate-limit counters, sticky sessions relied on for correctness.
• Retry storms — every layer retrying 3× turns one slow call into 27; use retry budgets, backoff with jitter and deadlines.

16. Real-world technologies

17. Interview corner

18. Check your understanding

Related lessons

• API Design — REST vs gRPC, idempotency keys and versioning for service APIs
• Message Queues & Kafka — the event bus behind choreography and the outbox
• Load Balancing — why stateless instances make balancing simple; sticky sessions
• API Gateway — the front door that routes to services and composes responses
• Resilience Patterns — timeouts, retries, circuit breakers between services
• Observability — distributed tracing across many services
• Databases — choosing a store per service
• CAP & Consistency — why cross-service data is eventually consistent
• Media Upload — a full saga-like flow with a post state machine and outbox