1. The problem: why is uploading media hard?

Posting a photo looks trivial: tap Share, done. Underneath, Instagram must accept a few megabytes (photo) to hundreds of megabytes (a long Reel) from a phone on an unreliable cellular network, store it durably forever, and do so for well over a hundred million media items a day without ever losing one or creating duplicates. Specifically:

• Size and duration. A 60 MB video over a 5 Mbps uplink takes ~96 seconds. Mobile connections drop, switch from Wi-Fi to 4G, or the user locks the phone mid-upload.
• Retries create duplicates. If the response is lost after the server saved the post, the app retries — and naive code creates a second post.
• Scale and cost. Ingest is tens of gigabytes per second at peak. Routing those bytes through Python web servers would waste enormous compute.
• Trust. Uploaded files come from the internet: wrong extensions, gigantic dimensions, malware, illegal content.
• Multiple steps. Stored bytes alone are not a post: metadata, renditions, moderation, feed fan-out and search indexing must all follow, each able to fail independently.

2. Why not just stream bytes through the app servers?

The naive design is POST /posts with a multipart body containing the photo; the web server receives it and writes it to S3. It works for a prototype and fails at scale:

Object stores like S3 are built to ingest enormous parallel byte streams from anywhere with high durability. The right design keeps the control plane (who may upload what, metadata, state) in your service and moves the data plane (the bytes) directly between client and storage.

3. Pre-signed URLs

3.1 What they are

A pre-signed URL is a URL to a specific object in a bucket that embeds an expiry time and a cryptographic signature made with the service's secret credentials. Anyone holding it may perform exactly the operation it was signed for (e.g. PUT this key) until it expires — without having any storage credentials themselves.

PUT https://media-uploads.s3.amazonaws.com/staging/7/3141592653589793238/orig
    ?X-Amz-Algorithm=AWS4-HMAC-SHA256
    &X-Amz-Credential=AKIA.../20261023/us-east-1/s3/aws4_request
    &X-Amz-Date=20261023T091500Z
    &X-Amz-Expires=900                       # valid for 15 minutes
    &X-Amz-SignedHeaders=content-type;host
    &X-Amz-Signature=5f2b9c...e81

3.2 How the signature works

1. The Upload service builds a canonical request: method (PUT), object key, expiry, and selected headers (e.g. Content-Type: image/jpeg).
2. It computes an HMAC-SHA256 over that string using a key derived from its secret access key, and appends the result as X-Amz-Signature.
3. When the client uses the URL, S3 recomputes the HMAC with the same secret. If anything was changed — another key, method, content type, or a later time — the signatures don't match and S3 returns 403.

No call to S3 is needed to create the URL — it's pure local cryptography, so the Upload service can mint thousands per second cheaply.

3.3 Restricting what may be uploaded

• Short expiry (5–15 minutes; per-part URLs for multipart), so leaked URLs are useless quickly.
• Server-chosen object key — the client never picks the path, so it can't overwrite someone else's object.
• Signed headers: Content-Type and, where supported, a checksum header (x-amz-checksum-sha256) so S3 rejects bytes that don't match the declared hash.
• Size limits: a pre-signed POST policy can include content-length-range (e.g. 1 byte – 30 MB). With pre-signed PUT, sign Content-Length or verify size at completion.
• Staging location: uploads land in a staging/ prefix (or bucket) that nothing serves publicly until validated.

4. The end-to-end upload flow

 App                    Upload service              Metadata DB        S3 (staging)     Kafka
  |  (1) POST /uploads      |                            |                  |              |
  |  {idempotency_key,      |                            |                  |              |
  |   type, size, sha256}   |                            |                  |              |
  |-----------------------> | (2) auth, quota, validate  |                  |              |
  |                         |     post_id = new ID       |                  |              |
  |                         |---- INSERT post(UPLOADING) |                  |              |
  |                         |     + upload session ----->|                  |              |
  | <-- (3) upload_id,      |                            |                  |              |
  |     pre-signed URL(s)   |                            |                  |              |
  |                                                                         |              |
  |  (4) PUT bytes directly (multipart, resumable) ------------------------>|              |
  | <------------------------------------------------------- 200 + ETags --|              |
  |                                                                         |              |
  |  (5) POST /uploads/{id}/complete {parts, caption, ...}                  |              |
  |-----------------------> | (6) HEAD object: size, checksum ok? ------->|              |
  |                         | (7) one transaction:       |                  |              |
  |                         |   post -> PROCESSING       |                  |              |
  |                         |   outbox += MediaUploaded->|                  |              |
  | <-- 202 Accepted        |                            |                  |              |
  |     {post_id, PROCESSING}                            | (8) relay ------------------->  |
  |                                                      |                  |  (9) media workers,
  |                                                      |                  |  moderation consume
  |                         | (10) MediaReady -> post PUBLISHED + outbox PostCreated ----> |
  |                                                                       fan-out, search,
  |                                                                       notifications

1. Request an upload. The app calls POST /uploads with an idempotency key, media type, byte size and (optionally) a SHA-256 of the file. The gateway authenticates the user and applies rate limits.
2. Create the upload session. The Upload service checks quotas and declared limits, generates a post ID (section 15), records a post row in state UPLOADING plus an upload session (expected size, parts, expiry, object key).
3. Return pre-signed URL(s) — one for a small photo, or one per part for multipart.
4. Direct upload. The phone PUTs bytes to object storage; S3 returns an ETag per part.
5. Completion callback. The app calls POST /uploads/{id}/complete with part ETags plus caption, tags, location and audience. (Alternatively, S3 event notifications can tell the service an object arrived; many designs use both — the client callback carries the metadata, the storage event acts as a safety net.)
6. Verify. The service completes the multipart upload and checks the object really exists with the expected size and checksum. It never trusts the client's claim.
7. Metadata write + event, atomically. In one local transaction: post → PROCESSING, caption stored, and a MediaUploaded event inserted into the outbox. Respond 202 Accepted.
8. Relay publishes outbox rows to Kafka.
9. Media processing creates renditions, thumbnails and video segments; moderation classifiers run.
10. Publish. On MediaReady (and passing checks) the post becomes PUBLISHED and PostCreated is emitted via the outbox, driving feed fan-out, search indexing and notifications.

5. Multipart / chunked uploads

Sending a 200 MB video as one HTTP request is fragile: any blip restarts it from zero, and a single TCP stream rarely saturates a fast link. Multipart upload splits the file into parts uploaded independently (and in parallel), then stitched together server-side.

1. CreateMultipartUpload(key)            → upload_id
2. UploadPart(upload_id, part=1..N, bytes) → ETag per part   (each via its own pre-signed URL)
3. CompleteMultipartUpload(upload_id, [(1,ETag1), (2,ETag2), ...])  → final object
   (or AbortMultipartUpload to discard)

• S3 limits: parts are 5 MiB–5 GiB (except the last), up to 10,000 parts, objects up to 5 TiB. Other object stores have similar rules.
• Parts can be uploaded in parallel (2–4 concurrent streams on mobile) and in any order; a failed part is retried alone.
• Parts uploaded but never completed still cost storage until aborted (see section 12).

Adaptive clients pick chunk size from measured throughput: bigger on Wi-Fi, smaller on weak cellular.

6. Resumable uploads

Multipart helps, but mobile needs more: if the app is killed or the network vanishes for 10 minutes, the upload must continue from where it stopped, not restart. A resumable protocol revolves around an upload session stored server-side:

 App                                   Upload endpoint (session store)
  | POST /uploads {size=48MB, sha256}  ->  session u_81 {received=0, expires=+24h}
  | PUT /uploads/u_81  Content-Range: bytes 0-8388607/50331648        -> 200 received=8MB
  | PUT /uploads/u_81  Content-Range: bytes 8388608-16777215/50331648 -> 200 received=16MB
  | PUT ... bytes 16777216-...   ✖ network lost (tunnel, app killed)
  |
  |  ... 10 minutes later, connectivity back ...
  | HEAD /uploads/u_81                     -> 200  Upload-Offset: 16777216
  | PUT /uploads/u_81  Content-Range: bytes 16777216-25165823/50331648 -> received=24MB
  | ...                                    -> received=48MB, complete ✔ checksum verified

• Upload session ID — created once, persisted on the device together with the file path and idempotency key, so it survives app restarts.
• Offset query — before resuming, the client asks "how many bytes do you have?" (the server's answer is authoritative; the client may have sent bytes that never arrived).
• Chunk integrity — each chunk can carry a checksum; the final object is verified against the full-file hash.
• Session expiry — sessions live hours to days; expired sessions are garbage-collected.
• Background transfer — iOS background URLSession and Android WorkManager let the OS continue uploads when the app isn't in the foreground.

Standard protocols: tus (open resumable upload protocol), Google Cloud Storage resumable uploads, and S3 multipart (resume by listing already-uploaded parts). Meta's public Graph API offers a resumable upload endpoint for Instagram video/Reels publishing, a sign that the same idea is used across its apps.

7. Client-side work before the upload

• Resize and compress. Phone cameras produce 12–48 MP images (5–15 MB). Instagram displays photos at up to 1080 px wide, so the app resizes to that and re-encodes (JPEG/HEIC/WebP) to a few hundred KB–~1.5 MB — a 5–10× saving on upload time, ingest bandwidth and storage. Videos are transcoded on device to a target resolution/bitrate.
• Strip sensitive metadata. EXIF may contain GPS coordinates; strip it (on device and again on the server) unless the user explicitly adds a location.
• Compute the content hash for dedup and integrity.
• Start early. Instagram's early engineers publicly described a classic trick: begin uploading as soon as the user picks and filters the photo, while they're still typing the caption. By the time Share is tapped the bytes are often already there, so posting feels instant. If the user cancels, the orphaned upload is cleaned up later.

8. Validation and safety

Objects stay in a quarantined staging prefix that no CDN can read until validation passes; then they are promoted (copied or referenced) into the permanent media location.

9. Deduplication via content hash

Identical files are common: the same meme reposted, the same video re-uploaded after a failed attempt, an app retry that re-sends everything. Content-addressed storage uses a cryptographic hash (SHA-256) of the bytes as identity:

POST /uploads {sha256: "9f86d0...", size: 1843221, ...}
server: blob = blob_index.get("9f86d0...")
        if blob exists and blob.size == size:
            # skip the byte upload entirely
            create post referencing blob_id; refcount += 1
            return {upload_needed: false, post_id}
        else:
            return pre-signed URL for key "blobs/9f86d0..."

• Saves upload time, bandwidth and storage; the post metadata just points to an existing blob.
• Reference counting: a blob may be shared by many posts; delete it only when the last reference goes (and respect legal holds).
• Verify, don't trust: if the client claims a hash, the server must still verify the bytes it receives hash to that value, or an attacker could "claim" someone else's blob.
• Privacy side channel: global cross-user dedup can leak whether a file already exists ("upload is instant → someone has this file"). Mitigations: per-user dedup only, or always require a real upload for small files.
• Limits: exact hashes break with a single changed byte (a re-encode or crop). Near-duplicate detection uses perceptual hashes — useful for moderation and ranking, not storage dedup.
• Retries within one upload are better handled by the idempotency key (section 11) than by hashing.

10. The post state machine

Because publishing spans several asynchronous steps, a post is modelled as an explicit state machine. Every component checks and advances the state; nothing else decides whether followers can see a post.

Transitions are conditional updates (compare-and-set), so duplicate or out-of-order events can't corrupt state:

UPDATE posts SET status = 'PUBLISHED', published_at = now()
WHERE post_id = 42 AND status = 'PROCESSING';
-- 1 row updated → we won: insert PostCreated into outbox in the same transaction
-- 0 rows updated → already published (duplicate MediaReady) or deleted: do nothing

A watchdog also alerts on posts stuck in PROCESSING longer than expected, and re-drives them.

11. Idempotency on retries

Mobile clients retry constantly: timeouts, app restarts, flaky radios. The dangerous case: the server did create the post, but the response was lost. Without protection, the retry creates a duplicate post. The fix is an idempotency key: a UUID the client generates once per user action ("share this post") and sends with every retry of that action.

def create_upload(user_id, idem_key, req):
    existing = idem_store.get((user_id, idem_key))
    if existing:
        return existing.response                  # replay the original answer
    with db.transaction():
        # unique constraint on (user_id, idem_key) protects against two concurrent first attempts
        post_id = id_gen.next()
        db.insert_post(post_id, user_id, status="UPLOADING", idem_key=idem_key)
        resp = {"post_id": post_id, "upload_id": ..., "urls": sign_urls(post_id)}
        idem_store.put((user_id, idem_key), resp, ttl=24h)
    return resp

Downstream consumers are idempotent too (dedupe on event ID, conditional state transitions), because Kafka delivery is at-least-once. See API design and resilience patterns.

12. Cleaning up orphaned blobs

Direct-to-storage uploads inevitably leave orphans: bytes in object storage with no published post pointing at them. Sources include users who cancel after the early upload started, apps killed mid-upload, multipart uploads never completed, posts that FAILED, and deleted posts. At hundreds of millions of uploads a day, even a 2% orphan rate is petabytes per year of paid-for garbage.

{ "Rules": [
  { "ID": "abort-stale-multipart", "Filter": { "Prefix": "" }, "Status": "Enabled",
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 2 } },
  { "ID": "expire-staging", "Filter": { "Prefix": "staging/" }, "Status": "Enabled",
    "Expiration": { "Days": 3 } } ] }

13. Carousels: multiple media per post

A carousel post contains several photos and/or videos (Instagram raised the limit from 10 to 20 items in 2024). Model it as one post with an ordered list of media:

posts       (post_id PK, author_id, caption, status, media_count, created_at, ...)
post_media  (post_id, position, media_id, media_type, width, height, status)  PK (post_id, position)
media       (media_id PK, blob_key, sha256, renditions{...}, duration_ms, status)

1. The client requests one upload session per item (or a single batch call returning N pre-signed URL sets) — all under the post's idempotency key.
2. Items upload in parallel (bounded concurrency, e.g. 2–3 at a time on mobile), each with its own resumable session.
3. /complete sends the ordered list of media IDs plus caption; the post enters PROCESSING.
4. Each item is processed independently; the post becomes PUBLISHED only when all items are ready (a per-post counter or a check of all post_media.status values, done with a conditional update so exactly one worker flips it).
5. If one item fails, the post is not published half-complete; the author is asked to retry or remove that item.

Carousels multiply upload volume (average items per post > 1), and the feed must render the first item fast while lazily loading the rest — so the first item's renditions are prioritised in processing and prefetching.

14. Reels and large videos

• Size: a few minutes of 1080p video is tens to hundreds of MB even after on-device compression. Always multipart + resumable, with background transfer.
• Upload close to the user: large providers terminate uploads at a nearby edge PoP and forward over their private backbone (the same idea as S3 Transfer Acceleration), cutting round-trip time on each chunk.
• Process while uploading: Facebook's published Streaming Video Engine (SOSP 2017) splits video into segments as they arrive and transcodes them in parallel, so processing overlaps the upload instead of starting after it — dramatically reducing time-to-publish.
• Progressive availability: publish once a baseline rendition (e.g. 480p) and the first segments exist; higher-quality renditions for adaptive bitrate streaming are added later.
• Extra metadata: cover frame, audio track/music licensing, duration, aspect ratio, captions — all part of the PROCESSING step.
• Priority queues: short photo jobs shouldn't wait behind long transcodes; separate worker pools/queues per media type.

15. Generating the post ID

The Upload service needs the post's ID at step 2 — before any bytes arrive — because the ID becomes part of the object key, the idempotency record, and later every event. Requirements: globally unique across all shards, generated without a central bottleneck, and roughly time-sortable so feeds and profile grids can sort by ID.

Instagram's publicly described scheme (2012 engineering blog) packs a 64-bit ID as:

 | 41 bits: milliseconds since custom epoch | 13 bits: logical shard ID | 10 bits: per-shard sequence |
   ~69 years of timestamps                    8,192 logical shards         1,024 IDs / ms / shard

It was generated inside each PostgreSQL logical shard by a PL/pgSQL function, so the ID also tells you which shard holds the row. See Unique ID generation for Snowflake, UUIDs and trade-offs.

16. Publishing PostCreated reliably

Everything downstream — follower feeds, search, notifications, analytics — hinges on the PostCreated event. If the post commits but the event is lost, followers never see it; if the event goes out but the commit fails, consumers process a ghost post. So the event is written with the transactional outbox pattern (explained fully in Microservices): the state change to PUBLISHED and the outbox insert happen in one local transaction, and a relay (poller or CDC) publishes outbox rows to Kafka.

{
  "event_id":   "evt-01HF7Q...",          // consumers dedupe on this
  "type":       "PostCreated",
  "post_id":    3141592653589793238,
  "author_id":  7,
  "created_at": "2026-10-23T09:16:02Z",
  "media":      [{"media_id": 991, "type": "photo"}, {"media_id": 992, "type": "video"}],
  "audience":   "followers",              // or close_friends, public...
  "caption_lang": "en",
  "schema_version": 3
}

Consumers must tolerate duplicates (at-least-once) and the event carries enough data (author, audience, media types) that most consumers needn't call back to the Upload service.

17. Estimating upload volume

Assumptions (not official figures): 500M users post feed media or stories on a given day; 150M feed/Reels media items (carousels included) and 400M story items per day; after client compression photos average 1 MB and videos 15 MB; 30% of items are video.

Items/day         = 150M + 400M                 = 550M
Average upload QPS = 550M / 86,400 s            ≈ 6,400 uploads/s   (peak ×3 ≈ 19,000/s)
Avg item size     = 0.7 × 1 MB + 0.3 × 15 MB    = 5.2 MB
Ingest/day        = 550M × 5.2 MB               ≈ 2.9 PB/day
Ingest bandwidth  = 2.9 PB / 86,400 s           ≈ 33 GB/s ≈ 265 Gbps average (~800 Gbps peak)
Control-plane API = ~2–3 calls per item         ≈ 15k–60k req/s peak — tiny vs the byte stream
Renditions        ≈ +50–100% storage on top of originals, then × replication/erasure coding

The control plane is a modest web service; the data plane is a firehose that only object storage and the network edge should touch. That asymmetry is the whole argument for pre-signed direct uploads. Redo this with your own numbers using Back-of-the-envelope estimation.

18. How Instagram uses the Upload service

• Stories use the same path with a different post type and TTL — see Stories.
• DM media uses the same direct-upload idea, with access restricted to the thread's participants.
• Profile photos are small, but still go through validation and renditions.

19. Trade-offs & common pitfalls

Common pitfalls

• Letting the client choose the object key → overwriting other users' media.
• Long-lived upload URLs or logging them in full → anyone can upload into your bucket.
• Trusting Content-Type or file extension instead of decoding the file.
• No idempotency key → duplicate posts on retries and double-taps.
• "Write DB then publish to Kafka" without an outbox → posts that never reach followers.
• Forgetting incomplete multipart uploads → invisible, billed storage growing forever.
• Publishing before processing finishes → followers see broken images.

20. Real-world technologies

21. Interview corner

22. Check your understanding

Related lessons

• Object Storage — where the bytes land; durability, lifecycle rules
• Media Processing — what happens after upload: renditions, transcoding, moderation
• Unique ID Generation — Instagram's 64-bit post IDs
• Microservices — sagas and the transactional outbox in depth
• API Design — idempotency keys and two-phase APIs
• Message Queues & Kafka — how PostCreated reaches consumers
• News Feed — the biggest consumer of PostCreated
• CDN — how the processed media is served back to viewers
• Security — signed URLs, malware scanning and abuse prevention