1. The problem: why do we need object storage?

Every Instagram post starts as a multi-megabyte photo or video. Each upload becomes many files — the original plus several resized images and a ladder of video bitrates and segments (see Media Processing). Assume, purely for practice, 100 million photos and 20 million videos uploaded per day: that's hundreds of terabytes per day and hundreds of petabytes per year, kept essentially forever, any of which might be viewed at any moment.

Where could all that live?

• In the database as BLOB columns? Databases are optimised for small rows, indexes and transactions. Multi-MB values bloat the buffer pool, slow replication and backups, and make the DB 10–50× more expensive per GB. The metadata (who posted, caption, timestamps) belongs in a database; the bytes don't.
• On a big network file system (NAS/NFS)? File systems maintain a directory tree, permissions, in-place writes and locking — all of which cost metadata operations. At billions of files the metadata itself becomes the bottleneck (Facebook hit exactly this; see Haystack below).
• On local disks of app servers? Then a server failure loses photos and every server would need to know which server has which file.

Object storage is the answer: a giant, cheap, durable, HTTP-addressable key → bytes store that scales horizontally by adding disks.

2. Block vs file vs object storage

There are three fundamental ways to expose storage. Understanding the difference is a classic interview warm-up.

3. The object model

Flat namespace. There are no real directories. The key p/3049/1080.jpg is just a string containing slashes. When a console shows "folders", it is running LIST with prefix=p/3049/ and delimiter=/ and grouping keys that share a prefix. This is a big reason object stores scale: there's no directory tree to lock, no rename of a directory (renaming means copying every object), and the index is just a giant sorted key → location map that can be partitioned by key range.

Immutability. You can't append or overwrite bytes 100–200; a PUT replaces the entire object atomically — readers see either the old or the new object, never a mix. With versioning enabled, an overwrite creates a new version and a delete inserts a "delete marker", so older versions remain recoverable (protecting against bugs and ransomware, at the cost of storing every version until a lifecycle rule expires them).

For Instagram this model is a perfect fit: processed media is write-once, read-many and never edited. An "edit" (e.g. a new filter) produces new renditions under new keys.

4. The S3-style API

4.1 Basic operations

PUT    /ig-media/p/3049215820194820101/1080.jpg     (body = bytes, Content-Type: image/jpeg)
GET    /ig-media/p/3049215820194820101/1080.jpg     (optional Range: bytes=0-1048575)
HEAD   /ig-media/p/3049215820194820101/1080.jpg     (metadata only: size, ETag, type)
DELETE /ig-media/p/3049215820194820101/1080.jpg
GET    /ig-media?list-type=2&prefix=p/3049215820194820101/   (LIST keys by prefix, paginated, 1,000 per page)

Range GETs fetch part of an object — essential for video players seeking within a file and for parallel downloads. ETag is a checksum-like version identifier useful for conditional requests and CDN revalidation.

4.2 Multipart upload

A single PUT is fine for a 3 MB photo but terrible for a 2 GB video on a mobile network: one dropped connection at 95% means starting over. Multipart upload splits an object into parts uploaded independently and in parallel:

1. CreateMultipartUpload → returns an UploadId.
2. UploadPart (partNumber 1…10,000, each 5 MB–5 GB except the last) — in parallel, in any order, retrying only failed parts. Each returns an ETag.
3. CompleteMultipartUpload with the list of (partNumber, ETag) → the store stitches parts into one object, atomically visible.
4. Or AbortMultipartUpload. Incomplete uploads keep their parts (and keep costing money) until aborted — add a lifecycle rule AbortIncompleteMultipartUpload: 1 day.

Benefits: resumability on flaky mobile networks, higher throughput through parallelism, and the ability to start uploading before the whole file exists (e.g. while the phone is still encoding). See Media Upload for the client side.

4.3 Pre-signed URLs: let the client talk to storage directly

Routing upload bytes through the Upload Service would make those servers a bandwidth bottleneck. Instead the service issues a pre-signed URL: a URL for one specific operation on one specific key, containing an expiry and an HMAC signature computed with the service's secret credentials. Anyone holding the URL can perform exactly that operation until it expires — no AWS credentials on the phone.

 Phone                    Upload Service                 Object Storage (S3)          Kafka
   |-- POST /uploads ------------>|                               |                      |
   |   (auth token, size, type)   | authZ, quota, pick key        |                      |
   |                              | sign PUT URL (expires 15 min) |                      |
   |<-- {upload_url, media_id} ---|                               |                      |
   |-- PUT bytes (multipart) -----------------------------------> |                      |
   |<------------------------------------------------- 200 OK --- |                      |
   |                              |<-- ObjectCreated event -------|                      |
   |-- POST /posts {media_id,...}>|-- "media.uploaded" ------------------------------>  |
   |                              |                               |   media workers resize/transcode

import boto3
s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "ig-uploads-raw", "Key": f"raw/{media_id}",
            "ContentType": "image/jpeg"},
    ExpiresIn=900)            # 15 minutes; signature covers bucket, key, method, headers, expiry
# The phone PUTs to `url`. The signature can't be reused for another key or for GET.

Security details: keep expiries short; sign the content type and (with POST policies) a maximum size; upload into a separate raw/quarantine bucket that is never served publicly until the media pipeline validates it; generate keys server-side so clients can't overwrite others' objects. Pre-signed GET URLs work the same way for private downloads, though at Instagram's scale downloads go through CDN-signed URLs (see CDN).

5. Inside an object store

A distributed object store has three logical layers. Understanding them explains its performance and consistency behaviour.

• Front-end tier — stateless HTTP servers that authenticate requests (checking signatures and bucket policies), split large uploads, and stream bytes. Scaled horizontally behind load balancers.
• Metadata / index service — a distributed, strongly consistent key-value index mapping (bucket, key, version) → size, checksum and the list of chunk locations. It's partitioned by key range (so LIST prefix=… is a range scan) and replicated with a consensus protocol. This index, not the disks, is what limits request rate per key prefix.
• Storage nodes — servers packed with large HDDs (cheap per TB) that store chunks, often appended into large container files, each protected by a checksum. Background processes scrub (re-read and verify checksums), repair lost chunks from redundancy, and rebalance when nodes are added.

A PUT streams the data to several storage nodes (replicas or erasure-coded fragments), waits until enough have durably written it, then commits the index entry — only then does it return 200. That ordering is why an acknowledged write survives failures.

6. Durability: replication and erasure coding

Durability is the probability that data you stored is still there (uncorrupted) later. It's different from availability (can I read it right now?). S3 Standard is designed for 99.999999999% ("11 nines") annual durability and 99.99% availability.

What does 11 nines mean? The annual probability of losing a given object is about 10⁻¹¹. If you store 10 million objects, you'd expect to lose one object about once every 10,000 years. At Instagram's scale (say 10¹² objects) the same rate still means ~10 objects per year — which is why hyperscalers add cross-region redundancy on top.

Disks fail constantly at scale: with a ~1–2% annual failure rate, a fleet of 500,000 drives loses on the order of 15–25 drives per day. Durability comes from storing redundant data in independent failure domains (disk, server, rack, power, availability zone) and repairing faster than further failures arrive.

6.1 Replication

Store N full copies (typically 3) on different nodes, ideally in different availability zones. Simple, fast to read (any copy) and fast to repair (just copy), but it costs 3× the raw storage — 200% overhead. For hundreds of petabytes that's hundreds of petabytes of extra disks.

6.2 Erasure coding — the intuition

Erasure coding is "RAID for the data center". Split the object into k data fragments, compute m parity fragments from them, and store all k + m fragments on different nodes. The magic property of a Reed-Solomon code: any k of the k + m fragments are enough to reconstruct the original. You survive the loss of any m fragments while storing only (k + m) / k times the data.

Simplest case — XOR parity (k = 2, m = 1):

D1 = 1011        D2 = 0110
P  = D1 XOR D2 = 1101             (store D1, D2, P on three different nodes)

Lose D2?  D2 = D1 XOR P = 1011 XOR 1101 = 0110  ✔
Lose D1?  D1 = D2 XOR P = 0110 XOR 1101 = 1011  ✔
Overhead: 3/2 = 1.5×, survives any 1 loss (replication would need 2× for the same).

XOR only survives one loss. Reed-Solomon generalises this: treat the k data fragments as coefficients of a polynomial over a finite field (Galois field arithmetic, so values stay in a byte range); the k + m fragments are the polynomial evaluated at k + m different points. A polynomial of degree k − 1 is uniquely determined by any k points (just as any 2 points determine a line), so any k surviving fragments let you recover all the data. Real systems use parameters like RS(6,3) (1.5×, survives 3 losses), RS(10,4) (1.4×, survives 4 losses) or wider.

Practical systems often combine both: replicate fresh, hot data (fast writes, cheap repairs) and erasure-code it once it cools — exactly the Haystack → f4 journey in section 9. Durability also needs end-to-end checksums (detect silent bit-rot), background scrubbing, and fast repair: the shorter the window between a failure and its repair, the less likely a second and third failure land in the same stripe.

7. Consistency

For years S3 offered read-after-write consistency for new objects but only eventual consistency for overwrites, deletes and LIST — a reader might see an old version or a list missing a just-written key. Since December 2020, S3 provides strong read-after-write consistency for all PUTs and DELETEs of objects and for LIST operations, at no extra cost. Google Cloud Storage and Azure Blob Storage are also strongly consistent for object operations.

• What "strong" means here: after a successful PUT returns, any subsequent GET or LIST (from anywhere) sees the new object/version.
• Concurrent writers: two simultaneous PUTs to the same key don't merge; the one with the latest timestamp wins (last-writer-wins). There is no locking. S3 has added conditional writes (e.g. If-None-Match: * to create-only-if-absent, and If-Match on ETag), which let you build compare-and-swap; otherwise coordinate via a database.
• No multi-object transactions: writing an object and its metadata row in your DB are separate; design for the gap (write the object first, then the DB row that references it; garbage-collect orphans).
• Caches downstream are not strongly consistent: the CDN may still serve an old copy — another reason to use immutable, versioned keys.
• Cross-region replicas are asynchronous, so a replica bucket is eventually consistent with its source (see section 13).

See CAP & Consistency Models for the vocabulary.

8. Storage classes, lifecycle policies and cost

Not all bytes are equal. A photo posted today might be fetched (by the CDN) thousands of times this week and almost never next year. Object stores offer storage classes that trade lower storage price for higher retrieval cost or latency. Approximate S3 US list prices at the time of writing (check current pricing; at Meta's scale, custom hardware replaces these):

Lifecycle policies move or delete objects automatically based on age, prefix or tags:

{
  "Rules": [
    { "ID": "renditions-cool-down", "Filter": { "Prefix": "p/" }, "Status": "Enabled",
      "Transitions": [ { "Days": 60,  "StorageClass": "STANDARD_IA" },
                       { "Days": 365, "StorageClass": "GLACIER_IR" } ] },
    { "ID": "raw-uploads-cleanup", "Filter": { "Prefix": "raw/" }, "Status": "Enabled",
      "Expiration": { "Days": 7 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } },
    { "ID": "old-versions", "Filter": {}, "Status": "Enabled",
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30 } }
  ]
}

8.1 Cost reasoning: when does a cooler tier pay off?

Moving from Standard ($0.023) to Standard-IA ($0.0125) saves $0.0105 per GB-month but adds ~$0.01 per GB retrieved. So IA wins only if each byte is fetched from the origin less than about once a month. That's true for most old Instagram media precisely because the CDN absorbs repeat views. Other traps:

• Small objects: IA bills each object as at least 128 KB — a 20 KB thumbnail costs 6× its size. Keep tiny thumbnails in Standard (or pack them).
• Minimum durations: deleting a Glacier object after 10 days still bills 90 days. Don't archive Stories that expire in 24 hours.
• Archive tiers are not for user-facing media: nobody will wait 12 hours for a 2015 photo to load. Archive is for disaster-recovery copies of originals, compliance and logs.
• Request and transition fees add up with billions of objects: transitioning 10 billion objects at ~$0.01 per 1,000 requests is ~$100K by itself.

9. Lessons from Facebook's Haystack and f4

Facebook's photo storage papers — Finding a needle in Haystack (OSDI 2010) and f4: Facebook's Warm BLOB Storage System (OSDI 2014) — are the best public explanation of how a social network stores billions of small media files. Instagram today runs on Meta's infrastructure, so these ideas are directly relevant.

9.1 The problem: per-file metadata overhead

Facebook originally stored each photo as a file on NFS-mounted network storage. The Haystack paper (which reported ~260 billion images and ~20 PB at the time) explains why that broke down:

• Reading one photo from a regular file system takes several disk operations: translate the path to an inode (directory lookups), read the inode, then read the data. Directories with thousands of files made it worse; the paper reports about 3 disk operations per photo in practice.
• The obvious fix — keep all metadata in RAM — doesn't fit: a Linux XFS inode structure is hundreds of bytes (the paper cites 536 bytes), times billions of files, and most of it (permissions, owner, timestamps) is useless for photos.
• The CDN absorbs popular photos, but social networks have a long tail: many requests are for old, unpopular photos that miss the CDN and hit storage. Those reads were dominated by metadata seeks, not by reading the photo bytes.

Lesson: at billions of small objects, metadata, not bandwidth, is the bottleneck. The goal became "one disk operation per photo read".

9.2 The Haystack design: needles in big volume files

• Physical volumes: each Store machine holds ~100 large files (~100 GB each). Millions of photos are appended into each file as needles (header, cookie, 64-bit key, alternate key for the size variant, flags, data, checksum).
• Logical volumes: a write is appended synchronously to the same logical volume on 3 physical volumes on different machines (replication).
• In-memory index: each Store keeps a map (key, alt key) → (flags, offset, size) — on the order of 10 bytes per photo — so reading is: look up RAM, one pread() at the offset. An index file checkpoint speeds up restarts.
• Haystack Directory: maps logical volumes to physical machines, load-balances writes over writable volumes, and marks full volumes read-only. URLs encode the CDN, Cache, machine and logical volume + photo ID, so serving needs no extra lookup.
• Haystack Cache: an internal cache in front of the Stores, mainly for recently uploaded photos (the hottest ones) that the CDN hasn't absorbed yet.
• Deletes just set a flag; a background compaction copies live needles to a new file and drops deleted ones — the same trick as LSM-tree compaction (see Indexing).

Modern object stores use the same core idea internally: pack small objects into large append-only containers and keep a compact index — it's why S3 can store trillions of objects.

9.3 f4: warm storage with erasure coding

Haystack's replication (3 copies, each on RAID-6 disks) gave an effective replication factor of about 3.6×. But request rates for photos and videos drop sharply with age: the newest content is "hot", while the vast majority of bytes are "warm" — still occasionally viewed, still needing low latency, but rarely. Paying 3.6× for warm data is wasteful. The f4 paper describes moving BLOBs whose volumes have cooled (after some months) from Haystack into f4:

• Inside a cell, data is Reed-Solomon(10,4) erasure-coded → 1.4×, tolerating 4 failures in a stripe, with fragments spread across racks.
• For region-level failures, rather than a second full copy (2 × 1.4 = 2.8×), f4 XORs blocks from two different regions and stores the XOR in a third region, bringing the effective factor down to about 2.1×.
• Warm data is read-only (no updates), which makes erasure coding simple; deletes are handled by encrypting each BLOB with its own key and deleting the key.

Takeaway: tier by temperature. Hot: replicated, fast, maybe cached. Warm: erasure-coded, still online. Cold: archived. The physical storage cost difference between 3.6× and 2.1× across tens of petabytes is enormous.

10. Key naming design

Keys look trivial but affect performance, security, operations and cost.

Good:   {hash4}/p/{post_id}/{version}/{rendition}.{ext}
        e.g. 7c2e/p/3049215820194820101/v2/1080.webp
Bad:    photos/2026/10/23/ana_k_beach_holiday.jpg

• Spread load across the keyspace. The metadata index is partitioned by key range. S3 supports at least ~3,500 writes and ~5,500 reads per second per partitioned prefix and splits busy ranges automatically, but keys that all start with today's date (or with a time-ordered ID!) concentrate all new uploads on one range. Prefixing with a short hash of the ID (7c2e/) spreads writes evenly. Note that Instagram-style IDs are time-sortable (see ID Generation), so p/{post_id} alone would cluster new writes.
• Immutable, versioned keys. Never overwrite; a changed rendition gets a new version segment → no CDN purge needed and caches can use immutable.
• Unguessable and PII-free. Don't put usernames or captions in keys (they leak and change); use IDs plus a random component or signed URLs so private photos can't be enumerated (Haystack's "cookie" serves this purpose).
• Group by lifecycle. Prefixes or separate buckets per data class (raw/, p/, stories/, backups/) let lifecycle rules, replication and permissions target them precisely.
• Derivable keys. If the rendition key can be computed from post_id + size + format, the DB stores one media ID instead of a dozen URLs.
• Content addressing (optional). Keying by sha256(content) gives free deduplication (the same meme uploaded 10,000 times is stored once) but complicates deletion (reference counting) and leaks "has this file been uploaded before?".

11. Cost and capacity math for Instagram

All inputs are assumptions for practice (see Back-of-the-Envelope Estimation):

Photos: 100M/day × (2 MB original + 0.5 MB of renditions: 1080/640/320/thumbnail, JPEG+WebP/AVIF)
        = 100M × 2.5 MB = 250 TB/day
Videos: 20M/day × 40 MB (original + bitrate ladder of HLS/DASH segments)
        = 800 TB/day
Total logical ≈ 1.05 PB/day  ≈ 380 PB/year  (and it accumulates forever)

Ingress bandwidth: 1.05 PB / 86,400 s ≈ 12 GB/s ≈ 100 Gbps average (peaks 2–3×)

Physical storage for one year of uploads:
  3× replication     → 380 × 3   ≈ 1,140 PB
  RS(10,4) (1.4×)    → 380 × 1.4 ≈   530 PB
  20 TB drives       → ~57,000 drives vs ~26,600 drives — erasure coding saves ~30,000 drives/year

At cloud list prices (just for intuition):
  380 PB in Standard  ≈ 380M GB × $0.023 ≈ $8.7M per month
  Tiered (most bytes in IA after 60 days, avg ≈ $0.0135) ≈ $5.1M per month
  … for ONE year of uploads; year 2 adds another 380 PB.

Egress if users downloaded directly from the bucket: assume 10 PB/day × ~$0.05/GB
  ≈ $500K per DAY → serving through a CDN / own edge network is mandatory.

These numbers explain why Meta builds its own storage (Haystack, f4 and successors) and edge network rather than renting: at exabyte scale, replication factor, disk density and egress pricing dominate the bill.

12. Deletion, privacy and GDPR

Deleting is harder than storing. When a user deletes a post (or their whole account), every copy must eventually disappear: all renditions, all replicas and erasure-coded fragments, CDN copies, cross-region replicas, old versions, and backups. Regulations such as the EU's GDPR ("right to erasure") require doing this without undue delay; Meta's public policy says deletion can take up to 90 days to fully complete across its systems.

1. Soft delete immediately: mark the post deleted in the metadata DB. The API stops returning it and stops issuing URLs. Signed CDN URLs already handed out expire within minutes to hours.
2. Purge caches: issue CDN purges for the media URLs (belt and braces; see CDN).
3. Enqueue a durable deletion job (e.g. a Kafka topic or deletion queue — see Message Queues) that enumerates every derived object — derivable keys or a manifest of renditions makes this reliable — and deletes them idempotently with retries.
4. Versioned buckets: a plain DELETE only adds a delete marker; old versions remain. Delete specific version IDs or rely on a short NoncurrentVersionExpiration rule.
5. Physical reclamation: in Haystack-style stores, compaction actually removes the bytes later; in erasure-coded warm storage, rewriting stripes is expensive, so f4 uses per-BLOB encryption keys and deletes the key ("crypto-shredding") — the remaining ciphertext is unreadable noise.
6. Backups age out on a fixed retention schedule, or are also crypto-shredded by deleting per-user keys.
7. Verify: deletion frameworks (Meta has published about its DELF framework) track every data type linked to a user so nothing is orphaned, and audit that deletion completed.

13. Cross-region replication

A single region can fail (power, network, natural disaster). Object stores offer cross-region replication (CRR): every new object written to a source bucket is copied asynchronously to a bucket in another region. S3 Replication Time Control offers an SLA of replicating 99.99% of objects within 15 minutes.

• Why: disaster recovery (RPO of minutes for media), lower-latency origin pulls for CDN shields in other continents, and compliance copies.
• Cost: a second full copy plus inter-region transfer (~$0.02/GB on AWS). At 1 PB/day that's ~$20K/day in transfer alone — so replicate selectively: originals and popular renditions; lesser renditions can be re-generated from the original by the media pipeline.
• Gotchas: replication is eventually consistent (a just-uploaded photo may 404 in the other region — fall back to the source region on miss); decide whether deletes (delete markers) replicate; avoid loops in bidirectional setups; respect data residency rules that forbid some data from leaving a jurisdiction.
• Alternatives: geo-distributed erasure coding (f4's XOR across regions) or dual-region buckets that write synchronously to two locations.

See Multi-Region & Disaster Recovery for RPO/RTO and failover.

14. Serving through the CDN

Object storage is optimised for durability and throughput, not for being hit by billions of phones. Users should never download directly from the bucket:

• The bucket is private; only the CDN (authenticated with an origin access identity/control or private network) can read it.
• The CDN does origin pull with an origin shield, so even a viral post causes a handful of origin reads (request collapsing).
• Objects are written with Cache-Control: public, max-age=31536000, immutable because keys are versioned.
• Apps receive signed CDN URLs (expiry + HMAC) from the API, not bucket URLs; the CDN validates them at the edge.
• Video is fetched as small HLS/DASH segments or with Range requests, which the CDN caches individually (see Media Processing).
• With a 95–99% CDN hit ratio, the origin serves only a few percent of media bytes — which is why cheaper, slower warm tiers are acceptable for old media.

15. How Instagram uses object storage

1. The Upload Service authenticates the user, creates a media ID (see ID Generation), and returns a pre-signed, short-lived upload URL. The phone uploads directly (multipart and resumable for video).
2. An object-created event lands on Kafka; Media Processing Workers validate the file, strip EXIF location data, run moderation, and write renditions under immutable, hash-prefixed keys.
3. The post's metadata row in Cassandra stores the media ID and available renditions — never the bytes.
4. The Feed returns signed CDN URLs; the CDN pulls from object storage on a miss.
5. As media cools, it moves from replicated hot storage to erasure-coded warm storage; raw uploads in the quarantine bucket expire after a few days; originals may be kept for re-processing into new formats (e.g. AVIF) later.

16. Trade-offs & common pitfalls

Common pitfalls

• Proxying uploads through app servers instead of pre-signed direct uploads → bandwidth bottleneck.
• Public buckets — the most common cause of real-world data leaks. Keep buckets private; serve through a CDN with signed URLs.
• Using LIST as a database ("list all photos of user X") — store an index in your DB instead.
• Overwriting keys → stale CDN copies and race conditions; use versioned keys.
• Forgetting incomplete multipart uploads and old versions — silent, growing storage bills.
• Archiving user-facing media into hours-latency tiers, or archiving tiny/short-lived objects that hit minimum-size/duration charges.
• Orphans: writing the object but failing to write the DB row (or vice versa); run a periodic reconciliation/garbage-collection job.
• Hot prefixes from date- or sequential-ID-first key names.

17. Real-world technologies

18. Interview corner

19. Check your understanding

Related lessons

• Media Upload Service & Pre-signed URLs — the client-side upload flow in detail
• Media Processing Pipeline — how renditions are produced and stored
• CDN — how stored media reaches users quickly
• Replication — the general theory behind copies and failure domains
• Multi-Region & DR — cross-region replication, RPO and RTO
• Security & Privacy — encryption at rest, private buckets, deletion obligations
• Back-of-the-Envelope Estimation — storage growth and cost math