S3 storage cost: what data lakes pay for files nobody reads

TL;DR. S3 storage cost for a data lake is $0.023 per GB-month on S3 Standard in US East for the first 50 TB, and the bytes that grow fastest can be bytes no query reads: table files from old versions, noncurrent object versions, staging dumps. In a measured run on one Delta table, about 2.7 million row updates in each run left 6,260 MB of unreferenced files when they were spread across the table and 413 MB when they landed in recent partitions: 15 times more, from where the updates fell, not how many there were. Rule of thumb: dead bytes are about retention days times bytes rewritten per day. Below: the S3 meters, the measurement, an Athena query to find prefixes nobody reads, a worked 100 TB lake, and the lifecycle moves that lose money.

S3 storage cost is a price per GB-month, so most guides stop at the storage class table. For a data lake, the larger question is which bytes you are paying for. Delta Lake and Iceberg never edit a file: an update writes new files and leaves the old ones on S3 until a maintenance job deletes them. S3 bills those old files exactly like live ones.

This post measures how much that leaves behind, and shows that the number of rows you change matters less than where they sit. The rates are from the AWS Price List API for US East (publication of October 15, 2026). The worked lake is hypothetical; the table measurement is real and reproducible.

The S3 storage meters a data lake pays

Source: AWS Price List, Amazon S3, us-east-1. Two rules from the S3 pricing page matter later: Standard-IA and Glacier Instant Retrieval have "a minimum billable object size of 128 KB", and "S3 Lifecycle Transition request pricing above represents requests to that storage class", so a transition costs what a PUT into the target class costs.

Storage is the line that grows without anyone deciding. Requests follow traffic; storage follows retention, and retention is usually a default nobody chose.

Old table versions bill as full-price S3 objects

Delta Lake removes old files with VACUUM, and its docs say plainly that "vacuum is not triggered automatically". When it does run, "The default retention threshold for the files is 7 days."

Iceberg works the same way. "Snapshots accumulate until they are expired by the expireSnapshots operation", and the project recommends expiring them regularly "to delete data files that are no longer needed". The default max age applied when expiry runs is 5 days (history.expire.max-snapshot-age-ms).

Retention now has a new knob. On October 28, 2026 Snowflake put Time Travel for Iceberg tables in preview, with version tags that keep a snapshot queryable "for a period that you choose, independent of the table's data retention period". A tag kept for a quarter keeps that snapshot's files, and their storage, for a quarter.

Measured: scattered updates leave 15 times more dead bytes

The setup, so anyone can rerun it: one Delta table written with delta-rs 1.6.6, pyarrow 26.0.0 and Python 3.13.5 on local disk. It starts with 30 daily partitions of 100,000 rows (id, event_day, customer_id, amount, status and a 32-character payload). Each simulated day appends one partition and runs one UPDATE on about 2% of the table's rows, for 30 days.

Two runs differ only in where the updated rows sit. In the clustered run they are in the last three partitions, like late-arriving corrections. In the scattered run they are spread across every partition, like corrections keyed on id rather than on the partition column. The clustered run updated 2,705,000 rows in total, the scattered run 2,730,000.

One updated row in a file rewrites the whole file. In the scattered run, every daily update touched every partition file, so each day added a full copy of the table to storage. The 7-day row sums the files removed in the last seven days, which is what a daily VACUUM at the default threshold would keep. A final VACUUM with zero retention brought both tables back to exactly their live size.

Dead bytes are about retention days times bytes rewritten per day

The two runs reduce to one formula: unreferenced bytes at steady state are about the retention window in days times the bytes rewritten per day. The scattered table rewrote its full size daily, so a 7-day window kept close to 7 copies (6.65 measured, because the table grew during the run).

Priced at $0.023 per GB-month, per TB of live table:

The live TB itself costs $23.55. A scattered-update table on default retention pays for storage about 7.7 times its size, and the fix is either a shorter window or updates that touch fewer files (partitioning or clustering on the update key).

Three more places a lake keeps bytes nobody reads

1. Noncurrent object versions. On a versioned bucket, AWS says "Normal Amazon S3 rates apply for every version of an object stored", and a delete only inserts a delete marker. A VACUUM on a versioned bucket turns dead table files into noncurrent versions; without a noncurrent expiration rule, the bill does not move.
2. Incomplete multipart uploads. Failed large writes leave parts behind, and AWS recommends an AbortIncompleteMultipartUpload rule "to minimize your storage costs". S3 Inventory sizes exclude them, so they do not show up in its byte counts.
3. Staging, export and backup dumps. Pipelines write intermediate copies and one-off exports that no job reads again. They sit under their own prefixes, which makes them the easiest bytes to find.

Find the prefixes nobody reads with S3 Inventory and access logs

S3 Inventory lists every object with size, last modified date, storage class and, when configured for all versions, version ID and IsLatest (docs). Server access logs record reads; delivery to a bucket has "no charge for log delivery" beyond storage, but records arrive "on a best-effort basis", so treat a missing read as a lead, not proof.

1. Turn on a daily Parquet inventory with all versions, and access logging, and wait a few weeks.
2. Create the Athena tables from AWS's definitions for inventory and access logs.
3. Run the query below, then review the prefixes at the top by owner.

-- Bytes per two-level prefix, noncurrent share, and last GET in 90 days.
-- Not tested against a live bucket; adjust table names, and add your log
-- table's date partition filter to limit the scan.
WITH inv AS (
  SELECT regexp_extract(key, '^([^/]+/[^/]+)/', 1) AS prefix, size, is_latest
  FROM s3_inventory
  WHERE dt = '2026-10-30-01-00'
),
reads AS (
  SELECT regexp_extract(regexp_replace(url_decode(key), '^/', ''), '^([^/]+/[^/]+)/', 1) AS prefix,
         max(parse_datetime(requestdatetime, 'dd/MMM/yyyy:HH:mm:ss Z')) AS last_get
  FROM s3_access_logs
  WHERE operation = 'REST.GET.OBJECT'
    AND parse_datetime(requestdatetime, 'dd/MMM/yyyy:HH:mm:ss Z')
        > current_timestamp - INTERVAL '90' DAY
  GROUP BY 1
)
SELECT i.prefix,
       round(sum(i.size) / power(1024, 4), 2) AS tb,
       round(sum(CASE WHEN NOT i.is_latest THEN i.size ELSE 0 END) / power(1024, 4), 2) AS noncurrent_tb,
       max(r.last_get) AS last_get
FROM inv i
LEFT JOIN reads r ON r.prefix = i.prefix
GROUP BY i.prefix
ORDER BY tb DESC;

Never put an age-based expiration rule on a table prefix. Inside a table, a live file from an old partition and a dead file from last week's update look the same to S3: same prefix, any age. Only the Delta log or Iceberg metadata knows which files the current version uses, so table cleanup belongs to VACUUM or snapshot expiry, and S3 lifecycle belongs to staging, exports, logs and noncurrent versions.

Worked example: a 100 TB lake cut to 48 TB

Consider a hypothetical 100 TB lake on S3 Standard in US East:

Before: 50 TB at $0.023 plus 50 TB at $0.022 is $2,304.00 a month. After removing 52 TB, the remaining 48 TB is all in the first tier: $1,130.50. The saving is $1,173.50 a month, priced at the marginal rate: the first 50 TB removed come off the $0.022 tier, only the last 2 TB off the $0.023 tier.

Lifecycle transitions that lose money

Lifecycle transitions are where good intentions cost money. Take the 8 TB of logs:

• In 1 MiB objects: 8,388,608 objects cost $167.77 to move to Glacier Instant Retrieval at $0.02 per 1,000. Storage drops from $188.42 to $32.77 a month, so the move pays back in about 1.1 months.
• In 32 KiB objects: 268,435,456 objects cost $5,368.71 to move, and each is billed as 128 KB, so 8 TB bills as 32 TB at $0.004: $131.07 a month. The saving is $57.34 a month, a payback of about 94 months.

S3 already guards against the second case: lifecycle configurations created or changed since September 2024 by default do not transition objects "smaller than 128 KB", and AWS warns "the transition costs can outweigh the storage savings". The loss comes from overriding that default with a size filter without doing the arithmetic. Compacting small log files into larger objects before archiving fixes the math.

Standard-IA has the read-side version of the trap. 10 TB in Standard-IA costs $128.00 a month instead of $235.52, but one full scan adds $102.40 in retrieval, and two scans a month cost more than Standard. Table files that dashboards scan belong in Standard, or in Intelligent-Tiering, which has no retrieval charges.

The objection: storage is cheap, compute is the bill

The strongest counterargument is that S3 is the cheapest line on a data platform and warehouse compute dwarfs it. That is often true per GB, and an old version also has value: Time Travel and rollback are why retention exists.

The measurement answers part of it. The scattered pattern copied 49 rows for every row it changed, and that rewrite is compute paid every day before it becomes storage. Engines with deletion vectors change the math: Databricks says that "Without deletion vectors, modifying a single row requires rewriting the entire Parquet file", and with them the rewrite moves to later compaction. Retention should be a choice per table, sized to how far back anyone restores, rather than a default multiplied across every table.

Limits of this measurement: one table, local disk, delta-rs rewriting whole files on UPDATE (the files-rewritten row above), 100,000-row partitions of about 4.6 MB, and no compaction or OPTIMIZE. Engines that write deletion vectors rewrite less per update until compaction runs; this run did not test them. The 100 TB lake is hypothetical, and the Athena query was not run against a live bucket.

Where Datapace fits

The S3 bill says how many bytes each prefix holds; it does not say which table, pipeline or reader stands behind them, which is what deciding a retention window needs. Datapace is building one map of your data estate with usage and cost attached, and agents that propose changes as pull requests your team reviews and merges. For the backup side of the same problem, read AWS snapshot cost; for the wider waste picture, read cloud waste in data infrastructure.

Sources

1. AWS, Price List API, Amazon S3, us-east-1, publication October 15, 2026, read October 31, 2026 (storage, request, retrieval, Intelligent-Tiering and Inventory rates).
2. AWS, Amazon S3 pricing (128 KB minimum billable size, transition request pricing).
3. AWS, Transitioning objects using Amazon S3 Lifecycle (128 KB default since September 2024).
4. AWS, S3 Versioning.
5. AWS, Deleting incomplete multipart uploads with lifecycle.
6. AWS, S3 Inventory and server access logging.
7. Delta Lake, Table utility commands: VACUUM.
8. Apache Iceberg, Maintenance and Configuration.
9. Snowflake, Time Travel for Apache Iceberg tables (preview), October 28, 2026.
10. Databricks, Deletion vectors.