Data pipeline cost: how to get the cost per pipeline

TL;DR. Data pipeline cost is orchestration, the compute it triggers, connector fees, intermediate storage and egress, and nearly all of it is compute. Priced at list for one hypothetical nightly pipeline (an Azure Data Factory copy and data flow, then dbt models on a Snowflake Medium warehouse), it comes to $160 a month. Orchestration is $0.15 of that. Data Factory's billing by pipeline shows 17.5% of the total, a Snowflake query tag 45%, and the remaining 37.5% is warehouse idle time no tag reaches. Cost per pipeline only exists once you add the three up, and it only tells you something once you join it to who reads the table the pipeline feeds.

Data pipeline cost is billed by the tool, never by the pipeline. Data Factory bills activity runs and DIU-hours, Glue bills DPU-hours, Snowflake bills warehouse credits, Fivetran bills rows per connection. None of those invoices has a line called "orders pipeline", and none says whether anyone reads its output.

Below, one pipeline priced from each vendor's list prices (read October 27, 2026), where each tool records a run's cost, and a checklist for the pipelines nobody reads. It is a worked model, not a measured benchmark: no pipeline ran.

A data pipeline is billed in five places

Every pipeline pays for the same five things, on whichever tools it touches:

Sources: Azure's retail price API for Data Factory in East US, the AWS Glue price file for US East, Fivetran's usage-based pricing and Azure bandwidth pricing.

Orchestration is the line most "pipeline pricing" pages explain first. It is also the line that matters least. Microsoft counts it as "activity runs, trigger executions, and debug runs", at a tenth of a cent each.

One nightly pipeline costs $160 a month at list price

Consider a hypothetical orders pipeline that runs every night, 30 runs a month:

1. Data Factory, Azure integration runtime, East US. A Lookup reads the watermark, a Copy activity moves the night's changes at 4 DIUs (the default for a single file and for "other scenarios") for 12 minutes, a mapping data flow cleans them on the minimum 8-vCore cluster for 20 minutes, and a stored procedure activity flags the load. Four activity runs plus one trigger execution.
2. dbt on Snowflake. Forty models build on a standard Medium warehouse (4 credits an hour, $3 a credit on Enterprise in AWS US East; Gen 2 warehouses burn more) for 12 minutes of queries. The warehouse keeps the default AUTO_SUSPEND of 600 seconds, so it stays on 10 more minutes after the last model.

Data Factory rounds by the minute: "a data copy taking 41 minutes 23 seconds of compute time will result in a charge for 42 minutes", and "The minimum cluster size to run a Data Flow is 8 vCores". Copy is charged "# of used DIUs * copy duration * unit price/DIU-hour". Snowflake's credit price and warehouse sizes come from its Service Consumption Table; the 600-second default from CREATE WAREHOUSE.

Orchestration is 0.1% of the month. The warehouse is about 82%. The pipeline people call "the ADF pipeline" spends most of its money on a tool whose bill never mentions it.

No billing view shows more than 45% of the pipeline

Each tool attributes its own share to a pipeline, and none attributes the rest.

Data Factory: turn on billing by pipeline

Microsoft documents it in Plan and manage costs: "Data Factory can separate out billing charges for each pipeline." Under Manage, open Factory settings, select Showing billing report by pipeline and publish. Three things to know before you rely on it:

• It is not retroactive. "The change only impacts how bills are emitted going forward, and does not change past charges."
• It is per factory and not in your templates. "You need to opt in for each factory that you want detailed billing for", and the setting "is not included in the exported ARM templates from your factory", so a factory redeployed from CI loses it.
• Some charges stay on a factory line. Operations charges, SSIS nodes and, "If you have Time to Live (TTL) configured for Azure Integration Runtime (Azure IR), Data Flow activities run on these IR won't file under individual pipelines." Teams that set a TTL to skip cluster start-up lose per-pipeline data flow cost.

For one run, the Consumption button on the pipeline run and billableDuration in each activity's output give usage, not price: the view "doesn't show the actual price charged".

AWS Glue: DPU-seconds per job run, and tags

Glue bills DPU-hours "per second with a 1-minute minimum (Glue version 2.0 to 5.1)", with "A single DPU provides 4 vCPU and 16 GB of memory." AWS's own example: "6 DPU * 0.25 hour * $0.44, or $0.66." Had our transform run as a Glue job on the default 10 DPUs for 15 minutes, it would cost $1.10 a run, $33 a month, or $23.10 at the Glue 6.0 rate.

Per run, GetJobRuns returns ExecutionTime and the workers allocated. For Flex runs or with Auto Scaling, it also returns DPUSeconds, "the total time each executor ran during the lifecycle of a job run in seconds, multiplied by a DPU factor":

aws glue get-job-runs --job-name orders \
  --query 'JobRuns[].[StartedOn,ExecutionTime,WorkerType,NumberOfWorkers,DPUSeconds]' \
  --output table

Cost per run is DPUSeconds / 3600 x price when the field is set, otherwise ExecutionTime / 3600 x workers x DPUs per worker x price. For the monthly bill by pipeline, tag the job: AWS says Glue tags "can be used to create cost accounting reports", once the tag key is activated: "For tags to appear on your billing reports, you must activate them."

dbt on Snowflake: query tags, then allocate the idle time

dbt sets a Snowflake query tag per model or project: "You can set more precise values (and override the default) for subsets of models by setting a query_tag model config". Snowflake's QUERY_ATTRIBUTION_HISTORY then prices each query, and Snowflake's cost attribution guide gives the query:

SELECT
    COALESCE(NULLIF(query_tag, ''), 'untagged') AS tag,
    SUM(credits_attributed_compute) AS compute_credits,
    SUM(credits_used_query_acceleration) AS qas
  FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY
  WHERE start_time >= DATEADD(MONTH, -1, CURRENT_DATE)
  GROUP BY tag
  ORDER BY compute_credits DESC;

This is the $72, not the $132. The view "doesn't include any warehouse idle time". The same guide shows how to spread a warehouse's metered credits over tags in proportion to their query credits; on a warehouse used only by this pipeline at night, all $60 of idle belongs to it. If you report only the query-tag number, a nightly job on a warehouse with the default timeout looks 45% cheaper than it is. The fix for that idle tail is in Snowflake cost optimization.

The waste is in pipelines nobody owns

Three patterns cost money without producing value, and none looks like an anomaly on any one bill.

Two pipelines loading one source. One team copies orders through Data Factory; another added a Fivetran connection to the same database for its own schema. Fivetran bills "the instance of the connection", not the source, so the second copy costs its own rows and its own base charge, and its dbt models rebuild their own tables on the warehouse. In our model, a duplicate dbt project on the same warehouse schedule would add another $132 a month with no new reader.

Full reloads where an incremental load works. If the copy reloaded the whole table in 2 hours instead of the night's changes in 12 minutes, the copy line alone would go from $6 to $60 a month, before the warehouse rebuilds the larger table. Each tool has the incremental path documented: Data Factory's watermark and change tracking patterns, Glue job bookmarks that "prevent the reprocessing of old data", and dbt incremental models, which "limits the amount of data that needs to be transformed".

Pipelines whose output no one reads. The dashboard was retired, the analyst left, the export now goes to a new system. The pipeline keeps running at $160 a month, and every bill above reports it as healthy usage. Cost per pipeline is only actionable once it is joined to the readers of the table the pipeline writes. We cover the same pattern across warehouses, BI and dev environments in cloud waste in data infrastructure.

Find the pipelines with no downstream reader

1. List each pipeline's output tables. From dbt, the models in the pipeline's selection; from Data Factory and Glue, the sink datasets and target tables.
2. List what wrote them, by tag. On Snowflake Enterprise or higher, ACCESS_HISTORY records objects_modified per query; join on query_id to QUERY_HISTORY for the tag.
3. List who read them in 90 days, excluding the pipeline itself. direct_objects_accessed in the same view. The query below does steps 2 and 3 for dbt-tagged writes.
4. Walk intermediates to their leaves. A staging table read only by the next model in the same pipeline shows up as unread; what matters is whether the last table in the chain has a reader.
5. Check readers outside the warehouse. BI extracts and exports read through service accounts; a BI read counts only if someone opens the dashboard.
6. Price what you found with the three views above, and propose the cheapest change first: a lower schedule, an incremental load, or switching it off.

WITH written AS (
  SELECT DISTINCT obj.value:"objectName"::STRING AS table_name
  FROM snowflake.account_usage.access_history ah,
       LATERAL FLATTEN(input => ah.objects_modified) obj,
       snowflake.account_usage.query_history qh
  WHERE qh.query_id = ah.query_id
    AND ah.query_start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
    AND qh.query_tag ILIKE 'dbt%'
),
read_by_others AS (
  SELECT DISTINCT obj.value:"objectName"::STRING AS table_name
  FROM snowflake.account_usage.access_history ah,
       LATERAL FLATTEN(input => ah.direct_objects_accessed) obj,
       snowflake.account_usage.query_history qh
  WHERE qh.query_id = ah.query_id
    AND ah.query_start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP())
    AND COALESCE(qh.query_tag, '') NOT ILIKE 'dbt%'
)
SELECT w.table_name
FROM written w
LEFT JOIN read_by_others r ON r.table_name = w.table_name
WHERE r.table_name IS NULL
ORDER BY 1;

The dbt% filter assumes your tags start with dbt. On BigQuery, the same join uses destination_table and referenced_tables in INFORMATION_SCHEMA.JOBS, which keeps 180 days of history.

The objection: per-pipeline cost is overhead for pennies

The strongest case against all this: orchestration is noise, so tune the warehouse and skip attribution. Our model half agrees. Tuning AUTO_SUSPEND removes most of the $60 tail without knowing which pipeline caused it.

The other half: the warehouse minutes are the pipeline. Of the $132 on Snowflake, a timeout fix saves the idle part; only knowing that orders feeds a table nobody reads saves the $72 of queries and the $28 upstream. Tuning makes a pipeline cheaper; attribution tells you which ones to stop. The objection wins for a small estate where one person knows every pipeline. It loses once pipelines outnumber the people who remember why they exist.

What this model does not tell you

• List price, read October 27, 2026. Reservations, Snowflake capacity contracts and Fivetran's per-connection curves change every row.
• The durations are assumptions. No pipeline ran; a data flow's real billed time depends on the integration runtime, cluster start-up and data volume.
• Attribution has gaps. QUERY_ATTRIBUTION_HISTORY leaves out "Short-running queries (<= ~100ms)" and has up to eight hours of latency; dbt warns that "build failures midway through a materialization may result in subsequent queries running with an incorrect tag".
• Readers are harder than writers. ACCESS_HISTORY needs Enterprise Edition, and reads outside the warehouse (files, exports, BI caches) need their own check.
• Not covered: Microsoft Fabric Data Factory, self-hosted integration runtimes and streaming pipelines.

For the warehouse side of this bill on four platforms, see data warehouse cost.

Where Datapace fits

A pipeline's cost sits in three or four bills, and the question that decides whether to keep it, who reads what it writes, sits in none of them. Datapace is building one map of your data estate (databases, warehouses, pipelines and what consumes them) with usage and cost attached, and agents that propose changes as pull requests your team reviews and merges. If the idle tail above looks like your warehouse, start with the queries in Snowflake cost optimization.

Sources

1. Microsoft, Azure Retail Prices API, Azure Data Factory v2, East US and Data Pipeline pricing.
2. Microsoft Learn, Plan and manage costs for Azure Data Factory, Copy activity performance features and Incrementally copy data.
3. Microsoft, Bandwidth pricing.
4. AWS, AWS Glue pricing, AWS Glue price file for US East, JobRun API, Tags in AWS Glue, Activating cost allocation tags and Job bookmarks.
5. Snowflake, Service Consumption Table, CREATE WAREHOUSE, QUERY_ATTRIBUTION_HISTORY, Attributing cost and ACCESS_HISTORY.
6. dbt, Snowflake configurations and Incremental models.
7. Google Cloud, INFORMATION_SCHEMA.JOBS view.
8. Fivetran, Usage-based pricing.