All field notes

Cost per feature

The cost of building a feature isn’t the cost of running it.

Why one AI-spend number can’t describe a password reset and an itinerary agent.

Agent Watch4 min read

A dark office at night. On a desk lit by a single lamp, one small receipt printer has produced a short slip, while a second printer beside a laptop feeds out a long paper ribbon that runs off the desk to the floor.
Two features from the same sprint. Only one calls a model when customers use it.

Take two features from the same sprint.

The first is “Reset password”. An engineer asks a coding agent to build it. The agent reads the auth code, writes the flow, fixes a test, and opens a pull request. Someone reviews it and merges it. There’s no model in the product, so using the feature doesn’t trigger a model call. Later fixes and reviews can still involve a coding agent.

The second is “Trip draft”, an agent inside the product that writes a traveller’s itinerary. It starts the same way, with a coding agent writing it. Then things diverge.

Same sprint, same team, same coding agent. Two very different cost shapes.

A vendor invoice asks, “What did this customer consume from us?” It can tell you which seat or model you paid for. It cannot tell you what your engineering team built.

The useful unit of AI spend is the feature, not the vendor. That means keeping the original vendor and cost information, but tying it to the work it contributed to. Even then, a feature total needs context: was this the cost of building it, or of running it?

The four stages of an LLM feature

An LLM feature spends money at four points in its life.

Coding. A coding agent writes it, just like any other feature. This is the stage every feature shares.

Local runs. The developer runs the feature on their own machine, over and over, to see whether the itineraries are any good. Every try is a model call. None of it is the coding agent; it’s the feature itself, running before anyone else has seen it.

CI. An LLM feature needs evals the way a checkout page needs unit tests. So it runs in CI, often on every change, against a set of example trips. Each run is more model calls, made by the feature, not by the agent that wrote it.

Production. Then it ships, and every user who asks for a trip triggers a call. The model-call bill continues as people use the feature.

“Reset password” has AI costs for building and maintaining it, but no production model-call bill. “Trip draft” has all four stages. As people use it, the running cost can overtake what it cost to build.

Diagram of four stages: coding, local runs, CI and production. A password reset feature has a building cost in coding only. An itinerary feature has the same building cost, then a running cost in local runs, CI and production that grows at each stage.
Same coding agent, two bills. Illustrative shape, not measured cost.

What one number does to them

A clear glass box on a desk under a lamp, overflowing with crumpled receipts that spill across the desk, next to a receipt spike.
Put every feature in one column and you can no longer tell which receipt belongs to which.

Most AI-cost reporting gives you one number. Per developer, per seat, or per month. It treats every feature as if it had the same life.

Put “Reset password” and “Trip draft” in one “AI spend” column and two things go wrong.

The static feature looks incomplete. It has no production model-call cost. Next to a feature with four stages it looks like something is missing. Nothing is missing. That’s simply the kind of feature it is.

The LLM feature’s real bill gets hidden. Its costs continue beyond the coding agent: in local runs, in CI evals and, for as long as it’s live, in production. A monthly total mixes that in with everything else, so ongoing model calls are hard to distinguish from the cost of building and maintaining code.

Per-developer views make it worse, not better. The engineer who built “Trip draft” looks expensive in the month they wrote it and cheap every month after, even though the feature they shipped keeps generating model calls as people use it. The number follows the person. The cost follows the feature.

A static feature can still pick up AI spend in CI, for example from a review bot or a coding agent running in the pipeline. That’s still the cost of building or maintaining it. It isn’t the feature calling a model, and it isn’t a production model-call bill.

Building isn’t waste

Coding-agent spend is a cost of building, not evidence of waste. Repeating the same session without changing the feature is worth investigating. So is an agent looping on a failing test overnight. But a large CI eval bill may help prevent production failures, and a large production bill may mean people use the feature. The total alone cannot tell you whether the work was worthwhile.

What a cost-per-feature number is for

We build Agent Watch to attribute AI spend to a feature, from the coding agent through to production. Along the way we’ve come to believe a cost-per-feature number only means something once you know which shape the feature is.

For a static feature, the AI bill covers building and maintaining it. It has no model-call cost when customers use it.

For an LLM feature, it has to answer two: what did this cost us to build, and what does it cost us to run?

If your reporting can’t tell those apart, it can’t answer either of them.