Instrumenting AI cost per run — and keeping it off every client screen
If you sell AI-generated deliverables, cost per run is a number you have to know precisely and a number your client must never see. Those two requirements pull in opposite directions, and the second one is easier to violate than it looks.
What this system does
Agentora sells fixed-price deliverables that are produced by language-model calls. This case study covers how the cost and provenance of each of those calls is recorded, and how that information is kept away from the people buying the deliverable.
The problem
Selling a generated artifact at a fixed price makes the cost of generating it a margin question, not a curiosity. Without per-run attribution you cannot tell which capability is expensive, whether a price is sustainable, or whether a change to a prompt made a deliverable cheaper or quietly more expensive.
Aggregate spend from a provider's billing console does not answer any of that. It tells you the total, not which capability, which client, or which prompt version produced it, and by the time it is visible the run is long gone.
The reproducibility problem sits alongside it. When a client questions a generated document weeks later, the useful question is what exactly produced it — which model, and which version of which prompt. Without that recorded per run, the answer is reconstructed from deployment history and guesswork.
Then there is the requirement that runs the other way. Cost of goods — raw model spend, token counts, model names — is internal. On a client-facing screen it is at best confusing and at worst commercially damaging, because a client reading the input cost of a fixed-price deliverable is being invited to price it themselves. The risk is not only that it is displayed deliberately; it is that a panel built for internal use gets rendered somewhere a client can see it.
Constraints
The boundaries the design had to respect, before any solution was chosen.
- Every model-backed run must record what it cost and what produced it.
- Cost estimation must be consistent across both implementation languages in the platform.
- Non-model runs must be representable without pretending to have usage data.
- Cost of goods must not be reachable from any client-facing surface.
The architecture
Usage and provenance are recorded as one value object attached to every run, and the surfaces that display it are kept structurally separate from the ones clients use.
- 1
One usage record per run
Each capability reports a usage record covering the model, input and output token counts, the estimated cost, the prompt version and a hash of the rendered prompt. It is stored on the run row, so cost and provenance are properties of the run rather than of a separate log to correlate later.
- 2
An explicit empty case
Capabilities that make no model call — deterministic work such as provisioning — report an explicit empty usage record rather than fabricating zero-cost model metadata. A run with no model is distinguishable from a run whose usage was not captured.
- 3
Cost estimation kept consistent across languages
The platform generates from both a TypeScript service and a Java engine. Token-to-cost estimation is implemented to mirror the other side deliberately, so a figure means the same thing regardless of which pipeline produced it and totals can be compared across both.
- 4
Prompt hash for reproducibility
Alongside the prompt version, a hash of the fully rendered prompt is recorded. Since prompts are assembled from templates plus injected context, the version alone does not uniquely identify what the model actually received. The hash does.
- 5
Cost surfaces kept off the client portal entirely
The cost and usage panel is not rendered in the client portal at all — not even behind an administrator check. It exists for a separate admin-gated route, and the client-facing panel carries an explicit instruction not to reintroduce it, with the reason recorded.
Key decisions and their trade-offs
Every decision below cost something. The trade-off is stated alongside the reasoning.
Attach cost and provenance to the run rather than to a separate log
Why
Anything stored separately has to be correlated later, and correlation fails exactly when it is most needed. Making usage part of the run record means the question "what did this cost and what produced it" is a field lookup rather than an investigation.
Trade-off
The run row carries columns that are meaningless for non-model runs, which is why the empty case is explicit rather than implied by nulls.
Exclude cost from the client portal entirely rather than gating it by role
Why
A role check is a correct-looking control that fails in an ordinary situation: an administrator walking a client through the portal on a screen share. Removing the surface from that application removes the whole class of accidental disclosure, rather than relying on nobody being logged in as an admin at the wrong moment.
Trade-off
Administrators cannot see cost in the context where they are already looking at the work, and have to use a separate view — deliberately less convenient.
Record a prompt hash in addition to a version
Why
A version identifies the template, not the request. When context is injected into a template, two runs on the same version can differ, so a hash of what was actually sent is what makes a run reproducible.
Trade-off
A hash is opaque — it proves two runs differed but does not show how, so diagnosing a difference still needs the inputs.
Estimate cost in-platform rather than reading it from billing
Why
An estimate available at the moment of the run can be attributed to a capability, a client and a prompt version. Authoritative billing data arrives later and aggregated, which is the wrong shape for margin analysis per deliverable.
Trade-off
The figures are estimates from a maintained rate table, so they drift from the invoice when provider pricing changes and the table has not been updated.
Technologies used
Engine
- Java
- Spring Boot
- Value-object usage records
Application
- TypeScript
- Next.js
- Admin-gated routes
Data
- PostgreSQL
- Per-run usage rows
Outcome
- Every model-backed run records its model, token counts, estimated cost, prompt version and prompt hash, so cost and provenance are answerable per run rather than per month.
- Cost figures mean the same thing across both the TypeScript and Java pipelines, so totals are comparable across the whole platform.
- Cost of goods is absent from the client portal by construction rather than hidden by a permission check, which removes the screen-share disclosure risk rather than mitigating it.
Known limitations
What this design does not do. Stated because an architecture without documented trade-offs has usually not been examined closely enough.
- Costs are estimates from a maintained rate table, not reconciled billing. They drift from the invoice when provider pricing changes before the table is updated.
- The prompt hash proves that two runs differed without showing how, so diagnosing a difference still requires the original inputs.
- Separating the cost view from the work view is deliberately inconvenient for internal users, and that friction is the point rather than a defect to fix.
Want this level of rigour on your AI initiative?
Start with a free AI Readiness Assessment, or book a Discovery Workshop to get a scored, costed roadmap.