Splitting one AI capability into four because the JSON ran out of room
A model asked for structured output does not stop cleanly when it hits its token ceiling. It stops mid-string, and what you get back is unparseable. The fix was partly a bigger budget and mostly a smaller ask.
What this system does
Agentora's delivery engine runs a set of generation capabilities, each of which asks a language model for a structured document and parses the result into a typed object. This case study covers what happens when the requested document is larger than the response budget allows.
The problem
The architecture design capability produced the largest output in the system: a target architecture, a set of architecture decision records, and a multi-kilobyte infrastructure-as-code scaffold, all in one structured response.
When a response of that size reaches the model's output ceiling, it does not degrade gracefully. There is no truncation marker and no partial-but-valid document. The response stops in the middle of a string, so the JSON never closes and parsing fails outright. A capability that worked for one target cloud would fail for another simply because the more verbose provider pushed the same document past the limit.
The default failure surfaced badly, too. A raw deserialization exception became the job's error message, which tells an operator that the JSON was malformed but not the one thing that matters: that it was malformed because it was cut off, and that the fix is a larger budget or a smaller request.
The tempting response is to raise the token limit globally. That is expensive across every capability that does not need it, and it only postpones the problem — the ceiling moves, the largest document keeps growing toward it, and nothing has changed structurally.
Constraints
The boundaries the design had to respect, before any solution was chosen.
- A single global token limit is wrong: capabilities differ by an order of magnitude in output size.
- Raising the ceiling for everything wastes budget on the many capabilities that produce small documents.
- A truncation failure must produce an actionable message, not a raw parser exception.
- Each capability's budget has to be tunable in configuration, without a code change.
The architecture
Two changes, in order of importance: make the request smaller by decomposing it, and make the budget a per-capability property rather than a global one.
- 1
A per-capability token budget
The capability interface declares a max-output-tokens value, defaulting to zero, which means "use the engine default". Heavy capabilities override it to request a larger budget for their own call only. The value is applied at the call site, so each capability's model call carries its own ceiling.
- 2
Budgets set in configuration, not code
Each override reads from a named configuration property with a sensible default, so a budget can be tuned per environment without editing or redeploying the service. The heavy capabilities — architecture design, low-level design, channel integration, site generation — each declare their own.
- 3
Decompose the largest capability
The more important fix was structural. The single architecture capability was split into separate sizing, high-level design and low-level design capabilities, each with its own prompt, its own artifact, and its own budget. Every response is now bounded by construction rather than by hoping it fits.
- 4
Each part earns its own place in the product
The split was not purely defensive. Sizing — how many services the target architecture decomposes into, and the estimated effort for each — is the input to quoting the build phase. Separating it made it independently useful and independently regenerable, rather than something buried inside a larger document.
- 5
Turn a parse failure into an instruction
When conversion fails, the capability does not propagate the deserialization exception. It fails with a message that names the likely cause — truncation at the token ceiling — and the action to take. The operator reads a diagnosis instead of a stack trace.
Key decisions and their trade-offs
Every decision below cost something. The trade-off is stated alongside the reasoning.
Make the token budget a per-capability property rather than a global setting
Why
Output sizes across capabilities differ enormously. One global ceiling is either too small for the largest document or wastefully large for everything else, and it makes the smallest capability pay for the biggest one's worst case.
Trade-off
There are now several budgets to understand and keep sensible instead of one number, and a capability that grows past its own budget fails until someone notices and raises it.
Split the largest capability rather than only raising its ceiling
Why
A bigger budget treats the symptom; the request being too large is the cause. Decomposition bounds each response structurally, which also means each part can be regenerated on its own rather than rebuilding the entire document to change one section.
Trade-off
More capabilities, more prompts and more artifacts to maintain, and the parts have to stay coherent with each other now that no single call sees the whole picture.
Default the budget to zero meaning "engine default"
Why
Most capabilities should not have an opinion about token limits. A zero default keeps the common case free of configuration and makes the override an explicit, visible statement that this capability is unusual.
Trade-off
Zero-as-a-sentinel is less self-explanatory than an explicit optional value, so its meaning has to be documented on the interface.
Convert the parse failure into a named, actionable error
Why
The parser's exception is true but useless: it describes the symptom. Naming truncation as the likely cause and stating the remedy turns a confusing job failure into a five-second fix.
Trade-off
The message asserts a probable cause, so a genuinely malformed response for some other reason will be mislabeled as truncation.
Technologies used
Engine
- Java
- Spring Boot
- Spring AI
Structured output
- Typed output converters
- JSON schema-guided generation
AI
- Anthropic Claude
Outcome
- Each capability carries its own output budget, so the heavy generators get the room they need without every other call paying for it.
- The largest capability is decomposed into bounded parts, and each part can be regenerated independently rather than requiring a full rebuild.
- A truncated response now fails with a message naming the cause and the remedy instead of a raw deserialization error.
Known limitations
What this design does not do. Stated because an architecture without documented trade-offs has usually not been examined closely enough.
- Budgets are static configuration, not adaptive. A capability whose output grows past its budget still fails until the number is raised.
- The truncation diagnosis is a heuristic: any unparseable response is reported as probable truncation, which will occasionally mislabel a different malformation.
- Decomposition shifts coherence work onto the design: no single call now sees the whole architecture, so consistency between the parts is a property of the prompts rather than of one response.
Want this level of rigour on your AI initiative?
Start with a free AI Readiness Assessment, or book a Discovery Workshop to get a scored, costed roadmap.