A closed review loop that refuses to close itself
Review finds problems, a model fixes them, re-review, repeat until clean. It is the obvious automation — and in a regulated delivery it is the design that should worry you, because the cheapest way to make findings disappear is to weaken them.
What this system does
Agentora's delivery packs go through generated QA and security reviews, which produce findings. Those findings then have to be fixed and re-checked. This case study covers the loop that closes between review and rollout.
The problem
The automation is easy to imagine and easy to build: feed the findings back into regeneration, re-run the review, and iterate until the report comes back clean. Every part of that exists already.
The problem is what the loop optimises for. An automated loop is rewarded for producing a clean report, and the cheapest path to a clean report is not always a genuine fix — it is a weaker finding. Nothing in the mechanism distinguishes a resolved risk from a re-scored one, and the artifact that comes out the far end asserts that a problem was addressed with no human willing to say so.
For a regulated delivery that is the wrong shape entirely. When a bank's technology committee asks who determined that a blocking security finding was resolved, the answer cannot be that the loop stopped flagging it.
Two further problems follow. If each pass overwrites the previous artifact, there is no way to show what changed between pass one and pass two — which is precisely what an auditor asks for. And an unbounded loop on a genuinely hard finding will iterate indefinitely, burning model spend while hiding a real blocker behind repeated attempts.
Constraints
The boundaries the design had to respect, before any solution was chosen.
- A human must decide when a finding is genuinely resolved — the system may not conclude that on its own.
- Rollout must be impossible while blocking findings are open, enforced somewhere stronger than the interface.
- Each pass must leave a record rather than overwriting the previous artifact.
- The loop must terminate and escalate rather than iterating forever.
The architecture
The loop is built so that its final step is a person. Everything else is automated; advancing the state is not.
- 1
Explicit states between review and rollout
Three states sit between review and rollout — remediation pending, remediating, and remediated — inside a transition graph that declares every legal move and rejects anything not listed. The lifecycle is data, not a set of booleans that accumulate special cases.
- 2
Rollout hard-gated on open blockers
Entering the rolling-out state is additionally gated on there being no open blocker findings, and that gate lives in the state machine on the server. The interface also derives a blocker count and disables its button, but the interface is the convenience, not the control.
- 3
Remediation as an advisory plan
The generated output is a plan, not an edit: a concrete proposed fix for each actionable finding, with the artifact it targets and a priority. It is persisted as an artifact row with a remediation kind, reusing the existing artifact store rather than introducing a new table.
- 4
Targets drive regeneration
Each remediation item names the artifact it applies to, and those targets determine which artifacts are regenerated on the next pass. Regeneration is scoped to what the findings actually touched rather than rebuilding the whole pack.
- 5
A person advances the state
Marking items remediated is an explicit human action through its own endpoint, and it is the only thing that moves the project forward. The loop can prepare everything and still cannot conclude that a risk was addressed.
- 6
Bounded, with escalation
The loop does not run indefinitely. Passes are limited, and an issue that survives them escalates for human attention rather than being retried forever.
Key decisions and their trade-offs
Every decision below cost something. The trade-off is stated alongside the reasoning.
Keep a human gate instead of fully automating the loop
Why
An automated loop is incentivised toward a clean report rather than a genuine fix, and in a regulated delivery someone has to own the assertion that a risk was addressed. Making the final step a deliberate human action is the whole point of the design, not a limitation of it.
Trade-off
Throughput. The loop stops and waits for a person, so a pack cannot go from review to rollout unattended — which is the intended cost.
Put the rollout gate in the state machine, not only in the UI
Why
A disabled button is a suggestion that any direct API call ignores. A transition graph that refuses the move is enforcement, and it holds regardless of which client is talking to the service.
Trade-off
A legitimate exception — a finding a client accepts as a known risk — requires a real state change rather than someone clicking through, so the escape hatch is deliberately less convenient.
Version the artifact per pass instead of overwriting it
Why
The question that gets asked later is what changed between passes. Retaining each pass makes that answerable, which matters far more in a regulated setting than saving storage.
Trade-off
Storage grows, and the read path has to distinguish the latest version from the history, which is more complex than always reading one row.
Reuse the artifact table with a kind discriminator rather than adding a table
Why
The remediation plan is another generated artifact with the same lifecycle needs as the others, so it inherits the existing storage, versioning and read path instead of duplicating them.
Trade-off
One table carries heterogeneous document shapes, so its schema is necessarily loose about what any given row contains.
Bound the loop and escalate
Why
An unbounded retry on a hard finding spends model budget to produce the appearance of progress while concealing a real blocker. A limit converts that into a visible escalation.
Trade-off
A finding that would have been resolved on a later pass gets escalated to a person instead.
Technologies used
Engine
- Java
- Spring Boot
- Explicit transition graph
Data
- PostgreSQL
- Versioned artifact rows
Portal
- Next.js
- React
- Server-enforced gating
Outcome
- Rollout cannot proceed while blocking findings are open, and the gate is enforced server-side in the transition graph rather than only by a disabled control.
- Each remediation pass is recorded as its own artifact version, so what changed between passes can be shown rather than reconstructed.
- Advancing out of remediation requires an explicit human action, so no generated output asserts that a risk was resolved without someone standing behind it.
Known limitations
What this design does not do. Stated because an architecture without documented trade-offs has usually not been examined closely enough.
- This is recently landed work. The full loop's production rollout is staged rather than long-proven in the field.
- Pack version badges and a pass-to-pass diff inside the exported PDF are not implemented, so comparing passes is currently easier in the portal than in the document.
- Throughput is deliberately limited by the human gate. This design is a poor fit for anything that needs to close a review loop unattended.
Want this level of rigour on your AI initiative?
Start with a free AI Readiness Assessment, or book a Discovery Workshop to get a scored, costed roadmap.