Moving long-running AI generation off the request path
LLM report generation ran inside the serverless request that confirmed payment — so the slowest possible operation sat on the most important path in the product. Here is how we moved it to a Postgres-backed job queue and an out-of-band container worker.
What this system does
Agentora generates a scored AI readiness report for each paid engagement. The report is produced by a large language model from the client's structured discovery responses and uploaded documents, then reviewed by a human before delivery. This case study covers how that generation step is scheduled and executed.
The problem
The first working version did the obvious thing: when a payment settled, the same serverless function that confirmed the payment also generated the report, then returned. It was simple, it was easy to reason about, and it was wrong.
Generating a long, structured report from an LLM takes minutes, not milliseconds. The hosting platform's serverless functions have a hard maximum execution ceiling — 300 seconds on the plan in use. Generation was approaching and sometimes exceeding it, which meant the function was killed mid-flight and the caller received a 504.
The failure landed at the worst possible moment in the product: immediately after a customer paid. The money had been captured, but the request that was supposed to acknowledge it timed out. From the customer's side, they had just paid and the page had broken.
There is also a structural problem beyond the timeout. Holding an HTTP connection open for the duration of an expensive, retry-prone LLM call couples two things that should be independent: the durability of the payment and the success of the generation. A transient model error should never put the payment record at risk, and a slow model should never decide how long a customer stares at a loading spinner.
Constraints
The boundaries the design had to respect, before any solution was chosen.
- The money path must never lose a payment, and must stay fast — payment confirmation cannot wait on generation.
- Payment settlement arrives twice by design: the browser's verify call and the payment provider's webhook can both fire for the same transaction. Neither may cause duplicate work.
- A worker can crash mid-generation. Jobs must not be stranded in a running state forever.
- A genuinely broken job must fail visibly and stop, rather than retrying indefinitely.
- No always-on infrastructure cost for a workload that is idle most of the day.
The architecture
The fix was to stop treating generation as part of the request and start treating it as a durable job. Payment confirmation now records intent and returns; a separate worker performs the work. The queue lives in the same Postgres database as the rest of the domain data.
- 1
Payment settles → enqueue, don't generate
The money path inserts a row into a report_jobs table and returns immediately. Generation is no longer on the request path at all, so the 300-second ceiling stops being relevant to it. The customer gets a fast, reliable acknowledgement of their payment.
- 2
Idempotent enqueue via a partial unique index
A partial unique index (uq_report_jobs_active) covers engagement_id where status is pending or running. When both the verify call and the webhook fire for the same settlement, the second insert raises a unique violation — Postgres error 23505 — which the enqueue helper treats as the intended no-op rather than an error. Deduplication is enforced by a database constraint rather than by application logic that has to remember to check.
- 3
Atomic claim in SQL
The worker calls a claim_next_report_job() SQL function that selects one pending job and marks it running in a single atomic step. Because the select-and-mark happens inside the database, two workers can never claim the same job, and the correctness argument lives in one place instead of being spread across application code.
- 4
Crash recovery via a sweep
A sweep_stuck_report_jobs() function returns jobs that have been stranded in running — a worker that died mid-generation — back to pending so they can be retried. Each job carries attempts and max_attempts, so a poison job stops after a bounded number of tries and surfaces as failed rather than cycling forever.
- 5
Out-of-band container worker
A containerized TypeScript worker runs on AWS ECS Fargate. It claims a job, performs the generation, writes the result, and notifies. Because it is a long-running container rather than a serverless function, it has no execution ceiling to design around.
Key decisions and their trade-offs
Every decision below cost something. The trade-off is stated alongside the reasoning.
Use Postgres as the queue instead of a dedicated queue service
Why
The engagement and payment data already lives in Postgres, so a job row can be reasoned about transactionally alongside the records it relates to. It is also one less system to operate, secure, and pay for at the current stage of the product.
Trade-off
The worker polls rather than being pushed to, which adds latency between enqueue and pickup, and a database-backed queue has a much lower throughput ceiling than a purpose-built broker. Both are acceptable at current volume; neither would be at very high throughput, at which point the claim/sweep interface is narrow enough to swap the implementation behind it.
Enforce idempotency with a database constraint, not application checks
Why
The duplicate-settlement case is a certainty, not an edge case. A partial unique index makes the duplicate physically impossible to insert, so correctness does not depend on every future call site remembering to check first.
Trade-off
The calling code has to understand and deliberately swallow a specific Postgres error code, which reads as unusual unless the intent is commented — so the reason is documented at the call site.
Separate the claim from the sweep
Why
They answer different questions. Claiming asks what to work on next; sweeping asks what was abandoned. Keeping them as distinct SQL functions means recovery behaviour can change without touching the hot path.
Trade-off
Recovery is bounded by how often the sweep runs, so a crashed job is not retried instantly.
Technologies used
Application
- TypeScript
- Next.js
- Node.js
Data & queue
- PostgreSQL (Supabase)
- SQL functions / RPC
- Partial unique indexes
Compute
- Docker
- AWS ECS Fargate
AI
- Anthropic Claude
Outcome
- Generation is off the request path. Payment confirmation now returns as soon as the job is recorded, so the operation that used to time out no longer runs inside a request with an execution ceiling.
- The duplicate-settlement case is handled by construction: the second insert is a no-op enforced by the database, not by a check that could be forgotten.
- The queue schema and its claim/sweep functions were applied to production and the enqueue path validated against live payment settlement.
Known limitations
What this design does not do. Stated because an architecture without documented trade-offs has usually not been examined closely enough.
- The worker polls, so there is a delay between a job being enqueued and being picked up.
- A Postgres-backed queue has a throughput ceiling well below a dedicated broker. This is a deliberate trade for operational simplicity at the current stage, not a claim that it scales indefinitely.
- Crash recovery is only as fast as the sweep interval.
Want this level of rigour on your AI initiative?
Start with a free AI Readiness Assessment, or book a Discovery Workshop to get a scored, costed roadmap.