Event-driven scale-to-zero for bursty AI workloads
Three container services that do real work for minutes a day were sized to be available all day. Rather than guess at capacity from metrics, we scale them from the business events that actually cause the work — and back to zero when the queue drains.
What this system does
Agentora runs three containerized backend services: a report generation worker, the delivery engine that produces architecture and implementation deliverables, and a retrieval service. All three do intense work in short bursts and then nothing at all. This case study covers how their capacity is scheduled.
The problem
Container platforms bill for running tasks. A service with a desired count of one is paid for every hour of the month, whether or not it did anything. That is the correct model for a user-facing API that must answer instantly, and the wrong one for a worker that generates a document for a few minutes after a customer takes a specific action.
The shape of the workload is the whole point: it is bursty and event-driven. Work arrives when a payment settles or when someone clicks generate — not on a smooth curve. Between those events, three services were idling at full readiness.
Conventional autoscaling does not fit well either. Scaling on a metric like queue depth means waiting for a metric to be published, aggregated, and to breach an alarm threshold before anything happens — minutes of detection latency for a job that was knowable the instant the payment settled. It also costs money to observe and adds another moving part that can misfire.
There was a second, less obvious risk. In a scale-to-zero design, a scale-up that fails to scale back down does not break anything visibly — it just quietly bills every hour until someone notices. Reliability of the return-to-zero path matters more than the speed of the scale-up.
Constraints
The boundaries the design had to respect, before any solution was chosen.
- The work is asynchronous, so cold-start latency on the first job after an idle period is acceptable.
- Scaling must be triggered by a real business event, not inferred from a proxy metric.
- Returning to zero must be reliable — a stuck scale-up is a silent recurring cost, not a visible outage.
- The infrastructure already existed, hand-created and live. Codifying it could not mean tearing it down and rebuilding it.
The architecture
Instead of asking the infrastructure to infer demand, the application tells it. The events that cause work are published to an event bus, and small functions translate those events into a desired task count.
- 1
Business events on an event bus
The events that cause work — a completed payment, a delivery generation request — are published to Amazon EventBridge. This is the earliest and most precise possible signal: it is the cause of the work, not a downstream symptom of it.
- 2
Rules invoke small scaling functions
EventBridge rules invoke Lambda functions whose only job is to set the desired count on the relevant ECS services. Keeping the scaling logic in a tiny, single-purpose function means the workers themselves know nothing about capacity management.
- 3
One function, many targets
Rather than a Lambda per service, the scaling functions read a services field from the event payload to decide which services to move, defaulting to all three when the field is absent. One code path handles every case, so there is a single place where scaling behaviour is defined.
- 4
Drain, then return to zero
When the queue is drained, the services scale back to zero. For the report worker this is driven by the drain event itself; for the delivery and retrieval services an interim scheduled function handles idle scale-down until an equivalent drain signal is wired through.
- 5
Codified with Terraform import
The whole arrangement — IAM roles, rules, functions, service definitions — is written as Terraform in an infra/terraform directory, using import blocks to adopt the already-live resources one-for-one instead of recreating them. The infrastructure became reproducible without a rebuild.
Key decisions and their trade-offs
Every decision below cost something. The trade-off is stated alongside the reasoning.
Scale on business events rather than on metrics
Why
The business event is the cause of the work and is available immediately. A queue-depth alarm is a downstream symptom that has to be published, aggregated, and breach a threshold first, which adds minutes of latency and its own cost to something already known.
Trade-off
The application and the infrastructure are now coupled through an event contract: if a new code path creates work without publishing the event, nothing scales up to serve it. That contract has to be honoured deliberately.
Scale to zero rather than keeping a minimum of one task
Why
For asynchronous work, paying for idle readiness buys nothing a user can perceive. Zero idle capacity is the correct default when nobody is waiting on a synchronous response.
Trade-off
The first job after an idle period pays a container cold start. This would be the wrong trade for a user-facing, latency-sensitive endpoint — it is right here precisely because the work is already asynchronous.
Select scaling targets from the event payload
Why
One function that reads which services to move is simpler to reason about and change than three near-identical functions that drift apart over time.
Trade-off
A malformed or unexpected event payload affects all targets rather than one, so the default behaviour when the field is missing has to be chosen carefully.
Adopt live infrastructure with Terraform import instead of recreating it
Why
The resources were already serving production. Importing them makes the estate reproducible and reviewable with no downtime and no orphaned duplicates left behind.
Trade-off
The Terraform has to describe reality exactly before the first apply, or the plan proposes destructive changes to live infrastructure. This front-loads careful work that a greenfield definition would not require.
Technologies used
Events & compute
- Amazon EventBridge
- AWS Lambda (Python)
- AWS ECS Fargate
Infrastructure as code
- Terraform
- Terraform import blocks
- AWS IAM
Observability
- Amazon CloudWatch Logs
Outcome
- The flow is live and was tested end to end: a completed payment scales the report worker up, a delivery generation request scales the delivery and retrieval services up, and draining the queue returns them to zero.
- Idle capacity is no longer paid for on services that work in bursts, because the default state between events is zero running tasks.
- The arrangement is codified in Terraform, so the estate is reproducible and reviewable rather than existing only as console state.
Known limitations
What this design does not do. Stated because an architecture without documented trade-offs has usually not been examined closely enough.
- The first job after an idle period pays a container cold start. Acceptable for asynchronous work; not a pattern to copy for latency-sensitive endpoints.
- Scale-down for the delivery and retrieval services is currently an interim scheduled sweep rather than a true drain-triggered event.
- The design assumes every code path that creates work also publishes its event. That coupling is a real maintenance obligation.
- Single-region. Multi-region behaviour has not been designed or tested.
Want this level of rigour on your AI initiative?
Start with a free AI Readiness Assessment, or book a Discovery Workshop to get a scored, costed roadmap.