A machine for every user (part 1)

One per user, for months

Kinetik is an AI agent for social media and marketing that lives in a messenger.

An agent is not a request handler. It's a long-running process — our TypeScript runtime, an OpenClaw gateway, a stack of skills — on a real filesystem with a real shell, holding one person's working state for months: their scraped profile, their content analysis, their live tokens for Instagram and Notion and Slack. So the unit of our infrastructure isn't a request. It's a machine, one per user, for as long as they're a user — an unnatural thing to ask a cloud for.

And because it lives in a messenger, there's no spinner to hide behind: someone sent a message and is staring at an empty thread. Every number below is measured against that.

A Helm release per human

The first version ran on Kubernetes, because everything ran on Kubernetes. Each worker was a Helm release reconciled by Flux (the operator that turns declared manifests into running objects) — a pod, a service, a 10Gi volume, an nginx sidecar, an ingress on its own subdomain. Per user. And each one requested two CPUs and 4Gi of memory, with ceilings at double that — requested meaning held by the scheduler whether or not anyone was typing.

Nobody was booting a pod from scratch per message. There was a warm pool, and it had roughly the shape we would later rebuild on Modal: a leader-elected reconciler holding a target number of idle releases ready, a claim labelling one with a projectId.

The pool was not the problem. The price of a slot was: the target ran at 30, so sixty CPUs and 120Gi sat permanently parked. A small datacenter, idling, as a patience subsidy.

And the subsidy only covered the hits. The acquire path carried a ten-minute timeout.

None of which was Kubernetes failing at its job. Kubernetes is a magnificent piece of engineering for running services: long-lived, interchangeable, reconciled toward a declared state. We weren't running services. We were running disposable laptops, one per person.

The three options

Google Cloud Run is shaped for a different problem: instances serve requests, get recycled between them, and nothing routes a returning user back to the box holding their agent's state. The quota math agrees — 100 instances per revision by default, then your regional vCPU quota.

Kubernetes in a dedicated cluster was the realistic option, and we came close. A warm slot is still a whole reserved pod, every session is still a reconciliation loop with a volume attached, and now there are node pools to autoscale, volumes to garbage-collect and an on-call rotation for the lot: a sandbox platform assembled out of parts designed for something else, forever.

What we wanted was the third thing: ephemeral, strongly isolated, sub-second sandboxes behind an API. Modal is a serverless compute platform out of the Python and ML world, and its Sandbox primitive is the piece that matters: one API call gives you an isolated container with a real filesystem and a real shell, in seconds, billed by the second. We picked it not because Modal is great (it is) but because a platform designed for the shape of your problem makes every later decision cheaper.

Modal, and the bill that came with it

The pitch: Sandbox.create gives you an isolated machine in a few seconds. True. What we measured on our own image was less romantic:

• Cold start: ~2 minutes, almost all of it pulling our agent image out of the registry.
• Warm start: ~7 seconds — 2 for the sandbox, 5 for the agent runtime to boot inside it.

Seven seconds is a fine number for a batch job and a terrible one for a chat thread. Two minutes isn't a number, it's an uninstall. We went after that two minutes later and briefly halved it, then had to hand the win back; it is still two minutes.

So: a warm pool again. The idea didn't get cleverer in the move — a slot just got eight times cheaper and a refill went from minutes to seconds.

The pool

Modal's docs have a sandbox pool example ↗; we started there and mostly stayed. A single modal.Queue of ready workers, ordered oldest-first. A claim takes one off the front. A cron once a minute keeps the queue honest.

A claim pulls two replacements in behind it

POST /workers fires a backfill and doesn't wait for it, then blocks up to fifteen seconds on the queue. The backfill spawns two sandboxes, concurrently. Scale-up is driven by consumption, not by a timer, so the pool leans into a burst as it happens.

Nothing is warm until it's proved warm

A pool entry carries the sandbox's URL and its connect token — the per-sandbox credential for Modal's edge — and the fill path has already made one authenticated call to the agent's health endpoint with it. That call is barely a health check; it's a tunnel pre-warm: the first request through a given connect token is the one that builds the edge path, and it is much slower than the ones after it. Skip it and you've saved a round-trip by charging it to the person waiting in the message thread.

Once a minute, forever, the cron destroys a perfectly healthy sandbox

That last one looks like a bug. It isn't — it's in Modal's example too, and it's the queue deciding the algorithm.

The cron's job is to clear dead and stale-image entries out of the front of the queue. But modal.Queue has no peek. get() removes. So the only way to learn whether the entry at the head is still good is to take it out — and the moment it turns out to be good, you are holding a live, healthy, fully warmed sandbox with nowhere to put it, because put() appends to the tail.

Putting it back is worse than throwing it away. The whole design rests on the queue being sorted by age: the cron walks from the front and stops at the first good entry, assuming everything behind it is younger and at least as healthy. Re-append one old worker and that assumption is gone — dead entries can shelter behind fresh ones indefinitely. One put() and the queue is no longer a queue, it's a bag.

So the cron terminates it, exactly as Modal's example does. One warm sandbox a minute buys an invariant that makes every other part of the pool trivial. It's the cheapest lock we've ever bought.

Then it resizes: below the floor it fires backfills asynchronously, so the cron never blocks on a Sandbox.create, and max_containers=1 keeps it from overlapping itself.

Image rollouts fall out of the same machinery for free

A deploy doesn't restart anything. It spawns one sandbox on the new image, health-checks it, and only then flips the pointer saying which image is current. From that instant every entry in the queue is stale, and the next cron pass terminates all of it — including the freshly validated sandbox, which is now the first good worker the cron meets, and we know what happens to those. There is a few-second window where the pool is empty and a claim waits on a spawn; it has never woken anyone up. No coordinated restart, no manifest to reconcile, and an image that can't boot never becomes current.

Where it landed

• One user, one machine. Every agent runs in its own gVisor-isolated sandbox — a user-space kernel intercepting its syscalls.
• A pool hit is p50 744ms, and a refill takes about seven seconds — where Kubernetes budgeted ten minutes for a miss. Under Modal's own 592-claim burst benchmark, p99 was ~2.1s at 100% success, and closer to 70% without retry handling, because Modal rate-limits Sandbox.create to 5/s per account.
• Sixty permanently-reserved CPUs became a pool that bills for what it uses — a pool sandbox requests 0.25 CPU and 512MB, with ceilings at 4 CPU and 8GB, billed on max(request, actual): an agent idle between messages costs roughly nothing, and one mid-burst gets sixteen times the CPU it reserved.

The platform bet paid, and it still cost a couple of weeks — spent less on learning Modal's limits than on discovering which of our own assumptions had only ever been true by accident: a timeout here, a setting nobody needed there, none of it written down anywhere, because none of it was ever a decision. Those weeks appear in no number above, and they are the ones to budget for.

What's next

A sandbox is ephemeral and a creator's work is not. Someone's agent holds six months of their content history, and the machine holding it can vanish mid-sentence — rotated, killed at its twelve-hour ceiling, or just unlucky. A fresh one has to pick up a conversation it was never part of. Worker memory is not a place to keep things.

Which means snapshots, project-state export, and the general problem of giving a very long memory to a machine designed to have none at all. That's part two.