Serverless vs serverful GPU inference

The useful question is not "serverless or serverful?" It is "which tier is cheapest for the next request without making latency unacceptable?"

app.nz uses the same GPU router for Comfy deployments and cogs. It observes request rate in a five-minute sliding window and chooses among four tiers:

The router is small enough to reason about, which is the point.

The default policy

The default policy has three key numbers:

The thresholds are environment-configurable. The shape matters more than the exact values: promote at a higher rate, demote at a lower rate. That hysteresis keeps a warm pod from bouncing between tiers whenever traffic hovers near the boundary.

The decision tree

For each request, the router sees:

• deployment key,
• declared VRAM requirement,
• current tier,
• whether serverless is available,
• whether local Docker has headroom,
• whether the user pinned a dedicated instance,
• any forced mode override.

Then:

1. forced mode wins,
2. dedicated instance wins,
3. local wins when it fits policy and live VRAM,
4. no serverless support means pod,
5. warm pod stays pod until traffic falls below demote RPS,
6. otherwise promote to pod above promote RPS, serverless below it.

This is deliberately boring control-plane code. The hard part is measuring the right input, not making the branch clever.

Why serverless wins at low traffic

Serverless endpoints are created with workersMin zero, a bounded workersMax, queue-delay scaling, and an idle timeout. The endpoint can exist without an idle GPU. For sporadic jobs, that is exactly what you want: pay when a request runs, accept cold starts, and scale down quickly.

That is a good fit for:

• demos,
• admin tools,
• once-per-hour media jobs,
• user-upload workflows,
• experiments where idle cost dominates.

The downside is tail latency. A cold serverless worker may still pull an image, initialize CUDA, and load weights. If the user experience needs consistently low latency, serverless may be the wrong answer even if it is cheaper.

Why pods win at sustained traffic

A dedicated pod pays idle time, but amortizes cold start across requests. Once a model is getting enough traffic to keep a GPU useful, the per-request cost can be lower and latency becomes predictable.

Pods also make sense when:

• the model has huge weights,
• startup compiles kernels,
• the workload needs provider HTTP proxy behavior,
• concurrency is steady,
• the user explicitly wants a warm deployment.

On app.nz, cogs using the pod path are still scale-to-zero. They are serverful while warm, then terminated after their idle window. That is the hybrid shape we like: pod economics during bursts, zero cost after the burst.

Single-flight cold starts

One subtle bug in GPU platforms is request amplification. Ten simultaneous requests to an idle model should not create ten pods. app.nz wraps cold starts in warmFlights, a per-deployment single-flight. The first request provisions the endpoint; the rest wait for the same result.

That protects both cost and provider quota.

The rule of thumb

Use serverless when idle time dominates. Use pods when cold starts dominate. Use local when it fits and is free. Use dedicated when the user wants control more than automatic optimization.

The platform can make that decision because deployments declare their VRAM floor and every request updates the same RPS tracker. Once traffic becomes a signal, "serverless vs serverful" stops being a philosophy argument and becomes routing policy.