RunPod serverless endpoints under the hood

RunPod serverless is a good fit for GPU workloads that are valuable when they run but mostly idle the rest of the day. app.nz uses it as one tier in the GPU routing system, not as the only execution model.

The goal is to pay for work, not for silence.

Endpoint shape

When app.nz creates a serverless endpoint, the defaults are tuned for scale-to-zero:

• workersMin is zero,
• workersMax is bounded,
• queue-delay scaling is enabled,
• idle timeout is set,
• the endpoint points at the model image and environment.

workersMin zero is the product promise. If nothing is running, the endpoint should not burn GPU time.

Submitting work

Serverless requests become provider jobs. app.nz sends the input payload, receives a job id, and polls until the job finishes or fails. That is different from a warm pod where app.nz can proxy HTTP directly to a model server.

This difference leaks into UX. Serverless is naturally job-shaped:

submit queued running completed or failed

For image and video workflows, that is fine. For low-latency chat-like inference, it may feel wrong.

Why queue delay matters

Queue-delay scaling tells the provider when to add workers. If the queue starts waiting too long, scale up within the configured bounds. If traffic disappears, workers scale down.

The platform should set bounds because unbounded GPU fan-out is dangerous. A viral endpoint should not create unlimited spend just because the user shared a link.

Endpoint reuse

Creating endpoints has its own overhead. app.nz can keep the serverless endpoint object around while allowing workers to scale to zero. That way the next request avoids the control-plane setup cost but still does not keep a GPU warm.

When the deployment image changes, the old endpoint should be retired or replaced. Reusing an endpoint for the wrong image is worse than a cold start.

Failure modes

Serverless can fail in ways a pod does not:

• job queue timeout,
• worker never becomes healthy,
• provider capacity unavailable,
• image pull auth failure,
• payload too large,
• endpoint stuck on an old image.

Those failures need to show up as deployment or prediction errors, not generic 500s.

When app.nz chooses it

Serverless is best when idle cost dominates:

• sporadic generation jobs,
• demos,
• internal admin tools,
• user-triggered media tasks,
• workloads with acceptable queue latency.

When a model receives steady traffic, app.nz promotes to a pod. When a user wants maximum control, they can pin dedicated capacity.

Serverless is not magic. It is a cost shape. app.nz's job is to use that shape when it fits and leave it when it stops fitting.