How app.nz runs GPUs on demand

The app.nz GPU stack is built around one rule: a model that is not serving traffic should not be burning GPU money.

That sounds obvious, but it forces the control plane to own the whole lifecycle: choose hardware, create the machine, pull the image, wait until the model is genuinely ready, proxy the request, meter the run, and tear the machine down when it goes idle.

Hardware is a catalogue, not a string

The GPU catalogue lives in the server as structured machine types: provider, vCPU, RAM, GPU name, VRAM, CUDA compute capability, and hourly price. Compute capability matters. A container built for one CUDA/Torch target can fail on another card even when VRAM looks sufficient.

That is why the catalogue carries entries like:

For cogs and Comfy deployments, declared VRAM becomes a placement input. cheapestGPUForVRAM picks the smallest priced card that fits the workload.

The Cog lifecycle

A Cog model starts as an idle DB row:

cog_models.status = idle
instance_id = ''
endpoint = ''

The first prediction calls ensureCogWarm. That path is deliberately single-flighted: if ten requests arrive at the same idle model, app.nz provisions one machine, not ten. The warm path then:

1. inserts a cloud_instances row,
2. chooses local Docker when allowed and there is live VRAM headroom,
3. otherwise creates a RunPod pod with the configured GPU,
4. attaches registry auth for private or gated images,
5. exposes port 5000 over the provider HTTP proxy,
6. polls /health-check, /healthz, or /openapi.json,
7. introspects the schema if the model did not provide one,
8. marks the model ready and proxies POST /predictions.

The readiness poll is important. A pod being "running" is not the same as a model being ready. Python may still be importing torch, CUDA may still be initializing, weights may still be loading, and the model may still be compiling kernels. app.nz keeps the model in starting until the container says it can serve.

Local headroom first, cloud overflow second

There is an opportunistic local path for low-volume GPU workloads. If APPNZ_LOCAL_DOCKER_COGS=1), the model's hardware policy matches, and nvidia-smi` reports enough free VRAM plus reserve, app.nz can run the Cog image on the prod host's local Docker runtime.

That path mounts:

• a per-instance workspace at /workspace,
• an image-keyed model cache at /models,
• NVIDIA_DRIVER_CAPABILITIES=all,
• a dynamic localhost port mapped to the Cog HTTP port.

If local launch fails, the control plane records the failure and falls back to the cloud GPU. Local capacity is an accelerator, not a dependency.

Scale to zero is two mechanisms

Dedicated pods scale down through the Cog idle reaper. Each successful prediction stamps last_used_at and arms a timer. If nothing touches the model within its idle window, default 120 seconds, sleepCogModel terminates the cloud instance and clears the endpoint.

Serverless workloads use RunPod serverless endpoints. Those are created with workersMin=0, bounded workersMax, queue-delay scaling, and an idle timeout. They cost nothing while idle, and app.nz can retire the endpoint when a deployment changes image.

What users see

The product surface is simple:

• register an image and hardware,
• optionally declare inputs and output kind,
• click predict,
• get a typed UI and a stored prediction record.

The control plane handles the unpleasant parts: registry credentials, provider ids, proxy URLs, health checks, per-request timing, cost calculation, and teardown. That is the real feature. GPUs on demand are not just GPU rental; they are lifecycle ownership.