Cheaper GPUs by scheduling: multi-cloud arbitrage, hidden cold starts, and cogs that learn their own footprint
How app.nz cog hosting cut serving costs: community-first multi-cloud provisioning under a fixed price, serverless hedging that hides pod cold starts, and per-model learned stats (cold-start EMA, peak VRAM) that feed routing and future GPU bin-packing.
Serving GPU models cheaply is mostly a scheduling problem. The GPU itself has one price; what you actually pay is shaped by three inefficiencies: paying for idle time, paying cold starts on every burst, and paying one provider's list price when another has the same silicon cheaper. This post covers how app.nz's cog hosting now attacks all three — automatically, per model.
1. Multi-cloud underneath a fixed price
Every GPU tier on app.nz has one advertised price. Underneath it, provisioning now consults a provider price matrix — the live list prices for the same GPU across clouds — and chases the cheapest capacity that can actually run the workload:
GET /api/gpus/pricing
h100: appnz $X/hr | runpod community $2.39 | runpod secure $2.99 | lambda $2.49* | vast $1.65*
(* = quote-tracked, integration pending)The first integrated arbitrage is RunPod's own two-tier market: community cloud runs the same GPUs ~30-40% below secure cloud. Cog pods now deploy with cloudType: AUTO — try community first, fall back to secure when no community host has capacity. No customer-visible change; the same request just costs us less to serve. Vast and Lambda rows sit in the matrix as tracked quotes so we always know how much margin the next integration unlocks.
2. Hiding cold starts behind serverless
A dedicated pod is the cheapest way to serve sustained traffic; serverless workers are the cheapest way to serve a trickle. The expensive part is the transition: the first request after a quiet period used to eat the pod cold start — image pull plus weight load, minutes for a big diffusion or LLM image.
The router now hedges: when traffic crosses the promote threshold but the pod is still cold, the request runs on a serverless worker immediately while the pod warms in the background. Callers never see the cold start; by the time the burst is a few requests deep, the warm pod takes over and the serverless premium stops.
request → pod warm? → pod (cheapest, no latency)
→ pod cold, low RPS → serverless (no pod spend at all)
→ pod cold, rising RPS→ serverless NOW + warm pod in background ← new
→ pod dies mid-flight → retry on serverless ← new3. Every cog learns its own scaling profile
None of the thresholds above are guesses anymore. Every cog now records:
- cold start EMA — measured from provision call to first healthy response, so the router knows whether hiding a cold start is worth a serverless premium for this model (an 8s TTS boot doesn't need hedging; a 4-minute video model always does);
- per-tier run times — pod vs serverless latency for the same model;
- peak VRAM and RAM — probed from the running container (any image exposing an omniserve-style
/statusendpoint reports real memory numbers).
The VRAM peaks are the interesting long game: a cog that declares a 24GB tier but never exceeds 9GB observed peak is a bin-packing candidate — two of those share one GPU, halving the hardware cost of both. The stats table is exactly the dataset that scheduler needs.
The open-source side
The probe contract comes from omniserve, our open-source omni model server — one image that serves diffusion, video, LoRA and LLM families with VRAM-aware loading, sleeping and eviction, and reports its memory reality at /status. Run it yourself, or deploy it on app.nz with scale-to-zero billing.
The pricing matrix is public at /api/gpus/pricing, machines at /pricing. When the numbers move — providers reprice weekly — the matrix moves with them, and the router just keeps picking the cheapest floor.