Why Docker cold starts are slow

Docker cold starts are slow when the first request has to do all the work that should have happened earlier.

On app.nz we see cold start latency as a stack, not a single number:

queue / provider scheduling
  + image auth and image pull
  + container create
  + Python imports
  + CUDA context init
  + model weight download
  + weight load to VRAM
  + optional torch.compile / graph warmup
  + health check
  + first prediction

Only one of those lines is "Docker starts a container." For AI workloads, it is rarely the biggest one.

The expensive parts

Image pull. A multi-GB CUDA image over a remote registry is already a cold start. If the image is private or gated and auth is missing, the pod may stop before the app even starts. app.nz checks image accessibility and attaches RunPod registry auth so failures happen with actionable errors.

Python import and CUDA init. Importing torch, loading kernels, and creating the first CUDA context can take seconds before any model code runs.

Weights. Downloading a model from Hugging Face during boot is the classic cold-start mistake. It turns every scale-from-zero event into a network benchmark.

Compilation. torch.compile, TensorRT plans, Triton kernels, and attention backend selection can make steady-state inference faster while making the first request painful.

Lazy readiness. If /health-check returns ready before weights are loaded, the user's first prediction pays setup time. That is worse than an honest cold start because it looks like inference latency.

What app.nz does about it

The Cog contract asks containers to expose /health-check with SETUP until they are actually ready. The control plane polls /health-check, /healthz, or /openapi.json before marking a model ready. For real pods, the cold-start wait has a generous cap because model images can legitimately pull large layers and load weights.

For local Docker cogs, app.nz mounts an image-keyed model cache at /models. A model that scales to zero can come back and reattach its cache instead of downloading the same weights again.

For productized workers such as splat capture and fast video, the deploy checklist pushes us toward R2 model mirrors. Workers pull from an app.nz-controlled mirror first, with parallel range/resume logic where the worker supports it, and fall back to upstream model hosts only when needed.

For serverless RunPod endpoints, app.nz keeps workersMin at zero and lets the provider scale workers in seconds for low-RPS jobs. That accepts cold starts as the price of zero idle cost. For sustained traffic, the router promotes to a pod where the cold start is amortized across many requests.

The rules we use

1. Bake or mirror weights. Do not download the largest artifact from a third-party host on every boot.
2. Keep runtime images boring. Build tools belong in the builder stage, not the serving image.
3. Report setup honestly. SETUP until the model can serve. A fast fake health check moves latency into the user's request.
4. Cache across warm/sleep cycles. Local /models caches and provider volumes are worth more than clever retry loops.
5. Separate cold-start latency from inference latency. Store started, ready, prediction, and finished times separately.
6. Choose serverless only when idle cost matters more than tail latency.

The product tradeoff

There is no universal fix. Keeping a pod warm solves cold starts by paying for idle GPU time. Serverless solves idle cost by accepting cold starts. Local headroom solves both only when the model is small and the prod GPU is free.

The right answer is routing, not ideology. app.nz's GPU router has three practical tiers: local headroom, serverless workers, and dedicated pods. Cold starts are slow, but they are also measurable. Once they are measured, the platform can decide whether this request should pay them.