Keeping training GPUs busy without wasting memory

The expensive part of hosted training is not the API request. It is the GPU sitting under a trainer container. The control plane can pick the right machine, but the trainer has to keep that machine fed and release memory as soon as large phases are over.

This is the checklist we use for app.nz trainer images and adjacent GPU workers.

Fit first, then go fast

A run that OOMs at step 20 is the worst possible utilization. The model catalog gives every trainable model three VRAM floors:

• LoRA training VRAM
• full fine-tune VRAM
• inference VRAM

Those are conservative floors, not leaderboard claims. They assume bf16 where possible, fp16 fallback where needed, and memory-saving optimizer choices for larger runs. Once a job fits, the trainer can spend remaining memory on batch size, workers, compile, and checkpointing choices.

Defaults that reduce GPU seconds

Trainer images built on app-trainer-sdk advertise these defaults through GET /api/training/models:

The goal is not to expose every PyTorch flag. The goal is to make the no-config path good, then leave explicit hyperparams for users who know their workload.

Memory should be released by phase

Many GPU jobs have phases:

1. Load a large helper model.
2. Extract features, poses, latents, captions, or embeddings.
3. Train the actual small adapter or downstream representation.
4. Export artifacts.

If phase 1 keeps tensors alive while phase 3 starts, utilization gets worse even if the code eventually calls torch.cuda.empty_cache(). The cache can only release blocks that no live tensor still references.

The pattern we want is:

with torch.inference_mode(), torch.autocast("cuda", dtype=dtype):
    outputs = helper_model(inputs_cuda)

cpu_outputs = move_only_needed_outputs_to_cpu(outputs)
helper_model.cpu()
del outputs, inputs_cuda
torch.cuda.empty_cache()
start_training(cpu_outputs)

That is especially important for pipelines such as splat generation, where a large vision model estimates geometry and then a separate optimizer needs the GPU for Gaussian training.

Utilization is also I/O

GPU utilization drops when the trainer waits on the network or CPU. The hosted system handles the obvious control-plane pieces:

• mirror base weights to R2 where possible
• use ranged, resumable, parallel downloads
• use presigned upload URLs so the worker uploads directly
• avoid routing large artifacts through the web server
• keep progress callbacks small

Inside the trainer, the equivalent is to keep dataset decoding predictable. A bigger dataloader is only useful if CPU, memory, and storage can keep up. For small image LoRAs, too many workers can add overhead; for large video datasets, too few can starve the GPU.

Compile is a throughput trade

torch.compile can make steady-state steps faster, but it moves time into startup. That is good for long training runs and bad for tiny smoke tests. app.nz exposes compile-related hyperparams rather than forcing one answer:

• torch_compile
• compile_mode
• gradient_checkpointing
• gradient_accumulation_steps
• dataloader_workers
• optim_8bit

The right default is workload-dependent. The product default should be fast enough for common runs; the escape hatch should be explicit for teams doing serious tuning.

The control plane still matters

Even perfect PyTorch code wastes money if the control plane is vague. app.nz keeps the job lifecycle explicit, cancels provider jobs on context cancellation, stamps terminal cost once, and stores output size for storage billing.

That is the boring work behind better cloud training: pick hardware conservatively, move bytes directly, make callbacks idempotent, free memory between phases, and expose the few knobs that actually change GPU seconds.