Five-minute training runs: capped QLoRA slices and autoresearch on RunPod

Our roleplay model (a QLoRA fine-tune of Gemma 4 E4B) is trained, merged, and serving in production at 264 tok/s with fp8 weights and MTP-2 speculative decoding. Getting there taught us an expensive lesson: long training runs on shared infrastructure fail in expensive ways. A crashed sweep took the serving GPU down twice in one day, and a forgotten cloud pod burns credits until someone notices.

So we rebuilt training around one rule: no run may exceed five minutes.

Chained micro-runs instead of long jobs

A 5-minute cap sounds like a toy until you make runs resumable. Our QLoRA micro-runner loads the latest checkpoint from a persistent network volume, trains until the wall clock says stop, saves, and exits. An epoch is just a chain of cheap slices:

python runpod_launch.py --job qlora --volume-id <vol> \
  --base-model google/gemma-4-E4B-it --dataset /workspace/data/train.jsonl
# run it again -> resumes exactly where the last slice stopped

Every slice is a full transaction: resume, train, checkpoint, die. If anything crashes, you lose at most five minutes of compute. There is no long-running job to babysit and no state that only exists in GPU memory.

Three layers of credit safety

Cloud GPU bills grow in the gaps between what you launched and what you remembered. The launcher closes those gaps three ways:

1. In-process cap — the trainer stops itself at the wall-clock limit.
2. Pod TTL — every pod carries its kill-time in an environment variable;

the launcher terminates it in a \finally\ block even when polling fails.

1. The reaper — \runpod_launch.py --reap\ finds any pod tagged

\appnz-training\ past its TTL and terminates it. Safe to run from cron.

Above all of that sits a budget ledger: a daily GPU-minute cap charged before pod creation. When the day's 60 minutes are spent, the launcher refuses to start — raising the cap is a deliberate edit, not a default.

Autoresearch: hyperparameter search, speedrun rules

Identically time-boxed runs have a second benefit: val loss becomes a fair, compute-matched metric. A config that learns more in five minutes is simply better — the nanoGPT speedrun rule (optimize learning per wall-clock second, not per step).

Our autoresearch loop exploits that. Each iteration mutates one axis of the current best config (learning rate, warmup, weight decay, batch size, width, depth, dropout, schedule), runs a capped training of a small single-file GPT, logs the result to \experiments.jsonl\, and accepts the mutation only if val loss improves. A leaderboard and the running best config regenerate after every run.

Even the smoke test finds wins: two 20-second CPU iterations took baseline char-level val loss from 2.60 to 2.47 by raising the learning rate. On a community-cloud RTX 4090 at ~$0.30/hour, a full 8-iteration research session costs about two cents.

What runs where

• Prod GPU box: serving only. The training scripts now refuse to run on

it — the fastest way to protect an inference SLA is to make training contention impossible.

• RunPod community cloud: every training and research run, 5-minute

capped, budget-metered, reaper-protected.

• Network volume: datasets, checkpoints, and the metrics log — the only

state that outlives a pod.

The harness lives in \app-site/training/\ — four small files, no framework.