How app.nz hosted training works

Hosted training on app.nz is a control plane around trainer containers. The important design choice is that the web server does not run PyTorch. It validates the request, chooses hardware, creates a durable job row, hands a signed spec to a GPU worker, and then treats progress callbacks and artifacts as the source of truth.

That separation keeps training jobs from becoming long HTTP requests. A LoRA run can take minutes, a full fine-tune can take hours, and the user should be able to close the browser without changing the job lifecycle.

The request

A job starts with four user-visible inputs:

The model catalog lives in server/training.go. Each entry carries the Hugging Face repo, modality, supported methods, LoRA/full/inference VRAM floors, and trainer image. The API uses those fields to compute offers and to avoid scheduling a job onto a GPU that cannot fit it.

Pricing before scheduling

Training pricing is computed before the job is created:

raw GPU hourly price
  x training margin
  x serverless premium when using serverless
  x estimated hours

The API returns both tiers for each model and method:

• Serverless: higher hourly equivalent, no pooled pod startup amortization, good for short jobs.
• Pool: lower hourly price, pays startup overhead, good for longer jobs.

The recommended offer is whichever has the lower estimated total. Actual compute is metered from started_at to finished_at by wall-clock seconds, and finished weights accrue storage at the R2-backed checkpoint rate.

The trainer contract

The worker input is intentionally small:

{
  "model": "Qwen/Qwen3.5-4B",
  "model_id": "qwen3-5-4b",
  "method": "lora",
  "dataset_url": "https://...",
  "hyperparams": {},
  "callback_url": "https://app.nz/api/training/jobs/<id>/events",
  "job_token": "<secret>",
  "upload_urls": {
    "weights.tar": "https://...",
    "adapter_model.safetensors": "https://...",
    "adapter_config.json": "https://..."
  }
}

The trainer reports progress back with a bearer job token. The server records those events as logs and updates the durable job state:

queued -> provisioning -> training -> uploading -> succeeded
                                      -> failed
                                      -> canceled

That means the UI, CLI, and desktop app all read the same state. They do not need to understand the worker's internal loop.

Artifacts and deploys

The output weights are uploaded through presigned R2 PUT URLs. On success, the job stores the public output URL and probes the content length so storage billing can be recomputed idempotently.

From there the user has three clean exits:

• Download the weights directly.
• Publish them into an app.nz git repo with a README and training.json lineage.
• Deploy them to a dedicated Cog endpoint with weights_url wired into the generated schema.

The publish step is deliberately boring. A model card that records base model, dataset URL, hyperparams, hardware, tier, job id, and finished time is more useful than a magic dashboard screenshot when somebody needs to audit where a checkpoint came from.

Why callbacks win

RunPod returns a terminal job output, but relying only on that would make the UI blind while the worker is training. app.nz accepts worker callbacks during the run and still has the runner path as a fallback. The terminal update is idempotent, so a final callback and a final poll result cannot race the job into two different states.

That is the shape we want for cloud training in general: durable rows, explicit terminal states, signed upload URLs, signed progress callbacks, and trainer images that can evolve independently from the web server.