Anatomy of an AI compute marketplace: tokens, GPU-seconds, and agent-hours on one set of rails

Yesterday we published our ten-year thesis: minds become products, markets go machine-speed, and the winners own their stack. This post is the other half of that argument — the tech that exists today. app.nz is, mechanically, a marketplace for AI compute: three exchanges (tokens, GPU-seconds, agent-hours) behind one API, one prepaid credit, and a router whose whole job is to land every workload on the cheapest hardware that can actually run it.

Everything below ships in production. File names included, because the repo is the data room.

Exchange 1: tokens

The gateway is an OpenAI-compatible surface over 232 models from 24 providers — one catalogue drives alias resolution, pricing, capability flags, and failover for every request (how routing works). The marketplace mechanics:

• Price-ordered failover. A request carries a fallback chain; by default fallbacks are sorted cheapest-first by list price, and provider errors (401/402/408/429/5xx…) fall through to the next venue while genuine bad requests stop immediately. An exhausted chain is recorded for the auto-fix monitor instead of vanishing into a 502.
• Auto-routing. auto models classify each prompt (vision, code, reasoning, short-cheap) and pick the tier — so callers can buy "answer this well" instead of naming a vendor. The classifier is deterministic today and designed to be swapped for an embedding model without rewiring.
• Self-healing params. Provider APIs drift weekly — new model generations reject parameters older ones required. A static dialect table handles the known quirks (max_tokens vs max_completion_tokens, reasoning models rejecting sampling params), and a reactive layer parses structured 400s, renames or drops the offending parameter, and retries. Model launches stop being integration projects.
• Bring your own supply. BYOK keys (encrypted at rest) take precedence over platform keys per provider, and subscription-plan OAuth accounts are a first-class capacity tier rotated LRU with circuit-breaking to API keys.

Metering is stream-native: responses tee to the client while the usage frame is scanned, and cost settles into prepaid credits at $0.001 each — the same unit that prices GPU-seconds, training, and everything else (metering deep dive).

Exchange 2: GPU-seconds

GPU serving on app.nz is one advertised price per tier with floating supply underneath (multi-cloud economics). Three mechanisms do the work:

Hardware selection is a price query. Every deploy — cogs, Comfy, training, an MCP call — resolves hardware the same way: the cheapest GPU whose VRAM clears the model's floor. Nobody picks a GPU by name unless they want to.

Supply is arbitraged live. A provider price matrix tracks the same silicon across venues — RunPod's community cloud runs ~30-40% under its secure cloud, so pods deploy cloudType: AUTO and chase community capacity first. Unintegrated venues (Vast, Lambda) sit in the matrix as tracked quotes, so the margin the next integration unlocks is a number we watch, not a guess. TPU capacity defaults to spot. The matrix is public at /api/gpus/pricing.

Placement is learned per model. Each cog records its own cold-start EMA, per-tier latencies, and observed peak VRAM. The router uses them to walk a three-tier ladder — spare capacity on our own GPUs (free), serverless workers, dedicated pods — with hysteresis thresholds, and hedges: when a pod is cold and the learned cold start is long, the request runs serverless immediately while the pod warms behind it. Callers never see a cold start; idle models cost zero (scale-to-zero mechanics). Observed-VRAM data also feeds the long game: a cog that declares 24GB but peaks at 9GB is a bin-packing candidate — two models, one GPU, half the hardware cost.

Interactive workloads get the same treatment at one-second resolution: websocket sessions proxy straight into the container and bill wall-clock GPU-seconds from open to close.

Proof: the market produces things

Infrastructure claims are easy to make abstract. These are preserved outputs from ManifoldGen, which uses the same app.nz Cog and provider routes described above. The files below are served from this post, not embedded from a provider demo page.

Music: a minute of audio in 20.592 seconds

This MiniMax Music3 job produced 60.07 seconds of stereo audio in 20.592 seconds on an H200: a 0.343 realtime factor, with zero clipped samples. The manifest retains route, seed, GPU, warm-start state, timing, loudness, hash, and every enabled optimization. Listen to the actual result:

Video: two production-route canaries

The first H3 Max canary is text-to-video: a fixed-seed, five-second 480P result. It completed in 8.138 seconds wall time, with 0.897 seconds reported model inference. The second is image-to-video, preserving the source observatory while adding a slow push, haze, and cloud motion; it completed in 11.653 seconds wall time, with 0.672 seconds reported inference. We keep wall and provider inference separate because queue, upload, and result retrieval are real parts of the product.

That is the marketplace in miniature: an agent chooses a model, the router finds supply, metering records the work, and a durable artifact comes back. The same shape works for code, images, speech, 3D, training, and interactive worlds.

Margins are unit tests

Two invariants run in CI on every commit:

1. For every GPU tier, the cheapest routable upstream price must sit below what we bill — the spread can never silently invert.
2. Every billed GPU price must undercut Replicate's published rates for the same silicon.

Pricing bugs in a marketplace are existential, so the margin is an asserted property of the codebase, not a spreadsheet. The other half of margin discipline is not paying for mistakes: a reaper reconciles the live cloud account against our database (pod names embed the database row id, so orphans are provable), instances that never became ready bill zero by construction, and every training or agent run carries spend caps and TTLs. That engineering was paid for by a real $954 phantom-billing incident; it now cannot recur by design.

Exchange 3: agent-hours

The agents layer is what makes the first two exchanges liquid — agents are both the product and, increasingly, the customer buying tokens and GPU-seconds at machine frequency.

One durable lease queue (crash-recovering, priority-ordered by plan) feeds interchangeable execution backends: sandboxed per-task jobs on our k3s fleet, an in-process serverless engine whose text-based tool protocol runs on any model — including ones with no native function calling — and, via `app worker`, the user's own machine as a supply venue (how worker jobs run). Every backend produces identical step, diff, and PR records; diffs snapshot to CDN so reviewing an agent's work never waits on a pod. Skills are packaged, versioned competencies injected per task, and the whole platform — deploys, research, media generation, repos — is exposed as ~50 metered MCP tools, so third-party agents transact on the same rails at the same prices.

The continuous version is already concrete. Codex Infinity and the app.nz CLI expose --auto-next-steps for obvious follow-on work and --auto-next-idea for new improvement loops:

app agent run --cloud "Improve this product and prove every change" \
  --repo owner/repo --auto-next-steps --auto-next-idea \
  --spend-cap 20 --timeout 86400

"Run forever" cannot mean "spend forever." Durable leases, task timeouts, spend caps, isolated workspaces, resumable traces, and human-reviewable diffs are the product. Autonomy becomes useful when continuation and control are designed together.

The compounding loop investors should care about: agents buy compute → routing data improves → margins hold while prices fall → cheaper compute attracts more agents. Every optimization we write lands in our unit economics because we own the serving stack.

Two-sided, on purpose

Supply-side onramps already exist: model providers apply to list on the gateway (OpenAI-compatible API, machine-readable pricing, staged rollout behind the parity harness), and builders selling software on hosted apps get Stripe Connect checkout with plan-tiered platform fees of 3-5% — deliberately exchange-like, not app-store-like — plus one-click conversion of sales earnings into hosting credits, which closes the loop entirely inside the platform.

Honest roadmap

What's schema-today, shipping-next: GitHub auto-deploy hooks and managed addon provisioning for hosted apps, interactive agent sessions over ACP, and runtime enforcement of the under-Replicate guard (today it's a CI assertion against published prices). We publish the gaps because the investor room is versioned in the repo — every claim above has a file name, and the ones that don't yet are listed here.

Sources

• RunPod pricing (secure vs community cloud) · Vast.ai · Lambda
• Replicate pricing — the public benchmark our GPU prices are tested against
• Stripe Connect direct charges
• omniserve — our open-source VRAM-aware model server (the /status probe contract)
• Our own deep dives: dynamic routing · multi-cloud GPU economics · metering · agent worker jobs · the ten-year thesis