Anatomy of an AI compute marketplace: tokens, GPU-seconds, and agent-hours on one set of rails
A deep tech dive on app.nz as a compute marketplace: 232 models across 24 providers with price-ordered self-healing failover, GPU placement that learns cold starts and arbitrages clouds under a fixed price, margins enforced as unit tests, and an agent layer that makes the whole market liquid.
Yesterday we published our ten-year thesis: minds become products, markets go machine-speed, and the winners own their stack. This post is the other half of that argument — the tech that exists today. app.nz is, mechanically, a marketplace for AI compute: three exchanges (tokens, GPU-seconds, agent-hours) behind one API, one prepaid credit, and a router whose whole job is to land every workload on the cheapest hardware that can actually run it.
Everything below ships in production. File names included, because the repo is the data room.
Exchange 1: tokens
The gateway is an OpenAI-compatible surface over 232 models from 24 providers — one catalogue drives alias resolution, pricing, capability flags, and failover for every request (how routing works). The marketplace mechanics:
- Price-ordered failover. A request carries a fallback chain; by default
fallbacks are sorted cheapest-first by list price, and provider errors (401/402/408/429/5xx…) fall through to the next venue while genuine bad requests stop immediately. An exhausted chain is recorded for the auto-fix monitor instead of vanishing into a 502.
- Auto-routing.
automodels classify each prompt (vision, code,
reasoning, short-cheap) and pick the tier — so callers can buy "answer this well" instead of naming a vendor. The classifier is deterministic today and designed to be swapped for an embedding model without rewiring.
- Self-healing params. Provider APIs drift weekly — new model
generations reject parameters older ones required. A static dialect table handles the known quirks (max_tokens vs max_completion_tokens, reasoning models rejecting sampling params), and a reactive layer parses structured 400s, renames or drops the offending parameter, and retries. Model launches stop being integration projects.
- Bring your own supply. BYOK keys (encrypted at rest) take precedence
over platform keys per provider, and subscription-plan OAuth accounts are a first-class capacity tier rotated LRU with circuit-breaking to API keys.
Metering is stream-native: responses tee to the client while the usage frame is scanned, and cost settles into prepaid credits at $0.001 each — the same unit that prices GPU-seconds, training, and everything else (metering deep dive).
Exchange 2: GPU-seconds
GPU serving on app.nz is one advertised price per tier with floating supply underneath (multi-cloud economics). Three mechanisms do the work:
Hardware selection is a price query. Every deploy — cogs, Comfy, training, an MCP call — resolves hardware the same way: the cheapest GPU whose VRAM clears the model's floor. Nobody picks a GPU by name unless they want to.
Supply is arbitraged live. A provider price matrix tracks the same silicon across venues — RunPod's community cloud runs ~30-40% under its secure cloud, so pods deploy cloudType: AUTO and chase community capacity first. Unintegrated venues (Vast, Lambda) sit in the matrix as tracked quotes, so the margin the next integration unlocks is a number we watch, not a guess. TPU capacity defaults to spot. The matrix is public at /api/gpus/pricing.
Placement is learned per model. Each cog records its own cold-start EMA, per-tier latencies, and observed peak VRAM. The router uses them to walk a three-tier ladder — spare capacity on our own GPUs (free), serverless workers, dedicated pods — with hysteresis thresholds, and hedges: when a pod is cold and the learned cold start is long, the request runs serverless immediately while the pod warms behind it. Callers never see a cold start; idle models cost zero (scale-to-zero mechanics). Observed-VRAM data also feeds the long game: a cog that declares 24GB but peaks at 9GB is a bin-packing candidate — two models, one GPU, half the hardware cost.
Interactive workloads get the same treatment at one-second resolution: websocket sessions proxy straight into the container and bill wall-clock GPU-seconds from open to close.
Margins are unit tests
Two invariants run in CI on every commit:
- For every GPU tier, the cheapest routable upstream price must sit below
what we bill — the spread can never silently invert.
- Every billed GPU price must undercut
Replicate's published rates for the same silicon.
Pricing bugs in a marketplace are existential, so the margin is an asserted property of the codebase, not a spreadsheet. The other half of margin discipline is not paying for mistakes: a reaper reconciles the live cloud account against our database (pod names embed the database row id, so orphans are provable), instances that never became ready bill zero by construction, and every training or agent run carries spend caps and TTLs. That engineering was paid for by a real $954 phantom-billing incident; it now cannot recur by design.
Exchange 3: agent-hours
The agents layer is what makes the first two exchanges liquid — agents are both the product and, increasingly, the customer buying tokens and GPU-seconds at machine frequency.
One durable lease queue (crash-recovering, priority-ordered by plan) feeds interchangeable execution backends: sandboxed per-task jobs on our k3s fleet, an in-process serverless engine whose text-based tool protocol runs on any model — including ones with no native function calling — and, via `app worker`, the user's own machine as a supply venue (how worker jobs run). Every backend produces identical step, diff, and PR records; diffs snapshot to CDN so reviewing an agent's work never waits on a pod. Skills are packaged, versioned competencies injected per task, and the whole platform — deploys, research, media generation, repos — is exposed as ~50 metered MCP tools, so third-party agents transact on the same rails at the same prices.
The compounding loop investors should care about: agents buy compute → routing data improves → margins hold while prices fall → cheaper compute attracts more agents. Every optimization we write lands in our unit economics because we own the serving stack.
Two-sided, on purpose
Supply-side onramps already exist: model providers apply to list on the gateway (OpenAI-compatible API, machine-readable pricing, staged rollout behind the parity harness), and builders selling software on hosted apps get Stripe Connect checkout with plan-tiered platform fees of 3-5% — deliberately exchange-like, not app-store-like — plus one-click conversion of sales earnings into hosting credits, which closes the loop entirely inside the platform.
Honest roadmap
What's schema-today, shipping-next: GitHub auto-deploy hooks and managed addon provisioning for hosted apps, interactive agent sessions over ACP, and runtime enforcement of the under-Replicate guard (today it's a CI assertion against published prices). We publish the gaps because the investor room is versioned in the repo — every claim above has a file name, and the ones that don't yet are listed here.
Sources
- RunPod pricing (secure vs community cloud) · Vast.ai · Lambda
- Replicate pricing — the public benchmark our GPU prices are tested against
- Stripe Connect direct charges
- omniserve — our open-source VRAM-aware model server (the /status probe contract)
- Our own deep dives: dynamic routing · multi-cloud GPU economics · metering · agent worker jobs · the ten-year thesis