Dynamic routing across model providers

The app.nz gateway has one job: let callers ask for a model by intent, while the control plane handles provider-specific details, keys, fallbacks, billing, and weird API differences.

The public interface stays boring:

{
  "model": "app/auto-code",
  "messages": [{"role": "user", "content": "Fix this failing test"}],
  "stream": true
}

Behind that, a request goes through a routing pipeline.

Step 1: resolve names and aliases

The gateway loads model_catalog.json when present and falls back to an embedded seed catalogue when running in a fresh clone or CI. Each catalogue model has:

• an app.nz id,
• provider,
• provider model id,
• aliases,
• fallback models,
• prices,
• context window,
• modality flags.

Lookups are forgiving. normalizeModelName lowercases names and collapses spaces and underscores to hyphens. The app/ and appnz/ brand namespaces map onto the canonical app.nz/openpaths auto routes, so clients can say app/auto without caring about the internal catalogue prefix.

Step 2: classify bare auto

A bare auto request is refined before provider resolution. The current classifier is deterministic and intentionally easy to audit:

reasoning_effort: "auto" uses the same classification: reasoning prompts get high thinking, code gets medium, ordinary prompts get low, and cheap/vision requests drop extended thinking.

This is deliberately not magic. The classifier can later become embedding- or eval-driven without changing the gateway contract.

Step 3: build the fallback chain

The resolved model contributes its fallback list. The request can also set a routing strategy:

Before trying the chain, app.nz applies model IAM. If a user or org policy blocks a model, it is removed up front, including from fallback candidates. An allowed primary should not silently fail over to a disallowed fallback.

Step 4: choose the key

For each candidate provider, the gateway picks credentials in this order:

1. caller BYOK key for that provider,
2. platform environment key,
3. skip the provider if neither exists.

BYOK keys are stored in the app.nz DB and can be encrypted at rest with the gateway secret key. Usage analytics remain per user/provider/model either way.

Step 5: adapt the provider

Most providers are OpenAI-compatible enough to receive /v1/chat/completions, but not all. Google uses a different OpenAI-compatible path. Anthropic has /v1/messages. fal media models use queue APIs. Cursor composer models run as async jobs. Some providers reject fields that others ignore, so the gateway strips or rewrites provider-specific fields such as reasoning_effort or image size when needed.

The response includes routing headers:

X-Gateway-Provider: ...
X-Gateway-Model: ...

That makes routing observable without exposing secrets.

Step 6: fail over only on the right failures

The gateway falls through to the next candidate for provider unavailability:

401, 402, 403, 404, 408, 409, 425, 429, and 5xx

Those statuses usually mean auth, quota, rate limit, missing model, or provider fault. A malformed request, such as a 400/422 schema error, is terminal because sending the same bad body elsewhere will probably fail there too.

Step 7: meter after success

For token models, app.nz reads upstream usage fields and bills against the catalogue's per-token prices plus markup. For streaming, the gateway tees SSE chunks to the client while scanning for the usage chunk. For images and media, metering uses the model's per-image, per-second, or job pricing shape.

The important part is that routing and billing share the same catalogue. The price shown on /gateway is the price the meter uses.

Why dynamic routing matters

Provider APIs are unstable in small ways: models move, balances expire, rate limits hit, auth scopes change, and one provider's "OpenAI-compatible" field is another provider's 400. Hard-coding one provider turns every one of those into an outage.

Dynamic routing makes the gateway less brittle. The user can pin an exact model when they need determinism, or ask for app/auto-code and let app.nz choose the best available path for that request.