Metering the model gateway without guessing

Routing a model request is only half the gateway. The other half is metering it correctly after it succeeds. app.nz uses the same model catalogue for routing, prices, and billing so the number users see on the gateway page is the number the meter applies.

That sounds obvious. It is easy to get wrong when providers return different response shapes.

Token models

For ordinary chat completions, the provider response usually includes usage:

• prompt tokens,
• completion tokens,
• total tokens.

The gateway reads those fields and multiplies them by the model's input and output prices. The result is stored as micro-dollars and charged as credits with the platform markup applied.

This is why catalogue quality matters. A model entry is not just UI metadata. It is the billing contract.

Streaming models

Streaming complicates metering because the gateway cannot wait for the full response before sending chunks to the client. app.nz tees server-sent events to the caller while scanning the stream for the usage chunk.

The happy path is:

upstream SSE chunk arrives write chunk to client inspect chunk for usage keep forwarding stream ends bill once from captured usage

The important rule is bill once. A retry or reconnect should not double-charge a finished stream. Logging and billing are tied to the request lifecycle, not to each chunk.

Image and media models

Image APIs often do not return token usage. The meter needs a different unit:

• price per image,
• price per video second,
• price per audio second,
• price per queued job.

The gateway already knows the resolved catalogue model, so it can choose the right meter. For fal queue media, the provider returns a job envelope. For Google video and similar APIs, request parameters such as duration and output count matter. For image generation, count the returned images or requested count.

One catalogue, multiple meters.

BYOK still needs analytics

When a user brings their own provider key, app.nz may not pay the upstream provider. But the request still needs logging:

• provider,
• resolved model,
• status,
• cost estimate,
• timestamp,
• user or org.

BYOK users care about analytics too. They need to know which models are being used and where failures happen. The gateway should not become blind just because the upstream invoice goes elsewhere.

Failure accounting

The gateway logs failures with status and provider/model context. That data is operationally important:

• a provider key is missing,
• a provider is rate-limiting,
• a fallback model is misconfigured,
• a model was removed upstream,
• a request shape is invalid.

Billing happens only on successful responses with billable usage. Observability happens on every route attempt.

The invariant

Every model request should answer four questions:

1. What did the caller ask for?
2. What provider and model actually served it?
3. What did it cost?
4. If it failed, where did the chain stop?

Dynamic routing without metering is an unreliable proxy. Metering without routing context is an accounting black box. app.nz keeps them together.