A token-efficient coding agent: app agent vs Codex, measured

We put a coding agent inside the app CLI. Not a client that queues a cloud task — the tool loop runs in your terminal, against the app.nz model gateway, and it keeps working toward a goal on its own.

Two things came out of building it that are worth writing down: a head-to-head measurement against Codex on the same small tasks, and the specific design choices that make an agent cheap enough to leave running.

The measurement

Two fixture repos, both with a failing node --test suite and a one-line brief. One is a bug fix (a slugify that mangles separators), one is a blank implementation (histogram(values, binCount)). Five trials each, per agent. A run only counts as a pass if the suite goes green and the agent did not touch the tests — the harness checks git diff on test/ and scores an edited test suite as a failure.

Totals across all twenty runs:

Both agents solved every task. app agent finished in about 73% of the wall time on roughly 43% of the tokens. Note the spread: the same agent on the same task ranges 19k to 78k tokens depending on how many turns it takes to converge. Medians over ten runs, not a hero number from one.

What this does and does not show

Read it as a comparison of two stacks, not two harnesses in a vacuum:

• The models differ. app agent ran on claude-sonnet-latest through the

app.nz gateway; codex exec ran on its own default model on a ChatGPT Pro plan. gpt-5-codex is rejected for ChatGPT-account auth, so "codex default" is the honest out-of-the-box configuration.

• Codex ran with --ignore-user-config. With the operator's real config, one

run pulled a GitHub MCP server into context and burned 473,422 input tokens listing repositories before it read a single file. That is a fair warning about MCP servers in an agent's default context, but it is not a measurement of Codex.

• Sandboxing was off for both. This host cannot run bubblewrap, so Codex's

sandbox fails before every filesystem operation; both agents got the same free hand instead.

• n is still small: two tasks, five trials, one machine, one afternoon.

The harness is in the repo (cli/bench/) so you can disagree with it precisely:

cli/bench/run.sh --trials 5 --tasks slugify,statsbin --agents app,codex

Where the tokens actually go

The interesting number in an agent is not the answer's length. It is the system prompt, because you pay for it on every single turn, and a coding task is usually 4 to 18 turns.

Our first version inlined the whole skill catalogue — 184 skills discovered across the project, the user's Claude and Codex homes, and a Hermes checkout — into the prompt. That is 5,018 tokens of pure preamble, resent every turn.

We changed it to inline the 25 skills most relevant to the actual task, with a single line pointing at skill(action:"search") for the rest. Same library, same discoverability, one ranking pass. The system prompt fell to 1,085 tokens, and the end-to-end cost of the benchmark tasks fell with it:

A 2.4× reduction from one change, with no loss of capability. The general rule: anything you inline unconditionally is multiplied by your turn count. Put it behind a tool call and you pay for it only when it matters.

The infinite run: planner, manager, verify, commit

A single-shot agent is easy. The hard part is leaving one running for an hour without it drifting, looping, or cheerfully reporting success on a broken build.

app agent auto runs three models in different roles:

        ┌─────────────┐   next step   ┌──────────────┐
        │   planner   │ ────────────▶ │    worker    │  tools, edits, tests
        │ (cheap, no  │               │ (your model) │
        │   tools)    │ ◀──────────── └──────┬───────┘
        └─────────────┘   trace digest       │
                                             ▼
        ┌─────────────┐              ┌──────────────┐
        │   manager   │ ◀─────────── │   verify     │  npm test / go test
        │ (different  │  same digest └──────┬───────┘
        │   model)    │                     │ pass
        └──────┬──────┘                     ▼
               │ stop / steer         git commit (only files it touched)
               ▼

The planner decides the next step. After each iteration it reads a bounded digest — the plan, the files changed, the last 40 trace events, tokens spent — and returns JSON: done, or the single next instruction. It never sees the full transcript, so its cost is flat no matter how long the run goes.

The manager is the part we had not seen elsewhere and the part that earns its keep. Every N iterations a different model reads the same digest and returns one of three verdicts: continue, steer (with a replacement instruction), or stop. It is prompted to be sceptical of the worker's own claims and to trust verification output over prose. A worker that has convinced itself it is done does not get to be its own judge — and a worker looping on the same file gets stopped by something that is not invested in its story so far.

Verify is the ground truth. When a run is given --verify "npm test", a failing suite is not a reason to stop; the failure output becomes the next instruction verbatim. No commit happens until it passes.

Commit stages only the files the agent touched in that iteration, never git add -A. An autonomous run must not sweep up whatever else was in your working tree — that is a test we wrote before we wrote the feature.

What a million tokens taught us

The first real autonomous runs — closing the nested-symbol gap in animflow, an open-source Adobe Animate alternative — shipped working code in verified commits, and cost 1.1M tokens to do it. The trace showed exactly where the tokens went, and none of it was thinking hard:

• One iteration was allowed to run forty tool-calling turns. Every turn

re-sends the whole transcript, so cost grows roughly as transcript size × turns. Iterations now cap at 12 turns and hand control back to the planner.

• Compaction only triggered near the model's context limit. It now triggers at

48k estimated tokens: the transcript is kept small deliberately, not merely kept legal.

• Each iteration inherited the previous one's transcript. Iterations now start

from the same compact handover the planner reads, unless you pass --carry-context.

• The agent re-read the same files repeatedly. An unchanged file it has already

read in this conversation now returns a one-line pointer instead of its contents, and the cache is cleared whenever the transcript is reset — so it can only ever point at text that is genuinely still in context.

Together those four changes took the benchmark's median run from 45,006 to 31,691 tokens on identical tasks with an identical pass rate.

Then a fifth, worse bug surfaced — one the small benchmark tasks could never have caught. When a model asked to read three files at once, all three tool calls arrived in a single streamed delta with index: 0, because the gateway had translated them from another provider's protocol. Our parser keyed fragments by index, so the three calls merged into one whose arguments were three JSON objects glued together: {"path":"a.js"}{"path":"b.js"}. Every parallel read failed to parse, the model retried, and an entire iteration went by with nothing written. Five iterations and 818,000 tokens produced zero file changes.

Keying tool-call fragments by id when the provider sends one — falling back to index only for continuation fragments that carry no id — fixed it. The next run on the same goal produced three verified commits in four iterations: a symbol model, stage rendering with per-instance transforms and start frames, nesting, and project serialization.

It also shipped two defects worth naming, because they are the honest shape of autonomous work. It committed a test file that was zero bytes — an empty suite passes, so --verify was satisfied and the manager saw a green run — and two commit messages claimed documentation updates that were never written. The fix for the first is a rule, not a prompt: the loop now refuses to commit a file the agent left empty, and the commit body lists the files actually changed so the subject's claim is checkable against it. The second still needs a human, or a reviewer model reading the diff rather than the digest. We wrote the real tests and the docs by hand afterwards.

The lesson generalises twice over: in an agent, the expensive mistakes are structural rather than clever, and a benchmark of small tasks will not find the ones that only appear at scale. Watch the token graph of a long run; anomalies there are bugs, not appetite.

Budgets are first-class, because "run until done" needs a floor:

app agent auto "close the symbol/instance parity gap" \
  --verify "npm test" --commit \
  --manager-every 2 --token-budget 900k --time-budget 45m

Running with no disk

One more design note, because it changed the architecture more than expected.

Agent state — session journals, memories, history — lives in a store that probes every plausible root (state dir, other mounts, /var/tmp, tmpfs) with a real write-and-read-back test, then fails over automatically when a disk fills up or goes read-only: next disk, then your private app.nz artifact filesystem, then memory. Child processes get a TMPDIR inside whichever root won, so builds do not die on a full /tmp either.

A session started on a machine with zero writable disk still resumes on another machine. app agent storage prints the chain and every failover it has made.

Try it

curl -fsSL https://app.nz/cli.sh | sh
app login
app agent                      # interactive, in the current directory
app agent "why is this test flaky?"
app agent auto "make the build reproducible" --verify "make test" --commit
app agent doctor               # storage, model routes, plan sync, skills

If you have a ChatGPT plan, app agent auth sync points the gateway at it, and model calls from every app.nz surface bill to your plan's quota before they touch credits.

The CLI is open source and the benchmark harness ships with it. If your numbers disagree with ours, the fixtures are three files each — send them.