Half the tokens for the same 36 tool calls: optimizing a coding agent

app agent is the coding agent in the app.nz CLI: the tool loop runs in your terminal, against the app.nz model gateway. We benchmarked it against Codex a few days ago and it came out cheaper. Then we went looking for the rest.

This is what that pass found, including the change that made things worse and the measurement that caught it.

The one fact that matters

A chat completion API is stateless. Every turn re-sends the entire conversation: system prompt, tool schemas, every file the agent has read, every command's output. So the cost of a run is roughly

transcript size x turns

Not "transcript size". Not "turns". The product. A file read on turn 3 of a 40-turn run is paid for 37 more times. That single fact is where every real optimization in an agent comes from, and it is also why intuitions from ordinary programming mislead: the expensive thing is not the work, it is the residue the work leaves behind in the transcript.

Why the benchmark could not guide this

Our existing harness runs two fixture repos with failing test suites through the agent and scores only whether the suite goes green without the tests being edited. It is the right way to measure outcomes, and it is nearly useless for attributing cost:

• The model chooses how many turns to take. Same agent, same task, same

prompt: 19k tokens one run, 78k the next.

• Those fixtures are solved in three or four turns, so nothing about long-run

context management is even exercised.

• Every run costs real money and real minutes, so you cannot iterate on a design

by re-running it.

So we built the missing half: a harness with no model in it at all.

Weighing the wire

TestLongRunWireCost scripts a fixed sequence of 36 tool calls — the shape of a real exploration-and-repair session, including the parts nobody designs for:

cycle := []model.ToolCall{
    {Name: "search",    Arguments: `{"pattern":"line 1","path":"handler.go"}`},
    {Name: "read_file", Arguments: `{"path":"handler.go"}`},
    {Name: "read_file", Arguments: `{"path":"store.go"}`},
    {Name: "shell",     Arguments: `{"command":"cat test.log"}`},
    {Name: "read_file", Arguments: `{"path":"handler.go"}`},   // re-read
    {Name: "search",    Arguments: `{"pattern":"line 1","path":"handler.go"}`}, // re-search
    {Name: "shell",     Arguments: `{"command":"cat bundle.min.js"}`},
    {Name: "read_file", Arguments: `{"path":"router.go"}`},
    {Name: "shell",     Arguments: `{"command":"cat test.log"}`},   // re-run
}

A fake gateway answers each request with the next scripted call and weighs the request body. Four cycles, 37 turns, one number out the other end: how many bytes the harness put on the wire. Every tool call is deterministic, so the number is identical run to run — which turns out to matter more than we expected.

go test ./internal/agent -run TestLongRunWireCost -v

Baseline, before any of this work:

wire cost: 4157949 bytes (~1039487 tokens) over 37 turns + 1 compaction call, 112377 bytes/turn

A million tokens for 36 tool calls against four small files. That is the real target, and no benchmark of three-turn tasks was ever going to show it to us.

Round one: the obvious things

Terser tool schemas. The whole tool block ships with every request, so a sentence in a tool description is paid once per turn, not once per session — it is the most expensive prose in the program. Most of it was restating parameter names: "path": "File path.", "new_string": "Replacement text.". Deleting a description entirely rather than serialising an empty one, and cutting the rest to what actually changes model behaviour, took the block from 5,193 to 4,556 bytes — about 160 tokens a turn. A test now fails the build if it grows back:

tool schemas are 5104 bytes (~1276 tokens per turn), over the 5000 byte budget;
shorten a description or drop one that restates a parameter name

Workspace-relative paths. ripgrep was being run with an absolute target, so every one of up to 80 match lines carried the same 60-character prefix. Eighty copies of a useless string, re-sent every later turn. Same for the wrote /abs/path/… confirmations. This one also made the terminal output legible, which is how we noticed it.

Per-line truncation. Output was capped in total bytes, so a single minified bundle or base64 blob could consume the whole budget and push out the lines that mattered. Now over-long lines are truncated first — every line survives, only the pathological ones get shortened.

Collapse repeated tool results. Agents re-run git status, re-grep the same pattern, and re-run the same test command constantly. When a result is byte-identical to one already in the transcript, we replace it with a pointer. That is not hiding information: "this has not changed since you looked" is information.

Count the fixed cost in the budget. The context budget only counted the transcript, not the system prompt and tool schemas — which are re-sent in full every turn too. A run that thought it was at 48k was really at 50k.

Round two: the change that made it worse

The last piece was compaction. When the transcript crosses the budget, the agent asks a model to summarise the older half. That call is not free, and its output is strictly less precise than the original file paths and error text.

So: trim stale tool output in place instead. Free, no model call, keeps recent turns exact. We wrote it, the unit tests passed, and the harness said:

before: 4293186 bytes after: 4751562 bytes

Ten percent worse. Two mistakes, compounding.

The first was hovering. The trim ran only when the transcript crossed the ceiling, and it stopped as soon as it was back under the ceiling. So every subsequent turn sat at the ceiling and paid full price for it — where the old summarisation path, crude as it was, cut the transcript to a fraction and bought many cheap turns afterwards. A reduction has to reach a low-water mark, not just clear the bar.

The second was that the two new optimizations cancelled each other. The repeat collapse remembered a hash of what it had sent; trimming the transcript invalidated that memory, because a pointer to output that has been trimmed away is a lie. Every trim therefore threw away the dedupe. Fixing it turned out to simplify the code: stop keeping a cache and check the transcript itself.

for _, m := range a.Session.Messages {
    if m.Role == "tool" && m.Name == tool && m.Content == output {
        return "(identical to the output of your earlier " + tool + " call ...)"
    }
}

No cache, no invalidation, and exactly correct by construction: if the bytes are still visible, point at them; if they are not, send them again.

Round three: trim early, not at the ceiling

The real fix was to stop treating reduction as an emergency. Tool output older than a working window — the last 12 messages, roughly six turns, which is what the agent is actually reasoning over — is trimmed to 400 bytes of head and tail on every turn. It is free, and it stops the transcript from ever drifting up toward the budget.

400 bytes is not arbitrary: it is exactly what the summarisation path already kept of a tool message when building its digest. A trim loses nothing that summarisation would have preserved anyway — and it keeps the real file paths and the real error text instead of a model's paraphrase of them.

Summarisation is still there, as the third rung: trim the window, then trim harder, then — only if the transcript is still over budget — pay for a summary, and compact to half the budget rather than to just under it.

Results

All four configurations on the same deterministic script, 37 turns, same fixture:

Half the cost for the same 36 tool calls, and the summarisation model call the baseline needed is gone entirely.

The window is the dial between cheap and remembers-everything-it-saw, and cost is close to linear in it:

We kept the default at 12 deliberately. Below that the agent starts re-reading files it has just seen, which costs a turn and — more importantly — is the kind of regression this harness cannot detect.

What this does not show

Being straight about the limits, because the numbers above are strong enough to be misused:

• The harness measures cost, not quality. It replays a fixed script; it cannot

tell you whether a trimmed transcript makes the model take more turns. That is the obvious way this whole approach could be wrong, and it is not measured here.

• The real benchmark says "no regression", but it is not sensitive to this.

Re-running the two fixture repos after the pass: 9/10 passed at 19,696 median tokens, against 9/10 at 20,189 before. Same run within noise — because those tasks finish in three or four turns and never touch the window at all. One failure per arm, and neither was a context problem: a 300-second gateway stall before, a model insisting the tests already passed after.

• Bytes are not tokens. We divide by four. Fine for comparing two runs of the

same content; not a billing statement.

• One script, one shape of work. Heavy web use, or a repo with a 6,000-line

AGENTS.md, would weight these differently.

Things worth stealing

If you are building an agent, in any language:

1. Build the model-free harness first. Ours changed three design decisions in

an afternoon and cost nothing to run. A benchmark that needs a real model is for outcomes, not for iteration.

1. Make it deterministic, and be suspicious until it is. The first version

searched three files, so ripgrep's file order varied and the byte total moved by 5% between runs — enough to hide a real regression and enough to invent a fake improvement. Pinning it to one file is what made the "10% worse" result trustworthy instead of arguable.

1. Reduce to a low-water mark. Anything that stops as soon as it is under the

limit leaves you sitting at the limit, paying for it every turn.

1. Prefer checking reality to caching a claim about it. The dedupe cache

needed invalidation, was wrong when it went stale, and was slower in aggregate than just looking at the transcript.

1. Put a budget test on your prompt. Tool descriptions and system prompts grow

one reasonable sentence at a time, and each one is multiplied by every turn of every run. Ours fails the build now.

Try it

curl -fsSL https://app.nz/cli.sh | sh
app login
app agent                         # interactive, in the current directory
app agent "why is this test flaky?"
APP_AGENT_WINDOW=6 app agent      # cheaper, shorter memory
APP_AGENT_SKILL_LIMIT=10 app agent

The CLI is MIT licensed and both harnesses ship with it — bench/run.sh for outcomes, go test ./internal/agent -run TestLongRunWireCost -v for cost. If your numbers disagree with ours, they are three files each; send them.