1ms web search: gobed int8 embeddings, a self-improving index, and agent tools for flash-tier models

app.nz/search answers most repeat queries in about a millisecond, on one box, from an index measured in megabytes — and it gets better every time someone searches or runs deep research. This post is how it works: gobed int8 embeddings, a learned index that compresses documents to ~600 bytes, and an agentic search loop cheap enough to run on flash-tier models like DeepSeek. It doubles as a field guide to building agent tools that models actually use well.

The architecture: three tiers, one merge

A search on app.nz fans out to three sources and merges by URL:

1. Learned index (in-RAM, ~µs) — everything past searches taught us.
2. Compact crawl index (SQLite FTS + gobed rerank, ~1ms warm) — our

webfiddle crawl, stored as compressed readable chunks.

1. Live web (Serper → Brave → keyless DuckDuckGo, ~300ms) — freshness,

and the teacher for tier 1.

The trick that makes the whole thing feel instant is that tiers 1 and 2 are free and local, so we always run them, and the paid tier only fills the gaps. Every result the paid tier returns is immediately embedded and written into the learned index, so the next similar query never pays for it again.

Int8 embeddings: the compression that matters

We tried compressing the index files themselves (zstd on SQLite pages, etc.) and it's the wrong layer. The win is compressing the representation:

• A float32 512-dim embedding is 2KB per document.
• gobed's int8 quantization stores vec[i] * scale ≈ f32[i]:

512 bytes + one float scale per document. 4x smaller, and the dot product runs on int8 lanes, which is faster too.

• Title + snippet + URL round a learned doc out to roughly 600 bytes.

50,000 learned documents — our eviction cap — is ~30MB on disk and in RAM.

At that size you don't need an ANN library, a vector database, or a GPU. We benchmarked the worst case — every one of 50k docs matching the query — at 4.2ms on a 2016-era Xeon, with typical learned-index sizes coming in well under a millisecond. Getting there was three concrete fixes worth stealing: the first naive scan was 117ms. Precomputing lowercased match text per doc (no per-query ToLower/concat allocations) and sharding the scan across cores took it to 14.7ms; quantizing the query to int8 too — so the hot loop is int8×int8 with int32 accumulate and zero float conversions — plus per-shard top-k instead of a full sort took it to 4.2ms. 28x from profiling one loop. gobed does have GPU CAGRA indexing for the millions-of-vectors regime, but at this scale RAM beats shipping vectors to a GPU and back. Quantize first, distribute never, GPU only when brute force actually loses.

The embedder itself is the reason this is cheap: gobed's SimpleInt8Model512 is a pure-Go int8 embedding model that loads in milliseconds and embeds in microseconds — no Python sidecar, no ONNX runtime, no cold start. We memoize embeddings (chunk snippets repeat across queries), so a repeat query is: one cached query embed, one map scan, one sort.

The index that learns

The learned tier is deliberately boring:

CREATE TABLE learned_docs (
  url TEXT PRIMARY KEY, title TEXT, snippet TEXT,
  vec BLOB,          -- 512 x int8
  scale REAL,        -- dequantization factor
  hits INTEGER, updated_at INTEGER
);

SQLite is persistence only; the working set lives in a Go map. Every live-web result from a search, an agent run, or a deep-research job is upserted with its quantized embedding. Popular documents accumulate hits; when the index outgrows its cap we evict the least-hit, oldest 10%. Scoring is dequantized cosine plus a token-overlap term — the overlap term doubles as the fallback when the embedding model isn't loaded, so search never hard-depends on it.

This is the "every search makes search better" loop, and it costs nothing: the results were already paid for. You're just refusing to throw them away.

Agentic search on flash-tier models

Deep research on app.nz runs on cheap fast models (DeepSeek-class, flash-tier). That only works because the tools are shaped for small models. The billed agent endpoint returns its own trace:

POST /api/search {"query": "...", "depth": 2}
{
  "engine": "webfiddle+gobed+learned+serper",
  "results": [...],
  "steps": [
    {"phase": "retrieve", "source": "index", "count": 8},
    {"phase": "search",   "source": "web",   "count": 10},
    {"phase": "rank",     "source": "gobed", "count": 15}
  ],
  "charged": 2, "balanceAfter": 981,
  "resultsUrl": "/api/search/results?q=..."
}

A small model doesn't have spare capacity to infer what happened — so the tool tells it: what ran, what it cost, and a plain GET URL to re-fetch the same results without paying again.

How to build agent tools (what we've learned)

• Never fail decoratively. Our search returns an empty result set with

engine: "unavailable" instead of a 500 or an "API key missing" error. Agents retry errors in loops; they handle empty lists fine.

• Split the charge from the read. Billed POST to act, free idempotent GET

(resultsUrl) to re-read. Agents re-request constantly; make the repeat path free and cached.

• Label provenance. Every result carries source: "index" | "web" and

the response names its engine chain. Models cite better and hallucinate less when the tool states where facts came from.

• Return steps, not vibes. The steps array is the model's reasoning

scaffold — a flash-tier model can narrate a research run by reading it.

• Degrade in tiers, price in depth. depth: 1–5 scales both the fan-out

and the charge linearly. Small models handle one dial well; they handle five booleans badly.

• Feed the loop. If your tool fetches anything remote, index what it

fetched. Agent traffic is a free crawl of exactly the content your users care about.

Share it

Any search is a URL — /search?q=… with mode variants /search/web, /search/index, /search/agent — and every page ships the same share control (native share sheet, clipboard fallback). A shared search link replays against the learned index, so the person you send it to usually gets the ~1ms answer you didn't.

Try it: app.nz/search · agent API at POST /api/search · embeddings: github.com/lee101/gobed.