gobed: GPU vector search as a service
The search engine we built for app.nz, as an addon: a 15 MB int8 embedding model, exact and IVF/HNSW indexes on CPU, cuVS CAGRA graph search on GPU, three HTTP endpoints, one injected env var. It powers character, art, audio, and repo search here.
why it is fast
Static embeddings, one index per scale
gobed embeds with static retrieval embeddings (token vectors + mean pooling, no attention pass), quantized to int8 at 512 dimensions: a 15 MB model embedding ~150K texts/sec on CPU and using roughly 600 bytes per stored vector, so a million documents fit in ~600 MB. The index is picked per corpus:
Flat
Exact brute-force with SIMD (AVX2/arm64) dot products. Default under ~1.5K vectors, zero recall loss.
IVF + HNSW routing
K-means inverted lists with an HNSW graph over centroids. ~3× faster at scale for ~10% recall trade, tunable via nprobe.
CAGRA on GPU
NVIDIA cuVS CAGRA graph index with custom fused CUDA kernels, GPU k-means, and a pooled memory manager. Auto-enabled when CUDA is present.
| Corpus | Index | Latency | Throughput / quality |
|---|---|---|---|
| 1K docs | flat (exact) | 357 µs | ~2,800 QPS |
| 100K docs | flat (exact) | 2.23 ms | NDCG@10 > 0.99 |
| 100K docs | IVF | 417 µs | ~2,400 QPS |
| 500K docs | IVF | ~1.4 ms | ~715 QPS |
| 1M docs | GPU batch | — | ~1,050 QPS sustained |
Measured on our benchmark suite (RTX 3090 for GPU rows); methodology ships in the repo. Sub-millisecond CAGRA at 100K+ docs is the current target, not a shipped measurement.
the addon
Three endpoints, one env var
| Endpoint | Purpose |
|---|---|
| POST /search | single or multi query, k, timeout_ms → results with similarity scores |
| POST /batch_search | high-throughput query batches |
| POST /index | index documents [{id, text}], sync or async |
| POST /batch_index | bulk ingestion |
| GET /health · /metrics · /gpu_stats | liveness, Prometheus metrics, GPU utilization |
Injected environment
GOBED_SEARCH_URLHTTP endpoint of your provisioned gobed instance (POST /search, /batch_search, /index)
GOBED_API_KEYBearer token scoped to this addon · secret
GOBED_INDEXDefault index/namespace name for this app
curl "$GOBED_SEARCH_URL/search" \
-H "Authorization: Bearer $GOBED_API_KEY" \
-d '{"query": "cozy sci-fi reading nook", "k": 5}'Shared CPU
metered
int8 CPU engine on shared workers, per-request billing
Dedicated CPU
40 credits/mo
pinned instance, ~600 bytes/vector, up to ~5M vectors
GPU (CAGRA)
from $0.228/hr
per-second billed GPU pod with cuVS CAGRA graph index
dogfood
It runs this site
The in-process int8 engine indexes app.nz characters, art, audio, documents, and repo code. The GPU sidecar serves /api/search-ais semantic character search, with a SQLite fallback when absent. The addon gives you that: GPU fast path, SQLite fallback, one env var.
Metered per index, no platform fee
No per-seat pricing, no minimum clusters: a metered index on shared CPU, or a per-second GPU pod for CAGRA.