Throughput on a shared GPU: fused kernels and a priority scheduler
How we serve chat, image, video, TTS, STT and a LoRA army on shared GPUs — 2× LLM tok/s (fp8+MTP), a fused multi-LoRA kernel (up to 2.4×), a 10.4× admission gate, tier-based VRAM arbitration, and cheaper building blocks (916k embeds/sec, Gemini STT).
We run every model family — chat LLMs, image diffusion, video, text-to-speech, speech-to-text, and a long tail of LoRAs — on shared GPUs. When many workloads compete for one card, throughput is won in two places: the hot kernels each model runs, and the scheduler that decides who gets the GPU. Here's what moved the numbers.
Serving the chat model 2× faster, losslessly
Our roleplay LLM (a QLoRA fine-tune of Gemma 4 E4B) serves at 264 tok/s single-stream — 2× the bf16 baseline — using fp8 weights, an fp8 KV cache, and MTP-2 speculative decoding. Speculative decoding is lossless: a small drafter proposes tokens the full model verifies, so accepted tokens are free speed with byte-identical output. Under concurrency, deeper speculation (MTP-3) reaches ~660 tok/s in our sweeps — the next unlock is a drafter adapted to the tuned model so its acceptance rate stays high on our own distribution.
A fused multi-LoRA merge kernel
Serving an army of LoRA adapters, the bottleneck isn't sampling — it's merging adapters into the weights. The stock path runs one B@A GEMM per adapter plus an add; for N adapters on a module that's N kernel launches. We collapse them into one batched GEMM by concatenating the low-rank factors and folding each adapter's scale into one side:
| shape (dim × rank × adapters) | stock | fused | speedup |
|---|---|---|---|
| 1536 × 32 × 4 | 0.99 ms | 0.62 ms | 1.6× |
| 3072 × 32 × 4 | 1.47 ms | 0.63 ms | 2.3× |
| 3072 × 64 × 4 | 2.64 ms | 1.11 ms | 2.4× |
| 3072 × 64 × 8 | 4.97 ms | 2.57 ms | 1.9× |
The default path is bit-for-bit identical to the per-adapter loop; a bf16 compute mode trades ~4e-3 error for up to 3.8× where tolerance allows.
A scheduler that pays off under load
The GPU only serves what flows through the front door, so the front door's admission gate has to be fast and fair. Rewriting it from a broadcast-wakeup design to per-waiter events took admission throughput from 588 to 6,121 requests/sec (10.4×) and cut p99 wait from 58 ms to 5.7 ms — no more thundering herd when a slot frees.
On top sits a priority economy. Every request carries a tier, and VRAM is arbitrated by tier, not by model:
- paid API traffic (any modality) is highest and can push a lower-tier model
out of VRAM;
- subscribers come next;
- free traffic waits behind them;
- background/batch jobs run only on an otherwise-idle GPU and always yield.
Because the tier rides on the request, a paid image API call and a batch image-generation script hit the same server at different priorities — the paid call wins the card, the batch job steps aside. A recently-served higher tier is protected, so a free request can't evict a paid one; it gets a clean retry.
Cheaper building blocks
Two supporting pieces punch above their weight:
- cbed — a C port of our static int8 embedding search does **916,000
embeds/sec on one core** (~0.001 ms each) with exact parity to the reference, used for semantic LoRA selection and retrieval.
- Speech-to-text via Gemini audio — a ~1.5 s, near-perfect remote
transcription replaced two local GPU models (a Parakeet ASR and a local multimodal transcriber), freeing VRAM and simplifying the box.
The theme
Every win here is either a fused kernel (fewer launches, same math) or a smarter scheduler (the right request on the GPU at the right time). Neither requires a bigger card — just being deliberate about where the cycles go. More is coming: a Triton kernel that fuses the LoRA concat-and-scale step, an adapted speculative drafter, and continuous batching for diffusion.