Serving tiny ONNX models serverlessly, in-process

Most inference infrastructure is sized for big models: a dedicated model server (Triton, TorchServe, a vLLM box) with its own container, its own GPU reservation, its own network hop. For a 5MB sentence classifier or a small embedding model, that stack is absurd. polyserve's Python host takes the opposite approach: import the model in-process and let the platform's normal serverless machinery — per-second billing, scale-to-zero, hot swap — do the rest. Background on the platform is in the announcement.

The Python host

polyserve runs one asyncio host process (default port 8441) that imports each space's ASGI app or handle() function in-process. There is no per-space interpreter, no per-space container. Your space is a Python module; onnxruntime is just a library you import, and the loaded session lives in the host's memory alongside hundreds of other spaces.

Why in-process beats a model server for small models

• No extra hop. A model server means serializing the request, crossing the network, and deserializing on the other side — often more time than the inference itself for tiny models. In-process, the request is dispatched to your handler inside the same runtime.
• Warmth is nearly free, and metered honestly. A small ONNX session held in memory costs megabytes. polyserve bills residency as warm_s and compute as busy_ms — for a tiny model, both round to almost nothing.
• Scale-to-zero is safe. If nobody calls the model for POLYSERVE_IDLE_TTL (default 900s), the module is released. The cold-start budget for a Python import is under 500ms — a re-import plus ONNX session load, not a container pull.
• Deploys are hot swaps. A new model version is loaded in a fresh module namespace and the old namespace is dropped after the switch, same as every other space.

The python-onnx-tiny example

The repo ships examples/python-onnx-tiny: a tiny ONNX model served in-process. Deploys go through the same protocol as every host — module is a source file or tar:

# build and start the hosts (Python host defaults to :8441)
make build

# deploy the example (module = source file or tar)
tar -cf onnx-tiny.tar -C examples/python-onnx-tiny .
curl -X POST localhost:8441/_polyserve/deploy \
  -F space=lee101/onnx-tiny \
  -F [email protected]

# call it
curl localhost:8441/s/lee101/onnx-tiny/

Usage shows up in the same billing counters as every other space:

curl localhost:8441/_polyserve/usage
{"spaces":{"lee101/onnx-tiny":{"requests":42,"busy_ms":310,"warm_s":120,"loaded":true}}}

When the model is not tiny: RunPod GPU workers

In-process CPU serving has a ceiling. That is what the autoscaler is for: it scrapes /_polyserve/usage across workers, computes aggregate busy-ratio, and scales providers between MIN_WORKERS and MAX_WORKERS — Hetzner for cheap CPU workers, RunPod for GPU workers when bigger models need them. Small models stay packed on CPU hosts; heavy inference gets GPU capacity that also scales back down when idle.

The point

Model serving for small models should look like the rest of your serverless stack, not like a second ops discipline. One host process, one deploy curl, honest per-second metering, and GPUs only when you need them. The Go host deep dive covers the compiled-binary side, and the deploy docs have quickstarts for all three languages. Source: github.com/lee101/polyserve, mirrored at app.nz/lee101/polyserve.