polyserve: in-process serverless that packs hundreds of apps into one process

We are open-sourcing polyserve, the extreme-lightweight serverless hosting engine that runs app.nz. One long-lived host process per language serves many apps ("spaces") with per-second usage metering, hot-swap deploys, and scale-to-zero. It is cheap enough to run yourself on a $4 VPS.

Containers per function are wasteful

The default serverless architecture spins up a container (or microVM) per function. Every one carries a full runtime, its own memory floor, its own cold start, and its own bill. For the long tail of apps serving a few requests a minute, you pay for isolation you rarely need at a price that only makes sense at hyperscaler margins.

• Memory: hundreds of MB per idle container, times every deployed function
• Cold starts: image pull, runtime boot, framework import, on every scale-from-zero
• Money: the overhead is passed straight to you as minimum billing units

The in-process model

polyserve inverts this: one host process per language, packing hundreds of spaces into a single long-lived runtime.

• Go — a fasthttp host; your compiled binary slots in behind a unix socket (or a .so plugin fully in-process). Microsecond-level proxy overhead.
• Python — an asyncio host imports your ASGI app / handle() in-process. Perfect for tiny ONNX model serving.
• Bun/JS — Bun.serve imports your fetch handler in-process.

All hosts speak the same small control protocol, so the control plane treats them uniformly:

POST /_polyserve/deploy   {"space":"lee101/hello","artifact":"..."}   # hot swap
GET  /_polyserve/usage    per-space busy_ms / warm_s / requests       # per-second billing
GET  /_polyserve/health

Requests route by path /s/<owner>/<space>/... or by Host header when a custom domain is mapped.

Per-second billing: busy_ms + warm_s

Because everything lives in one process, metering is exact and nearly free. Each host tracks two monotonic counters per space:

• busy_ms — cumulative wall time with at least one in-flight request: billable compute
• warm_s — cumulative seconds loaded in memory: billable residency

The control plane scrapes /_polyserve/usage and diffs counters between scrapes; hosts also append usage once per minute to usage.jsonl for crash-safe billing. No per-invocation billing events, no 100ms rounding, no minimum instance charges.

Scale-to-zero that actually costs zero

A space with no requests for POLYSERVE_IDLE_TTL (default 900s) is unloaded: the Go process is stopped, the Python or Bun module released. State stays in spaces.json, so the next request cold-starts it transparently — and in-process cold starts are small: Go binary under 100ms, Python import under 500ms, Bun import under 100ms.

Hot-swap deploys

Deploys are a multipart POST. For Go, the new binary is saved to a fresh generation slot, started on a fresh unix socket, traffic is atomically switched, and the old process gets a 5 second drain before being killed. Zero dropped requests. Python and Bun load the new module in a fresh namespace and drop the old one after the switch. The Go host deep dive walks through the mechanics.

Autoscaling to Hetzner and RunPod GPUs

Above the hosts sits one tiny Go control loop. The autoscaler scrapes /_polyserve/usage from every worker, computes an aggregate busy-ratio, and scales a provider between MIN_WORKERS and MAX_WORKERS. There is a Hetzner implementation for cheap CPU workers and a RunPod implementation for GPU workers when models outgrow in-process CPU serving — see serving tiny ONNX models in-process. Workers run hosts under systemd; the scaler only needs each worker's address list. Nothing else to maintain.

Addons via injected env

Spaces get environment injected per-space at deploy time. On app.nz the control plane provisions addons (Postgres, MongoDB, ...) and passes DATABASE_URL and friends through the deploy call. Locally it is just -F env=DATABASE_URL=... on the deploy curl.

Try it

make build               # builds all hosts
make e2e                 # deploys every example against real hosts and asserts
./hosts/go/polyserve-go  # serve Go spaces on :8440

Then deploy a Go space in two commands — the deploy docs cover Go, Python, and Bun quickstarts.

polyserve is MIT licensed. Source on GitHub and mirrored at app.nz/lee101/polyserve.