CI service containers without shared state

CI looks easy until tests need Postgres, Redis, Mongo, environment variables, isolated networks, timeouts, logs, and cleanup. app.nz CI keeps the model simple: a run owns a Docker network, services live on that network, and every step runs in a short-lived container.

That gives users the shape they expect from hosted CI without turning the runner into a full cluster scheduler.

Services are part of the run

A pipeline can request services. The runner starts each service container once per run with a deterministic alias:

The step container does not need to know the container id. It connects to postgres:5432 or redis:6379 over the run network.

This is better than a shared test database because each run gets fresh state. Parallel CI runs cannot race through the same schema.

Step containers are disposable

Each command runs in its own container with the repo mounted at /work. The workspace persists through the mounted directory, but the process tree does not.

That is a useful compromise:

• npm install can write node_modules for later steps,
• cargo can write target for later steps,
• a background process started by one step cannot silently survive into another,
• the runner can timeout and remove the container cleanly.

The current runner uses a 15-minute timeout per step and a 45-minute timeout for the whole run. Those numbers are product defaults, not security boundaries, but they stop stuck jobs from becoming permanent capacity leaks.

Readiness matters

Starting a Postgres container is not the same as Postgres accepting connections. The runner waits for service readiness before executing steps that depend on it. Without that wait, CI becomes flaky under load because tests race service boot.

Good CI infrastructure is mostly removing races users should not have to think about.

Labels are the cleanup API

Every run-owned container gets a label. Teardown can sweep by label even if a particular code path forgot a container id. The network name also includes the run id, so removal is deterministic.

That label strategy matters for failure cases:

• checkout failed after services started,
• a step timed out,
• a service health check never passed,
• the runner process returned early,
• a test script spawned children.

The cleanup code should not need a perfect memory of how far the run got. It should ask Docker what still belongs to the run and remove it.

Why not run everything in one big container?

One container per run is simpler, but it hides process leaks and couples unrelated steps. One container per command gives the runner a hard boundary around step lifetime while still sharing the filesystem.

The extra docker run overhead is acceptable because CI is dominated by dependency install, compile, test, and service startup time. The isolation and debuggability are worth it.

The product result

Users see a pipeline with steps and services. Internally, app.nz sees named resources with owners:

run id docker network service containers step container mounted workspace logs timeout cleanup label

That accounting is the difference between "we execute shell commands" and "we operate CI."