How we built isolated containers for agents and CI

app.nz runs untrusted-ish work: CI commands, build jobs, coding-agent commands, and model containers. The isolation model is intentionally layered. A Docker container is useful, but it is not the whole security story. We treat it as a per-job execution boundary, then add network scoping, credential scoping, timeouts, labels, cleanup, and provider-level machines around it.

The clearest example is the CI runner.

One run, one Docker network

When APPNZ_CI_DOCKER_RUNNER=1, a CI run creates a Docker network named from the run id:

appnz-ci-<run>

Every job container and service container for that CI run joins that network. Postgres, Redis, and Mongo are started once per run with aliases like postgres and redis, then the runner injects DATABASE_URL, REDIS_URL, or MONGO_URL into each step container.

That gives each run a private DNS island:

job container -> postgres:5432
job container -> redis:6379
other CI run -> different network, different containers

There is no global shared database service for tests to accidentally mutate.

One step, one container

Each CI step runs with:

docker run --rm
  --label appnz-ci-run=<run>
  --network appnz-ci-<run>
  -v <workdir>:/work
  -w /work
  -e CI=true
  -e APPNZ_CI=1
  <job image> /bin/sh -ce <step>

The workspace is shared across steps through the mounted workdir, but the process tree is not. A step that starts a background process does not get to quietly live forever after the container exits. Each step also gets a 15-minute timeout, while the whole run has a 45-minute context.

That is a pragmatic CI isolation line: steps can share checked-out source and build artifacts, but not ambient processes.

Cleanup is label-driven

Every container created for a run carries the same label. Teardown does not need to remember every container id perfectly:

docker ps -aq --filter label=appnz-ci-run=<run>
docker rm -f ...
docker network rm appnz-ci-<run>

This matters because failure paths are where leaks happen. If checkout fails, a service fails to become ready, a command times out, or the runner itself returns early, label cleanup still has a simple sweep.

The build runner uses the same philosophy. It unpacks a context into a temp directory, writes the supplied Dockerfile as Dockerfile.appnz, runs docker build, pushes to the private registry, and deletes the temp directory afterward. Docker auth is scoped by setting DOCKER_CONFIG to a temp directory, so a registry login for one build cannot overwrite the host's normal Docker config.

What containers do not solve

Docker containers are not a perfect sandbox for hostile code. A container can still:

• consume CPU, memory, disk, and network until the host enforces limits,
• exploit kernel or Docker daemon vulnerabilities,
• exfiltrate anything mounted into it,
• abuse network egress unless egress is filtered.

So app.nz uses containers as the unit of execution, not the only trust boundary. Higher-risk work can be handed to remote workers or provider machines. Agent nodes and builders run on Hetzner; GPU cogs run on RunPod or a local Docker path only when explicitly enabled. That means a bad job can be isolated by machine lifecycle too: create a worker, run the work, collect logs/artifacts, terminate the worker.

The isolation checklist

The useful pattern is repeatable:

This is not exotic infrastructure. It is careful accounting. Most isolation bugs are bookkeeping bugs: a token leaked into logs, a container left running, a network reused, a cleanup path skipped. The engineering work is making every unit of work have a name, a label, a timeout, and an owner.