6h. Tenant builds in CI: one leg per hash, as a user of its own

Since 2026-09-29 this is the fallback's path, not the everyday one. Customer deploys, staging's test fleet and previews are deployed by the deployer Container from the builder's stored bytes (§6i): staging.yml and preview.yml enqueue and wait, and build nothing for it. What follows is what a runner pass does when the deploys queue's consumer is set back to dispatcher, or when deploys.yml is dispatched by hand — and what staging.yml's build, collect, rehearse and land jobs still run.

A tenant's build is that tenant's code, so no job that holds a secret builds one (deploys.yml, and staging.yml's fallback jobs). A pass plans with the platform's token, builds each planned hash on a matrix leg of its own holding nothing, collects the legs in a job that runs none of them, and deploys from a job that builds nothing.

On a leg, the build runs as a throwaway Unix user — divvent prebuild --isolate, which needs Linux and sudo without a password: a GitHub-hosted Ubuntu runner, never a contributor's machine (prebuild without --isolate builds as you, as divvent deploy does). The user builds under /opt/dvb, root's and mode 711, outside the runner's HOME: node is copied there, the checkout is mounted there read-only, and pnpm is reached through it. The user is locked and kept out of cron and at, and everything it runs is killed and counted twice before its output is read. Before the build starts it is probed as itself, and the leg fails — not the tenant — if it can sudo, is in docker, sudo or adm, can write the docker socket, open another user's /proc/<pid>/environ, read the runner's credential files or write the step's command files, pnpm, the linked packages, the toolchain or any PATH directory. The probe only tests whether; it never reads any of them.

Before the first build user exists, the runner is prepared (prepareRunner in packages/cli/src/isolate.ts), as root, and the leg's log says what it did in two runner lines:

• Every directory the probe asks about is closed to all but its owner — extended ACLs removed where setfacl is installed, then go-w — and a toolchain is closed whole: /usr/local, the tool cache, pipx's, each …/bin's prefix. The list is the probe's own, so they cannot disagree. Nothing is chowned; what only the others' bit let the runner write (a root-owned 777 tool cache) is root's alone afterwards. That is safe because a hosted VM is discarded when its job ends, and no step after a build writes the toolchain. /tmp and the other shared temp directories are never touched — one on a PATH fails the probe instead.
• Build users cannot reach the metadata service: build users are made with uids 400000–400999 only, and an iptables/ip6tables chain, divvent-build, refuses that range 169.254.0.0/16, 168.63.129.16 (bar its DNS), fe80::/10 and fd00:ec2::254. No iptables means no build: the leg fails before any user is made.

The first dispatched diagnosis (run 36273250829) is why: the hosted image left /usr/local/bin, the tool cache's node, /opt/pipx_bin and ghcup's bin writable by every user, and 169.254.169.254:80 answered a build user.

Checking it on a hosted runner. Dispatch deploys.yml with diagnose ticked. It plans and deploys nothing; its one job makes a build user, prints what that user can and cannot do — uid and groups, sudo, the docker socket, the Runner.Worker's environment, the cloud metadata endpoints, the kernel's ptrace and protected settings, cron and at, processes left after the reclaim, the user gone after — and is red if any of it is unsafe. It prepares the runner exactly as a leg does first, so a green diagnosis is the preparation checked on a hosted runner. Run it after a runner image change, or when a leg fails with "build user … could …".

Reading a failed pass.

• A leg is red, the rest green. That tenant's build failed, hung past its twenty minutes, or left a link in its output; collect records it as that hash's failure and the deploy fails that request with the reason. Every other request deployed. Nothing to do but read the reason.
• Every leg is red with "build user … could …" or "can sudo". The runner image changed underneath the isolation. Nothing was built for anyone. Do not remove --isolate to get past it: dispatch the diagnosis above, and tell Ant with both logs.
• *collect is red with "is not the plan the plan job wrote".* The plan artifact changed between jobs. Nothing was deployed; re-run the workflow.

A request raised after a pass read the queue is not claimed by that pass: the manifest names the requests the plan read, and the deploy claims those only. It lands on the next pass.

A land that changes production's engine is rehearsed between two passes (DESIGN.md §17). The first landing pass (tenants in staging.yml, run in deploys.yml) deploys the rehearsal from the same build and stages its bundle; a rehearse leg per bundle runs its generated suites as a user of its own, holding nothing (divvent rehearse --only <request> --isolate); land is the second pass, with the first's secrets, and lands what passed. *A rehearse leg red* is a leg that could not run — its request waits, and an hour after it was staged the next pass rehearses it again (the hourly pass asks for it then, if nothing else has), and blocks it, saying why, only after the third unanswered (REHEARSAL_ATTEMPTS). The deployer stores the bundle and the verdict between its steps (§6i, decision 99), so a delivery that ends between them is resumed by the next; the hour is its floor, for a bundle nothing stored; a failed suite is not a red leg but a verdict, and the request is blocked with the tests' names.

**The canary's verification is three jobs in staging.yml**: land reads the deployed schema and stages a signed bundle (fleet verify --read-schema), the canary job runs the scenarios holding nothing (--scenarios --isolate), and canary-record records the verdict with the control plane's D1 alone (--record). *canary-record red* is a failure recorded — the scenarios failed, never reported, or their verdict was not the staging's — and the rollout is frozen until a later pass or resumeRollout. deploys.yml does not verify; a canary that lands there is verified by the next staging run.

After the canary, the rings (docs/proposals/64-flags-rings.md; DESIGN.md §6, "Rollout rings"). The pass moves through internal (our own companies), early (a tenth of customers' companies, at most fifty) and everyone, each opened when the one before has landed healthy on the new engine and baked — an hour for ours, a day for the early share (half an hour for an urgent update). The Fleet screen's Rings table says which ring is open and what holds it ("waiting for a healthy reading of … taken after it landed", "baking for …") and Where each target sits says which ring every production target is in. A ring that will not complete is almost always a landed target the health pass cannot reach: look at its reading before anything else. The rollout frozen by a ring member reads "… answered unhealthy after its upgrade to engine … landed": look at that application, then resumeRollout with what was wrong — the resume lets every failure before it past, and the ring goes on once its members answer healthy. A customer asking why they cannot take an update yet is in a ring that has not opened; their fourteen days start when it does.

Platform flags are set on the same screen, with why (setPlatformFlag, audited platform.flag.set): on for everybody, or for the rings ticked. A flag for internal is on for our own companies only; early follows the same companies the rollout's early ring holds.