9. The first production release — BY HAND, once, in this order

9.0 Everything done by hand on staging, and whether production has it

Kept current as each is done, so production is set up from one list rather than from memory of what staging needed. Each row links to its steps.

production.yml runs on a v* tag: the disaster-recovery drill against staging, then — behind the production environment's approval — the promote of staging's artifact by hash into "Divvent — production", the perimeter, and a smoke from outside. It has never run. acceptance/test/first-release.test.ts runs its stage lists and its environment against an empty fake account, and workflow-stages.test.ts walks it step by step; what follows is everything a test cannot do, because each is a person in somebody else's console.

Nothing below is optional except where it says so. A step skipped is a release that stops at it — most of them loudly, and the ones that do not are marked.

9.1 The Cloudflare account — "Divvent — production"

1. The account exists (bootstrap step 1) and is on the plan staging is on. Containers need Workers Paid.
2. R2 enabled in the dashboard — terms and a billing method. No token permission substitutes, and the deploy stops at the bucket without it.
3. **divvent.co.uk is a zone in this account** (bootstrap step 4: move it once the registrar lock lifts). The dns stage attaches divvent.co.uk, www.divvent.co.uk, app.divvent.co.uk and console.divvent.co.uk to their Workers and creates no zone, so the release stops at dns without it.
4. **Zero Trust enabled with the team name divvent**, so its auth domain is divvent.cloudflareaccess.com — the manifest's team.prod, which the Google client below is registered against. Pick the Free plan if asked.
5. An Access service token, in Zero Trust → Access → Service Auth, named divvent-scheduler. Its id and secret go in the GitHub environment below as DIVVENT_ACCESS_CLIENT_ID and DIVVENT_ACCESS_CLIENT_SECRET — the values only, not the CF-Access-Client-Id: prefix the dashboard shows beside them. Needed before the first tag, because the manifest declares the pair for prod and the workers stage refuses without it. (It is created here rather than by the deploy because its secret is shown once, and the deploy has nowhere to put it but a log.) Do not add a policy for it: the access stage puts the Service Auth policy on the console's application itself, named scheduler service token and naming this token only (STAGE2.md §2.4), and it refuses — naming the two variables — if the token is not in the account. It then renders the console through the perimeter with the token, and waits up to a minute, asking every five seconds, for Access to take a policy it has just written; "turned away" after that is a real failure. Only that check waits: the one asking with no credential is answered once.
6. One account-scoped deploy token, with exactly the rows .dev.vars.example lists for the control plane — the tenant four (Workers Scripts, D1, Workers R2 Storage, Account API Tokens), then Queues, Workers Containers, Cloudchamber, Access: Apps and Policies, Access: Organizations, Identity Providers, and Groups and Zero Trust, all Edit, and Access: Service Tokens at Read, and Turnstile: Edit (the sign-up widget) — **plus three zone rows on divvent.co.uk**, Workers Routes: Edit, Zone: Read and Zone WAF: Edit (the rate-limiting rules), which the editor puts in a separate section. Copy staging's token's rows rather than reading this list, if the two ever disagree: the file is what was read off staging's token, and staging deploys.

   Check it from a terminal holding it, before the tag:

   divvent control-plane token --env prod

9.1a The scheduler's door, on staging — BY HAND, once

The scheduler had never completed a pass: Access refused its token and the console refused its assertion (STAGE2.md §2.4). The code for both halves is in; three things on the account and the repository are Ant's:

1. **Add Access: Service Tokens → Read to staging's deploy token** (DIVVENT_DEPLOY_API_TOKEN in the staging environment). Without it the access stage refuses in its preflight, names that row and changes nothing. divvent control-plane token --env staging shows it before a push.
2. **Check that DIVVENT_ACCESS_CLIENT_ID / _SECRET in the staging environment are a live service token in the staging account** (Zero Trust → Access → Service credentials). If they are unset, or the token was deleted or made elsewhere, create one called divvent-scheduler and set both — values only. The access stage refuses, naming both variables, when the id matches no token in the account.
3. Delete any Service Auth policy added to "Divvent console" by hand, which .dev.vars.example used to ask for. The deploy owns the one called scheduler service token at precedence 2; a second one would be a second door nobody reviews, and one at the same precedence may collide with it.
4. Optional: delete the DIVVENT_ACCESS_SERVICE_TOKEN secret in the production environment. health.yml no longer reads it — the fallback presents the scheduler's pair, as the Worker does.
5. Optional: delete the DIVVENT_CONSOLE_URL variable in the production environment. scripts/fleet-health.mjs composes the console from infra/control-plane.json now, because that is where the pair is sent, and refuses to run if the variable names any other console.

Nothing else. DIVVENT_SYSTEM_CLIENT_ID on the console is set by the deploy from the runner's DIVVENT_ACCESS_CLIENT_ID — it is a var the deploy writes, not one to set. The first push after (1)–(3) carries the client id to the console in its deploy step and reconciles the policy in its last step; from the next tick the console's fault list stops saying Error on cache-warm and divvent doctor's cache-warm section stops saying never pinged.

Production is the same three at its first release: 9.1 step 5 and step 6 above.

9.2 Google — the OAuth client Access signs operators in with

In the Google Cloud console of the divvent.co.uk Workspace, by a Workspace admin:

1. An OAuth client, type Web application, with one authorised redirect URI: https://divvent.cloudflareaccess.com/cdn-cgi/access/callback.
2. The consent screen Internal, and the Admin SDK API enabled on the same project — without it Cloudflare cannot read Workspace groups, the operators rule matches nobody, and the perimeter looks configured and admits no one.
3. The client id and secret go in the GitHub environment as PRODUCTION_GOOGLE_CLIENT_ID and PRODUCTION_GOOGLE_CLIENT_SECRET. Optional for the first tag: without them the access stage skips and says so, and the console has no door until a later release runs it.

9.3 GitHub — the production environment

Settings → Environments → **production**:

1. A required reviewer — Ant. That is the approval; without it the environment: production line gates nothing.
2. **Deployment branches and tags: tags matching v*.**
3. Secrets, spelled exactly as .dev.vars spells them, with production's own values — never staging's:

   Secret	What
   DIVVENT_DEPLOY_API_TOKEN	the token from 9.1 step 6
   DIVVENT_DEPLOY_ACCOUNT_ID	"Divvent — production"'s account id
   BETTER_AUTH_SECRET	the portal's session key
   DIVVENT_OIDC_SIGNING_KEY	the portal's issuer key
   CREDENTIAL_SEALING_KEY	32 bytes, base64
   DEPLOYER_OPENING_KEY	the same value as the sealing key
   RESEND_API_KEY	the platform's Resend key, with divvent.co.uk verified (bootstrap step 6)
   DIVVENT_ACCESS_CLIENT_ID, DIVVENT_ACCESS_CLIENT_SECRET	9.1 step 5
   PRODUCTION_GOOGLE_CLIENT_ID, PRODUCTION_GOOGLE_CLIENT_SECRET	9.2
   STAGING_REGISTRY_ACCOUNT_ID, STAGING_REGISTRY_API_TOKEN	staging's account id, and a token in staging's account with Workers Containers (or Cloudchamber) and nothing else — not staging's deploy token. Be clear what it can do: that permission mints registry credentials, push as well as pull, and manages staging's containers. So it is on one step of the release alone, which mints a fifteen-minute pull-only credential and hands that to the copy (§9.6); the step that copies never holds the token

   Variables: DIVVENT_DEPLOY_TARGET = sub-account, DIVVENT_OPERATOR = whoever is releasing. Leave DIVVENT_OIDC_KEY_ID unset.

   Replacing one of these replaces it on the Workers at the next deploy. control-plane deploy writes every secret it is given to every Worker that holds it, on every run, so the environment is the record of what the control plane runs with — a value changed here, or leaked and changed here, is gone from Cloudflare when the next run finishes. (Before 2026-09-26 it put a secret only where a Worker had none, and a replaced value never arrived.) Two of them are not rotations to make casually: CREDENTIAL_SEALING_KEY and DEPLOYER_OPENING_KEY are one key under two names, replaced together or not at all, and every credential sealed under the old one is unreadable after. A pair that is two keys is refused wherever both are present, before anything is sealed or opened: control-plane deploy refuses before it uploads, and every command that reaches a control plane (deploys, deploy --via, tenant register) before it asks it anything — by sealing a random probe with one and opening it with the other, in memory, and saying neither value. divvent secrets check asks the same question on its own, from this shell's environment or .dev.vars, after setting or rotating them; BETTER_AUTH_SECRET signs every portal session out and makes each portal member's authenticator enrolment undecryptable. Run control-plane deploys from the environment's workflow: one run from a terminal whose .dev.vars differs puts those values on the Workers, until the workflow's next run puts the environment's back.

   Make them in a checkout of their own, so staging's .dev.vars is not overwritten. divvent secrets init asks for the account id, the token, the operator and the Resend key, and generates the auth secret, the signing key and the sealing pair in the shapes the code parses. divvent secrets export --github-env production --script production-secrets.ps1 then writes a PowerShell script that reads that .dev.vars when it runs and hands each value to gh secret set on stdin — no value in a command line, a log or the script. Delete that .dev.vars afterwards; the environment is where production's values live. The script also sets names this release does not read (the account registry, the namespace), which is harmless, and names the Google pair DIVVENT_GOOGLE_* — set PRODUCTION_GOOGLE_* by hand, the only two production spells differently.

4. A tag ruleset, so only Ant can make a release tag. Settings → Rules → Rulesets → New ruleset → New tag ruleset: name releases, enforcement Active, target Include by pattern v*, and tick Restrict creations, Restrict updates and Restrict deletions. Bypass list: Repository admin (Ant) and nobody else — not a team, not an App. The production environment admits v* tags, so without this anybody who can push can make the thing it admits; the approval is still Ant's, but it is an approval of whatever a tag was pushed on. production.yml refuses a tag whose commit is not on main before any secret is in reach, and promotes only a push run of staging.yml on main — this ruleset is the half a workflow cannot do, which is who may push the tag at all. By hand, because a ruleset is repository settings and no workflow here holds a token that can write them.

9.3a The sealing key: an offline copy in two shares, and rotating it — BY HAND

CREDENTIAL_SEALING_KEY and DEPLOYER_OPENING_KEY are one AES key (§9.3's table). It opens every tenant's Cloudflare token and every target's BETTER_AUTH_SECRET, which encrypts its users' authenticator secrets. GitHub environment secrets cannot be read back, so until this section is done the key has no copy anywhere: lose it and every tenant is registered again by hand, and every end user of every application enrols their authenticator again (the full analysis, S4). Do this per environment, staging and production each with its own key.

The custodians are Ant's to name. Two people, not the same person, and not both reachable by one compromise — for example Ant and a named second operator under NDA (the analysis's A5 asks for that person anyway), or Ant and a solicitor's safe. Neither can use their share alone: each is 32 random bytes. Until two are named, this section is not done, and PROGRESS.md says so.

Making the offline copy — on a trusted machine, never a runner:

1. Put the environment's key in this shell as DEPLOYER_OPENING_KEY (from where it was generated, or joined from an earlier pair of shares). Never as an argument.
2. divvent secrets split --out-a share-a.txt --out-b share-b.txt. It prints the key's fingerprint — sixteen hex characters that name the key and are no part of it — and writes each share with the same fingerprint on it.
3. Give share-a.txt to the first custodian and share-b.txt to the second, each offline: printed and sealed, or on a hardware-encrypted drive. Never both in one place, one email, one password manager or one cloud drive.
4. Delete both files from the machine, and write the fingerprint, the date and the two custodians' names in the operations log.
5. Prove it before you need it: with both shares, divvent secrets join --a share-a.txt --b share-b.txt --out joined.key must print the same fingerprint. Delete joined.key. Once a year, and after any change of custodian.

The recovery key the off-site backups are sealed to (§3d) is held the same way, by the same custodians: put its contents in this shell as DIVVENT_RECOVERY_PRIVATE_KEY and run divvent secrets split --key-env DIVVENT_RECOVERY_PRIVATE_KEY --out-a … --out-b …. Its public half is a variable and needs no custody.

Rotating the key — after a leak (docs/runbooks/compromised-sealing-key.md), after a custodian leaves, or yearly:

1. Generate the new key: node -e "console.log(require('crypto').randomBytes(32).toString('base64'))" in a shell whose history is off, or divvent secrets init into a scratch directory.
2. In this shell (or .dev.vars): DIVVENT_RESEAL_FROM_KEY = the old key, DIVVENT_RESEAL_TO_KEY = the new one, plus the platform token and account id, and both CREDENTIAL_SEALING_KEY and DEPLOYER_OPENING_KEY = the old key (the CLI holds the pair to each other).
3. Pause what opens: no deploys.yml dispatch, and the schedule will fail its opens harmlessly until step 5 (a deploy that cannot open retries on the next pass).
4. divvent secrets reseal --env <env> — every sealed value moved to the new key, audited per column (secrets.reseal). It prints fingerprints and counts, never a value. A value somebody re-sealed while it ran is left and counted.
5. At once, set CREDENTIAL_SEALING_KEY and DEPLOYER_OPENING_KEY to the new key everywhere they are held: the environment's GitHub secrets and the control plane's Workers (a staging.yml run, or production.yml for production, puts them), and your own shell.
6. divvent secrets reseal --env <env> again — it catches anything the control plane sealed under the old key between steps 4 and 5 — then divvent secrets reseal --env <env> --check, which fails while anything is left under the old key and says "nothing is under the old key" when done.
7. Split the new key for the custodians (above), have them destroy the old shares, and record the new fingerprint. Destroy every copy of the old key.

Nothing downstream changes: a tenant's token is the same token and a target's auth secret the same secret, so no customer re-registers and no end user re-enrols.

9.4 GitHub — the staging environment

The release's first job, the recovery drill, runs in **staging** with staging's own DIVVENT_DEPLOY_API_TOKEN, DIVVENT_DEPLOY_ACCOUNT_ID and DEPLOYER_OPENING_KEY. **If staging restricts which refs may deploy, add tags matching v*** beside main — otherwise the drill is refused and the release never reaches the approval, which is right, and confusing the first time.

The domains pass (deploys.yml, "Reconcile customer domains") reads one more from staging, set once (§9.0): the variable DIVVENT_REGISTRANT_CONTACT. Each company's registrar credential is its own, sealed in the control plane (§1.3a); without one, a company whose customers verify a domain, and whose own domain is not already a zone in its account, waits — said by name, and failing that run.

The drill is also runnable by hand from a terminal holding staging's values:

divvent control-plane recover --env staging

9.4a The audit digest — BY HAND, once per environment

The console seals the audit log into its hash chain every hour and writes the chain's head once a day to divvent-<environment>-audit-digests, which the deploy creates (review 2026-09-28, S7; SECURITY §12, "The audit log"). Two things only a person does:

1. Lock the bucket. In the dashboard, R2 → the bucket → Settings → Bucket lock rules: one rule, the whole bucket (no prefix), retention indefinite (or the longest the company is willing to hold). An object under a lock rule cannot be overwritten or deleted, which is the whole of what the digest is for — without it, somebody holding the account can rewrite a day's digest to agree with a rewritten log. The deploy does not set it: a lock cannot be taken back, and that is a decision, not a step.
2. Give the console a signing key. An Ed25519 key, PKCS#8, standard base64, as AUDIT_DIGEST_SIGNING_KEY in the environment's GitHub secrets — the console's alone (optionalSecrets):

   node -e "const k=require('crypto').generateKeyPairSync('ed25519');console.log(k.privateKey.export({type:'pkcs8',format:'der'}).toString('base64'));console.error(k.publicKey.export({type:'spki',format:'pem'}))"

   The first line is the secret; keep the public half (the second) with the release records, since a digest is verified with it and names it (keyId, ed25519: and sixteen hex of a SHA-256 of its x). Unset, the digest is written unsigned and every hourly pass says so in the scheduler's log.

Check it: the console's auditIntegrity answers ok: true with nothing unsealed after the next hour, and the bucket holds audit-digest/<environment>/<day>.json after the next day.

**The build image, after fix-auth-audit.** The base image is now pinned by digest (packages/build/container/Dockerfile), and that branch changes the compiler, the runtime and the model the image carries — so pnpm preflight fails until an image.yml run from the merged commit and the three pins (containers.BUILD, containers.DEPLOYER, containers.SUITES) are moved to it, as in §6i step 2. Confirm the digest against Docker Hub first: docker buildx imagetools inspect node:24.11.1-slim must print sha256:48abc13a19400ca3985071e287bd405a1d99306770eb81d61202fb6b65cf0b57 (it was read from two mirrors; Docker Hub rate-limited the check).

9.5 Anthropic — optional for the first tag

A service account divvent-production (Developer, Default workspace), a federation issuer https://app.divvent.co.uk, and a rule for subject divvent:prod:portal. Then the four ids — ANTHROPIC_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, ANTHROPIC_WORKSPACE_ID — go under vars.prod in infra/control-plane.json, as staging's are, in a commit. Until then production's portal serves and its chat says no Anthropic credential is configured, naming these four.

9.6 The one thing that is a decision, not a step

The build image. containers[].image pins divvent-build@sha256:f95b…, a digest pushed to staging's registry by image.yml (staging only). Cloudflare's registry is per account, the builder's upload names the image in production's namespace, and a rebuild does not reproduce a digest. So on the first release the builder Worker names an image production's account does not hold — what Cloudflare does with that upload has not been observed, and the likely answer is a refusal at workers. Either production copies staging's image by digest (pull and push unchanged, which keeps the digest; one job holding both accounts' tokens), or the manifest pins per environment. Decided by Ant, 2026-09-24: production copies staging's image by digest, in one release job behind the production environment's approval (DESIGN.md §17, decision 1). Built (2026-09-25): production.yml's image job, between the drill and the deploy, runs

divvent control-plane copy-image --to prod --tag "$TAG"

which copies containers[].image's own digest from staging's registry into production's with skopeo copy --all --preserve-digests, reads production's manifest back and hashes it (stdout alone, so a warning on stderr is not hashed with it), and fails before anything is deployed if the digest moved. The step before it, divvent control-plane registry-pull, is the only one holding staging's token: it mints a pull-only credential, masks it, and hands it on through $GITHUB_OUTPUT. Every password is masked before skopeo runs. The manifest's one pin is then right for both accounts. The job needs the two STAGING_REGISTRY_* secrets in §9.3, and it is a second approval of the same release — approve image, then production.

9.7 The release

1. Staging is green on the commit: staging.yml succeeded for exactly that SHA, including its last step, because that step uploads the hash a release promotes. A docs-only push has no staging run and cannot be tagged.
2. git tag v0.1.0 <sha> and git push origin v0.1.0.
3. The recovery job runs against staging, unattended. Red here is a finding about the dump, not a flake: it names the table and never a value.
4. Approve the image job in Actions — it copies staging's build image into production's registry by digest (§9.6) — and then the production job, which waits on it.
5. It checks first that the tagged commit is on main — both jobs, before any step holds a secret — then builds, finds staging's push run on main for the SHA, refuses before deploying unless this build hashes to what staging deployed, then deploys every stage staging does, then the perimeter, then smokes app.divvent.co.uk and divvent.co.uk. Each secret is on the step that reads it, never the job, so the install and the build never hold one.

9.8 After the first release

1. Complete the Google Workspace provider's test in Zero Trust → Settings → Authentication. The access stage registers the provider and says it cannot finish it.
2. Add a Service Auth policy naming divvent-scheduler to the "Divvent console" Access application the release created. The deploy reconciles its own policies by name and leaves this one alone. Until it exists the scheduler's fleet and cache passes are refused at the perimeter.
3. Read what is there, from a terminal holding production's values:

   divvent doctor --env prod

1. Tag again (v0.1.1 on the same commit is enough). The criterion is that a second release is a no-op, and this is the only place it can be seen.

Not built, and stated so nobody expects it: a runner for production's deploy requests. deploys.yml claims staging's queue only, so production has no dispatcher configured (vars.prod deliberately leaves its three names out) and a customer's change approved in production's portal waits in the queue. That is the next piece after this list, and it needs its own GitHub environment and its own decision about which credential claims production's requests.