For Divvent’s own team. It is generated from the operations document in our repository, which is the one to edit. If you use Divvent, the help centre is for you.
3d. Off the account: the nightly copy, and the control plane's export
A dump in the tenant's own bucket does not survive the account: a suspended, compromised or deleted Cloudflare account takes it too. So each night's dump is also sealed and put outside Cloudflare, and so, weekly, is the control plane's own database (recovery.yml, control-plane export).
Sealed so the writer cannot read it. Each copy is sealed to the recovery public key (X25519, packages/deployer/src/offsite.ts). The runner holds only that and a write-only storage key; the private key is offline, with the custodians (§9.3a). A leaked runner, storage key or bucket reads nothing, and cannot delete a backup either. A control-plane copy's credentials are still sealed under the sealing key inside it, so opening one needs both keys.
Set up once — BY HAND (awaiting Ant; docs/proposals/59-fix-ops-recovery.md):
- A bucket in an S3-compatible store outside Cloudflare, in the EU. Recommended: Backblaze B2, region
eu-central-003(Amsterdam), with Object Lock on. Add the provider to the sub-processor list. - Lifecycle rules by prefix, because nothing here can delete:
365d/expires after 365 days (DESIGN's "a year by default"), and one rule per otherretention: { dumps: N }an application declares (Nd/). A copy's key is<days>d/tenants/<company>/<app>/<env>/<date>.sql.sealedor365d/control-plane/<env>/<date>.sql.sealed. - An application key that can write files and nothing else — no read, no list, no delete — scoped to that bucket.
divvent backups keygen --out recovery.keyon a trusted machine. The private key goes to the file, readable by you alone: split it between the custodians with the sealing key's shares (§9.3a) and delete the file. The public key is printed.- GitHub,
stagingenvironment — variables:DIVVENT_OFFSITE_ENDPOINT(https://s3.eu-central-003.backblazeb2.com),DIVVENT_OFFSITE_BUCKET,DIVVENT_OFFSITE_REGION(eu-central-003),DIVVENT_RECOVERY_PUBLIC_KEY; secrets:DIVVENT_OFFSITE_ACCESS_KEY_ID,DIVVENT_OFFSITE_SECRET_ACCESS_KEY.
Until those exist both runs fail, and say which names are missing. That is deliberate: a backup that stays inside the account it protects is not the one that survives it.
Restoring from an off-site copy:
- Download the copy from the store with a read key made for the purpose (the runner's cannot read), onto a trusted machine.
- Join the recovery key from the custodians' shares (§9.3a); put it in this shell as
DIVVENT_RECOVERY_PRIVATE_KEY, never as an argument. divvent backups open --in <copy> --out dump.sql— refused on any changed byte. The output is the customer's data, readable by you alone.- Into a database migrated to the revision the application ran: empty the tables the dump names, then run it (the recovery drill does exactly this, and
acceptance/test/backups-chain.test.tsrestores an off-site copy that way). Tell the customer before, as the controller: §4's rule. - Delete
dump.sqland the key from the machine.
Not yet copied off the account: R2 media. Its only copy is the tenant's own bucket.