6k. Proactive monitoring: incidents, and what to do in a provider outage

The scheduler's hourly pass (23 * * * *) calls watchFleet after billing and before the fleet update (DESIGN.md §17, Stage 5, "Proactive monitoring"; docs/proposals/38-proactive-monitoring.md). Nothing here is by hand: it runs on the next scheduler deploy, and the first pass reads the providers' status pages and the console's own D1.

What it does. From each production application's pushed numbers — codes, counts, durations, pins and trace ids, never a row — it detects a new failing class, an error- or denial-rate jump against the application's own last seven days, a failing flow step, a p95 regression, and a failed health reading within a day of a deploy. Each is an incident, classed before anything is spent as external (the provider), engine (ours), app (the application's model or its latest change) or unknown. Engine and app incidents are diagnosed automatically — 3 per application a day, 10 across the fleet an hour, on the platform's AI line — and a carried fix is a change request held for an operator.

The operator flow.

1. The console's home says so: Incidents in the work list, and the headline when a platform incident froze the rollout or a fix is waiting.
2. Incidents lists the open ones, each with its cause in words and the reasons behind it, then the providers' status, then the last resolved.
3. An incident's page has the evidence (the code, the counts, the trace ids, the applications), its diagnosis, and — when a fix is waiting — Approve the fix / Decline, which are the Requests queue's own actions: read what it would do first. Approving confirms the fix's regression test and writes the revision; the build, rehearsal and canary are the normal path's.
4. An engine incident across two or more accounts after an engine change freezes the rollout (Fleet says why). Fix the engine, then Resume on Fleet with what was done — as for any freeze.
5. The caps are at the foot of Incidents; zero turns automatic diagnosis off. Every change is audited with its reason.

In a provider outage (Cloudflare, Stripe, Resend). The watch does the first three things itself; check that it did.

1. Incidents shows one fleet incident naming the provider and every affected application, cause the provider's, and the reason — its status page and/or our own canary. No diagnosis is spent and no fix is proposed.
2. The rollout is paused, not frozen: Fleet's pass says paused while … has an incident. Do not resume it — there is nothing to resume; the first pass after the incident resolves raises upgrades again by itself.
3. Operators are mailed once, when it opens (the operatorEmail setting, as the health digest is).
4. Customers are not told while it lasts. After it resolves, each account is told — by default only where it reached their users — that it was the hosting provider and not their application or their changes.
5. It resolves when the provider's page stops saying degraded or outage, our canaries pass, and the applications' numbers are back to their baseline. A page that does not answer is unknown and never counts as recovered on its own; if it stays unreachable, the incident resolves once the canaries pass and the numbers clear.
6. If the watch has not classed it external — the page was silent and no canary failed, so it reads unknown — it waits one more pass before any diagnosis. Set the fleet cap to 0 on Incidents if you want to be certain nothing is spent while you look, and set it back afterwards.

**What reads unknown and why.** R2 and Queues canaries read unknown until a probe bucket and a probe queue are bound to the console (a probe on BUILDS or DEPLOYS would be work their consumers act on). Stripe's and Resend's pages are read at their Statuspage summary.json; a provider whose page is not that shape reads unknown, which is safe, and is worth checking once against the live URL before production.