apac-east-jp service disruption

A code change to the regional monitoring agent's /readyz health endpoint, intended to make it tolerate brief main-server connectivity blips, added an extra field to the endpoint's JSON success response. This broke an external monitor's literal string-match assertion, causing a false "down" detection for apac-east-jp even though the agent was fully healthy throughout. No actual service disruption.

Public status page showed apac-east-jp as Outage for ~6h20m (23:10 UTC 2026-09-28 → ~05:30 UTC 2026-09-29). Zero real impact. The monitoring agent was healthy and correctly serving checks the entire time. Impact limited to a false public status entry.

/readyz's 200 response was changed from {"status":"ok"} to {"status":"ok","using_monitors_cache":false} as part of an unrelated reliability fix, deployed fleet-wide to all 14 region agents. An external monitor asserted on the literal substring {"status":"ok"} (expects the object to close right after "ok"); the added field broke that match. The check first failed for apac-east-jp when its check cycle landed after the rollout, tripping quorum and auto-creating the incident — while the endpoint was returning HTTP 200 and every other health signal was clean, which made initial SSH investigation look like a dead end.

• Reverted /readyz's success body to the exact original {"status":"ok"}; moved the cache-usage diagnostic field to /healthz instead, and left it on the /readyz 503 (failure) body only, which no external check matches literally.
• Rebuilt and redeployed the fixed binary to all 14 region agents (make agent-update-all), verified byte-for-byte /readyz response on the affected host and one other.

• [23:10 UTC Sep 28] Incident detected; quorum confirms DOWN, auto-incident created
• [23:15 UTC Sep 28] Investigation via SSH: agent process, logs, all HTTP 200s clean; no host-side issue found
• [05:15 UTC Sep 29] Root cause identified, fix implemented and verified locally
• [05:34 UTC Sep 29] Fix redeployed to all 14 region agents
• [05:35 UTC Sep 29] Incident mitigated; /readyz confirmed back to exact {"status":"ok"}
• [05:35 UTC Sep 29] Incident resolved