Code patches are not root-cause analysis
ORCA-bench, submitted July 30, asks whether general coding agents can diagnose production incidents. Agents start from ambiguous user-facing reports and inspect six days of metrics, logs, traces, and the full source tree.
The testbed combines an OpenTelemetry-instrumented microservice system, Prometheus, Jaeger and OpenSearch interfaces, roughly 50 GB of telemetry, and 1,079 RCA tasks. Expert SREs approved ground-truth symptoms, while human rescoring of the LLM judge reached a reported weighted Cohen's kappa of 0.90.
Low success under realistic input
Across five frontier agents, the best accuracy was 25.3% on Medium tasks and 10.0% on Hard tasks. The weakest model produced an implausible cause in 40% of incident reports, and removing source-code access hurt every metric, according to the authors.
Operational meaning and limits
AI can summarize telemetry and draft hypotheses or checks, but these results do not support unattended remediation, deployments, or traffic changes. Evidence links, reproducible commands, human approval and rollback should remain gates.
This is a preprint on one public testbed; real systems are larger and organization-specific. Teams should evaluate false diagnoses, missed causes and recovery time on their own incidents instead of treating the model ranking as universal.