Sonnet 5.5
Anthropic
100%
20 right · 0 wrong
AI model comparisons / OpenTelemetry Demo
How AI handles
an incident.
The AI model leaderboard for incident response. Compare Claude Haiku and Sonnet on investigation accuracy, confidence, response time and cost, using live faults in the OpenTelemetry Demo. Built for SREs exploring AI-powered root cause analysis and observability.
Which ones to test next? Drop me a mail or open a GitHub issue.
Methodology & data ↓01 / Answer quality
Did it get the answer right?
RightWrong
Haiku 5.5
Anthropic
75%
15 right · 5 wrong
Haiku 4.5
Anthropic
31.25%
5 right · 11 wrong
Each square is one graded finding.
02 / Confidence vs. correctness
How sure was it?
RightWrong
One dot per finding, placed at the model’s own confidence. Hover or tap for details.
0 = unsure / 1 = certain03 / Fault by fault
Which scenarios did it handle?
✓Sound explanation×Main error
Cells show right / total findings.
04 / Response time
How long did it take?
MedianMeanP95
05 / Cost
What did the runs cost?
06 / Response pipeline
Where did it stumble?
Method & dataHow we score · scope · sources
Scoring
We stage faults in the OpenTelemetry Demo and grade observable primary findings against the evidence. “Right” can retain cause uncertainty and auxiliary caveats. “Sound explanation” is qualified Stop B; Stop A means a material main error. No-alert cases are unscored. Model confidence is self-reported.
Comparison scope
These are live runs with different builds, inputs and finding counts. Shared families align scenario coverage, not the experiment. Sonnet has two payment-failure runs. The full historical view includes Haiku’s additional ad-failure finding. Reviews were performed by the evaluating agent, not independently or blind.
Speed, cost & reliability
Response durations pool logged rows, including errors. Logged cost includes recorded cache categories and mixed request types; missing usage is unknown. The cost chart uses each model’s available estimate, with the scope labeled per row. Historical triage cost covers full historical graded runs, excluding assessments and caching. Error, assessment-rejection and required-key counts are separate. Source recovery is not model repair.
Warmups, pilots and incidental harness findings do not enter accuracy or confidence. These small selected samples do not establish universal accuracy, deployment readiness or a model-only ranking. Sonnet has no comparable recorded-work cost, response-duration or pipeline-error measurements in this dataset.
Exact numbers & per-finding data
Live lab runs · updated 8 Oct 2026