Haiku vs Sonnet: which one is better for incident alerts?

AlertINT investigates alerts with whichever AI model you plug in. One question decides whether that's useful: does the model find the real cause of a failure, or only sound like it did?

Production can't answer that, because nobody knows the right answer for certain. So we stage failures ourselves, where the cause is known before the model starts looking, and grade each conclusion against it.

First two models tested: Claude Haiku 4.5 and Claude Sonnet 5.5.

Why this setup#

• The OpenTelemetry Demo, not a toy app. It's an open-source shop of a dozen-plus services with real metrics, logs and traces, so a failure in one service shows up in others, as it does in production. Anyone can run it.
• Feature flags, not killed containers. A dead container is too easy to spot. The demo's own flags cause partial failures that look like real bugs: some payments fail, some orders stall. The model never sees the flag, so it has to reason from what monitoring shows.
• Live runs, not replayed prompts. Real monitoring and live model APIs test what a team would actually run. The price is that no two runs are identical: alerts fire a little differently each time, so the models didn't always get the same investigations, 17 for Haiku and 20 for Sonnet.
• One variable. In each scenario, both models ran on the same AlertINT build and settings from a fresh database. When a run showed AlertINT dropping evidence, we fixed AlertINT and scored both models on the fixed build. We're grading the model, not our bugs.
• Cheapest model first. If the cheaper model is good enough, there's no reason to pay for a bigger one. Haiku ran each scenario first, and Sonnet came in wherever Haiku got it wrong with the right evidence in front of it.
• Grading the reasoning, not just the service named. A model can point at the right service for the wrong reason, and the wrong reason is what misleads whoever is on call. Each scenario has a written answer key, and we graded every investigation against it by hand.

How answers are graded:

• Right: matches the answer key on the main point.
• Overstates: right, but one claim goes further than the evidence. Counts as right.
• Wrong: a main conclusion the evidence doesn't support, or a misreading of the evidence itself.

A run where no alert fired isn't scored, and isn't a pass. A model clears a scenario when every investigation in it is right. Of 12 scenarios, 3 raised no alert in any run, which leaves 9 graded scenarios, 18 runs and 37 investigations.

Results#

	Claude Sonnet 5.5	Claude Haiku 4.5
Scenarios cleared	8 of 8	0 of 9
Investigations right	20 of 20	5 of 17
Right, but Overstates	6	5
Confidence when right	0.30 to 0.70, mean 0.47	0.72 to 0.87, mean 0.79
Confidence when wrong	no wrong answers	0.58 to 0.85, mean 0.75

Confidence didn't separate right from wrong#

Every investigation ends with the model's own confidence, from 0 to 1. Haiku's barely moved with correctness: a mean of 0.79 when right and 0.75 when wrong. Ten of its 12 wrong answers were more confident than any answer Sonnet gave. Sonnet stayed between 0.30 and 0.70 and used phrases like "likely" and "cause not established".

Each dot is one graded investigation, placed at the confidence the model gave it.

A model that sounds sure is not the same as a model that's right.

A plain rule did better than the model's confidence. AlertINT labels a finding Unconfirmed when none of its checks tested the claimed cause, or when its logs can't confirm a claimed affected group. That label was on 8 of Haiku's 12 wrong answers and on none of its 5 right ones. Of the 4 wrong answers it missed, one shows a gap in the rule: in the email scenario, Haiku re-measured memory under different metric names, and the rule let it through.

How the wrong answers went wrong#

Most wrong answers got the symptom right, then took one step the evidence didn't support. The same patterns kept coming up:

• Empty read as healthy. An empty lookup of AlertINT's incident history, which starts empty every run, taken as proof the failure was contained.
• An average hiding the slow few. Only orders going abroad were delayed, but a 0.0067-second average shipping time, counted since the service started, was read as current.
• Units. CPU time called GHz. 442,368 bytes read as 442. Errors per second reported as percentages.
• Recovery read as continuity. Payments were coming back by the time payment's alert fired, and that was read as traffic having continued all along. The counters showed no payments during the fault.
• Direction. Product catalog's failures blamed on the services that call it.
• Arithmetic. 642 seconds of failed calls in 15 minutes, divided by all 259 failures since startup, gave 2.5 seconds each. Only 3 had failed in that window, so each took about 214 seconds.

Cost#

This covers investigation (triage) calls only, at Anthropic's list prices in October 2026, without prompt caching. AlertINT's follow-up assessment calls aren't included.

	Claude Sonnet 5.5	Claude Haiku 4.5
Price per million tokens, in / out	$2 / $10	$1 / $5
Tokens in / out	284,865 / 46,789	183,512 / 25,335
All graded runs	$1.04	$0.31
Per investigation	5.2¢	1.8¢
Per right answer	5.2¢	6.2¢

Sonnet costs twice as much per token and used about 35% more tokens per investigation (16,600 against 12,300). Per right answer, it was still cheaper. Because the runs weren't identical, read this as an order of magnitude, not a price comparison: with either model, an investigation costs cents.

Conclusion#

On these runs, Sonnet 5.5 was the better investigator. It was right in all 20 investigations, stated its uncertainty, and cost less per right answer. Haiku 4.5 usually found the symptom, then drew unsupported conclusions with high confidence. It's a small case study, not a benchmark: one or two runs per model per scenario, graded by us, on a demo shop that's simpler than production.

In fairness, Haiku is still on 4.5 while Sonnet and Opus have moved on to 5.5. It's about time Haiku got a 5. Anthropic, get your s**t together!

The broader lesson matches our last post: don't treat a model's confidence as a trust signal. Use rules that check what the evidence actually supports.

Want AlertINT investigating your own alerts? We set it up on your most critical services and keep it running: try it on your alerts. Or run it yourself from GitHub. And if there's a failure you'd like us to add to the next round, email me at ernests@alertint.com.

Ernests