Verification round

Why: triage evidence is scoped to the Incident's own alerts. A draft like "regional infrastructure event" has nothing in its prompt to contradict it, and the same one-sided evidence would confirm it again on every recurrence.

What: one bounded round of read-only contrast evidence, meaning facts chosen to disprove the draft, then a re-judgment. It is not an open-ended investigation; that happens over MCP, with you or your agent driving.

The Evidence tab of the Situation workflow shows this round as a diagram.

How it runs#

Step	What happens
1. Draft	Call 1 drafts the finding and proposes read-only checks that could disprove it.
2. Check	AlertINT runs the floor, the model's checks, and any checks from an operator correction.
3. Re-judge	Call 2 sees the draft plus every result. Results outrank the draft, the evidence pack, and recalled memory.
4. Persist	The re-judged result becomes the Finding. Confidence caps and memory verdicts apply to it, not to the draft.

What gets checked#

Check	When	What
Floor	Every round. The model can't skip or shrink it.	Prometheus: peer-scope up ratio (up 34/37 in namespace=checkout). Zabbix: host reachability and maintenance, host-group neighbors. Always: other Incidents firing in the window.
Model checks	Up to max_queries (default 4)	Read-only Prometheus queries or another incidents-in-window lookup
Correction checks	Only when an operator correction governs the group	Up to 5: the correction's queries plus one probe per named cause series

• A closed set of query kinds: never raw SQL, never a write.
• Same limits as enrichment: prometheus.max_series and per-query timeouts.
• Peer scope comes from the alerts' own namespace / service / job labels; with none shared, it falls back to a global ratio.

Outcomes#

Result	Condition	On the Finding
Supported / revised	Every floor check returned (empty counts), including at least one real source: Prometheus or Zabbix	The Finding, with no caveat. Neither label proves the cause.
Unverified	A floor check failed, no real source was checked, or call 2 failed	A ⚠ unverified caveat. Confidence can't rise above the draft; no memory verdict.
Invalid query	A PromQL check failed parsing	Its own caveat. Treated as inconclusive.

The caveat names the cause:

Caveat	Cause
⚠ unverified — checks unavailable	Floor incomplete, or no real source configured
⚠ unverified — verification call failed; confidence is the draft's	Call 2 failed
⚠ unverified — verification reply unreadable; confidence is the draft's	Call 2 returned an unusable reply
⚠ unverified — no live evidence checks ran (expected for a drill: its labels are fictional)	A drill with no real source checked

• A failed or invalid model check alone doesn't mark the round unverified. The floor is the promised minimum.
• The Finding never shows a draft-vs-final story, only the result and, if needed, the caveat.

Invalid PromQL#

• Parsed locally with Prometheus's own parser, compiled into the binary. No dependency on your Prometheus version.
• Model queries get one repair call: all invalid ones are batched, capped at 512 output tokens. A repair may fix syntax only; it can't change a metric name, label matcher, function, or literal.
• Still invalid → never sent. There is no second repair.
• Operator and correction queries are never repaired. They are marked invalid.

Query outcome	Meaning
invalid	Never ran: failed local parsing, or Prometheus rejected the query itself (bad_data on query)
empty	Ran and matched nothing. Counts as an answer.
failed / degraded	Backend error / timeout

Cost#

Path	LLM calls
Verification off	1
Normal	2
Model proposed invalid PromQL	3 (adds one small repair call)

• Call 2 is cheap: it reuses call 1's prompt as a cache prefix, read back at about 0.10× the input price.
• Caching needs size: the prefix must clear the model's minimum cacheable size. Small Incidents on claude-haiku-4-5 often don't; AlertINT logs a warning when call 2 reads no cache.
• Measure it: llm.response audit rows carry cache_creation_input_tokens and cache_read_input_tokens. Effective input cost = input + 1.25 × creation + 0.10 × read.
• Recurrences don't pay: recurrence collapse already removes re-fires, so the extra call applies only to new or escalated conditions.

Configuration#

Under triage.verification. The defaults are shown below; omit the block to accept them.

triage:
  verification:
    enabled: true              # false = single-call triage
    max_queries: 4             # cap on model checks; the floor always runs
    query_timeout_seconds: 10  # query-phase budget, sliced per query
    max_rounds: 1              # only 1 is accepted

enabled: false means one call, no floor, and no caveat. Correlation, enrichment, recurrence, and notifications are unchanged.

In a drill#

• Same flow: alertint drill has no drill-specific triage path.
• Fictional labels: the peer-scope ratio usually returns no data. Empty still counts, so the round isn't degraded.
• No real source configured: the Finding carries the drill caveat above.
• Inspect it: the drill Finding's verification section is available over MCP.