Concepts

Incident memory

Repeats cost no LLM call, new analyses start with what was concluded before, and your corrections are tested.

AlertINT remembers what it concluded, so a nightly disk-fill or a noisy quota doesn't cost a fresh analysis every time.

Mechanism When Effect Extra LLM cost
Collapse The same condition re-fires while its Situation is open Counted, not re-analyzed None
Recall A new Incident matches a past analysis The past Finding is given to the model as a hypothesis None
Verdicts You corrected a past Finding The next triage must test your correction None
Near-match classifier A past Finding is one label away Optional check: "same condition?" ~250 tokens, off by default

Collapse, recall, and verdicts are always on.

Collapse: repeats cost nothing#

A re-fire attaches as an occurrence to the analyzed Incident when all of these hold:

  • the group key is identical
  • the owning Situation is still open (not yet Recovered or Recovery unconfirmed)
  • it fires within the attach window (default 30 min) of the last occurrence; each re-fire slides the window

Some re-fires are marked as an escalation on the occurrence: a severity rise, a new alert type, a sudden faster cadence, 100 occurrences, or 4 hours since the last analysis. They still attach (see Limits).

What you see: the Situation's recurred ×N count goes up. At ×5, ×10, ×25, ×50, ×100, then every ×100, one quiet thread reply is added (see Slack). stdout gets one occurrence line.

After the Situation ends: a re-fire opens a new Incident and a new, linked Situation, and is triaged normally.

Recall: new analyses start with the past#

When a new Incident's group key matches a past analysis (within lookback_days), the model gets a short memory section:

Recalled Not recalled
Source Incident and analysis time Whole past Findings
Confidence and root-cause statement Raw alert labels
Recurrence count and cadence
  • A hypothesis, not evidence. It is labelled as such, and it can't lift confidence: a re-fire with only annotations stays under the metadata-only cap.
  • Self-correcting. Each analysis rules on the recalled cause: confirms, refutes, or silent. A cause refuted twice is demoted.
  • Error status check. If the past Finding named an application error, AlertINT checks it once: resolved-but-firing reads as a regression, ignored reads as known-tolerated.

Verdicts: your corrections are tested#

A correction captured over MCP (alertint_incident_capture_verdict) is not taken as fact. On the next triage of that group, its evidence runs in the verification round, and the model must rule:

Ruling Result
supported Your cause is adopted; confidence comes from the evidence
contradicted Not adopted; the Finding says what contradicted it
unverifiable Adopted as the leading hypothesis, confidence capped at 0.6, dated
Evidence never fetched Treated as untested; the Finding says so
  • A confirmation ends the steering: the Finding was right.
  • Corrections never expire by date. Live evidence can retire them; the calendar can't.
  • Notes (alertint_incident_annotate) are for the next person. They appear in the Situation's thread and in MCP (operator_history), and never enter the prompt.

Near matches (optional classifier)#

Recall needs an exact group key. When a past Finding differs by one label value (same cluster and namespace, different service), it's shown as a weak "one label off" hint. The classifier is one small Haiku call (~250 tokens, about $0.0003) that asks: same condition?

memory.classifier.mode Behaviour
off (default) No call
shadow Call runs, verdict goes to the audit log only; the prompt is unchanged
on A matched verdict tags the hint "probably related"

A timeout or unreadable reply never counts as a match.

When to switch shadow → on: after precision ≥ 0.90 over ≥ 100 matched verdicts, measured on your own audit log:

SELECT
  SUM(json_extract(v.payload_json, '$.verdict') = 'confirms') AS confirmed,
  SUM(json_extract(v.payload_json, '$.verdict') = 'refutes')  AS refuted,
  ROUND(1.0 * SUM(json_extract(v.payload_json, '$.verdict') = 'confirms')
        / NULLIF(COUNT(*), 0), 2) AS precision
FROM audit_log c
JOIN audit_log v
  ON  v.kind = 'incident.memory_recalled'
  AND json_extract(v.payload_json, '$.recalled')
      = json_extract(c.payload_json, '$.candidates[0]')
  AND json_extract(v.payload_json, '$.verdict') IN ('confirms', 'refutes')
WHERE c.kind = 'memory.classifier_verdict'
  AND json_extract(c.payload_json, '$.verdict') = 'matched';

Check it#

  • What the model saw: alertint_get_incident returns a memory block with the same facts the analysis got.

  • Analyses avoided:

    SELECT COUNT(*) FROM audit_log
    WHERE kind = 'incident.occurrence_attached'
      AND json_extract(payload_json, '$.trigger') = 'none';
    
  • How recalled causes held up (healthy memory confirms far more than it refutes):

    SELECT json_extract(payload_json, '$.verdict') AS verdict, COUNT(*) AS n
    FROM audit_log WHERE kind = 'incident.memory_recalled'
    GROUP BY verdict;
    

Configure#

Under memory:

Field Default Does
attach_window_minutes 30 Attach window: a re-fire within this time of the last occurrence attaches
judgment_ceiling_hours 4 Marks an escalation this long after the last analysis
occurrence_cap 100 Marks an escalation after this many occurrences
lookback_days 90 How far back recall and recurrence counts reach
classifier.mode off off / shadow / on. Quote the value: bare off and on are YAML booleans.
classifier.timeout_seconds 10 Classifier call timeout

Limits#

  • Escalations don't re-analyze. An escalation is recorded on the occurrence, but it doesn't start a new analysis while the Situation is open.
  • Keys are verbatim. Grouping on volatile labels (a pod name, a job id) means a condition rarely repeats. alertint validate warns about labels that look volatile.