Architecture

• One process: alertint serve
• One state file: local SQLite
• One config file: no external services to install

Vocabulary#

Term	What it is
Alert	One source observation from Alertmanager or Zabbix. Evidence, not a unit of attention.
Incident	Alerts sharing a group key, correlated for one bounded triage.
Situation	The episode you track: one or more Incidents with the same group key. Owns the lifecycle and the Slack message.
Phase	Where a Situation is: Observed · Correlating · Investigating · Monitoring · Confirming recovery · Recovered (or Recovery unconfirmed).
Finding	The verified triage result, shown on the Situation.
Verdict	Your confirmation or correction of a Finding, captured over MCP.

A Situation ends as Recovered or Recovery unconfirmed and never reopens. A later firing starts a new, linked Situation. Explore it in the Situation workflow.

Two feedback loops#

Loop	What it prevents
Verification round: every draft must survive contrast evidence before it becomes the Finding	A confident answer built only from the Incident's own alerts
Operator verdict: your correction steers the next triage of the same group	The same wrong answer twice

Phase 1: alerts to a Finding#

#	Step	What happens	Configure
1	Receive	Alertmanager, Zabbix, and change-event webhooks on one address, each with its own bearer token.	receivers.address (:9911)
2	Accept	Written to disk before the sender gets 204. Retried webhooks deduplicate.	Reliability
3	Correlate	Same group key within the window → one Incident. Rules run here.	correlator.window_seconds (90 s), correlator.group_labels
4	Track	The Incident joins its Situation. The Situation controller decides whether to investigate.	situations
5	Remember	A re-fire of an analyzed condition attaches as an occurrence, with no LLM call. New Incidents recall past Findings as hypotheses.	Incident memory
6	Gather evidence	Labels, timeline, and annotations, plus optional live reads from connected sources.	Integrations
7	Draft	The LLM drafts cause, severity, confidence, and next checks.	llm
8	Verify	Contrast checks try to disprove the draft; a second call re-judges.	Verification round
9	Publish	The Finding goes to the Situation history, stdout, and at most one Slack root per Situation.	Slack

Evidence sources are all read-only and optional:

Source	Adds
Prometheus	Live metric values, peer-scope health
Loki	Recent log lines
Sentry	Exceptions with file:line, releases
Zabbix	Runbook, trigger dependencies, flap history, host inventory, maintenance
Change events	Deploys, config edits, and flag flips just before the alert

With none configured, triage runs on labels and annotations alone.

Selected mobile notifications#

ntfy can run alongside Slack or without it. Selected Situation transitions commit frozen Markdown messages into their own SQLite delivery queue. A bounded HTTP worker publishes to one topic, retries failures and sends current-state catch-up after an outage or pause. Each push is a new message. Both destinations share the Situation's decisions and findings, so delivery adds no LLM calls. External delivery is at least once; an uncertain response can produce a duplicate on retry.

Phase 2: investigate with your agent#

#	Step	What happens	MCP tools
10	Connect	Your MCP client connects to the same binary: http://host:9912/mcp, bearer token.	See MCP clients
11	Read state	Situations, Incidents, alerts, evidence, and changes, served from local state.	alertint_list_situations, alertint_get_situation, alertint_list_incidents, alertint_get_incident, alertint_get_evidence_pack, alertint_search_alerts, alertint_recent_changes
12	Query telemetry	Read-only queries proxied to your backends.	prometheus_query, prometheus_query_range, loki_query_range, zabbix_metric_history, zabbix_host_problems, sentry_issues_list, sentry_issues_trace
13	Close the loop	Confirm or correct the Finding. A correction is tested against live evidence on the next triage before it's adopted.	alertint_incident_capture_verdict, alertint_incident_annotate

You decide the next action. AlertINT supplies the context.

Typical prompts:

List active AlertINT Situations.
Open the latest critical Situation and summarize the evidence.
Query Prometheus for CPU and memory around that window.
That root cause is wrong: it was the cache rollout. Capture that as a correction.

Guarantees#

• Read-only outward: connectors only query. Writes come only from operator MCP commands (verdicts, notes, expected-behaviour judgments) and land in AlertINT's own state, never in your infrastructure.
• No inline secrets: config fields ending in _env name the environment variable that holds the value.
• No silent config drift: unknown YAML keys are rejected at load.
• Tamper-evident: every action appends to a hash-chained audit log.
• No web UI in the self-hosted agent: config, MCP, and Slack are the interfaces.

Durability, retries, restart recovery, and LLM health: Reliability. What AlertINT won't do: Scope and limits.