Concepts
Reliability
What AlertINT guarantees when senders retry, the process restarts, or the LLM is down.
Ingest#
Every webhook answer means one thing:
| Response | Meaning | Sender should |
|---|---|---|
204 |
The delivery and its dispatch record are on disk. | Nothing |
400 |
Structurally invalid payload. Nothing was written. | Fix the payload |
503 |
Valid, but could not be stored (for example, SQLite unavailable). | Retry the same body |
- Retries are safe. A delivery's id is a digest of its normalized content, so a retried webhook resolves to the same delivery.
- Byte-identical repeats deduplicate. A Zabbix escalation step resending the same macros is one delivery, and it does not extend the recurrence collapse window. Any changed byte (a timestamp, a counter) makes a new delivery.
- Nothing is lost between accept and correlate. A background worker correlates accepted deliveries; on startup it drains pending ones before accepting new traffic.
5xxonly means "retry me". A payload AlertINT decided to reject always gets a4xx.
Triage attempts#
Each Incident gets at most five triage attempts. The Investigation tab of the Situation workflow shows the states.
| Attempt fails | Next attempt after |
|---|---|
| 1st | 30 s |
| 2nd | 2 min |
| 3rd | 8 min |
| 4th | 32 min |
| 5th | None: exhausted |
- A crash doesn't buy a free retry. An interrupted attempt counts toward the five.
- No burst after an upgrade. At startup, any attempt more than an hour overdue is closed without an LLM call (audited as
startup_retry_window_expired). - Late alerts join a retrying Incident. While an Incident waits in backoff, a new alert with the same group key joins it instead of opening another Incident.
- Exhaustion ends analysis, not monitoring. The Situation keeps tracking its source alerts.
LLM dependency health#
AlertINT tracks the configured LLM as one installation-wide dependency.
| State | Cause | Slack system message |
|---|---|---|
unavailable |
Draft or Situation-assessment calls fail, or the idle probe fails | "New Incident triage is retrying; correlation may be delayed." |
degraded |
Verification re-judgment calls fail | "Draft Findings continue with reduced confidence." |
- Idle probe: after five idle minutes, a metadata
GETchecks reachability. It never sends a prompt. - Where to see it:
/healthunder thellmkey (the HTTP status is unaffected), and oneAlertINT systemSlack message per outage, edited in place. - Never per Incident: an LLM outage does not become one message per stuck Incident.
Audit log#
Every action appends a hash-chained row:
hash = SHA256( ts FS actor FS kind FS canonical_json(payload) FS prev_hash )
FS is the ASCII unit separator 0x1f. Each row covers the previous one, so
tampering is detectable:
- CLI:
alertint verify-audit - MCP:
alertint_verify_audit