Concepts

Reliability

What AlertINT guarantees when senders retry, the process restarts, or the LLM is down.

Ingest#

Every webhook answer means one thing:

Response Meaning Sender should
204 The delivery and its dispatch record are on disk. Nothing
400 Structurally invalid payload. Nothing was written. Fix the payload
503 Valid, but could not be stored (for example, SQLite unavailable). Retry the same body
  • Retries are safe. A delivery's id is a digest of its normalized content, so a retried webhook resolves to the same delivery.
  • Byte-identical repeats deduplicate. A Zabbix escalation step resending the same macros is one delivery, and it does not extend the recurrence collapse window. Any changed byte (a timestamp, a counter) makes a new delivery.
  • Nothing is lost between accept and correlate. A background worker correlates accepted deliveries; on startup it drains pending ones before accepting new traffic.
  • 5xx only means "retry me". A payload AlertINT decided to reject always gets a 4xx.

Triage attempts#

Each Incident gets at most five triage attempts. The Investigation tab of the Situation workflow shows the states.

Attempt fails Next attempt after
1st 30 s
2nd 2 min
3rd 8 min
4th 32 min
5th None: exhausted
  • A crash doesn't buy a free retry. An interrupted attempt counts toward the five.
  • No burst after an upgrade. At startup, any attempt more than an hour overdue is closed without an LLM call (audited as startup_retry_window_expired).
  • Late alerts join a retrying Incident. While an Incident waits in backoff, a new alert with the same group key joins it instead of opening another Incident.
  • Exhaustion ends analysis, not monitoring. The Situation keeps tracking its source alerts.

LLM dependency health#

AlertINT tracks the configured LLM as one installation-wide dependency.

State Cause Slack system message
unavailable Draft or Situation-assessment calls fail, or the idle probe fails "New Incident triage is retrying; correlation may be delayed."
degraded Verification re-judgment calls fail "Draft Findings continue with reduced confidence."
  • Idle probe: after five idle minutes, a metadata GET checks reachability. It never sends a prompt.
  • Where to see it: /health under the llm key (the HTTP status is unaffected), and one AlertINT system Slack message per outage, edited in place.
  • Never per Incident: an LLM outage does not become one message per stuck Incident.

Audit log#

Every action appends a hash-chained row:

hash = SHA256( ts FS actor FS kind FS canonical_json(payload) FS prev_hash )

FS is the ASCII unit separator 0x1f. Each row covers the previous one, so tampering is detectable:

  • CLI: alertint verify-audit
  • MCP: alertint_verify_audit