Getting started

Configuration

Complete YAML configuration reference for the AlertINT agent.

One YAML file. Start from config.example.yaml at the repo root, or the minimal config in the Quickstart.
Secrets stay out of it. A field ending in _env holds the name of an environment variable; the value is read at startup.
Strict. Unknown keys are rejected, so a typo fails loudly instead of being ignored.

Before you deploy#

Validate#

alertint validate config.yaml

Checks every key and cross-field rule; exits 0 or 1 with actionable errors.
Skips checks that depend on the machine (the SQLite directory, the local rule-pack directory), so a pod config validates on a laptop or in CI.
Doesn't read environment variables. Missing secrets are caught at serve startup.

Secrets#

A starting .env:

# Required
ALERTINT_WEBHOOK_TOKEN=<openssl rand -hex 32>          # Alertmanager webhook
ALERTINT_CHANGES_WEBHOOK_TOKEN=<openssl rand -hex 32>  # change webhook
ALERTINT_MCP_TOKEN=<openssl rand -hex 32>              # MCP clients
ANTHROPIC_API_KEY=sk-ant-...                           # console.anthropic.com

# Optional integrations
# PROMETHEUS_BEARER_TOKEN=...
# SLACK_BOT_TOKEN=xoxb-...
# NTFY_TOKEN=tk_...
# LOKI_BEARER_TOKEN=...
# SENTRY_AUTH_TOKEN=...

Any long random string works for the ALERTINT_* tokens.
Save ALERTINT_MCP_TOKEN in your team's secret store. Webhook tokens are machine-to-machine, but every teammate pastes the MCP token into each new client.

Sections at a glance#

Sections follow the pipeline: inputs, processing, outputs, then operations.

Stage	Section	Configures	Needed?
Inputs	`receivers`	Webhook listen address	When any webhook is on
	`alertmanager`	Alertmanager webhook	On by default
	`zabbix`	Zabbix webhook and API	Optional
	`changes`	Change-event webhook	Optional
Processing	`correlator`	Grouping alerts into Incidents	Defaults work
	`memory`	Recurrence collapse and recall	Defaults work
	`rules`	Local rule pack	Optional
	`llm`	Model, provider, spend limits	Required
	`triage`	Verification round, query labels	Defaults work
	`situations`	Situation timing and Slack cooldown	Defaults work
Evidence	`prometheus`	Metric connector	Optional
	`logs`, `sentry`	See Loki, Sentry	Optional
Outputs	`notify`	stdout and Slack	stdout on by default
	`mcp`	MCP server	On when its token is set
Operations	`storage`	SQLite path, upgrades	Defaults work
	`health`	LLM health timers	Defaults work
	`telemetry`	OpenTelemetry traces	Off by default
	`log_level`, `log_format`	Logging	Defaults work

At least one of alertmanager, changes.ingress, or mcp must be enabled. Every integration has an enabled flag; once enabled, its required fields must be set or the agent won't start.

Inputs#

`receivers`#

All webhooks share one listen address.

Field	Type	Default	Description
`address`	string	`":9911"`	Address the webhook server binds to. Also serves `GET /health`.

`alertmanager`#

The Alertmanager webhook. The only integration enabled by default.

Field	Type	Default	Description
`enabled`	bool	`true`	Run the receiver
`webhook_token_env`	string	—	Required. Variable holding the bearer token. Wrong or missing token → `401`.

`zabbix`#

ingress receives Zabbix problems; api reads context from Zabbix. See Zabbix.

Field	Type	Default	Description
`instance_id`	string	—	Stable, non-secret ID for this Zabbix installation (up to 64 letters, digits, `.`, `_`, `-`). Needed to trust rule versions. Changing it creates a new source identity.
`ingress.enabled`	bool	`false`	Mount `POST /webhook/zabbix`
`ingress.webhook_token_env`	string	—	Required when enabled. Variable holding the bearer token
`api.enabled`	bool	auto	On when `api.base_url` is set; `false` forces off
`api.base_url`	string	—	Required for `api`. Zabbix frontend root
`api.api_token_env`	string	—	Required for `api`. Variable holding a read-only API token
`api.timeout_seconds`	int	`10`	Per-request timeout
`api.default_range_minutes`	int	`60`	Default look-back for `zabbix_metric_history`
`api.history_retention_days`	int	`7`	Older windows use hourly trends instead of raw history
`api.flap_window_hours`	int	`24`	Look-back for the trigger flap count
`api.host_label`	string	`host`	Alert label holding the Zabbix host name

`changes`#

Deploys, config edits, and flag flips. ingress receives them; enrichment adds matching changes to triage and the alertint_recent_changes MCP tool. See Change events.

Field	Type	Default	Description
`ingress.enabled`	bool	`false`	Mount `POST /webhook/change`
`ingress.webhook_token_env`	string	—	Required when enabled. Variable holding the bearer token
`enrichment.enabled`	bool	auto	On when a change source exists (the webhook or Sentry releases); `false` forces off
`enrichment.window_minutes`	int	`120`	How far before the first alert to look for changes
`enrichment.max_events`	int	`10`	Max changes added to a prompt
`retention_days`	int	`30`	Delete changes older than this

Processing#

`correlator`#

How alerts become Incidents.

Field	Type	Default	Description
`window_seconds`	int	`90`	Alerts with the same group key inside this window form one Incident
`min_alerts`	int	`1`	Fewer alerts than this → no triage. Keep `1`; control noise with `notify.slack.min_severity`.
`group_labels`	list	—	Build the group key from these labels instead of the source's own grouping

Default grouping: Alertmanager's groupLabels (as sorted key=value pairs); Zabbix's host.
Never empty: with no value for the chosen labels, the key falls back to alertname, then the fingerprint.
Reserved: label keys starting alertint_ are rejected.
Choosing the window: shorter gives a faster first Finding with fewer alerts grouped; longer gives more context, later.

`memory`#

Repeats and recall. See Incident memory.

Field	Type	Default	Description
`attach_window_minutes`	int	`30`	A re-fire within this time of the last occurrence attaches instead of re-triaging
`judgment_ceiling_hours`	int	`4`	Marks an escalation this long after the last analysis
`occurrence_cap`	int	`100`	Marks an escalation after this many occurrences
`lookback_days`	int	`90`	How far back recall and recurrence counts reach
`classifier.mode`	string	`off`	Near-match classifier: `off`, `shadow`, or `on`. Quote it: bare `off`/`on` are YAML booleans.
`classifier.timeout_seconds`	int	`10`	Classifier call timeout

Avoid volatile labels (pod, instance, job_name) in group_labels: the key rarely repeats, so memory rarely matches. alertint validate warns.

`rules`#

The built-in rule pack always loads. A local pack overrides entries with the same id or name. Layout: pack.yaml, rules/*.yaml, templates/*.md (see examples/rules/).

Field	Type	Default	Description
`local_pack_dir`	string	—	Path to a local pack; must contain `pack.yaml`

`llm`#

The model behind triage and Situation assessments.

Field	Type	Default	Description
`provider`	string	`anthropic`	`anthropic` or `openai-compatible` (see OpenAI-compatible endpoint)
`api_key_env`	string	—	Variable holding the API key. Required for `anthropic`.
`model`	string	`claude-sonnet-5`	Model ID, or the name your endpoint serves
`max_tokens`	int	`4096`	Reply ceiling. Raise it if a large Incident truncates; the log says `raise llm.max_tokens`.
`timeout_seconds`	int	`120`	Whole-request timeout. Local endpoints under load often need ~300.
`base_url`	string	—	`openai-compatible` only, required: endpoint root (a trailing `/v1` is fine)
`response_format`	string	`json_object`	`openai-compatible` only: `json_object` or `off`
`thinking`	bool	`false`	`openai-compatible` only: enable thinking on hybrid-reasoning models. Needs `max_tokens` 8000–16000.
`reasoning_effort`	string	—	`openai-compatible` only, with `thinking: true`. Passed to the model unchecked (e.g. `low`, `medium`, `xhigh`).
`budget.calls_per_hour`	int	`0`	Max LLM calls, retries included, in any rolling 60 minutes. `0` = unlimited.
`budget.total_tokens`	int	`0`	Lifetime token ceiling, never reset. `0` = unlimited.

Choosing a model: claude-sonnet-5 gives the strongest Findings; claude-haiku-4-5 costs less per Incident.
Every triaged Incident uses tokens. Watch spend in your provider console, or set a budget.
Hitting the hourly budget delays Situation assessments until the window frees up; nothing is lost.
Hitting the token budget, or unknown usage, stops token-budgeted calls. Affected Situations stay parked until you reconcile the budget with the agent stopped; raising the limit or restarting doesn't release them. config.example.yaml has the steps.
A declined request (4xx, 503, 529) counts its reported usage, or zero, and stops nothing. A budget stopped before v0.14.0 still needs the reconcile above.
Budgets are approximate for compatible servers that report usage inaccurately.

`triage`#

The verification round and which alert labels may enter evidence queries.

Field	Type	Default	Description
`verification.enabled`	bool	`true`	`false` = single-call triage, no verification
`verification.max_queries`	int	`4`	Max model-proposed checks; the floor always runs
`verification.query_timeout_seconds`	int	`10`	Time budget for all checks, split per query
`verification.max_rounds`	int	`1`	Only `1` is accepted
`extra_selector_labels`	list	—	Topology labels (`cluster`, `region`) to add to the built-in list (`namespace, service, job, pod, container, instance`) used in metric and log queries. See Prometheus.

`situations`#

How often Situations are re-checked, how long recovery must hold, and how often Slack may be interrupted. The defaults suit most installs. See the Situation workflow.

Most likely to tune:

Field	Type	Default	Description
`webhook_recovery_grace_seconds`	int	`120`	How long a Situation waits after its last firing alert before it can count as recovered, or end as unconfirmed
`slack.repage_cooldown_seconds`	int	`900`	Minimum gap before a changed required action may post to the channel again. First posts, escalations, and hand-offs skip it.
`cadence.fast_seconds`	int	`60`	Re-check interval for urgent Situations, those confirming recovery, and those with work running
`cadence.normal_seconds`	int	`300`	Re-check interval for Situations under investigation
`cadence.slow_seconds`	int	`900`	Re-check interval for quiet, waiting, or parked Situations. Must be `fast < normal < slow`.

Capacity and limits:

Field	Type	Default	Description
`workers`	int	`2`	Situations processed in parallel
`llm_concurrency`	int	`2`	Max Situation-assessment LLM calls at once
`attempt_wall_seconds`	int	`180`	Time limit for one Situation update
`preparation.max_source_calls_per_cycle`	int	`8`	Max requests to your sources while gathering evidence for one update (1–32)
`preparation.max_wall_seconds`	int	`20`	Time limit for gathering evidence (1–30, below `attempt_wall_seconds`)
`preparation.refresh_seconds`	int	`300`	Minimum time before the same check runs again (60–3600)
`semantic_profiles.workers`	int	`1`	Workers building advisory source profiles (1–4). Only one profile call runs at a time.
`semantic_profiles.max_attempts`	int	`3`	Attempts per profile (1–5)
`semantic_profiles.attempt_wall_seconds`	int	`30`	Time limit per profile call (1–60)

Advanced, leave at default:

Field	Type	Default	Description
`reconcile_poll_seconds`	int	`1`	How often the controller wakes on its own
`lease_seconds`	int	`300`	How long a claimed Situation or triage attempt stays claimed before another worker may take it
`heartbeat_seconds`	int	`30`	How often a claim is renewed while work runs
`retry.min_seconds`	int	`5`	First retry delay after a transient failure
`retry.max_seconds`	int	`300`	Longest retry delay
`retry.jitter_percent`	int	`20`	Random spread on retry delays
`max_l2_calls_per_attempt`	int	`2`	Fixed: one assessment call plus one retry for a malformed reply. Other values are rejected.
`max_work_attempts_per_input`	int	`5`	Fixed: attempts on an unchanged Situation before it's parked. Other values are rejected.

Not configurable: triage attempts (five, see Reliability); Slack retry timing and the delivery-gap threshold (see Slack).

Evidence#

`prometheus`#

Adds live metric values to triage and PromQL tools to MCP. See Prometheus.

Field	Type	Default	Description
`enabled`	bool	auto	On when `base_url` is set; `false` forces off
`base_url`	string	—	Prometheus API root, e.g. `http://localhost:9090`
`bearer_token_env`	string	—	Only if your Prometheus needs a bearer token
`org_id`	string	—	Tenant ID sent as `X-Scope-OrgID` (Mimir, Cortex)
`timeout_seconds`	int	`10`	Time budget for one Incident's metric fetch, split across queries
`default_range_minutes`	int	`60`	Default look-back for range queries
`max_series`	int	`1000`	Max series per query, so a broad selector can't flood a storm

A slow Prometheus shows as degraded (slow), not unreachable, and doesn't lower the Finding's confidence.

Outputs#

`notify`#

See ntfy setup for server, topic and credentials.

Field	Type	Default	Description
`stdout`	bool	`true`	Write events to stdout. Leave on.
`ntfy.enabled`	bool	`false`	Send selected Situation changes to one ntfy topic, independently of Slack
`ntfy.base_url`	string	`https://ntfy.sh`	Hosted or self-hosted server; no embedded credentials, query or fragment
`ntfy.topic`	string	—	Required when enabled; 1–64 letters, digits, underscores or hyphens
`ntfy.token_env`	string	—	Optional variable holding a publisher token; must be set when named
`ntfy.events`	list	five defaults	Omitted uses defaults; an explicit list replaces them; `[]` selects none. Event list
`slack.enabled`	bool	`false`	Post Situations to Slack
`slack.bot_token_env`	string	—	Required for Slack. Variable holding the `xoxb-` token (`chat:write` scope)
`slack.channel`	string	—	Required for Slack. Channel name or ID
`slack.min_severity`	string	`low`	Lowest interruption priority that may post to the channel. `low` posts everything; `critical` always passes.
`slack.recurrence_mode`	string	`change-gated`	`change-gated`: one quiet thread reply at recurrence milestones. `off`: count in the message only.

What stdout writes:

Line	When
`finding` JSON	Each Finding, only at `--log-level=debug`. At `info`, a one-line summary instead.
`occurrence`	A re-fire attached to an Incident
`annotation`	An operator note was added
`triage_exhausted`	Triage failed on every attempt
`situation.transition`	Every Situation change. Means saved, not sent to Slack. Deduplicate by `transition_id`.

The log also shows one notifiers ready line at startup and one notified line per delivery (notified · stdout=ok slack=ok), so a failed send is never silent. Slack setup: Slack. The independent ntfy worker logs ntfy notified or retained delivery failures. A saved transition does not prove external delivery. See ntfy setup and troubleshooting.

`mcp`#

The MCP server your agent connects to. Setting the token variable turns it on.

Field	Type	Default	Description
`enabled`	bool	auto	On when the `token_env` variable holds a token; `false` forces off
`addr`	string	`"0.0.0.0:9912"`	Listen address. Endpoint: `http://host:9912/mcp`.
`token_env`	string	`ALERTINT_MCP_TOKEN`	Variable holding the bearer token

Lost the token? Set a new value, restart, and update every client. Client setup: MCP clients.

Operations#

`storage`#

Field	Type	Default	Description
`sqlite_path`	string	`./alertint-agent.db`	Database file. Its directory must be writable.

Upgrades: every start applies pending schema migrations before accepting traffic.
Back up first: take a live backup before upgrading (Backup & restore).
Rollback means restore: there is no schema downgrade, and an older binary can't run on a newer database.

`health`#

Timers for LLM dependency health.

Field	Type	Default	Description
`llm_idle_probe_after_seconds`	int	`300`	Idle time without a real LLM call before a metadata probe runs (at most once a minute, never during a call)
`broadcast_after_seconds`	int	`300`	How long the LLM must stay unhealthy before the Slack system message is posted. State changes within one outage don't reset it.

`telemetry`#

OpenTelemetry traces for the Situation controller and triage worker. Off by default: nothing leaves the process. Spans carry IDs, counts, result classes, and durations, never prompts, responses, or secrets.

Field	Type	Default	Description
`otlp.enabled`	bool	`false`	Export traces
`otlp.endpoint`	string	—	Required when enabled. `host:port` (TLS unless `insecure`) or a URL (the scheme decides TLS; HTTP without a path uses `/v1/traces`)
`otlp.protocol`	string	`grpc`	`grpc` or `http`
`otlp.insecure`	bool	`false`	Plaintext for a `host:port` endpoint
`otlp.service_name`	string	`alertint-agent`	`service.name` for your backend. `OTEL_RESOURCE_ATTRIBUTES` is merged in.
`otlp.timeout_seconds`	int	`10`	Per export batch, and the final flush at shutdown

telemetry:
  otlp:
    enabled: true
    endpoint: otel-collector:4317
    insecure: true

Never blocks work. An unreachable collector is a log warning, not a startup failure.
Logs match spans. Each span also writes a log line with the same IDs plus trace_id and span_id.

`log_level`#

debug, info (default), warn, or error.

`log_format`#

Value	Output
`auto` (default)	`console` when stderr is a terminal, `json` otherwise
`console`	One readable line per record; colour only on a terminal with `NO_COLOR` unset
`json`	One JSON object per record, for log shipping

Precedence: --log-level / --log-format flags → config → defaults.
Findings as JSON: run with --log-level=debug (see notify).
Colour in docker logs: set CLICOLOR_FORCE=1. NO_COLOR still wins.

Full example#

receivers:
  address: "0.0.0.0:9911"

alertmanager:
  enabled: true
  webhook_token_env: ALERTINT_WEBHOOK_TOKEN

storage:
  sqlite_path: /var/lib/alertint/alertint.db

llm:
  provider: anthropic
  api_key_env: ANTHROPIC_API_KEY
  model: claude-sonnet-5

correlator:
  window_seconds: 90
  min_alerts: 1
  # group_labels: [cluster, namespace, service]   # optional override

notify:
  stdout: true
  slack:
    enabled: true
    bot_token_env: SLACK_BOT_TOKEN
    channel: "#alerts"

mcp:
  # on automatically when ALERTINT_MCP_TOKEN is set
  addr: "0.0.0.0:9912"
  token_env: ALERTINT_MCP_TOKEN

rules:
  local_pack_dir: /etc/alertint/rules

prometheus:
  base_url: http://localhost:9090
  bearer_token_env: PROMETHEUS_BEARER_TOKEN

log_level: info
log_format: auto

Integration health#

At startup, each enabled integration is probed (Prometheus with a trivial query, Slack with auth.test) and logged:

15:04:05 INFO  integration health: OK · integration=prometheus detail=http://prometheus:9090
15:04:05 WARN  integration health: FAILED · integration=slack detail=#alerts err=invalid_auth

GET /health returns the same, cached for 60 s:

status reflects only whether the agent is alive. A failing integration never makes an orchestrator restart it.
llm carries the LLM dependency state: healthy, degraded, or unavailable, with per-capability detail.

{"status":"ok",
 "integrations":[{"name":"prometheus","detail":"http://prometheus:9090","ok":true,"checked_at":"..."}],
 "llm":{"state":"unavailable","reason":"provider_unavailable","detail":"HTTP 503","unhealthy_since":"2026-08-27T12:00:00Z","in_flight":0,
        "capabilities":[{"capability":"triage_draft","healthy":false,"reason":"provider_unavailable","detail":"HTTP 503"}]}}

Configuration

Before you deploy#

Validate#

Secrets#

Sections at a glance#

Inputs#

receivers#

alertmanager#

zabbix#

changes#

Processing#

correlator#

memory#

rules#

llm#

triage#

situations#