Tutorial: AI triage for Zabbix alerts, on your own infra

Claude: We shouldn't do the integration yet, as Zabbix is not as widely used as Prometheus/Alertmanager...#

Me:#

Yet I built the integration anyway, as in reality people are asking for it.

Here's the setup: wire Zabbix into AlertINT in about 30 minutes: problems become correlated, LLM-triaged incidents your AI coding agent can read over MCP — read-only and self-hosted.

Pre-flight checklist#

Have these ready before you start:

• Zabbix 7.0.x LTS with super-admin access (you'll create a media type, a trigger action, and — for the optional context pull — a user role, user group, service user, and API token).
• A host for AlertINT (Docker, or a Linux/macOS box for the plain binary). The Zabbix server must be able to reach it on port 9911; your workstation must be able to reach it on port 9912 (MCP).
• AlertINT v0.12.0 or newer — Zabbix support first ships in 0.12.0. go install github.com/alertint/alertint-agent/cmd/alertint@latest, or grab a release binary.
• An LLM API key. An Anthropic API key is the reference setup; OpenAI keys and self-hosted endpoints (vLLM, Ollama, …) work via the OpenAI-compatible provider.
• An MCP client — Claude Code, Cursor, or Windsurf.
• If your Zabbix frontend runs under Apache and you want the context pull: you'll need to set CGIPassAuth On in the frontend's Apache config (details in Part 3).

Part 1 — Run AlertINT#

1. Create config.yaml:

   zabbix:
     ingress:
       enabled: true
       webhook_token_env: ALERTINT_ZABBIX_WEBHOOK_TOKEN
   
     # Optional context pull — fill in Part 3, or delete this block for now.
     # Setting base_url is what turns it on.
     api:
       base_url: https://your-zabbix.example.com   # frontend root, no /api_jsonrpc.php
       api_token_env: ZABBIX_API_TOKEN
   
   llm:
     provider: anthropic
     api_key_env: ANTHROPIC_API_KEY
   

   (Using OpenAI or a self-hosted endpoint instead? provider: openai-compatible plus base_url/model/api_key_env — see the OpenAI-compatible provider docs.)

2. Export secrets (never put them in the file — fields ending _env name an environment variable):

   export ALERTINT_ZABBIX_WEBHOOK_TOKEN="$(openssl rand -hex 32)"  # you invent this; Zabbix will present it
   export ALERTINT_MCP_TOKEN="$(openssl rand -hex 32)"             # setting it turns the MCP server on
   export ANTHROPIC_API_KEY="sk-ant-..."
   export ZABBIX_API_TOKEN="..."                                   # only if you kept the api: block (Part 3)
   

   Save ALERTINT_MCP_TOKEN somewhere durable (password manager) — it's what you paste into every MCP client later.

3. Check the config, then start:

   alertint validate config.yaml
   alertint serve --config config.yaml
   

4. Sanity check from the Zabbix server's network:

   curl http://<alertint-host>:9911/health
   

Part 2 — Point Zabbix at AlertINT (the push)#

1. Media type — import (fastest). Download zabbix-media-type.yaml, then Alerts → Media types → Import, pick the file, confirm. Open the imported AlertINT media type and set two parameters for your install:

   • url → http://<alertint-host>:9911/webhook/zabbix
   • token → the value of ALERTINT_ZABBIX_WEBHOOK_TOKEN

   Then skip to step 2 — the two "trip on" notes below the manual path still apply to you (the {$ZABBIX_URL} macro in particular).

   Media type — manual (if you'd rather see what's inside). Alerts → Media types → Create media type, type Webhook, with three parameters:

   Parameter	Value
   url	http://<alertint-host>:9911/webhook/zabbix
   token	the value of ALERTINT_ZABBIX_WEBHOOK_TOKEN
   payload	the JSON template below — paste exactly

   {"event_id":"{EVENT.ID}","status":"{EVENT.STATUS}","severity":"{EVENT.SEVERITY}","nseverity":"{EVENT.NSEVERITY}","host":"{HOST.HOST}","host_visible":"{HOST.NAME}","trigger_id":"{TRIGGER.ID}","trigger_name":"{TRIGGER.NAME}","item_key":"{ITEM.KEY}","item_value":"{ITEM.VALUE}","tags":{EVENT.TAGSJSON},"clock":"{EVENT.DATE} {EVENT.TIME}","recovery_clock":"{EVENT.RECOVERY.DATE} {EVENT.RECOVERY.TIME}","generator_url":"{$ZABBIX_URL}/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}"}
   

   Script:

   var params = JSON.parse(value);
   var req = new HttpRequest();
   req.addHeader('Content-Type: application/json');
   req.addHeader('Authorization: Bearer ' + params.token);
   var resp = req.post(params.url, params.payload);
   if (req.getStatus() >= 300) {
       throw 'alertint replied ' + req.getStatus() + ': ' + resp;
   }
   return 'OK';
   

   Two things people trip on:

   • {EVENT.TAGSJSON} must stay unquoted in the payload (it expands to a JSON array by itself — quoting it breaks the payload).
   • {$ZABBIX_URL} is a user macro you define (global macro holding your Zabbix frontend base URL). It only feeds the click-back link; an empty macro won't break ingestion.

2. Service user + media. Create (or pick) a service user for machine-to-machine delivery, and on its Media tab add this media type (Send to can be any placeholder, e.g. alertint — the webhook ignores it, but Zabbix requires the media entry to deliver anything).

3. Trigger action. Alerts → Actions → Trigger actions → Create action: operation "Send message" to that service user via this media type, and — important — enable recovery operations with the same media type, so RESOLVED events flow through and close incidents out.

4. Quick test. Use the media type's Test button (fill the parameters by hand — macros don't expand in tests; any numeric event_id works). Expect OK in Zabbix and an ingest line in the AlertINT log. A 401 means the token parameter doesn't match the exported env var.

Part 3 — Context pull (optional, recommended)#

This is the half that makes findings noticeably better on Zabbix infra: the LLM (and your MCP client) get the trigger's runbook, dependencies, flap count, host inventory/topology, maintenance state, other open problems on the host, and ack history. Read-only by construction.

1. Role. Users → User roles → Create role: set API access with an Allow list of exactly: host.get, trigger.get, problem.get, event.get, item.get, history.get, trend.get
2. User group. Grant Read on the host groups AlertINT should see.
3. User. Create a service user with that role and group.
4. API token. Users → API tokens → Create, scoped to that user. Export it as ZABBIX_API_TOKEN, make sure the zabbix.api block from Part 1 is in your config, restart AlertINT.

Apache-only gotcha: if the Zabbix frontend runs under Apache, add CGIPassAuth On to its Apache config — otherwise the Authorization header is silently stripped and every context fetch comes back empty/unauthorized while the health check stays green (nginx setups are unaffected). If context is mysteriously missing, check this first.

Part 4 — Verify end to end#

1. Trip a real trigger (or temporarily lower a threshold on a test host). Within seconds the AlertINT log shows the alert arriving.

2. Wait out the correlation window (90 s by default) — related alerts from the same host land in one incident, then triage runs. You'll see a JSON finding line on stdout when it's done.

3. Connect your MCP client — copy-paste configs for Claude Code, Cursor, Windsurf (server is http://<alertint-host>:9912/mcp, auth is ALERTINT_MCP_TOKEN). Then ask:

   List recent AlertINT incidents and summarize the most critical one.

4. Try the Zabbix MCP tools (if Part 3 is on):

   Show CPU history for <host> over the last 2 hours. List open problems on <host> with severity at least high.

5. Resolve the problem in Zabbix and confirm the RESOLVED event closes the alert on the incident (this proves recovery operations are wired).

What "working" looks like#

• Zabbix problems appear as incidents grouped per host with zero grouping config; severities rank correctly even if your install renamed severity display names.
• Findings on Zabbix incidents include the operator context sections (runbook, dependencies, host topology, ack history) when Part 3 is on.
• To correlate one condition across hosts (shared dependency down on many hosts), tag the trigger with a service tag — tags become labels and join grouping automatically.

Troubleshooting quick hits#

Symptom	Likely cause
Zabbix action log shows alertint replied 401	token media-type parameter ≠ ALERTINT_ZABBIX_WEBHOOK_TOKEN value
alertint replied 400 mentioning tags/JSON	{EVENT.TAGSJSON} got quoted in the payload — remove the quotes
Nothing arrives at all	Zabbix server can't reach <alertint-host>:9911 (firewall / wrong host), or the trigger action's conditions don't match your test problem
Health green, but context sections empty	Apache stripping the auth header (CGIPassAuth On), token's user lacks Read on the host group, or wrong base_url
Incident exists but no finding	Check the AlertINT log for the LLM call — usually a missing/invalid API key, or (for openai-compatible) a base_url that already ends in /v1/chat/completions (use the endpoint root)
llm: response truncated at max_tokens	Raise llm.max_tokens (default 4096)

That's the loop#

Zabbix keeps doing what it does; AlertINT sits beside it, read-only, and turns problem storms into investigated incidents that are already waiting in your editor. The full docs cover everything above in more depth.

Questions, rough edges, or a setup that didn't match this post? Open an issue on GitHub or mail me — early-tester feedback is exactly what this phase is for.