AlertINT logoAlertINT
BlogGitHub

Integrations

Prometheus

Connect Alertmanager as the alert source and Prometheus for live metric context.

AlertINT integrates with the Prometheus ecosystem in two places: Alertmanager is the alert source — it forwards a webhook copy of every alert to the agent — and the optional Prometheus connector enriches LLM triage with live metric values at incident time and powers PromQL tools for MCP-driven investigation. The connector only issues queries; it never writes metrics, creates recording rules, or modifies Prometheus state.

Alertmanager — the alert source#

Add an alertint-agent receiver to your alertmanager.yml and route alerts to it. A complete minimal config ships in the repo as examples/alertmanager.yml:

route:
  receiver: alertint-agent
  group_by: [alertname, cluster, namespace, service]
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 4h

receivers:
  - name: alertint-agent
    webhook_configs:
      - url: "http://<agent-host>:9911/webhook/alertmanager"
        send_resolved: true
        http_config:
          authorization:
            credentials_file: /etc/alertmanager/alertint_token

Notes:

  • The bearer token must match the value of the env var named by alertmanager.webhook_token_env in the agent config (default ALERTINT_WEBHOOK_TOKEN). Prefer credentials_file over an inline credential so the token is not stored in alertmanager.yml.
  • send_resolved: true is required for resolution tracking — it lets AlertINT close incidents and update Slack messages when alerts recover.
  • To keep your existing paging intact, add alertint-agent as a child route with continue: true instead of replacing your top-level receiver.

Prometheus connector — live metric context#

How it works#

When an incident is ready for analysis, AlertINT builds a generic PromQL selector from the alert group's shared labels — the same allowlist logs use (namespace, service, job, pod, container, instance) — and queries it at the incident start time. This is what makes Kubernetes-style alerts (labeled by namespace/pod/container, often with no instance at all) get live metrics instead of falling back to annotations-only: the old behavior queried {instance="X"} alone, which most K8s alerting rules never set.

Two refinements keep the selector from missing evidence:

  • Per-instance supplement. Any alert that does carry instance keeps at least that broad per-instance scope as an extra query, even when the shared selector narrows it away — non-Kubernetes stacks see the same coverage as before.
  • Physical-core retry. If the full selector matches zero series (a logical label like service or job that an alerting rule attaches but no series actually has), AlertINT retries once with only the physical-identity keys (namespace, pod, container, instance) before concluding the query is genuinely empty.

Up to 10 non-system metric series are kept per query, ranked by how many labels they share with the firing alerts (a series carrying the same pod as a member alert outranks an unrelated series in the same namespace) and appended to the LLM prompt as a Live metrics section. The model uses those values to calibrate severity and confidence — actual numbers take precedence over text annotations.

Scoping evidence queries: the selector allowlist#

When AlertINT builds metric queries for an incident, it uses the alert-label keys shared by every member alert, filtered to a selector allowlist that drops alert metadata (alertname, severity, …) no backend labels data by. The built-in allowlist is:

namespace, service, job, pod, container, instance

Multi-cluster setups: triage.extra_selector_labels#

If one Prometheus/Mimir serves alerts from more than one cluster, cluster is on every alert but is not in the built-in allowlist — so evidence queries would mix series from other clusters. Add topology labels to the allowlist with triage.extra_selector_labels. The examples below all use one incident: shared labels cluster=eu-west, namespace=payments, service=checkout, with member alerts also carrying region=eu where noted.

Example 1 — not configured (the default). Behavior is unchanged:

triage: {}                # or the key omitted entirely

metric primary:  {namespace="payments",service="checkout"}
log (pre-map):   {namespace="payments",service="checkout"}
floor up_ratio:  {namespace="payments",service="checkout"}

Example 2 — one topology label (the multi-cluster case above):

triage:
  extra_selector_labels: [cluster]

metric primary:   {cluster="eu-west",namespace="payments",service="checkout"}
zero-match retry: {cluster="eu-west",namespace="payments"}          # sheds service/job, keeps the extra
supplement:       {cluster="eu-west",instance="10.0.4.7:9100"}      # per member alert carrying instance
log (pre-map):    {cluster="eu-west",namespace="payments",service="checkout"}
floor up_ratio:   {cluster="eu-west",namespace="payments",service="checkout"}

Configured labels are never dropped by fallback queries: when the primary query matches nothing, the retry sheds rule-attached labels (service, job) but keeps your extras. A label that exists on alerts but not on your series therefore shows up as an empty enrichment with the exact query in the agent log — loud and diagnosable — rather than silently widening to the wrong cluster.

Example 3 — several extras; only keys shared by every member are used. Alerts carry cluster and region, but no datacenter — a configured key absent from any member alert simply never enters a query:

triage:
  extra_selector_labels: [cluster, region, datacenter]

metric primary: {cluster="eu-west",namespace="payments",region="eu",service="checkout"}

Example 4 — extra on metrics, dropped for logs. Mimir series carry cluster but the Loki streams don't; loki.label_map (see Loki) removes it from log queries only:

triage:
  extra_selector_labels: [cluster]
logs:
  loki:
    label_map:
      cluster: ""        # drop for LogQL; metric/floor queries keep it

metric primary:  {cluster="eu-west",namespace="payments",service="checkout"}
log (post-map):  {namespace="payments",service="checkout"}

Example 5 — rejected at startup. Fail-loud config validation catches bad syntax and built-in collisions before the agent ever runs a query:

triage:
  extra_selector_labels: [cluster-1, namespace]

triage: extra_selector_labels: "cluster-1": invalid label name (must match [a-zA-Z_][a-zA-Z0-9_]*)
triage: extra_selector_labels: "namespace": already in the built-in allowlist (namespace, service, job, pod, container, instance)

Use topology labels (cluster, region, datacenter), not identity labels (anything pod-like): a narrow extra would collapse the verification floor's peer scope to the incident's own targets. Run with --log-level=debug to see which shared alert labels the allowlist dropped for each incident.

Evidence line#

Every finding notification carries a per-source evidence summary — how many metrics, log lines, changes, and Sentry issues fed the triage, e.g. Prometheus 21 metrics · Loki 0 lines · Changes 2 · Sentry unreachable. A connector that could not be reached renders unreachable, distinct from a genuine 0, so a misconfigured or down connector is visible on every card instead of silently degrading confidence. See Slack notifications for where it appears on the card.

Configuration#

prometheus:
  base_url: http://localhost:9090            # setting this turns the connector ON
  # enabled: false                           # uncomment to force OFF despite base_url
  bearer_token_env: PROMETHEUS_BEARER_TOKEN  # optional
  # org_id: tenant-1                         # only for multi-tenant Mimir/Cortex
  timeout_seconds: 10                        # default
  default_range_minutes: 60                  # default

Enablement is presence-based: setting base_url turns the connector on automatically; an explicit enabled: false forces it off.

The connector speaks the standard Prometheus HTTP query API (/api/v1/query and /api/v1/query_range), so anything implementing that API works as base_url. A Thanos Querier is a drop-in — same requests, same responses, same authentication story (Thanos likewise has no built-in auth) — and extends query reach to the full retention window across all connected Prometheus instances. Note its default HTTP port is 10902, not 9090, and point base_url at the Querier component, not a Sidecar or Store Gateway.

Field Description
enabled Optional. Omitted = on when base_url is set; false forces off.
base_url Base URL of your Prometheus instance, e.g. http://localhost:9090.
bearer_token_env Optional. Name of the env var holding the Prometheus bearer token — see Authentication for when you need one and where it comes from.
org_id Optional. Tenant/org ID sent as the X-Scope-OrgID header on every query — required by multi-tenant Grafana Mimir and Cortex. Omit for vanilla Prometheus.
timeout_seconds HTTP timeout for Prometheus queries. Default: 10.
default_range_minutes Default lookback window for range queries. Default: 60.

Authentication#

Prometheus itself has no bearer-token authentication — there is nothing to obtain from Prometheus. A bearer token only exists when something in front of Prometheus checks it, so start by identifying your setup:

curl "http://<prometheus-host>:9090/api/v1/query?query=up"

Answers without credentials — plain Prometheus. The default for most self-hosted setups (including a vanilla kube-prometheus-stack inside the cluster network). Omit bearer_token_env; base_url is the whole configuration.

Rejected, and you run a reverse proxy (nginx, Traefik, oauth2-proxy, …) in front of Prometheus. The token is one you make up, exactly like the agent's own webhook tokens:

  1. Generate a long random secret (any generator works).
  2. Configure the proxy to require it as Authorization: Bearer <secret>.
  3. Set bearer_token_env: PROMETHEUS_BEARER_TOKEN in the agent config and export PROMETHEUS_BEARER_TOKEN=<secret> in the agent's environment.

Rejected, and the cluster's monitoring stack is managed (OpenShift cluster monitoring, or any Prometheus fronted by kube-rbac-proxy). The token is a ServiceAccount token issued by the cluster:

  1. Create a ServiceAccount and grant it metrics-read permission — on OpenShift, the cluster-monitoring-view cluster role.
  2. Mint a token: kubectl create token <sa> -n <namespace> --duration=8760h. The default duration is one hour — too short, because the agent reads the token once at startup.
  3. Point base_url at the authenticated query endpoint — on OpenShift that is Thanos Querier on port 9091, not Prometheus directly — and wire the token through bearer_token_env as above.

Whatever the source, verify the pair before starting the agent:

curl -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

The token is read once at serve startup; after rotating it, update the env var and restart the agent.

Multi-tenant Mimir and Cortex#

Grafana Mimir (and Cortex) are multi-tenant: every query must carry an X-Scope-OrgID header naming the tenant, or the request is rejected even with valid credentials. Set org_id to your tenant ID — it is an identifier, not a secret, so it lives inline in the config. It is independent of bearer_token_env: use either, both, or neither depending on what sits in front of your endpoint.

Verify the pair the same way:

curl -H "X-Scope-OrgID: <tenant>" \
  -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

MCP tools#

When Prometheus is enabled, two additional tools become available to your MCP client:

Tool Description
prometheus_query Instant PromQL query. Parameters: expr (required), time (optional ISO 8601).
prometheus_query_range Range PromQL query with auto-stepped resolution. Parameters: expr, start, end (ISO 8601), step (optional).

prometheus_query and prometheus_query_range are backend-native passthrough: whatever PromQL the connected agent sends goes straight to Prometheus. There is no local syntax check and no repair attempt on this path — a malformed expression comes back as whatever error Prometheus itself returns, verbatim, for the calling agent to read and correct. That's a deliberate boundary, not an oversight: this is a human or a connected agent driving an open-ended investigation, and the tool's job is to report exactly what the backend said, not to intercept or second-guess the query. It also means a non-vanilla backend's own extensions keep working — a VictoriaMetrics MetricsQL function like median_over_time, say, that the standard PromQL grammar doesn't recognize still runs fine through these tools, because nothing here parses it against that grammar first.

The bundled local validation described in Verification round is wider than just the model's own checks: it covers every PromQL expression the unattended pipeline runs as a check — the model's disprove-queries, a governing correction's operator-sourced steering checks, and the widen queries run once at verdict capture all go through the same local parse (the deterministic floor's own up-ratio and incidents-in-window checks build fixed queries of their own and carry no operator- or model-authored expression to validate, so this step never applies to them). The one-shot repair call is narrower: it only ever fires for locally-invalid model-proposed PromQL — a fixed, no-human-in-the-loop pipeline where a malformed query would otherwise either burn a Prometheus round-trip on something that can never succeed or silently drop a check with no chance to fix it. A locally-invalid operator-sourced or capture-widening query is marked invalid the same way but is never sent to a repair call — a human authored it, so the honest move is to surface the failure, not have the model guess at a fix. Neither local validation nor repair ever touches these two MCP tools: prometheus_query and prometheus_query_range queries go straight to Prometheus exactly as sent, and the verification round never calls either tool.

Example queries#

Ask your agent in natural language — AlertINT handles the PromQL via MCP:

Query CPU usage for instance api-1 right now.
Show me the error rate for the last 30 minutes.
What was the latency trend during this incident?

Or pass PromQL directly to the prometheus_query tool:

cpu_usage_percent{instance="api-1"}
rate(http_errors_total[5m])