AlertINT logoAlertINT
BlogGitHub

Integrations

Prometheus

Connect Alertmanager as the alert source and Prometheus for live metric context.

Two separate pieces, both read-only toward your stack:

Piece Role Gives you
Alertmanager Alert source A webhook copy of every alert, firing and resolved
Prometheus connector Evidence source Live metric values in triage, verification checks, and PromQL tools over MCP

The connector only queries. It never writes metrics, creates rules, or changes Prometheus.

Alertmanager: the alert source#

Add a receiver and route to it (full example: examples/alertmanager.yml):

route:
  receiver: alertint-agent
  group_by: [alertname, cluster, namespace, service]
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 4h

receivers:
  - name: alertint-agent
    webhook_configs:
      - url: "http://<agent-host>:9911/webhook/alertmanager"
        send_resolved: true
        http_config:
          authorization:
            credentials_file: /etc/alertmanager/alertint_token
  • Token: must match the variable named by alertmanager.webhook_token_env (default ALERTINT_WEBHOOK_TOKEN). Use credentials_file so it isn't stored in alertmanager.yml.
  • send_resolved: true is required. Resolutions are what move a Situation to Confirming recovery and Recovered.
  • Keep your paging: add alertint-agent as a child route with continue: true instead of replacing your top-level receiver.

Reusable expected schedules#

To mark a recurring Alertmanager alert as expected on a schedule, map it to its exact Prometheus rule:

alertmanager:
  instance_id: prod-alertmanager-eu
  rules:
    - alert_name: ReconciliationLoad
      group: batch-jobs
      rule: ReconciliationLoad
      scope_labels: [service]
prometheus:
  instance_id: prod-prometheus-eu
  • Needs the connector (prometheus.base_url) to prove the rule is active.
  • Unmapped alerts go through normal triage.
  • Change identities in a quiet window: a new Alertmanager instance_id starts a new delivery and episode namespace; a new Prometheus instance_id starts a new rule scope.
  • pending or firing in Prometheus both count as "the rule is active". The Alertmanager webhook is still what fires or updates a Situation.

Not available when the mapped rule is missing or ambiguous, its definition can't be read, it depends on a recording rule, or Prometheus has active alert_relabel_configs. AlertINT never infers a rule from generatorURL, externalURL, alert name, or fingerprint, and never claims which rule version produced an older delivery.

Prometheus connector: live metrics#

Configure#

prometheus:
  base_url: http://localhost:9090            # setting this turns the connector on
  # enabled: false                           # force off despite base_url
  bearer_token_env: PROMETHEUS_BEARER_TOKEN  # optional; see Authentication
  # org_id: tenant-1                         # multi-tenant Mimir/Cortex only
  timeout_seconds: 10                        # default
  default_range_minutes: 60                  # default
Field Default Does
enabled auto On when base_url is set; false forces off
base_url — Prometheus API root, e.g. http://localhost:9090
bearer_token_env — Only if something in front of Prometheus checks a token
org_id — Tenant sent as X-Scope-OrgID (Mimir, Cortex)
timeout_seconds 10 Query timeout
default_range_minutes 60 Default range-query look-back

Compatible backends: anything serving /api/v1/query and /api/v1/query_range.

  • Thanos: point base_url at the Querier (default port 10902), not a Sidecar or Store Gateway. You get the full retention across all connected Prometheus instances.

Authentication#

Prometheus has no token auth of its own; a token exists only if something in front of it checks one. Test first:

curl "http://<prometheus-host>:9090/api/v1/query?query=up"
Result Your setup Token
Answers Plain Prometheus (e.g. kube-prometheus-stack in-cluster) None: omit bearer_token_env
Rejected Your own reverse proxy (nginx, Traefik, oauth2-proxy) One you invent, as below
Rejected Managed stack (OpenShift monitoring, kube-rbac-proxy) A ServiceAccount token, as below

Your own proxy:

  1. Generate a long random secret.
  2. Make the proxy require Authorization: Bearer <secret>.
  3. Set bearer_token_env: PROMETHEUS_BEARER_TOKEN and export PROMETHEUS_BEARER_TOKEN=<secret>.

Managed stack:

  1. Create a ServiceAccount with metrics-read permission (on OpenShift, cluster-monitoring-view).
  2. Mint a long-lived token: kubectl create token <sa> -n <namespace> --duration=8760h. The default one hour is too short: AlertINT reads the token once at startup.
  3. Point base_url at the authenticated endpoint (on OpenShift, Thanos Querier on port 9091) and wire the token through bearer_token_env.

Verify before starting the agent:

curl -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

After rotating the token, update the variable and restart the agent.

Multi-tenant Mimir and Cortex#

Every query needs an X-Scope-OrgID header, or it's rejected even with valid credentials.

  • Set org_id to your tenant ID. It's not a secret, so it goes inline.
  • Independent of the token: use either, both, or neither.
curl -H "X-Scope-OrgID: <tenant>" \
  -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

Situation evidence and coverage limits#

For Alertmanager alerts, Situation preparation reads the configured Prometheus /api/v1/rules endpoint and looks for one uniquely named current alerting rule. When its expression yields exact metric names without recording-rule dependencies, those metrics are collected before broader diagnostics. Both reads preserve the same scope. The alert name remains available after a resolved delivery. If no firing member matches the scope, current-rule selection is explicitly marked as an advisory hint; it does not prove the historical rule, establish lifecycle state or grant reusable schedule authority.

  • Small background samples: optional diagnostics and Alertmanager scope fallback collect at most 32 series plus one overflow sentinel, preserving the original time resolution. Primary rule metrics keep the plan's series limit. Samples can omit whole metric families; a returned sentinel marks coverage incomplete. Large labels or primary queries can still reach the byte limits.
  • At most three source calls: rule lookup, primary metrics, then diagnostics. A two-call budget allows lookup and primary metrics; a one-call budget uses bounded scope collection.
  • Explicit fallback: ambiguous rules, unsupported expressions and lookup failures use bounded scope collection and record the reason.
  • Specific warnings: ntfy and Slack identify series-count, 16 KiB summary, and 1 MiB decoded-response limits with known counts and an observation reference. Overflow counts are lower bounds; rejected responses have unknown counts.
  • Earlier evidence survives a later diagnostic failure. Inspect details with alertint_list_observation_runs, using the Situation's id or handle. Older notifications say when the specific cause is unavailable.

How evidence queries are built#

When an Incident is ready for triage, AlertINT queries metrics at its start time:

  1. Selector from the labels all member alerts share, filtered to an allowlist: namespace, service, job, pod, container, instance. This is what gives Kubernetes alerts (often without instance) real metrics.
  2. Per-instance supplement: any alert with instance also gets its own per-instance query.
  3. Zero-match retry: if the selector matches nothing, retry once with only namespace, pod, container, instance.
  4. Top 10 non-system series per query, ranked by how many labels they share with the alerts, go into the prompt as Live metrics. Real numbers outrank annotation text.

Multi-cluster: triage.extra_selector_labels#

If one Prometheus or Mimir serves several clusters, cluster isn't in the allowlist, so queries mix clusters. Add topology labels:

triage:
  extra_selector_labels: [cluster]

For an Incident sharing cluster=eu-west, namespace=payments, service=checkout:

Query Default With [cluster]
Metrics {namespace="payments",service="checkout"} {cluster="eu-west",namespace="payments",service="checkout"}
Zero-match retry {namespace="payments"} {cluster="eu-west",namespace="payments"}
Per-instance supplement {instance="10.0.4.7:9100"} {cluster="eu-west",instance="10.0.4.7:9100"}
Logs (before label_map) {namespace="payments",service="checkout"} {cluster="eu-west",namespace="payments",service="checkout"}
Verification floor {namespace="payments",service="checkout"} {cluster="eu-west",namespace="payments",service="checkout"}
  • Extras are never dropped by the retry. If your series lack the label, you get an empty result with the exact query in the log, never a silent widening to the wrong cluster.
  • Only shared keys are used. A configured label missing from any member alert is left out.
  • Metrics but not logs? Drop it for Loki with logs.loki.label_map: {cluster: ""} (see Loki).
  • Topology, not identity: use cluster, region, datacenter. A pod-like label would shrink the verification floor to the Incident's own targets.
  • Checked at startup: invalid names (cluster-1) and built-in duplicates (namespace) are rejected.
  • Debug: --log-level=debug shows which alert labels the allowlist dropped.

Evidence line#

Every Finding carries a per-source evidence summary: how many metrics, log lines, changes, and Sentry issues fed the triage, e.g. Prometheus 21 metrics · Loki 0 lines · Changes 2 · Sentry unreachable. unreachable is distinct from a genuine 0, so a down connector is visible instead of silently lowering confidence.

MCP tools#

Tool Parameters
prometheus_query expr (required), time (optional, ISO 8601)
prometheus_query_range expr, start, end (ISO 8601), step (optional; auto by default)
  • Passthrough: your agent's PromQL goes to Prometheus exactly as sent, and Prometheus's own error comes back verbatim. No local check, no repair.
  • Backend extensions work: e.g. VictoriaMetrics MetricsQL functions like median_over_time.
  • Automated checks are different: PromQL the verification round runs is validated locally first; see Verification round. The round never calls these two tools.
Query CPU usage for instance api-1 right now.
Show me the error rate for the last 30 minutes.
What was the latency trend during this incident?

Or pass PromQL directly:

cpu_usage_percent{instance="api-1"}
rate(http_errors_total[5m])