Prometheus

Two separate pieces, both read-only toward your stack:

Piece	Role	Gives you
Alertmanager	Alert source	A webhook copy of every alert, firing and resolved
Prometheus connector	Evidence source	Live metric values in triage, verification checks, and PromQL tools over MCP

The connector only queries. It never writes metrics, creates rules, or changes Prometheus.

Alertmanager: the alert source#

Add a receiver and route to it (full example: examples/alertmanager.yml):

route:
  receiver: alertint-agent
  group_by: [alertname, cluster, namespace, service]
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 4h

receivers:
  - name: alertint-agent
    webhook_configs:
      - url: "http://<agent-host>:9911/webhook/alertmanager"
        send_resolved: true
        http_config:
          authorization:
            credentials_file: /etc/alertmanager/alertint_token

• Token: must match the variable named by alertmanager.webhook_token_env (default ALERTINT_WEBHOOK_TOKEN). Use credentials_file so it isn't stored in alertmanager.yml.
• send_resolved: true is required. Resolutions are what move a Situation to Confirming recovery and Recovered.
• Keep your paging: add alertint-agent as a child route with continue: true instead of replacing your top-level receiver.

Reusable expected schedules#

To mark a recurring Alertmanager alert as expected on a schedule, map it to its exact Prometheus rule:

alertmanager:
  instance_id: prod-alertmanager-eu
  rules:
    - alert_name: ReconciliationLoad
      group: batch-jobs
      rule: ReconciliationLoad
      scope_labels: [service]
prometheus:
  instance_id: prod-prometheus-eu

• Needs the connector (prometheus.base_url) to prove the rule is active.
• Unmapped alerts go through normal triage.
• Change identities in a quiet window: a new Alertmanager instance_id starts a new delivery and episode namespace; a new Prometheus instance_id starts a new rule scope.
• pending or firing in Prometheus both count as "the rule is active". The Alertmanager webhook is still what fires or updates a Situation.

Not available when the mapped rule is missing or ambiguous, its definition can't be read, it depends on a recording rule, or Prometheus has active alert_relabel_configs. AlertINT never infers a rule from generatorURL, externalURL, alert name, or fingerprint, and never claims which rule version produced an older delivery.

Prometheus connector: live metrics#

Configure#

prometheus:
  base_url: http://localhost:9090            # setting this turns the connector on
  # enabled: false                           # force off despite base_url
  bearer_token_env: PROMETHEUS_BEARER_TOKEN  # optional; see Authentication
  # org_id: tenant-1                         # multi-tenant Mimir/Cortex only
  timeout_seconds: 10                        # default
  default_range_minutes: 60                  # default

Field	Default	Does
enabled	auto	On when base_url is set; false forces off
base_url	—	Prometheus API root, e.g. http://localhost:9090
bearer_token_env	—	Only if something in front of Prometheus checks a token
org_id	—	Tenant sent as X-Scope-OrgID (Mimir, Cortex)
timeout_seconds	10	Query timeout
default_range_minutes	60	Default range-query look-back

Compatible backends: anything serving /api/v1/query and /api/v1/query_range.

• Thanos: point base_url at the Querier (default port 10902), not a Sidecar or Store Gateway. You get the full retention across all connected Prometheus instances.

Authentication#

Prometheus has no token auth of its own; a token exists only if something in front of it checks one. Test first:

curl "http://<prometheus-host>:9090/api/v1/query?query=up"

Result	Your setup	Token
Answers	Plain Prometheus (e.g. kube-prometheus-stack in-cluster)	None: omit bearer_token_env
Rejected	Your own reverse proxy (nginx, Traefik, oauth2-proxy)	One you invent, as below
Rejected	Managed stack (OpenShift monitoring, kube-rbac-proxy)	A ServiceAccount token, as below

Your own proxy:

1. Generate a long random secret.
2. Make the proxy require Authorization: Bearer <secret>.
3. Set bearer_token_env: PROMETHEUS_BEARER_TOKEN and export PROMETHEUS_BEARER_TOKEN=<secret>.

Managed stack:

1. Create a ServiceAccount with metrics-read permission (on OpenShift, cluster-monitoring-view).
2. Mint a long-lived token: kubectl create token <sa> -n <namespace> --duration=8760h. The default one hour is too short: AlertINT reads the token once at startup.
3. Point base_url at the authenticated endpoint (on OpenShift, Thanos Querier on port 9091) and wire the token through bearer_token_env.

Verify before starting the agent:

curl -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

After rotating the token, update the variable and restart the agent.

Multi-tenant Mimir and Cortex#

Every query needs an X-Scope-OrgID header, or it's rejected even with valid credentials.

• Set org_id to your tenant ID. It's not a secret, so it goes inline.
• Independent of the token: use either, both, or neither.

curl -H "X-Scope-OrgID: <tenant>" \
  -H "Authorization: Bearer $PROMETHEUS_BEARER_TOKEN" \
  "<base_url>/api/v1/query?query=up"

How evidence queries are built#

When an Incident is ready for triage, AlertINT queries metrics at its start time:

1. Selector from the labels all member alerts share, filtered to an allowlist: namespace, service, job, pod, container, instance. This is what gives Kubernetes alerts (often without instance) real metrics.
2. Per-instance supplement: any alert with instance also gets its own per-instance query.
3. Zero-match retry: if the selector matches nothing, retry once with only namespace, pod, container, instance.
4. Top 10 non-system series per query, ranked by how many labels they share with the alerts, go into the prompt as Live metrics. Real numbers outrank annotation text.

Multi-cluster: triage.extra_selector_labels#

If one Prometheus or Mimir serves several clusters, cluster isn't in the allowlist, so queries mix clusters. Add topology labels:

triage:
  extra_selector_labels: [cluster]

For an Incident sharing cluster=eu-west, namespace=payments, service=checkout:

Query	Default	With [cluster]
Metrics	{namespace="payments",service="checkout"}	{cluster="eu-west",namespace="payments",service="checkout"}
Zero-match retry	{namespace="payments"}	{cluster="eu-west",namespace="payments"}
Per-instance supplement	{instance="10.0.4.7:9100"}	{cluster="eu-west",instance="10.0.4.7:9100"}
Logs (before label_map)	{namespace="payments",service="checkout"}	{cluster="eu-west",namespace="payments",service="checkout"}
Verification floor	{namespace="payments",service="checkout"}	{cluster="eu-west",namespace="payments",service="checkout"}

• Extras are never dropped by the retry. If your series lack the label, you get an empty result with the exact query in the log, never a silent widening to the wrong cluster.
• Only shared keys are used. A configured label missing from any member alert is left out.
• Metrics but not logs? Drop it for Loki with logs.loki.label_map: {cluster: ""} (see Loki).
• Topology, not identity: use cluster, region, datacenter. A pod-like label would shrink the verification floor to the Incident's own targets.
• Checked at startup: invalid names (cluster-1) and built-in duplicates (namespace) are rejected.
• Debug: --log-level=debug shows which alert labels the allowlist dropped.

Evidence line#

Every Finding carries a per-source evidence summary: how many metrics, log lines, changes, and Sentry issues fed the triage, e.g. Prometheus 21 metrics · Loki 0 lines · Changes 2 · Sentry unreachable. unreachable is distinct from a genuine 0, so a down connector is visible instead of silently lowering confidence.

MCP tools#

Tool	Parameters
prometheus_query	expr (required), time (optional, ISO 8601)
prometheus_query_range	expr, start, end (ISO 8601), step (optional; auto by default)

• Passthrough: your agent's PromQL goes to Prometheus exactly as sent, and Prometheus's own error comes back verbatim. No local check, no repair.
• Backend extensions work: e.g. VictoriaMetrics MetricsQL functions like median_over_time.
• Automated checks are different: PromQL the verification round runs is validated locally first; see Verification round. The round never calls these two tools.

Query CPU usage for instance api-1 right now.
Show me the error rate for the last 30 minutes.
What was the latency trend during this incident?

Or pass PromQL directly:

cpu_usage_percent{instance="api-1"}
rate(http_errors_total[5m])