OpenAI-compatible endpoint

AlertINT can run triage against any self-hosted endpoint that speaks the OpenAI chat-completions wire format — SGLang, vLLM, Ollama, LM Studio, and anything else implementing POST /v1/chat/completions. Set llm.provider: openai-compatible and point llm.base_url at it: nothing leaves your network, completing the self-hosted story alongside your own Prometheus, Loki, and Alertmanager.

One provider serves an install. There is no routing, no fallback, and no mixing Anthropic and a local endpoint on the same deployment — pick one at config time (see scope and limits).

Configuration#

llm:
  provider: openai-compatible
  base_url: http://localhost:30000     # endpoint root; a trailing /v1 also works
  model: qwen3-32b                     # the model name your endpoint serves
  max_tokens: 4096
  timeout_seconds: 300                 # local decode is slower, and a storm's concurrent triages share the GPU
  # api_key_env: ALERTINT_LLM_API_KEY  # only if your endpoint requires auth
  # response_format: json_object       # default; set "off" if your runtime lacks enforced-JSON output
  # thinking: false                    # default. If you enable it, raise max_tokens to 8000–16000 —
  #                                    # thinking output competes with the reply for the token budget,
  #                                    # and an overrun fails triage with
  #                                    # "llm: response truncated at max_tokens" in the logs.
  # reasoning_effort: low               # adaptive-reasoning models only (e.g. Qwen3.8: xhigh/medium/low).
  #                                    # Requires thinking: true — it sizes the thinking budget, and the
  #                                    # pair {enable_thinking: false, reasoning_effort: "low"} is rejected
  #                                    # at config load rather than silently ignored by the model. Sent as
  #                                    # chat_template_kwargs.reasoning_effort alongside enable_thinking;
  #                                    # passed through unvalidated, so use whatever vocabulary your
  #                                    # model's chat template accepts. Empty (default) omits the field.

The memory shadow classifier (when enabled) follows the same endpoint and reuses llm.model — a single-model local install has nothing cheaper to route classifier calls to. With llm.thinking: true the classifier call automatically gets a larger completion ceiling (8192 tokens) so reasoning output cannot truncate the verdict; a truncated classifier reply is noted explicitly in the logs and fail-opens to an "unsure" verdict.

SGLang#

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-32B \
  --port 30000

SGLang serves an OpenAI-compatible API by default, including native response_format: json_object support, so the config block above works unmodified.

Ollama#

llm:
  provider: openai-compatible
  base_url: http://localhost:11434     # either spelling works: with or without /v1
  model: qwen3:32b                     # the model name as pulled, e.g. `ollama pull qwen3:32b`

Honest quality note#

Triage calibration — the verification round's contrast queries, confidence language, and the overall finding quality — is tuned against Claude. A self-hosted endpoint works with any instruct model your runtime serves, but we recommend a ≥30B-class instruct model for triage-grade reasoning. The Anthropic path remains the reference setup this project is built and tested against; treat a local endpoint as a self-hosted alternative with its own tuning curve, not a drop-in equivalent.

Prompt caching#

This wire format has no client-side cache_control breakpoint, so Prompt.CachePrefix is a no-op here — every call sends the full prompt. SGLang and vLLM both prefix-cache server-side automatically (radix-tree / KV-cache reuse across requests with a shared prefix), so the verification round's shared-prefix design still pays off; it just happens on the serving side instead of being billed as a discrete cache read like Anthropic's API.

Troubleshooting#

llm: api error: HTTP 400 ... (set llm.response_format: "off" ...) — your runtime does not support enforced-JSON output. Set llm.response_format: "off" and rely on the model's instruction-following instead.

llm: response truncated at max_tokens=N (raise llm.max_tokens) — the reply was cut off before the JSON closed. Raise llm.max_tokens, or if llm.thinking: true is set, either raise it further (8000–16000) or disable thinking — reasoning output competes with the JSON reply for the same token budget.

WARN llm: endpoint rejected chat_template_kwargs; … once after startup — the endpoint does not accept chat_template_kwargs, the field self-hosted runtimes read for the thinking toggle (some hosted APIs, such as Mistral's, answer 400 or 422). AlertINT resends that request without the field and, once the endpoint accepts it, leaves the field off until the next restart, so analysis carries on; llm.thinking and llm.reasoning_effort have no effect on such an endpoint. If llm.budget.calls_per_hour declines the resend, the next permitted request goes without the field instead: with a one-call budget, the first hour's call is spent on the refusal.

404 on the very first triage call — llm.base_url is pointed at the wrong path. The client always appends /v1/chat/completions itself; set base_url to the endpoint root (http://localhost:30000), not a path that already ends in /v1/chat/completions.