Jev and AlertINT learned the same lesson about AI

Ask a chatbot a simple question and you usually get a lovely paragraph back. For a person, that's great. For software, it's a problem.

Software doesn't want a paragraph. It wants yes or no. A number. One option out of three. So every time a program asks an AI something, someone has to write code that reads the paragraph and guesses what it meant. Sometimes the answer comes back in a slightly different shape and the code breaks. And the AI sounds just as sure of itself when it's guessing as when it actually knows.

Imagine a colleague who answers every yes-or-no question with a two-page memo, in the same confident tone whether they checked or not. Smart, sure. But not someone you'd build a process around.

Jev: an AI that fills in a form#

That's the problem TypeSafe went after with Jev, which launched on September 15. Their founder asked a good question: AI has been amazing at conversation for years, so why isn't more software actually running on it?

Their answer is a model that never writes a sentence. You describe the situation and ask a short question, with the possible answers decided in advance. Which of these three categories is this? How risky is it, on a scale? Jev picks one and tells you how sure it is.

That "how sure" part matters more than it sounds. TypeSafe says Jev's confidence is calibrated, which is a fancy way of saying honest: when it's more confident, it's more often right. Think of a weather forecast where "70% chance of rain" really means it rains on about 70% of those days.

Everything after that is up to normal, predictable code. Jev gives the answer, and the program decides what to do with it. In TypeSafe's words: "The surrounding code constrains their freedom."

Software doesn't need an essay. It needs a checked box, and an honest idea of how sure the AI is.

We learned the same thing, the slow way#

AlertINT helps the people who get woken up when a company's systems break. Alarms go off, and AlertINT works out what's probably going on and tells the person on call. (side-note: AlertINT doesn't use Jev. It runs on whichever AI model a team already uses.)

We started where most AI tools start: the AI reads the evidence and writes up what it thinks happened. Then, one surprise at a time, we ran into the same things TypeSafe describes.

The AI sounded just as sure whether it had real evidence or was guessing. Our first fix was to ask it nicely, in its instructions, to be less confident when it had nothing to go on. Five days later we gave up asking and wrote a rule instead: no evidence, capped confidence, whatever the AI says. Its answers also sometimes came back in the wrong shape and quietly broke things further down the line. So more and more of what it tells us became fixed choices instead of free text.

Put Jev's launch post next to our changelog from this summer and they land on the same few ideas:

• Ask the AI for answers you can check, not essays.
• Don't take its word for how sure it is.
• Let the AI advise, and let clear, predictable rules make the final call.

Where one good answer isn't enough#

Jev is very good at one question. But a lot of real problems aren't one question. They unfold over time.

Think of a patient in hospital overnight. One good diagnosis is great. What actually keeps the night safe is the chart: what's been decided, by whom, what's changed since, and which earlier notes no longer apply. Without a chart, every new doctor starts from zero, asks the same questions again, and might come to a different answer each time.

We saw exactly that in our own testing. An AI was asked about the same problem every minute, while nothing about the problem was changing. Without its earlier answer in front of it, it kept changing its mind. By the end of the morning it had posted 108 "conclusion changed" messages.

Every answer looked fine on its own. The problem was between the answers.

That's what AlertINT 0.14 is about. Every problem now gets its own chart: one running story that remembers what people decided, notices when a decision no longer fits the facts, and doesn't bother the AI again when nothing has changed. A few rules sit underneath, and none of them are clever AI:

• Not knowing isn't the same as "all fine". If the monitoring goes quiet, that's a gap, not good news.
• "We expected this" doesn't make a problem go away. It only changes how loudly we talk about it.
• A late answer doesn't overwrite newer facts.

Same belief, different ends#

Jev and AlertINT come at this from different ends. Jev makes a single AI decision reliable. AlertINT tries to keep a whole string of decisions reliable over hours. Underneath, it's the same belief: AI is at its best as an advisor working inside clear rules, not as an improviser left on its own.

The AI advises. The rules decide.

I think that's where a lot of useful AI is heading. Less "write me something clever", more "give me a straight answer I can check, and remember what we already know".

If you want to see what this looks like in practice, AlertINT is on GitHub, and the docs walk through the whole lifecycle of a problem. Questions, or a problem you think fits? Email me at ernests@alertint.com.