Stop using a 100B LLM to answer yes or no

For about a year I had a language model doing a job it quietly hated. Every hour it read a pile of incoming leads and decided, one by one, whether each was worth our time. Pass or reject. A coin flip with reasons attached.

It was good at the reading. It was slow and expensive at the flip. A 120 billion parameter model, the kind that can write you a sonnet about a stapler, spending three and a half seconds and a paragraph of its own reasoning to arrive at a single word: reject. I was paying a novelist to flip coins, and he insisted on explaining each flip.

Then I read about a different kind of model called Jev, from a company named TypeSafe, got curious enough to try it on a real workload, and it quietly changed how I think about half the "AI features" I have ever shipped.

Most AI in production is a decision, not an essay

Look at what your model actually returns. Route this ticket. Score this lead. Is this comment toxic. Which of these five buckets does it belong in. Does this clause need a lawyer. None of that is writing. All of it is a typed answer wearing a costume made of text.

We reach for a big text model because it is there and it understands the question. Then we throw away everything that makes it a text model and squeeze the answer back down to a boolean. We rent a Ferrari and use it to hold a door open.

A decision model does the opposite. Jev calls itself a System One model, the fast reflexive kind, as opposed to the slow deliberate System Two that a chat model runs on every token. It does not write. It takes your input and a set of typed questions and returns typed answers with calibrated confidence. Yes, at 0.94. A score. A choice from a list. That is the entire output. No prose, no preamble, no "certainly, here is".

The first time I ran one it answered in half a second for a twentieth of the cost, and I felt faintly robbed, the way you do when a specialist fixes in one minute the thing that ate your whole weekend.

The move that makes it work: let the model report, let the code decide

Here is the part I wish someone had told me on day one. Do not ask the model for the verdict. Ask it for the facts.

The tempting design is "read this and give me a score out of 100". The model then does two jobs at once, one well and one badly: it reads (well) and it does arithmetic in its head (badly, and differently every time). I once had two near identical inputs score 90 and 38 for the same stated reason, because the model was vibing the number rather than computing it.

The better design splits the work. The model reports typed facts. Is there real work here. Does this trip a hard blocker. Is the budget under the line. Yes or no, each one. Then plain code, a boring function you can unit test, adds it up. The model never sees a total. It just answers questions.

And the moment your task becomes a list of yes or no questions, you are standing exactly where a decision model lives. The refactor that makes your pipeline testable is the same refactor that lets you drop in a model twenty times cheaper. You do not get a two for one like that very often.

Then you tune, and the model does exactly what you asked, which is the whole problem

This is where I lost a day and most of my dignity, so let me spend them for you.

A typed question is a tiny prompt, and a loose one misfires in style. I had a question that meant "is this unpaid or grant funded work". I wrote it casually. The model cheerfully flagged it on half of everything, because a perfectly normal budget that simply was not spelled out looks, to a literal reader, like money that is not committed. The model was not wrong. My question was. I had asked "is money uncertain here" and gotten a correct, useless yes.

The fix is to write each question like a lawyer who assumes you are trying to trick them. True only if it explicitly says X. A stated range is not this. Default to no. The discomfort you feel writing "only if" is precisely the discomfort the model needed you to feel.

Measure against a human, not against the model you are replacing

The comfortable mistake is to grade the new model against the old one and throw a party when they agree. All that measures is whether your new hire copied your old hire's homework, wrong answers included.

Sit down and label a set by hand. That is your answer key. Grade both models against you. Do not skip this because it is tedious. It is the only number that means anything, and as a bonus it will tell you that a few of your own snap labels were wrong, which is humbling and exactly the point.

The small sample will lie to you, warmly

I tuned on twenty carefully chosen cases and got a lovely score. Ship it? No. I ran it across the whole pile, and the two models agreed only seventy percent of the time. My tidy little sample had been hiding the truth: the new model was far stricter.

For one heart stopping minute that reads like a disaster. Then you audit the disagreements by hand and the story flips inside out. On the cases where they clashed, the strict model was right about twice as often. It was rejecting the junk the old one had been waving through for a year. The thing that looked like a regression was the feature I had wanted all along.

The lesson is not "strict is better". The lesson is that a curated sample is a flattering mirror, and only the full run plus a hand audit of the disagreements tells you what you actually built.

Know the shape of your tool, then stop sanding it

Decision models are calibrated but coarse. Ask for a smooth zero to a hundred and you get something that grades in chunky steps, which will underrate a genuinely strong case by a hair and tip it over a threshold. That is not a bug you can prompt away. It is the shape of the thing. Use it for the crisp calls it is built for, and keep the soft gradient somewhere else.

And when you catch yourself rewording a question for the fourth time to rescue the same two stubborn cases, stop. Those two are almost always genuinely borderline, the ones where two careful humans would also disagree. Chasing a spotless score on a fixed set is not learning, it is memorising. Ship the honest ceiling and move on.

The shape I landed on

The decision model, Jev in my case, makes the call on every item, fast and nearly free. For the handful of items that actually reach a human, a cheap text model writes the one line of explanation, taking the decision as its input. The reflex does the deciding, the writer only writes, and the combined bill is still a rounding error next to one big model doing everything itself.

My novelist is back to writing. The coin flips went to something that was built to flip coins, and it does the job in half a second without ever telling me how it feels about the result.