Can you catch an AI’s mistakes by looking inside its head?

By Sanne Wielinga & Jorrit van der Laan, KickstartAI

Anyone building products on top of large language models (LLMs) runs into the same problem: LLMs can be unpredictable and do not always do what you want. They can either generate things that are not true (what we call “hallucinating”) or just not follow prompt instructions the way you ask them to. Battling this typically involves a second AI system checking every answer before it goes live (“guardrailing”), but this is both slow and expensive. And in privacy-sensitive settings, where models may need to run locally on more limited hardware, adding another model is not always practical.

That’s why a recent idea from AI research caught our attention: what if you could catch an undesired output before it was even generated?

The theory is that a language model’s internal state shows subtle signs of trouble when it’s about to make something up. You can imagine the internal state as the human equivalent of a series of thought signals before speaking. The idea is to extract the model’s internal signals and train a small system to distinguish between patterns associated with normal responses and with those that tend to appear before a hallucination. This is called probing a model.

If you can make the distinction, you can use this probe to catch errors before an answer even reaches the user. Fast, cheap, no second AI system required.

Not as obvious as it sounds

There is one big assumption behind this idea: that something changes inside the model when it is about to hallucinate.  You might imagine the model being somehow “in turmoil” when it hallucinates, and that this turmoil is visible in its internal state. However, that assumption is often contested in the research literature. A key argument is that an LLM considers a hallucination simply to be the truth, meaning that there’s no error  to detect. But other researchers argue that  the degree of confidence behind a  hallucination could still be probed.

We tested the probing approach ourselves in two very different settings: a prototype banking chatbot, and an on-premise medical AI assistant. In the end, both experiments led us to the same conclusion. 

The banking chatbot

Jorrit had been working on the prototype of a financial health chatbot as part of a controlled pilot setting. Since financial health is a sensitive topic, careful phrasing and consistent prompt adherence were especially important. This made the chatbot a relevant use case to explore whether probing could offer an additional way to detect potential prompt violations – without relying on guardrail to check every single response. Extracting internal states, however, requires access to a model’s raw mechanics. Closed models – such as Gemini, on which the chatbot ran – do not provide access to these layers, meaning probing only works on open-source models.

To still put the probe-theory to the test, Jorrit swapped Gemini for an open-source model. Further narrowing the scope, he focused on a single prompt instruction that was often violated during the experiment: “always ask a follow up question.” He created 50 different conversation starters, let the chatbot generate an answer for each of them, and recorded whether it would forget a follow-up question or not. Then he probed the internal state to see if it could predict which conversations would violate the instruction.

Contrary to what he was expecting, the probe could not distinguish between conversations that did and did not violate the prompt. One possible explanation is that the model simply had not decided yet whether it would ask a question at the start of each response. If that is the case, trying to predict the outcome is similar to trying to predict a coin flip. 

By then, it was clear that predicting violations directly from the internals was not easy. The same conversation could lead to a violation one time, but not the next. However, Jorrit figured that some conversations could intrinsically be more likely to result in a violation than others. So rather than predicting a violation directly, he set out to predict the probability of a violation for each conversation as a whole – eliminating the volatility of each individual answer.

Knowing how likely a violation is before generating an answer, makes it possible to use this information as some kind of pre-guardrail. Risky conversations would still go through the slower LLM-as-a-judge, while the rest could pass without the extra check.

To create such a probe, Jorrit generated five answers per conversation starter instead of only one like before. Again, he recorded whether the chatbot forgot to ask a follow-up question, giving each conversation a score from zero to five. This is the same approach Sanne later leans on: asking the model the same question many times and recording the variation in the answer.

In the graph below you can see the internals of the model that contain information about this probability. You can see that riskier conversations live on the right side, and safer ones on the left. If all the way to the left, it can pass through. More to the right, activate the real guardrail.

This is exactly what we need to put our probe in practice. If we now get a new conversation, we simply look where it lands in this graph.

Next to this detection use case, you could also use this approach to test what conversation style your prompt doesn't work well for. Simply generate different conversation styles, and see in what area of the brain they land. If you find a weak spot, update the prompt. 

That being said, training such a probe and, in particular, generating all the conversations is a lot of work, and for each prompt iteration you’d have to create a new probe. Also, proper research needs to be done before you can conclude this really works. And you definitely need an open source model which is suitable for your use case. 

Moreover, further development of the chatbot showed for the follow-up question issues, flow engineering is much more effective than probing. After answer generation, simply add a deterministic step that makes sure the question is always asked.

The medical AI

Sanne’s setting was a bit different. Imagine a Dutch hospital pilots an AI assistant powered by a small language model – small meaning far less powerful than the giants behind ChatGPT, but compact enough to run on a workstation inside the hospital. A patient on Lisinopril for high blood pressure asks if they can take Ibuprofen for their back pain. The AI confidently answers yes. 

The answer is wrong, and the consequences can be serious: Ibuprofen blunts the effect of ACE inhibitors (medicines like Lisinopril) and can strain kidney function. But the AI didn’t say “I’m not sure.” It just sounded convincing.

This is the problem Sanne has been working on: how do you detect when a medical AI is wrong, especially when the AI system itself seems sure?

First, why is the model small at all? Most of the impressive AI in the headlines runs in the cloud, on enormous models. Healthcare data, though, can’t easily go to the cloud. GDPR treats health information as a special category, and for a Dutch hospital the realistic path to AI assistance is often a small model running locally, with no patient data leaving the building. Smaller models are accessible, compliant, and within reach of any hospital’s budget and hardware. They are also more prone to hallucinations than the bigger ones, which makes detecting their mistakes more important.

Sanne tested two ways of asking the model what it already knows. The first - a similar idea Jorrit used to for the banking chatbot - poses the same question ten times: if the answers genuinely disagree in meaning, that's a sign of uncertainty. 

The second is the internal probe – a peek at the model’s internal state during the answer itself. Neither works well alone. The first fails exactly where you most need it: when the model is confidently wrong, giving the same wrong answer ten times in a row, no uncertainty in sight. The probe, on the other hand, catches much of that, but misses cases where the model is indecisive between plausible answers. The two signals fail in different situations however, which is the perfect condition for combining them. Across four models and two medical benchmarks, the combination outperformed either signal alone in six of the eight cases. 

The most interesting result here wasn’t a number. Working with a medical expert, Sanne sorted the errors into clinical categories, and an asymmetry appeared. Errors like the Ibuprofen example get flagged reliably, and the framework’s alarm signal for these sat, on average, more than three standard deviations above its normal range. But subtle conceptual errors are nearly invisible: an AI claiming that major depressive disorder requires two months of symptoms instead of two weeks looks, to the probe, just like an AI getting it right.

Clinically, this asymmetry matters. The framework is strongest exactly where the stakes are highest, such as errors that could cause immediate harm, like the Ibuprofen example. It is weakest on subtle mistakes, such as answers that swap one plausible detail for another, like two months versus two weeks. Those can still be harmful, which is exactly why the framework is a risk indicator and not a hallucination eliminator: a tool that tells a clinician “you should probably double-check this one”, instead of deciding on its own whether to suppress an answer. And back to the hospital chatbot: this does mean the AI can’t answer patients unsupervised.

What the framework changes is the economics of supervision - instead of a clinician re-checking every answer, the risk flags route only the suspect ones to review, and the highest risk errors are exactly the ones it flags most reliably.

What we both learned

Probing is not a free hallucination detector. On closed models, you can’t really probe at all. On open models, what the probe sees depends on where you look: risk in the context, or trouble in the answer itself.  And on small models in particular, one signal alone isn’t trustworthy, but combined signals, used with humans in the loop, are. 

Where it could help today: cheap pre-guardrails, finding risky conversation types while iterating on prompts, and calibrated risk flags that route answers to human review. Where it doesn’t help yet: closed-source models, confident errors that also look right to the probe, and subjective or context-dependent failures. 

That’s a less thrilling headline than “AI lie detector”, but probing can still be something you can actually build with. And in settings like healthcare and finance, it matters beyond convenience: small models can be made meaningfully safer with the right reliability layer on top. Not safe enough to remove the human, but safe enough to be worth deploying. 

Want the full methodology, numbers, and charts? A technical companion post will follow soon!/ Sanne’s underlying research was supervised by Dr. Clara Maathuis (Open University of the Netherlands) and Prof. Natasha Alechina (Open University of the Netherlands and Utrecht University). The paper has been accepted at AIES 2026. 

Help accelerate AI adoption in the Netherlands for societal impact. Join our community and follow us on LinkedIn.