Hallucination detection for small on-prem medical LLMs: what works, what doesn’t, and why
By Sanne Wielinga
If you’re building an LLM product in healthcare, sooner or later you have to stop sending data to OpenAI. GDPR doesn’t really care about how good GPT-5 is. Health data is a special category under EU law, and for many hospitals, especially in Europe, the realistic path forward is a small model running on-premise. Something that fits on a single workstation, with no patient data leaving the building.
The problem is that ~1B-parameter models hallucinate. A lot. In the benchmarks I worked with, the four small models I tested produced answers that failed a correctness check 58 – 91% of the time. Some of those are partial-credit failures, some are dangerous. A small medical model that confidently mixes up a contraindication is, notably, a safety problem, not a UX problem.
So if you want to actually deploy these things, you need a reliability layer on top: something that flags risky outputs for human review. The question is what that layer should look like when the model is small, the data is sensitive, and you can’t bring in cloud-based judges.
This post walks through the framework from our AIES 2026 Conference paper (to be presented in October): a hybrid hallucination detector that combines two complementary intrinsic signals, with the full methodology, real numbers, and the part of the result I find most interesting – a subtype-level detectability asymmetry that shapes how this kind of system should be deployed. The research is joint work with Clara Maathuis and Natasha Alechina at the Open University of the Netherlands.
The setup
I evaluated four small architectures: Llama-3.2-1B-Instruct, TinyLlama-1.1B-Chat-v1.0, its domain-adapted medical variant (TinyLlama-1.1b-chat-medical), and Granite-3.1-1B. The model choice was deliberate, as it includes four different lineages, to check that conclusions generalize across the 1B class rather than reflecting one model’s quirks.
Two benchmarks were used. First, BioASQ, containing expert-level biomedical QA, and MedQuAD, containing patient-facing QA. These capture two distinct clinical communication modalities: expert-to-expert and patient-to-system. Hardware included a single RTX 2080 Ti with 11 GB of VRAM. The whole pipeline, including generation and detection, runs on what a clinical IT team can realistically procure. If the framework needs an A100 cluster, it doesn’t solve the deployment problem.
Formalizing the problem
Let M be a language model that produces a response y given a medical query x. The objective is to learn a risk scoring function which can be used to abstain or escalate when risk exceeds a safety threshold.
S(x, y) ∈ [0, 1], interpreted as P(Incorrect | x, y)
For training, I need a correctness label. Collecting expert annotations at the scale of 11,750 MedQuAD questions is not feasible, so I use a proxy target z ∈ {0, 1} derived from automated metrics (z = 1 denotes “incorrect”). Primary results use a SQuAD F1 < 0.5 threshold against the reference, and BERTScore F1 < 0.85 serves as a sensitivity check.
These proxies are noisy, however. They conflate factual error with paraphrase, mark valid reworded answers as incorrect, and under-penalize answers that are semantically close but clinically wrong. The working assumption is that a supervised probe can still learn generalizable patterns of error under noisy supervision, and that absolute metrics should be read with caution while qualitative findings (especially subtype-level patterns) should be cross-checked against expert annotation.
Put together, the pipeline looks like this:
Two ways to look for trouble inside a model
Farquhar et al. (2024) ask the model the same question multiple times at a higher temperature, then cluster the answers by meaning (not surface form) using a bidirectional entailment check from an NLI model. The intuition is that a confident model says the same thing in different words; an uncertain model hedges across genuinely different answers.
Formally, generate N stochastic samples Y = {y⁽¹⁾, ..., y⁽ᴺ⁾} at high temperature (I used N = 10, T = 0.8). Define a semantic equivalence relation y⁽ⁱ⁾ ∼ y⁽ʲ⁾ iff the two responses mutually entail each other. This partitions Y into K semantic clusters {C₁, ..., Cₖ}. The probability mass of cluster Cₖ is estimated by marginalizing over its constituent sequences:
p(Cₖ) = Σ_{y∈Cₖ} p_θ(y|x) / Σ_{y′∈Y} p_θ(y′|x)
The Semantic Entropy is the Shannon entropy over the cluster distribution:
SE(x) = − Σₖ₌₁..K p(Cₖ) log p(Cₖ)
When K = 1 (all samples semantically agree), SE(x) = 0. For comparability across varying K, I also use the normalized form NSE(x) = SE(x) / log K when K > 1. Higher SE means more dispersion over distinct semantic answers, which means a higher likelihood of hallucination.
For the entailment check I used BiomedNLP-PubMedBERT fine-tuned for NLI. Domain-specific NLI matters here, as a general-purpose NLI model handles medical terminology poorly and produces noisy clustering.
SE fails predictably when a model is consistently wrong: the same incorrect answer ten times in a row yields K = 1, SE = 0, and a higher error rate simultaneously.
Internal Signal Probe (IS)
The complementary idea, from Azaria & Mitchell (2023) and Orgad et al. (2025): generate the answer once at low temperature, grab the final-layer hidden state at the last generated token, and train a lightweight classifier to predict whether the answer is wrong. The hypothesis is that the model’s internal state encodes information about correctness that the surface output doesn’t.
Concretely: let h_T ∈ ℝᵈ denote the activation vector of the final transformer layer at the last generated token T. Train a supervised probe f_φ : ℝᵈ → [0, 1] to predict the proxy correctness label z:
IS(x) = f_φ(h_T) = σ(wᵀ h_T + b)
where σ is the sigmoid. I used logistic regression, with stratified K-fold cross-validation and out-of-fold probabilities retained for each instance. The out-of-fold step matters because IS scores feed into a downstream meta-learner. Without this step, target information would be leaked into the meta-features and the meta-learner overfits.
IS fails in a different way than SE: when the answer is genuinely uncertain at the surface (the model would benefit from hedging) but the final-layer hidden state doesn’t encode that uncertainty linearly. It can also be miscalibrated if training labels are noisy.
Because these failures occur in different cases, the idea was to start combining them.
The hybrid
SE is an unbounded entropy. IS is a [0,1] probability. They can’t be averaged directly. The fusion step normalizes both within each cross-validation fold using min-max scaling, with (u_min, u_max) computed on the fold’s training split only:
norm(u; x) = (u(x) − u_min) / (u_max − u_min)
Two variants:
Hybrid (Avg) is the mean of the two normalized scores:
H_avg(x) = ½ norm(SE; x) + ½ norm(IS; x)
No additional learned parameters beyond the IS probe.
Hybrid (Meta) trains a meta-classifier m_ψ over a three-element feature vector:
v(x) = [norm(SE; x), norm(IS; x), norm(RL; x)]
H_meta(x) = m_ψ(v(x)) ∈ [0, 1]
RL is response length in tokens, an auxiliary heuristic, because smaller models empirically hallucinate more when they ramble. I tried both logistic regression and random forest as the meta-learner. Both produce probabilities that go into the final risk score. Out-of-fold predictions, again, to avoid leakage.
I kept both variants because they answer different questions. Avg is the workhorse for discrimination, and Meta lets me probe how the relative importance of signals shifts across datasets, and its learned coefficients enable a global feature importance read-out.
What actually worked
Below are the headline numbers on the SQuAD F1 proxy, with bold marking the highest AUROC per row:
Hybrid (Avg) wins in 6 of 8 configurations. The gain over the better individual signal is modest, typically 1–3 AUROC points, but it’s consistent. Where Hybrid (Avg) doesn’t win, one of the individual signals is already strong enough that combining adds noise.
Two methodological observations are worth detailing some more.
Simple averaging usually beats meta-learning. This surprised me. I went in assuming random forest over three normalized features would extract more signal than equal weighting. But with only a handful of features that are already calibrated and known to be complementary, the meta-learner has too much capacity and overfits the CV folds, giving up some discrimination. There’s a methodological lesson here: when your features are few, calibrated, and complementary, averaging is already a strong baseline.
AUROC isn’t the whole story. The IS probe consistently produces well-calibrated probabilities (Expected Calibration Error < 0.06 across all configurations, often < 0.03). Calibration matters more than raw discrimination for this use case because you need probability thresholds for abstention. “Flag for review if P(Incorrect) > 0.4” only makes sense if 0.4 means what it claims to. The meta-learner, while sometimes matching IS on discrimination, is noticeably worse on calibration (ECE 0.035–0.075). For production, I would use the IS probabilities for abstention thresholds, and the hybrid averaged score for ranking what gets reviewed first.
A separate caveat is that practical utility, measured by AURAC (area under the rejection-accuracy curve – how much accuracy improves as you abstain on the riskiest predictions), is more limited than the AUROC numbers suggest. Generally 0.28–0.34 on MedQuAD and 0.54–0.63 on BioASQ. The reason is base accuracy: when the model is wrong 80%+ of the time, even a perfect detector can only preserve so much correct output. This framework is not a fix for low base accuracy, but rather a risk indicator that routes flagged outputs to humans.
The asymmetry that actually matters in a clinical setting
To get at clinical risk beyond a single binary label, I worked with a medical expert (15+ years of experience) to develop a taxonomy of medical factuality hallucinations and annotate a 480-question subset from each benchmark, resulting in five different subtypes. The expert annotation is explicitly exploratory at this stage, with n=1, no inter-rater reliability possible, framed as a first stage of a human-centric validation program. Multi-speciality validation is future work.
For each subtype, I computed the mean Z-normalized uncertainty score the framework assigned. A more negative score means the framework systematically assigned higher uncertainty to this subtype, and therefore detected the subtype more reliably.
Across both datasets and all detectors I found the following pattern: indication/contraindication errors get flagged well. On MedQuAD, the IS probe assigns those errors a mean Z-score of -3.22, more than three standard deviations below the population mean. On BioASQ, the score is -1.32. These are errors that can directly harm a patient, and the framework detects them well.
However, diagnostic criteria/definition errors are nearly invisible. Same scale, they sit at +0.09 for BioASQ and +0.14 for MedQuAD, essentially indistinguishable from a correct answer. An AI that claims major depressive disorder requires two months of symptoms instead of two weeks looks, internally, just like an AI getting it right.
Mann-Whitney U tests on these different subtypes give p < 10⁻⁴ across all detectors, with rank-biserial effect sizes from -0.47 to -0.70 (medium-to-large). The asymmetry is statistically robust, not a small-N artifact.
This shapes how the framework should be deployed. The instinct is to imagine a single risk score and a single threshold. The data argues for a tiered system instead. For example, a high IS score likely means a concrete factual error, which should result in fast escalation. A high SE score likely means a conceptual uncertainty, which should be prompted for review. The taxonomy of which signal is firing tells the clinician what kind of problem to look for, which matters when triage time is scarce.
It also means there’s a known blind spot, as the framework is least useful for the subtle conceptual errors that look correct from the outside. Clinicians need to know that, but the right answer isn’t to hide the limitation, but to communicate it as part of how the system is used. Recent work on accountability in healthcare AI (Bagave et al. 2025) argues that this kind of decision-level transparency is what makes joint accountability possible between developers and clinicians, rather than dumping the burden on either side.
The “confidently wrong” case
There’s a particular failure mode that the above asymmetry illuminates. Consider an output where the model gives a concise, logically consistent, but factually incorrect “Yes” to a contraindication query. SE is low (the model agrees with itself across samples). IS is low (the internal state encodes nothing unusual). The detector misses it, and SHAP attribution in this case shows how both signals contribute negatively to the risk score, pushing it well below threshold.
A parametric failure like this means the error is baked into the model’s weights, not produced by a noisy sampling step. No intrinsic uncertainty signal will catch it, because by definition there’s no uncertainty to detect.
This has led to two practical takeaways for me. First, intrinsic methods have a hard ceiling here. To catch parametric errors, you eventually need extrinsic grounding, meaning retrieval against a verified medical knowledge base, and structured guideline checks. Second, explainability isn’t just a nice-to-have, but part of how clinicians know when not to trust the framework. If the SHAP breakdown shows low contributions from every signal, that’s the system communicating “I had nothing to go on,” which is different from “I’m confident this is fine.”
A numerical reasoning aside
I also added a synthetic dosage calculation task, consisting of weight-based pediatric dosing with the formula provided in the prompt. The idea behind this was to test detection on purely numerical errors with unambiguous ground truth.
However, base accuracy was very low, with TinyLlama solving 20.4% of the problems, and Llama-3.2-1B managing only 0.6%. This matches known limitations of small LLMs on precise arithmetic (Mirzadeh et al. 2024).
But the IS probe did a good job at detection, with AUROC > 0.96 across all models, while SE collapsed to 0.52–0.77. SE’s weakness here is clear: a model that applies an incorrect rule consistently produces semantically consistent wrong outputs, so entropy stays low. IS picks up the confidence anomaly in the final layer state that SE has no way to see.
The generalized takeaway of this is that the internal signal is the right tool when errors come from misapplying explicit rules rather than from semantic uncertainty. An encouraging thought for deploying these methods in clinical-calculation contexts, such as dosing, scoring, or conversions.
What this isn’t
A few things this framework is explicitly not, in case the post above oversold any of them:
It’s not a hallucination eliminator. It’s a risk indicator. The base rate AURAC numbers make that clear.
It’s not validated for deployment. The expert annotation is exploratory, the proxy labels are noisy, and the framework hasn’t been tested on longer-form clinical tasks (summarization, discharge instructions, guideline drafting). Establishing generalizability across languages, tasks, and model scales is necessary before anything goes near a real patient workflow.
It’s not a substitute for RAG. The strongest production setup likely combines intrinsic detection (this) with extrinsic grounding (retrieval against verified medical sources). Neither alone catches everything.
It doesn’t address fairness. If proxy labels carry demographic or linguistic bias, the detector inherits it. This is a real risk for underrepresented patient populations and a priority for future work.
What’s next
The roadmap from here, in roughly the order I would prioritize:
Multi-expert annotation. N=1 is not enough, and multi-specialty panels with inter-rater reliability are the next concrete step.
Scaling up, to see if the hybrid’s advantage persists at 3B and 7B. At what scale does SE alone become good enough that fusion stops paying off?
Longer-form tasks. Summarization and discharge instruction generation are where it gets interesting.
Combine with RAG. Intrinsic + extrinsic is most likely the ideal combination for clinical deployment, especially for the parametric failure cases.
Fairness monitoring, across demographic groups and language varieties.
If you want to try the approach, the building blocks are accessible: a 1B open-weights model, the BiomedNLP-PubMedBERT NLI model for entailment clustering, sklearn for logistic regression probes, and any benchmark with reference answers. The full methodology is in the AIES paper.
If you’re working on something adjacent, such as small models in regulated domains, hallucination detection in production, or calibrated risk indicators for human-in-the-loop systems, I’d like to compare notes!
Read the companion post. A general audience write-up of the same research, written for non-specialists, is also available: Can you catch an AI’s mistakes by looking inside its head?
—
The underlying research was supervised by Dr. Clara Maathuis (Open University of the Netherlands) and Prof. Natasha Alechina (Open University of the Netherlands and Utrecht University). The paper has been accepted at AIES 2026.
