The finding
Arbiter answers questions about a patient's documents, and every answer passes a gate before it is shown: it must cite an excerpt, contain no identifier, and score at least 0.7 from a small judge model asked whether the answer's claims are stated in the excerpts. In September, expanding the question set to 346 questions exposed a problem with that gate. Family-history questions passed only 10% of the time, and every failure was the gate rejecting the answer with a score of exactly 0.0 1.
The answers weren't wrong. Asked whether a patient has had a stroke, with a note that lists stroke only under family history, the right answer is "no, it is in the family history". The judge could find "family history of stroke" in the note. It could not find "the patient does not have stroke", because a note does not write down every condition a patient lacks. So it scored the correct answer exactly like a made-up one.
Today I re-ran the cases with the same local model and the same 0.7 bar:
Grounding score per answer, before and after the fix
Nine answers about one synthetic note; 0.7 is the pass mark. Same local model (qwen2.5 7B), temperature 0.
Source: Arbiter, measured 2026-11-02 with platform/eval/negation_cases.py 2
Show the data
| label | correct | kind | before | after | before_gate | after_outcome | tag |
|---|---|---|---|---|---|---|---|
| Stroke appears in the family history for this patient. | yes | family, stated positively | 1.00 | 1.00 | pass | delivered | correct |
| The record documents a family history of stroke. | yes | family, stated positively | 1.00 | 1.00 | pass | delivered | correct |
| No, the patient does not have stroke. Stroke is documented as family history. | yes | family, with a negation | 0.00 | 1.00 | reject | delivered, negation restated | correct |
| The patient does not have stroke. | yes | family, as a bare negation | 0.00 | 1.00 | reject | restated as not documented | correct |
| The patient does not have pneumonia. | yes | denial the note states | 1.00 | 1.00 | pass | delivered | correct |
| Yes, the patient has stroke. | no | wrong | 0.00 | 0.00 | reject | rejected | wrong |
| The patient was diagnosed with stroke during this admission. | no | wrong | 0.00 | 0.00 | reject | rejected | wrong |
| The patient has a family history of breast cancer. | no | wrong | 0.00 | 0.00 | reject | rejected | wrong |
| The patient has pneumonia. | no | wrong | 0.00 | 0.00 | reject | rejected | wrong |
Before, two of five correct answers scored 0.00, and so did all four wrong ones. A gate that scores a true "no" and a fabrication identically blocks errors only by also blocking correct answers. From outside, the two failures look the same. Meanwhile the aggregate accuracy still read 0.90 1.
The fix: change the question, not the model
A bigger model would probably score these cases better, but it would still be asked the wrong question. "Is this sentence stated in the text?" has no good answer for a sentence about absence. So the fix restructures the task:
- Split the answer into sentences, and class each one as positive or negative.
- Positive sentences go to the judge, as before.
- A negative sentence counts as supported only if the note states the negation itself. "No evidence of pneumonia" supports "the patient does not have pneumonia". That check is deterministic and needs no model.
- A negative sentence the note does not state is restated as what the system actually knows. The new wording is "the records searched do not document this either way; absence from these records is not proof of absence". The restatement makes no claim about the patient, so there is nothing for the judge to score.
After the change, every correct answer is either delivered or restated, and every wrong answer is still rejected. The rule is now part of the agent's policy as F-NEGATION. Its tests replay the original table.
It is not just our 7B model
Checking what is not in a text is a known weak spot for language models, and 2026 research shows it in clinical notes. Fox and colleagues built 500 pairs of AI-scribe notes, each with a single error, and tested eight judge designs 3. The judges were good at catching content that was added or changed. They were close to a coin flip at catching content that was left out, even though omission is the most common scribe error:
AI judges on clinical notes: what they catch, by error type
Paired discrimination; 0.5 is a coin flip. Range across 8 judge designs.
Source: Fox et al., 2026, preprint 3
Show the data
| label | before | after | tag |
|---|---|---|---|
| Added or altered content | 0.79 | 0.94 | 8 judge designs |
| Omitted content | 0.50 | 0.63 | 8 judge designs |
Automatic factuality metrics show a related sensitivity. Mujahid and colleagues rewrote long-document summaries into logically equivalent negations, and two widely used metrics roughly halved their scores 4:
Factuality metrics when a summary is rewritten as an equivalent negation
Mean score; the rewrites were machine-made and not checked by people.
Source: Mujahid et al., ACL 2026 4
Show the data
| label | before | after | tag |
|---|---|---|---|
| MiniCheck, LexAbSumm | 0.84 | 0.40 | factuality metric |
| UniEval, LexAbSumm | 0.82 | 0.39 | factuality metric |
| MiniCheck, SQuALITY | 0.56 | 0.30 | factuality metric |
| UniEval, SQuALITY | 0.68 | 0.32 | factuality metric |
That second result needs care. The rewrites were made by a model and were not checked by people, so some may have changed the meaning. What it does show is that these metrics react strongly to negation itself. Other work points the same way. Models find omitted content at about 70% F1 at best 5. Judges asked to say "not enough evidence" still give a verdict on 9% to 48% of such claims 6. Judges separating complete from incomplete medical chatbot answers do little better than chance 7.
What to check in your own pipeline
- Score accuracy per question type, not only overall. A 0.90 total hid a 0.10 on family history.
- Put true negatives in the gold set. Include questions whose right answer is "no" and questions whose answer is not in the record. Grade them by kind, because "the gold string appears in the answer" scores a fabricated value as correct.
- Look at what the gate rejects, not only what it passes. A gate that rejects correct answers looks like a cautious model.
Show your work
How this was done
The before-and-after scores were measured on 2 November 2026 with platform/eval/negation_cases.py. Nine answers were scored against one synthetic note: five correct, four wrong, written to mirror the cases in the original finding 1. "Before" is one call to the judge on the whole answer, the gate the query pipeline used. "After" is worker/grounding.py. It splits the answer into sentences. A negative sentence counts as supported only when the note itself negates one of its words in the same clause. A negative sentence the note does not state is restated as "the records searched do not document this", and only the positive sentences go to the judge. The model was qwen2.5:7b-instruct on a laptop via Ollama, at temperature 0, so the scores are deterministic for this model build.
The research figures are as published, read in each paper on 2 November 2026.
What this does not tell you
Nine cases and one note show the mechanism. They do not estimate a rate. The full 346-question eval is the measurement that does, and it has not been re-run since the fix. The negation check is lexical, so a note that negates a different word in the same clause could still support a claim it should not; the clause boundary and an eight-word window limit this, but do not remove it. Fox et al. is a preprint with one physician validator. Mujahid et al.'s negated rewrites were not checked by people, so their result shows that the metrics react to negation, not that they penalize correct negatives. We found no 2025-26 study that measures exactly our failure: a groundedness scorer marking a correct "the source does not say X" as unsupported.
Data
Every number in the charts, as CSV:
Agent policy
This read is evidence for these rules in the agent's policy:
Sources
- Arbiter changelog, 2026-09-08: "the validation gate cannot verify a negation"
- Arbiter, negation cases re-measured 2026-11-02 (CSV)
- Fox, Markham, Lail, Karotsieris, "LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It", arXiv 2608.31016 (2026, preprint)
- Mujahid, Wright, Augenstein, "Stress Testing Factual Consistency Metrics for Long-Document Summarization", ACL 2026
- Fu et al., "AbsenceBench: Language Models Can't Tell What's Missing", arXiv 2506.11440 (2025)
- Xu, "Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes", arXiv 2606.07834 (2026, preprint)
- DeLucia et al., "Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness", arXiv 2604.16383 (2026, preprint)
Corrections
None so far. Found an error? Tell me and it will be corrected here, with the date.