The 7B judge scored identical facts 0.0 or 1.0 depending on where the citation sat

The number

The validation node in worker/graph.py runs three checks on every answer before it can be re-identified: no PHI in the output (deid.contains_phi), at least one citation to a retrieved chunk, and a grounding score. The grounding score comes from llm.faithfulness_score, which asks the small model (qwen2.5:7b-instruct via Ollama, temperature 0) for a single number between 0 and 1, the fraction of the answer's claims that the excerpts support, and gates at 0.7 or above.

In run 1 on the synthetic corpus, validation rejected 5 of 40 answers. All five were the same question, "What is the patient's most recent HbA1c?", and all five were correct. A typical one: "The patient's most recent HbA1c is 9.0% [7]."

I re-issued the exact judge prompts by hand, changing only the answer text:

The patient's most recent HbA1c is 9.0% [7].   ->  0.0
The patient's most recent HbA1c is 9.0%. [7]   ->  1.0
The patient's most recent HbA1c is 9.0%.       ->  1.0
HbA1c 9.0% [7]                                 ->  1.0

Same facts, same context, same model, temperature 0. A citation marker before the full stop scored 0.0; after it, 1.0. The judge prompt already told the model to ignore citation markers like [12]. It did not.

Escalation made it worse, at about 8x the cost

after_validate in worker/graph.py sends a rejected answer back to generate once, on the "large" tier, before failing the job. On this machine there is no separate large model: LARGE_MODEL_URL is unset and worker/llm.py falls back to the small tier's URL and model. So escalation re-ran the same model at temperature 0, got the same "9.0% [7].", got the same 0.0 from the same judge, and discarded a correct answer for the second time.

The cost accounting made it look expensive rather than free, because the escalated call is metered at the large-tier rate: 0.0539 cents against 0.0066 cents for the small-tier attempt, at the placeholder rates in COST_PER_1K. About 8x, to reproduce a rejection.

Half the model traffic was never counted

While reading the judge's prompts I noticed their size: about 330 prompt tokens per call, the judge instructions plus every retrieved chunk plus the answer. The judge runs on every answer, and the answer call is roughly the same size, so the judge was about half of all small-tier traffic. faithfulness_score returned only the float and discarded the response's usage. None of it reached jobs.tokens_small, so the per-query cost in the eval report was about half the truth.

The fix

Both changes are in worker/llm.py. Citation markers are stripped before the answer reaches the judge, and the judge returns its token count so graph.validate can meter it:

_CITE = re.compile(r"\s*\[\d+\]")


def faithfulness_score(answer_text: str, chunks: list[dict]) -> tuple[float, int]:
    ctx = "\n".join(c["text"] for c in chunks)
    claim = _CITE.sub("", answer_text).strip()
    out, used = _chat("small", [
        {"role": "system", "content": JUDGE},
        {"role": "user", "content": f"CONTEXT:\n{ctx}\n\nANSWER:\n{claim}"}], max_tokens=8)
    ...
    return max(0.0, min(1.0, v)), used

# worker/graph.py, validate()
score, used = llm.faithfulness_score(state["answer_deid"], state["chunks"])
usage["small"] = usage.get("small", 0) + used      # judge calls are metered too

The judge prompt itself had already been rewritten once during bring-up: a vaguer "fraction of claims" wording scored a fully supported answer 0.5 on this model, below the gate. The current JUDGE text asks for 1.0 if every claim is stated in the context, 0.0 if none is, otherwise the fraction, and was measured at 1.0 supported and 0.0 unsupported. The lesson from the citation bug is that an instruction to ignore something is not the same as removing it. Normalize the input; do not ask a 7B model to.

Before and after

The small-tier count roughly doubling is the honest number, not a regression: nothing changed in the answer path, the judge's calls are simply counted now. Runs 3 and 4 came in at about 24,000. The one rejection that remained in run 2 was an answer that copied a Tesseract misread, "HbAtc 7.2%", verbatim from a scanned page; that is an OCR problem, tracked separately. On the regenerated synthetic corpus the QA eval reports 0 validation failures and accuracy 0.95.

What to check in your own pipeline

• Log the judge's verdict next to the answer for every rejection, and read the rejections. Five identical questions was the tell.
• Reproduce with the exact prompt and change one thing. Citation placement took four calls to isolate.
• Normalize the answer before judging: strip citations, placeholders and trailing punctuation. Do not rely on the prompt to do it.
• Meter every model call, including the ones that return one number. A judge that runs on every answer is not overhead; it is half the bill.
• Check what your "large" tier actually is. If it resolves to the same model, escalation is a retry at a higher price.
• Compare the cost of a rejection to the cost of an answer. Here a wrong rejection cost more than a right answer.

The other 12 misses in the same run were the de-identifier's fault, not the judge's: spaCy redacted the word "daily". Where validation sits in the query graph, and what it gates, is on the architecture page.

← All posts