spaCy redacted the word 'daily' — and broke 11 of 12 medication lists

The number

The first run of the eval harness (eval/run_eval.py) ingested 20 synthetic discharge summaries and asked 40 gold questions, two per patient. Answer accuracy came back at 0.575: 23 of 40. I expected the weak point to be retrieval or the 7B model. It was neither.

Grouping the 17 misses by question, 12 were "List the discharge medications." In 11 of those 12 the model had listed every drug with the right dose and left out the frequency. The answer was faithful to the chunk; the chunk was wrong. A stored chunk read:

Medications: sertraline 50 mg <DATE_TIME_2>; atorvastatin 40 mg <DATE_TIME_3>

The de-identifier had replaced "daily" and "nightly" with date tokens. Presidio's DATE_TIME recognizer inherits spaCy's DATE label, and spaCy's DATE covers durations and frequencies as well as calendar dates. Across the synthetic corpus, "daily" was scrubbed in 15 of the 19 documents that contained it and "nightly" in 4 of 8. The system prompt tells the model to keep placeholders exactly as written and to answer only from the excerpts, so it wrote "sertraline 50 mg" and stopped. It did what I asked.

The count that gave it away was phi_tokens grouped by entity type before the fix: DATE_TIME 84, PERSON 66, PHONE_NUMBER 20, MRN 20, LOCATION 18. Twenty synthetic documents with a date of birth, an admission date and a discharge date each is 60 date tokens. Eighty-four meant about two dozen were something else.

The fix

Safe Harbor removes dates tied to a person: birth, admission, discharge, death. A dosing frequency or a follow-up interval is clinical content, and there is no re-identification risk in the word "nightly". In worker/recognizers/date_filter.py (the filter lived in worker/deid.py when this was found) a DATE_TIME hit is now kept only if it contains something calendar-like:

_DATE_LIKE = re.compile(
    r"\b(?:jan|feb|mar|apr|may|jun|jul|aug|sep|sept|oct|nov|dec)[a-z]*\.?\b"  # month names
    r"|\b\d{1,2}[/.\-]\d{1,2}[/.\-]\d{2,4}\b"                                  # 3/5/2024, 03-05-24
    r"|\b\d{4}-\d{2}-\d{2}\b"                                                  # ISO
    r"|\b(?:19|20)\d{2}\b"                                                      # a year
    r"|\b\d{1,2}(?:st|nd|rd|th)\b",                                             # 5th (of March)
    re.IGNORECASE)


def _is_identifying_date(span: str) -> bool:
    return bool(_DATE_LIKE.search(span))

Scrubber.__call__ applies it as a filter on the analyzer's results before anything is replaced. "daily", "nightly" and "in 2 weeks" survive; "March 9", "03/05/2024" and "1985-03-12" do not. Two tests pin both directions: test_dosing_frequency_and_duration_are_not_redacted and test_calendar_dates_are_still_redacted in tests/test_deid.py.

The filter also uncovered something it had been masking. A bare ten-digit number after Tel: had only been caught because spaCy mislabelled it as a date; with the filter in place it would have survived. The phone recognizer gained a bare-NANP pattern that scores 0.35 on its own, below the 0.5 threshold, and clears it only when Presidio's context enhancer finds a phone-like word nearby. Removing a false positive can expose a false negative that was hiding behind it. Re-run recall after every relaxation.

Three more stock-recognizer gaps from the same pass

Running the de-identifier against the real Presidio engine in tests/test_deid.py and then against the synthetic corpus found three more places where the stock recognizers fall short of a clinical chart.

• MRNs were never scrubbed. The custom MRN pattern scored 0.4 against a 0.5 threshold, so every hit was discarded before it reached the resolver and the first test summary came back with its MRN intact. _mrn_recognizer() was rewritten: a labelled MRN: 1234567 now scores 0.85 and only the digits are replaced, so the model still sees MRN: <MRN_1>. test_labeled_mrn_forms covers six spellings of the label. That rewrite is why the table above shows exactly 20 MRN tokens for 20 documents.
• Phone numbers with extensions. Presidio's phonenumbers-based recognizer missed NANP numbers with extensions such as +1-921-899-3518x19093. _phone_recognizer() is a regex backstop: optional country code, area code with or without parentheses, and an extension group (?:\s*(?:x|ext\.?|extension)\s*\d{1,6})?. test_phone_forms_with_extensions covers five formats.
• Physician surnames that are common words. spaCy missed Dr. Fields, Dr. Lewis, Dr. Young and Dr. Holder: the four whole-string survivors in run 1, which is where the synthetic de-id recall of 0.967 (116/120) came from. _name_recognizer() anchors on a title or field label (Dr., Doctor, Attending:, Patient:, Name:) and replaces only the name, so Attending: Dr. <PERSON_2> keeps its title. Presidio compiles every pattern with IGNORECASE, so the name part is wrapped in (?-i:...) to stop at the first lowercase word; without that, "Dr. Priya Raghunathan-Okafor reviewed" swallowed "reviewed".

A later finding in the same family: after those fixes some charts read Attending: <PERSON_2>. <PERSON_1>. spaCy had tagged the bare title "Dr" as a person, and the model answered "the attending physician is Dr." _trim_person() now drops a span whose every word is a title or a field label, trims leading titles and trailing labels off the rest ("Dr Young" becomes "Young", "Joshua Duncan DOB" becomes "Joshua Duncan"), and drops the span if nothing is left. Tests: test_bare_title_is_never_a_person_token and test_person_span_is_trimmed_of_leading_title_and_trailing_label.

Before and after

The two remaining misses are Tesseract misreads on the synthetic scans that the model copied faithfully: "fisinopril" for lisinopril and "HbAtc" for HbA1c. Those need the vision route on a GPU box or a drug-name correction step, and they are tracked in TODO.md.

What to check in your own pipeline

• Count tokens by entity type, not just recall. 84 DATE_TIME tokens on 20 documents with three dates each was the tell.
• Diff one scrubbed document against its source and read the medication list by hand.
• Check that your date recognizer separates calendar dates from frequencies and durations. spaCy's DATE label does not.
• Assert in a test that every custom recognizer's score clears the threshold. A 0.4 pattern against a 0.5 gate is silently doing nothing.
• Scrub "Attending: Dr." on its own and check that no PERSON token comes out.
• After removing any false positive, re-run recall. Something may have been depending on it.

The validation side of the same run, where the judge threw away correct answers, is in the 7B judge scored identical facts 0.0 or 1.0. Where de-identification sits relative to the model call is on the architecture page.

← All posts