Our de-id recall read 0.967 while ZIP codes survived in 14 of 20 documents

The number

The harness in eval/run_eval.py is simple on purpose. The synthetic generator records every identifier it injects into each of 20 discharge summaries: name, date of birth, MRN, phone, address, attending physician. After ingest, the harness reads every stored chunk as the application role and checks whether each injected string appears verbatim. Run 1 reported a synthetic de-id recall of 0.967: 116 of 120 strings gone. The four survivors were all Dr. <common-word surname>, which spaCy's NER had missed and which a title-anchored recognizer fixed (that story is in the "daily" post).

That read as a near miss on the 0.99 gate. It was not. A stored chunk read:

Address: 1049 Mitchell Lights Suite 075, <LOCATION_1>, AR 29101

The injected identifier was the whole 40-character string, street through ZIP. The check asked whether that string appeared in the chunks. It did not, because the city was gone, so the metric counted the address as removed. The street number, street name, unit, state and ZIP were all sitting in the index.

Counting by component across the 20 synthetic documents: the ZIP survived in 14 of 20, the street line in 7 of 20, the city in 6 of 20, and the state in 20 of 20 (a state is allowed under Safe Harbor). By component, roughly 97 of 120 identifiers were actually gone: about 0.81, not 0.967.

The city was the only part being caught because Presidio ships no US street-address recognizer. spaCy's NER tags a city it recognizes, and nothing else on the line. The Faker-invented cities in the synthetic corpus (Blankenshipstad, Hernandezview, Kingland, New Amber) it mostly did not recognize either.

Fix 1: a metric that partial hits cannot flatter

deid_recall_strict in eval/run_eval.py splits each multi-part identifier on commas and counts any surviving component of four or more characters as a leak. It is reported alongside the old number, not instead of it, so the gap between the two is itself visible in every run.

parts = [(p, part.strip()) for p in phi for part in p.split(",") if len(part.strip()) >= 4]
leaked_parts = sorted({part for _, part in parts if part in text})
...
"deid_recall_strict": 1 - len(leaked_parts) / max(1, len({part for _, part in parts})),
"phi_component_survivors": leaked_parts[:8],

Fix 2: an address recognizer

The address recognizer (now its own module, worker/recognizers/address.py; it lived in worker/deid.py when this was found) adds four LOCATION patterns:

• A labelled Address: line, taken whole as street, city, ST ZIP (up to three comma segments) and stopping at a run of two spaces, the next Label: on the line, or end of line. Score 0.85.
• A street line: a number, one to four words, a USPS suffix (Street, Lights, Roads, and the rest of the list), an optional unit. Score 0.6.
• City, ST 12345, which needs no knowledge of city names at all. Score 0.6. This is the pattern that closed the Faker-city gap.
• A bare ZIP, \d{5}(?:-\d{4})?, at 0.35, below the 0.5 threshold, lifted only when address context words (address, street, zip, apt, suite) are nearby. It started at 0.6, and WBC 11000 was scrubbed as a ZIP code.

Tests: test_labelled_address_line_is_fully_redacted and test_city_state_zip_without_label_is_redacted.

Fix 3: overlapping spans merge into their union

With the city pattern in place a new failure appeared. In Kingland, TX 75001, spaCy tagged Kingland as LOCATION at 0.85 and my pattern tagged the whole Kingland, TX 75001 at 0.6. Presidio dropped the ZIP as a duplicate contained in the longer span, and the old _select() rule, "keep the best-scoring overlapping span, drop the rest", kept the shorter, higher-scoring city. The index read <LOCATION_3>, TX 75001.

_select() now merges overlapping candidates into their union, typed by the highest-scoring member:

for r in sorted((r for r in results if r.end > r.start), key=lambda r: (r.start, -(r.end - r.start))):
    if groups and r.start < groups[-1]["end"]:
        g = groups[-1]
        g["end"] = max(g["end"], r.end)          # the union, never the best fragment
        ...

The argument for union is that the two error types are not symmetric. Over-redaction is recoverable: the token maps back through phi_tokens and re-identification restores it for the authorized caller after validation (how it works). Under-redaction is a breach. So no fragment of any above-threshold candidate may survive, and union is the only safe policy. test_select_merges_overlaps_into_their_union_typed_by_the_best_member covers a phone number with an MRN-shaped run inside it, a year inside a full date, and a city inside a City, ST ZIP line.

The harness shape, and four iterations in a day

20 synthetic records, two gold questions each, 40 queries. One run ingests 20 documents, asks 40 questions, and reads the chunks: about 2 min 50 s of wall time and about half a cent at the placeholder per-token rates in worker/llm.py. That is cheap enough to run after every change, which is how the four iterations fit into one day.

The six strict survivors in run 2 were all Faker city names: Blankenshipstad, Hernandezview, Kingland, New Amber, New Brendaton, New Bryanhaven. Run 3's City, ST 12345 pattern removed them without knowing any of them.

What to check in your own pipeline

• If an identifier has parts, score the parts. A whole-string check passes when one word of an address is gone.
• Read a stored chunk. One SELECT text FROM chunks LIMIT 1 found this; the metric never would have.
• Do not rely on NER for geography. Add patterns for City, ST ZIP and street lines that need no city list.
• Resolve overlapping spans as a union. "Best span wins" leaves the rest of the line in the index.
• Run the leak check as the role RLS applies to. If the harness connects as the owner, it reads every tenant's chunks and proves nothing about isolation (the superuser post).
• Make one iteration cheap enough to run four times a day, and report the new metric next to the old one.

What the de-identifier guarantees, and what it does not, is written up on the security model page.

← All posts