Open reference implementation
Arbiter AI is an open reference implementation of HIPAA-safe document AI: de-identification before any model call, cited and verified answers, database-enforced patient isolation, and an append-only audit trail — with the evaluation harness that proves each of those properties holds.
Built and maintained by one engineer. Every number on this page comes from eval/run_eval.py or eval/run_extraction_eval.py, run on synthetic records. Nothing here has touched real PHI.
Four properties
Every path below is a link into the repository.
Every node that calls a model or writes to the index asserts that the state has been de-identified first. Presidio produces the placeholders, with recognizers I had to add for labelled record numbers, phone numbers with extensions, clinician names after a title, and street addresses.
deidentified assertions in worker/graph.py (annotate, chunk_embed, generate nodes) and Presidio in worker/deid.pytests/test_deid.py and the de-id recall eval in eval/run_eval.pyThe tenant comes from the API key, never from the request. Row-level security filters every PHI table on it, and retrieval adds a hard patient filter on top, so isolation does not depend on application code being bug-free.
tenant_isolation row-level-security policies in db/init.sql plus the WHERE tenant_id=… AND patient_id=… filter in worker/retrieval.pytests/test_retrieval.py::test_query_is_hard_filtered_by_tenant_and_patient and the cross-tenant probes (cross_patient_leaks in the eval, 0 in every run)An answer must cite a retrieved chunk by id, a judge must score it at or above 0.7 against those chunks, and a second de-identification pass must find nothing. Fail any of the three and no answer is returned.
worker/graph.py (cites_chunks, grounded, phi_leak) and the judge in worker/llm.pyqueries_failed_validation; the judge prompt measured on qwen2.5:7b at 1.0 for a supported answer and 0.0 for an unsupported oneEvery ingest, query and fact read writes an audit_log row with the credential, the action and the patient. The application role can insert rows and read them, and nothing else, so there is no code path that could edit history.
REVOKE UPDATE, DELETE ON audit_log FROM app_rw in db/init.sqlaudit_log as app_rw return permission deniedWhat it demonstrates
The corpus is one-page discharge summaries. The gold questions ask for the most recent HbA1c and the medication list, and an answer counts only if it cites the chunk it came from and clears the faithfulness judge. 38 of 40 in the last run; both misses are Tesseract misreads the model copied faithfully.
Half the records are rasterized, tilted and blurred, with no text layer. Those pages go to Tesseract, and when the router wants a vision model that is not configured, the job records vlm_wanted_pages instead of pretending the page was easy.
Problems, medications with dose and frequency, labs, vitals and allergies, extracted by rules, Med7, bc5cdr and medspacy ConText. "Denies pneumonia" is stored as absent and a family-history line as family, so a problem list never lists what the patient does not have.
Every fact links to the chunk with the largest span overlap, and GET /v1/patients/{external_id}/facts returns each one with its chunk and page. That is the primitive an evidence packet or a chart-review tool is built from.
The dev database holds two tenants; the second exists only to prove the first cannot see it. Key B gets 404 on tenant A's job, an app_rw session set to A sees zero of B's rows, and a cross-tenant INSERT violates the policy.
POST /v1/documents, POST /v1/queries, GET /v1/jobs/{id}. Idempotency keys make retries safe, and every job records its tokens and cost by model tier, judge calls included.
Benchmarks
Everything below is measured on synthetic records generated by scripts/make_synthetic_docs.py. The i2b2 row is the one that matters, and it stays empty until I hold credentialed access.
| Metric | Dataset | Value | Notes |
|---|---|---|---|
| De-identification recall, whole string | Synthetic 20 records | 0.967 → 1.000 | four iterations in a day |
| De-identification recall, strict per component | Synthetic 20 records | ≈0.81 → 1.000 | the metric that exposed surviving ZIP codes |
| ZIP codes surviving de-identification | Synthetic 20 records | 14/20 → 0/20 | |
| Answer accuracy, strict | Synthetic 40 gold questions | 0.575 → 0.975 | |
| Cross-patient leaks | Synthetic every run | 0 | |
| Clinical extraction F1: problems (assertion required) / medications (name, dose, frequency) / labs | Synthetic 20 records | 1.00 / 1.00 / 0.99 | |
| De-identification recall | i2b2 2014 test set (real, credentialed) | not yet run | the number that matters; pending credentialed access |
On the regenerated corpus that includes OCR'd scans, whole-string recall was 0.992 until 2026-09-05: one medical record number survived because Tesseract read its 'MRN:' label as 'MAN:'. The MRN recognizer now tolerates one OCR error in the label, and the same corpus scores 1.000 whole-string and 1.000 strict (20 records, 40 questions, accuracy 0.975, 0 leaks).
Before → after spans the four eval iterations of 2026-09-03; the repository's changelog records every run. How to reproduce them.
Failure modes I found
Each post leads with the bug and the number, then the fix, then what to check next time.
Presidio's default date recognizer took dosing frequencies for dates, so sertraline 50 mg <DATE_TIME_2> went into the index. The fix keeps a date hit only if it contains something calendar-like.
The whole-string metric checked the full address and never noticed that only the city had been tokenized. A strict per-component metric did, on the first run.
Read the post-mortem →The faithfulness judge rejected correct answers over citation placement, escalated at 8× the cost, and its calls were half of all model traffic and unmetered.
Read the post-mortem →Every row-level-security policy in the schema was inert, so the isolation the README described did not exist. Fixed with a plain app_rw role, then proven with cross-tenant probes.