Run it, then run the evals.

Run it locally

git clone https://github.com/yoonbo1/arbiterai && cd arbiterai

Every command below runs from the root of the clone; the targets are in the Makefile.

Prerequisites

• Docker (validated with Colima on Apple Silicon).
• Python 3.12. make venv creates .venv/; 3.14 is too new for spaCy and Presidio.
• A local model. On Apple Silicon: Ollama with qwen2.5:7b-instruct (brew install ollama, ollama pull qwen2.5:7b-instruct, then make llm in a second terminal). On NVIDIA: the vLLM gpu compose profile, and vlm for the vision model on a second GPU. Without a vision model, low-confidence scans fall back to OCR and the job records that one was wanted.

Bring the stack up

cp platform/.env.example platform/.env   # then set every change-me (openssl rand -hex 24)
make up              # postgres (pgvector), redis, embeddings, gateway, worker
make synth N=20      # 20 synthetic records with injected fake identifiers -> platform/data/synthetic/
make bootstrap       # dev tenant + one API key -> platform/.env.dev-tenant (git-ignored)

make bootstrap prints the tenant id and the key once. The gateway listens on 127.0.0.1:8080.

Ingest a document

curl -X POST http://127.0.0.1:8080/v1/documents \
  -H "Authorization: Bearer $API_KEY" \
  -H "Idempotency-Key: ingest-P00001-scan" \
  -H "Content-Type: application/json" \
  -d '{"patient_external_id":"P00001","doc_type":"discharge_summary",
       "storage_uri":"/data/synthetic/scan/P00001.pdf"}'

→ 202 {"job_id":"f9fc0185-…"}

Odd-numbered records are the scanned variants under scan/; the rest have a text layer under clean/. /data is the worker's mount of platform/data/.

Ask a question

curl -X POST http://127.0.0.1:8080/v1/queries \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"patient_external_id":"P00001","question":"What is the most recent HbA1c?"}'

→ 202 {"job_id":"…"}

Read the result

curl http://127.0.0.1:8080/v1/jobs/$JOB_ID -H "Authorization: Bearer $API_KEY"

{"job_id":"…","status":"done",
 "result":{"answer":"The most recent HbA1c is 5.7% [128].",
           "citations":[128],
           "validation":{"ok":true,"cites_chunks":true,"faithfulness":1.0,
                         "grounded":true,"phi_leak":false,"attempts":1,
                         "tokens_restored":0},
           "errors":[]},
 "finished_at":"…"}

Citation numbers are chunk ids. Each chunk carries its page and character offsets, so a reviewer can open the source at the right place. The answer is re-identified for the caller; everything stored stays de-identified. A rejected answer comes back as status: failed with the validation object that explains why, never as an unverified answer.

Run the evals

make eval LIMIT=20              # eval/run_eval.py: ingest, de-id recall, 40 gold questions, leaks, latency, cost
make eval-extraction LIMIT=20   # eval/run_extraction_eval.py: clinical facts against the manifest's gold_facts
make test                       # pytest, no Docker needed: 227 tests with the clinical extras, 172 without

Both evals read TENANT_ID, API_KEY, DATABASE_URL and GATEWAY from platform/.env.dev-tenant, written by make bootstrap. run_eval.py refuses to run as a superuser or a BYPASSRLS role, because a leak check that bypasses row-level security measures nothing. One run of twenty records and forty questions takes under three minutes on an M4.

What each metric means

The extraction eval reports precision, recall and F1 per fact kind (problem, medication, lab, vital, allergy). Problems must match on assertion; medications on name, dose and frequency. It also reports separately how many negated or family-history conditions were wrongly stored as present.

Adding gold questions

Gold questions live in the manifest. scripts/make_synthetic_docs.py writes a gold_qa list per record ({"q": …, "a": …}) next to injected_phi and gold_facts. Add a question there, regenerate with make synth N=20, and run_eval.py picks it up. The first gold item is what the strict metric checks verbatim, so put the most specific value first.

Data

Every record that ships with the reference implementation is synthetic. No real patient has ever been near it. scripts/make_synthetic_docs.py generates one-page discharge summaries with Faker, seeded so the corpus is reproducible, and plants six identifiers per record that the de-identification eval must remove: a name, a date of birth, a medical record number, a phone number with an extension, a street address, and an attending physician. Each record also carries the gold facts: two diagnoses, three medications, an HbA1c, a blood pressure, an LDL, two negated findings, one family-history condition, and an allergy with reaction or a note of no known allergies.

Half the corpus is rasterized as scans: every odd-numbered record's PDF is rendered, rotated up to 1.5 degrees, blurred, and saved as an image with no text layer, the way a fax arrives. --scan-mode bilevel gives 1-bit at 300 dpi for harsher fax realism. The manifest, platform/data/synthetic/manifest.json, is the ground truth: injected_phi, gold_qa and gold_facts per record.

Pointing the harness at i2b2 2014

The n2c2/i2b2 2014 de-identification corpus is the credible benchmark for this, and it needs a data use agreement. Once you hold credentialed access: render each note to a PDF (one per record, in the same layout the synthetic generator uses, or any layout Tesseract can read), and write a manifest with one entry per record whose injected_phi list is the gold PHI annotations for that note. run_eval.py --manifest then reports the same recall metrics against real annotations. The QA metrics need gold_qa, so give each record at least one question whose answer is a value in the note, or read only the recall fields.

API

Three endpoints for the reference implementation, plus admin routes for tenants and keys, all defined in gateway/main.py. There is no shared public API: <your-deployment> is wherever you run the gateway.

curl -X POST https://<your-deployment>/v1/documents \
  -H "Authorization: Bearer hipaa_live_…" \
  -H "Idempotency-Key: intake-0001" \
  -H "Content-Type: application/json" \
  -d '{"patient_external_id":"P00001","doc_type":"discharge_summary",
       "storage_uri":"/data/intake/P00001.pdf"}'

curl https://<your-deployment>/v1/patients/P00001/facts?kind=medication \
  -H "Authorization: Bearer hipaa_live_…"

{"patient_external_id":"P00001","count":3,"active_only":true,
 "facts":[{"kind":"medication","normalized":"lisinopril",
           "attributes":{"dose":"10 mg","frequency":"daily"},
           "assertion":"present","section":"medications",
           "chunk_id":128,"page":1,"confidence":0.95,"extractor":"rules"}, …]}

Concepts

Tenants and patients

Your tenant is identified by your credential. Patients are identified by an external_id you choose, typically your own record number. Queries are always scoped to one patient.

Idempotency

Send an Idempotency-Key on every POST. Retrying with the same key returns the original job instead of creating a duplicate, so a network failure never double-processes.

Job states

queued, processing, done, failed. A failed query includes a validation object explaining why it was rejected rather than returning an unverified answer.

Scopes and rate limits

A key carries scopes (ingest, query, admin) and a per-minute limit, both set when it is minted. Exceeding the limit returns 429. Revocation takes effect within 60 seconds.

Errors