Arbiter AI

How it works

From a scanned page to a cited answer, with a gate at every step.

One pipeline, two graphs. Ingest turns a document into de-identified chunks and structured facts; query turns a question into a cited, validated, re-identified answer. Every stage below names the module it lives in and the test or eval metric that covers it.

A real trace from the eval harnesssynthetic record P00001
Ingest1 scanned page
  1. Document received
    tenant resolved from the API key, job queued on Redis Streams
    gateway/main.py
  2. Text extracted
    route: ocr (no text layer), Tesseract
    worker/extract.py
  3. PHI removed
    9 placeholders, map encrypted per tenant
    worker/deid.py
  4. Clinical facts extracted
    12 facts, 39 ms
    worker/annotate.py
  5. Chunk embedded
    384-dim, chunk 128
    worker/store.py
  6. Audit written
    job.ingest.completed · wall time 2.1 s
    audit_log
Querymost recent HbA1c?
  1. Question de-identified
    same recognizers as the document
    worker/deid.py
  2. Retrieval scoped
    to (tenant, patient), vector + BM25
    worker/retrieval.py
  3. Answer drafted
    local 7B model, citation [128]
    worker/llm.py
  4. Validation ok
    cites_chunks true · faithfulness 1.0 · phi_leak false
    worker/graph.py
  5. Re-identified
    for the caller only, only the tokens the answer uses
    worker/reid.py
  6. Audit written
    job.query.completed · 3.6 s median
    audit_log
tenant: synthetic-test-clinicmodel: qwen2.5:7b-instruct, local

Pipeline

Ten stages, in the order they run.

Ingest is stages 1 to 4 and 10. A query runs 5 to 10. The deidentified flag set in stage 2 is asserted by every later stage that touches a model or the index.

  1. Text extraction, with native, OCR and vision routing

    PyMuPDF renders each page at 200 dpi and keeps any native text layer. A page with more than 200 characters of native text is read as is. Otherwise Tesseract runs and reports its mean word confidence: at 75 or above with at least 40 words the page is routed to OCR; below that it goes to a vision model (Qwen2.5-VL, behind the vlm compose profile) if one is configured, or to OCR with vlm_wanted flagged so the usage stats stay honest. storage_uri is client input, so it is confined to DATA_ROOT after symlink resolution.

    Lives in worker/extract.py · Covered by the scanned half of the synthetic corpus in make eval; every residual miss in the changelog traces back to a Tesseract misread on one of those pages

  2. Presidio de-identification, with a reversible per-tenant PHI map

    Presidio with spaCy en_core_web_lg, plus recognizers Presidio does not ship: labelled MRNs, NANP phone numbers with extensions, clinician names anchored on a title, labelled address lines and bare City, ST 12345, and a date filter that keeps dosing frequencies such as "daily". Pages are scrubbed with shared token state, so one person gets one placeholder across a document. Overlapping candidates merge into their union, so a fragment of any above-threshold span never survives. The placeholder-to-value map goes to phi_tokens through pgp_sym_encrypt under a per-tenant key; decrypting it with another tenant's key returns "Wrong key or corrupt data".

    Lives in worker/deid.py, map storage in worker/store.py · Covered by tests/test_deid.py (real Presidio); deid_recall and deid_recall_strict in eval/run_eval.py

  3. Clinical fact extraction

    Three layers over the de-identified text, cheapest first: regex rules for labs, vitals, allergies, medication lines, diagnosis lists, family history and follow-up; Med7 (medications with dose, route and frequency) and bc5cdr (problems) merged by character offset; medspacy sections and ConText assertion, so "Denies pneumonia" is stored as absent and a family-history line as family. Every fact links to the chunk with the largest span overlap. Median 36 ms per page inside the container.

    Lives in worker/annotate.py · Covered by tests/test_annotate.py; eval/run_extraction_eval.py (make eval-extraction), precision, recall and F1 per fact kind with assertion required for problems

  4. Index

    Chunks of up to 800 characters with 120 of overlap, split at headings and blank lines first so a medication list stays whole. Each chunk is embedded by the local embeddings service (bge-small-en-v1.5, 384 dimensions; fastembed on Apple Silicon, TEI on x86) and stored in chunks (pgvector, HNSW, cosine) with its page, extraction route and character offsets, under the same row-level security as every other PHI table. Only de-identified text is ever embedded.

    Lives in worker/store.py · Covered by the deidentified assertion in the chunk_embed node; the vector-literal and 384-dimension guards in tests/test_retrieval.py; deid_recall_strict, which scans every stored chunk for every injected identifier

  5. Retrieval scoped to (tenant, patient)

    The question is de-identified with the same recognizers, embedded, and matched against this tenant's and this patient's chunks only: WHERE tenant_id=%s AND patient_id=%s, on top of row-level security. The top 30 by cosine form a pool; BM25 over the pool catches exact terms such as drug names; reciprocal rank fusion (k=60) merges the two rankings and the top 6 go to the model. Placeholders from different documents are renumbered into one namespace per query, so the same label cannot mean two people inside one prompt.

    Lives in worker/retrieval.py, namespace in worker/reid.py · Covered by tests/test_retrieval.py::test_query_is_hard_filtered_by_tenant_and_patient and thirteen more; cross_patient_leaks in the eval, 0 in every run

  6. Answer

    The small tier (qwen2.5:7b-instruct through Ollama here; Qwen2.5-7B-Instruct-AWQ on vLLM with a GPU) drafts an answer from the retrieved chunks and must cite each one it uses as [chunk id]. No cloud endpoint is called at any stage.

    Lives in worker/llm.py · Covered by answer_accuracy (strict: the first gold item verbatim) and answer_accuracy_lenient (drug and dose, or the lab value, for every gold item)

  7. Validation

    Three checks on every draft. cites_chunks: at least one retrieved chunk id appears in the answer. grounded: a faithfulness judge scores the answer against the chunks and must reach 0.7. phi_leak: a second de-identification pass over the answer, with citation markers stripped first so [12] is never mistaken for an identifier. ok is all three. The judge's tokens are metered like any other call.

    Lives in the validate node of worker/graph.py, judge in worker/llm.py · Covered by queries_failed_validation in the eval; the judge prompt measured on qwen2.5:7b at 1.0 for a supported answer and 0.0 for an unsupported one

  8. Single escalation

    A draft that fails validation is regenerated once on the large tier. A PHI leak is never retried, and a second failure returns no answer rather than an unverified one. In the last eval run that was one escalation and one rejection in forty questions.

    Lives in after_validate in worker/graph.py · Covered by queries_escalated and queries_failed_validation

  9. Re-identification, for the authorized caller only

    After validation passes, only the tokens the answer uses are restored: question tokens from the in-memory map, document tokens decrypted from phi_tokens under the tenant key, using the per-query namespace index to find which document each came from. Nothing is decrypted before the model has answered, and nothing the answer does not use. Logs, checkpoints and metrics keep the tokenized form.

    Lives in worker/reid.py · Covered by tests/test_reid.py, including the case where a question about one clinician could name another

  10. Audit write

    Every job ends with an audit_log row: job.ingest.completed with pages, routes and fact counts; job.query.completed for a delivered answer, or job.query.rejected with the reasons (phi_leak, cites_chunks, grounded, no_chunks) when validation failed, each with the validation summary, whether an escalation ran, and whether anything was re-identified; facts.read on the facts endpoint. The application role holds INSERT and SELECT on the table and nothing else, and row-level security limits its SELECT to its own tenant.

    Lives in db/init.sql, writes in worker/store.py and gateway/main.py · Covered by REVOKE UPDATE, DELETE ON audit_log FROM app_rw; UPDATE and DELETE as app_rw return permission denied

Why an arbiter

The model is not in charge.

Language models are persuasive, and a clinical record is the wrong place to be persuaded. So there is a gate on each side of the model: de-identification between the document and the model, and validation between the model and the caller. Cite the source, clear the faithfulness threshold, leak nothing, or return no answer. The arbiter is the part that says no, and the eval harness is what shows it saying no for the right reasons.

Engineering note

Cost that scales with difficulty, not volume.

Native text is free: PyMuPDF reads the layer. Tesseract costs about 1.5 to 2 seconds per page on the Apple Silicon dev machine; the scanned-page ingest in the trace above is 2.1 s wall, annotation included. A vision model runs only when OCR confidence is under 75 or the page has fewer than 40 words, which means forms, tables and handwriting, not clean scans. On the answer side the 7B model goes first and the large tier runs at most once, after a validation failure: one escalation in forty questions in the last run. The routing is what lets the expensive checks run on every request.

Interface

Two endpoints and a job poll. Asynchronous by design.

Requests return a job id immediately and never block on model inference. The worker reads a Redis stream with at-least-once delivery, and ingest is idempotent so a redelivered message cannot double-process.

POST /v1/documents
Authorization: Bearer hipaa_live_…
Idempotency-Key: ingest-P00001-scan

{ "patient_external_id": "P00001",
  "doc_type": "discharge_summary",
  "storage_uri": "/data/synthetic/scan/P00001.pdf" }

→ 202 { "job_id": "f9fc0185-…" }
POST /v1/queries
Authorization: Bearer hipaa_live_…

{ "patient_external_id": "P00001",
  "question": "What is the most recent HbA1c?" }

→ 202 { "job_id": "…" }

GET /v1/jobs/…
→ { "status": "done",
    "result": { "answer": "The most recent HbA1c is 5.7% [128].",
                "citations": [128],
                "validation": { "ok": true, "cites_chunks": true,
                                "faithfulness": 1.0, "grounded": true,
                                "phi_leak": false } } }

Any HTTP client works. The full endpoint table, including GET /v1/patients/{external_id}/facts, is in the docs.

Observability

Every job records what it cost.

The jobs row for each ingest and query carries tokens_small, tokens_large and cost_cents at placeholder rates, judge calls included. That is how the eval reports tokens and cost per query, and how the judge's calls turned out to be half of all model traffic when they were still unmetered. The audit row beside it carries the extraction route mix per document, so the OCR and vision share of a corpus comes from one table.

Run it on your own machine.

Run it locallyRead the security model