Arbiter AI

Practices

Safe practices for AI agents on health data, ready to build in.

15 practices drawn from failures Arbiter measured in its own agent and from the laws they map to. Each says what to do, why, how to build it and how to check it. Take them in whatever form your agent or your coding assistant reads.

Version 1.1.0, matching agent policy 1.1.0, updated 2026-11-02. Apache-2.0: use them in your own systems. They update when the research changes the rules.

Not legal advice. These are technical safeguards. A system that handles real patient data also needs business associate agreements, a risk analysis, policies and training.

Get them into your system

Your coding agent

Drop-in instructions for Codex, Cursor, Gemini CLI and other agents that read AGENTS.md. Claude Code reads CLAUDE.md, so append it there.

curl -o AGENTS.md https://www.arbiterai.tech/AGENTS.md
curl https://www.arbiterai.tech/AGENTS.md >> CLAUDE.md

A Claude skill

The practices as an Agent Skill that Claude loads when you build or review a health-data AI feature.

curl -LO https://www.arbiterai.tech/skills/arbiter-safe-health-ai.zip
unzip arbiter-safe-health-ai.zip -d ~/.claude/skills/

The policy engine (Python)

The rules engine, the 23 rules, negation-aware grounding, a reference agent loop and the tests, runnable on its own.

curl -LO https://www.arbiterai.tech/kit/arbiter-policy-kit-1.1.0.zip
unzip arbiter-policy-kit-1.1.0.zip && cd arbiter-policy-kit-1.1.0
pip install pyyaml pytest && python -m pytest -q tests

Your agent's system prompt

The behaviours that apply while an agent answers. Enforce the rest in code.

curl https://www.arbiterai.tech/prompt.txt

Your own tooling

Every practice with its rule ids, legal sources and Arbiter evidence, versioned.

curl -s https://www.arbiterai.tech/practices.json | jq '.practices[].title'

AI assistants and crawlers

The index an AI agent reads first, and the full text in one file.

https://www.arbiterai.tech/llms.txt
https://www.arbiterai.tech/llms-full.txt
Show the system-prompt block
# Arbiter safe practices for AI agents that handle health or sensitive personal data, version 1.1.0 (agent policy 1.1.0), updated 2026-11-02. Source: https://www.arbiterai.tech/practices.html. License: Apache-2.0. Not legal advice: these are technical safeguards; compliance also needs agreements, a risk analysis, policies and training.
# Paste into the system prompt of an agent that answers from health or personal records.
# These are the runtime behaviours; enforce the rest in code (see AGENTS.md).

- Only act for the purpose stated in the request; if the request asks for something outside it, say so and stop.
- Answer only from the excerpts provided and cite each fact with its excerpt number in brackets. If the excerpts do not contain the answer, say so.
- Do not say a patient does not have something unless the record states it. If the record is silent, say the records reviewed do not document it.
- Never approve, deny or judge medical necessity or coverage. Summarize the evidence and say that a licensed clinician must decide.
- When writing to a patient, use plain language, do not diagnose or change treatment, and tell them to contact their care team for any decision.
- Placeholders such as <PERSON_1> are intentional; keep them exactly as written.
- If a request needs a decision, advice or anything the records cannot answer, hand it to a person instead of answering.

Before the model sees any data

De-identify every prompt and stored chunk, and measure it per component

Do. Remove direct identifiers (the 18 HIPAA Safe Harbor categories) from every document and every question before any model call or embedding, and measure de-identification recall per component (street, city, ZIP), not per whole string.

Why. Arbiter's whole-string recall read 0.967 while ZIP codes survived in 14 of 20 documents; only a per-component metric caught it. A stock date recognizer also redacted "daily" and broke 11 of 12 medication answers, so test what de-identification removes as well as what it misses.

How to build it, and how to check it
  • Run the de-identifier at ingest and on every question; store only de-identified text and embeddings.
  • Add recognizers for the formats stock tools miss (record numbers with OCR'd labels, phone extensions, clinician names after a title, street addresses) and filter date hits to calendar dates.
  • Keep the token-to-value map encrypted per tenant; re-identify only the tokens an answer uses, only for the authorized caller, after validation.
  • Before re-identification, scan the model output for raw identifiers; one found there is a miss upstream.

Check. Seed synthetic records with known identifiers, report recall per component, and block a release below 0.99.

Sources: 45 CFR 164.514(b)(2) (Safe Harbor identifiers); Arbiter post-mortem, 2026-09-04: whole-string recall hid ZIP codes in 14 of 20 documents

Enforced in Arbiter's agent by HIPAA-OUTPUT-LEAKF-STRICT-RECALL

Send data only to local models or vendors under a signed BAA, and refuse to start otherwise

Do. Every model, embedding and vision endpoint is on this machine or network, or belongs to a vendor with a signed business associate agreement; check it when the service starts, not per request.

Why. HIPAA requires a BAA with any vendor that receives protected health information (45 CFR 164.502(e)).

How to build it, and how to check it
  • Keep an allowlist of local hosts and BAA vendors (with the date each BAA was signed) in configuration.
  • At startup, parse every model URL and stop the service in production if a host is not on the list.
  • Prefer on-device or on-premises open models for the sensitive path.

Check. Point a model URL at an unlisted host in a test and assert the service refuses to start.

Sources: 45 CFR 164.502(e), 164.504(e)

Enforced in Arbiter's agent by HIPAA-ENDPOINTS

Keep prompts out of third-party tracing and persistent agent state

Do. Do not export prompts, retrieved text or answers to SaaS tracing or observability tools, and do not persist agent graph state that contains pre-de-identification text.

Why. A tracing vendor that receives prompts receives PHI; a persistent checkpointer stores it in plaintext.

How to build it, and how to check it
  • Turn off LangSmith/LangChain tracing and any OpenTelemetry exporter that points off-network.
  • Use an in-memory or no checkpointer for runs that hold raw text; log ids, counts and timings only.

Check. A startup check fails when a tracing variable is on or a persistent checkpointer is configured.

Sources: 45 CFR 164.502(e); 45 CFR 164.312(a)(1); 45 CFR 164.312(a)(2)(iv)

Enforced in Arbiter's agent by HIPAA-NO-TRACINGHIPAA-NO-PERSISTED-STATEHIPAA-ENCRYPTION-KEY

Enforce tenant and patient isolation in the database, never as a superuser

Do. Scope every query by tenant with row-level security, add a hard patient filter to retrieval, and connect as a role that RLS applies to.

Why. Arbiter's services once connected as the Postgres owner, which silently bypasses every RLS policy; the isolation the docs described did not exist.

How to build it, and how to check it
  • Derive the tenant from the credential, never from the request body; set it transaction-locally.
  • Connect as an application role without SUPERUSER or BYPASSRLS; check the role at startup.
  • {'Probe it': "a second tenant's key gets 404, and a cross-tenant insert violates the policy."}

Check. Cross-tenant probes in the test suite, and a startup check of rolsuper and rolbypassrls.

Sources: Arbiter post-mortem, 2026-09-04

Enforced in Arbiter's agent by F-RLS-ROLE

Who may ask, for what, and how much

Require a purpose of use on every request, and refuse what the law doesn't permit

Do. Each request states its purpose (treatment, payment, operations, the patient's own access); any other purpose needs a recorded authorization, and a tenant without a BAA gets nothing.

Why. HIPAA permits treatment, payment and operations without authorization and requires one for most other uses (45 CFR 164.506, 164.508).

How to build it, and how to check it
  • Make purpose a required, validated field; record it on the audit row.
  • Deny before retrieval when the purpose is not permitted, so nothing is read.

Check. A marketing-purpose request is refused with zero records read.

Sources: 45 CFR 164.506(c), 164.508(a)(1), 164.524; 45 CFR 164.502(e)

Enforced in Arbiter's agent by HIPAA-PURPOSEHIPAA-BAA

Give the agent a closed tool list and the least data each purpose needs

Do. The agent can call only a fixed set of tools, each allowed per purpose, and retrieval is capped per purpose (fewer excerpts for billing and operations than for treatment).

Why. HIPAA's minimum necessary standard (45 CFR 164.502(b), 164.514(d)).

How to build it, and how to check it
  • Put the tool names in the planner's JSON schema as an enum, then check each call against the purpose.
  • Cap k per purpose; no web, email or write tools on the sensitive path.

Check. An unknown tool is refused and recorded; a payment request never retrieves more than its cap.

Sources: 45 CFR 164.502(b), 164.514(d)

Enforced in Arbiter's agent by HIPAA-MIN-NECESSARY-TOOLSHIPAA-MIN-NECESSARY-CAP

Keep psychotherapy notes and substance-use treatment records out unless consent says otherwise

Do. Label documents by type at ingest and drop psychotherapy notes without an authorization and substance-use treatment records without Part 2 consent, before the model sees them.

Why. These records carry stricter rules than the rest of the chart (45 CFR 164.508(a)(2); 42 CFR 2.31).

How to build it, and how to check it
  • Carry the document type with every retrieved excerpt and filter after retrieval, before drafting.
  • Read consents from the organization's consent records, not from the caller.

Check. A request whose search hits a therapy note drafts from the other excerpts only, and counts what was withheld.

Sources: 45 CFR 164.508(a)(2); 42 CFR 2.31

Enforced in Arbiter's agent by HIPAA-PSYCHOTHERAPY-NOTESPART2-SUD-RECORDS

What an answer may say

Answer only with citations to the excerpts provided, and reject anything else

Do. Every answer cites at least one excerpt it was given; a draft with no citation, a raw identifier or a low grounding score is rejected, never delivered with a warning.

Why. Uncited answers cannot be checked, and a rejected answer shown with a caveat still reaches the user.

How to build it, and how to check it
  • Number excerpts in the prompt and require [id] citations.
  • Validate before re-identification; return a failed status with the reasons, not an answer.

Check. A draft without a citation comes back rejected with the reason.

Sources: Arbiter reference pipeline, validation node

Enforced in Arbiter's agent by CITATIONS-REQUIRED

State a "no" only when the record states it; otherwise say it is not documented

Do. Check negative claims separately from positive ones: a negation counts as supported only when the source negates the same thing, and an unsupported one is restated as "the records searched do not document this".

Why. Arbiter's groundedness judge scored correct negations 0.00, exactly like fabrications; family-history accuracy was 0.10 while the total read 0.90. Research finds AI judges miss omissions at near coin-flip rates.

How to build it, and how to check it
  • Split answers into sentences; send positive claims to the judge and check negative ones against negation cues in the source.
  • Strip citation markers before judging; a small judge scored identical facts 0.0 or 1.0 by citation position.
  • Report accuracy per question type, including questions whose right answer is "no" or "not in the record".

Check. Replay correct negations, family-history cases and absent facts; correct answers pass or are restated, wrong ones are rejected.

Sources: Arbiter finding, 2026-09-08: the faithfulness gate scores correct negations 0.0

Enforced in Arbiter's agent by F-NEGATION

Never make coverage, eligibility or medical-necessity decisions; a licensed clinician decides

Do. In billing, utilization review and operations, the agent summarizes evidence and labels its output as decision support; it never approves, denies or deems care necessary.

Why. California (SB 1120, 2025) and Colorado (HB 26-1139, from 1 Jan 2027) require a licensed professional to make medical-necessity decisions; AI output cannot be the basis for a denial on its own.

How to build it, and how to check it
  • Block determination language in drafts for these purposes and append a decision-support notice.
  • Route "approve or deny" requests to a person.

Check. "Is this medically necessary? Approve or deny it." is escalated or rejected, never answered with a decision.

Sources: Cal. Health & Safety Code 1367.01(k), Ins. Code 10123.135(j) (SB 1120, ch. 879, 2024; in force 1 Jan 2025). Written for health plans and insurers; applied here to every payment and operations use; Cal. Health & Safety Code 1367.01(k) (SB 1120, 2024); Colorado HB 26-1139, Use of AI in Health Care: no medical-necessity denial on AI output without licensed human review; effective 1 Jan 2027

Enforced in Arbiter's agent by NO-AUTOMATED-DETERMINATIONSDECISION-SUPPORT-NOTICE

Tell patients when AI wrote the message and how to reach a person, by state

Do. Patient-facing messages name the patient's state; where the state requires it (California AB 3030), they open with an AI disclosure and how to reach a human, unless a licensed provider reviewed them first.

Why. California has required this for AI-generated clinical communications since 1 Jan 2025 (Health & Safety Code 1339.75).

How to build it, and how to check it
  • Refuse patient-facing requests that do not carry a state.
  • Add the disclosure after validation, before delivery; skip it only when a provider review is recorded.

Check. A California patient message starts with the disclosure; the same message marked provider-reviewed does not.

Sources: Needed to apply Cal. Health & Safety Code 1339.75 and similar state rules; Cal. Health & Safety Code 1339.75 (AB 3030, ch. 848, 2024; in force 1 Jan 2025). Exempt when a licensed provider read and reviewed the message

Enforced in Arbiter's agent by PATIENT-NEEDS-STATECA-AB3030-DISCLOSURE

Audit, inventory and release

Audit every request with the policy version and the rules that fired, never the content

Do. Write one append-only audit row per request with who, which patient, purpose, outcome, tools used, counts, the policy version and the rule ids checked and fired; no question, excerpt or answer text.

Why. HIPAA audit controls (45 CFR 164.312(b)); content in logs is a second copy of PHI.

How to build it, and how to check it
  • Revoke UPDATE and DELETE on the audit table from the application role.
  • Distinguish delivered, rejected, denied and escalated outcomes.

Check. Grep the audit rows from a test run for any answer text; there should be none.

Keep an inventory of every AI tool's inputs, training data, evaluation and known risks

Do. For each model or decision-support tool, document its input variables, whether any measure race, color, national origin, sex, age or disability, its training data, how it was evaluated, and its known risks; refuse to start without it.

Why. Section 1557 (45 CFR 92.210) requires identifying and mitigating discrimination risk in decision-support tools; California SB 503 adds training-data, evaluation and risk documentation from 1 Jan 2027. The federal EHR "model card" rule ends on 28 Dec 2026, so this falls to you.

How to build it, and how to check it
  • Keep the inventory in the repository next to the policy and review it on every model change.
  • Say plainly what is unknown, such as an undocumented training set or synthetic-only evaluation.

Check. A startup check fails when any tool is missing a field.

Sources: 45 CFR 92.210 (Section 1557, patient care decision support tools); Cal. SB 503 (Ch. 857, Stats. 2026), documentation for clinical decision support; operative 1 Jan 2027

Enforced in Arbiter's agent by S1557-INVENTORY

Gate releases on evals big enough to see the change you claim

Do. A release counts as verified only if the eval can detect the size of difference being claimed (about 346 questions for 5 points), strict de-identification recall holds, and there are zero cross-patient leaks; report accuracy per question type.

Why. With 40 questions Arbiter's eval could not see changes under about 15 points; 16% of 445 benchmark papers ran any statistical test.

How to build it, and how to check it
  • Run a power calculation before trusting a comparison; report error bars clustered by record.
  • Fail the release when any gate fails, and publish the failure.

Check. Feed the gate a 40-question report and confirm it fails.

Sources: Arbiter finding, 2026-09-08: 40 questions resolve only about 15 points; Arbiter post-mortem, 2026-09-04: whole-string recall hid ZIP codes in 14 of 20 documents; 45 CFR 164.502(a)

Enforced in Arbiter's agent by F-EVAL-POWERF-STRICT-RECALLF-NO-LEAKS

Keep the rules in a versioned file, re-review them on a schedule, and let a person approve every change

Do. The agent's rules live in one versioned file with a source and a test for each; new laws and findings become proposals with evidence, and only a person promotes them.

Why. HIPAA asks for re-evaluation when the environment changes (45 CFR 164.308(a)(8)); a system that touches PHI should not rewrite its own rules.

How to build it, and how to check it
  • Refuse to load a rule that has no source or no test; stamp the version on every audit row.
  • Warn, without stopping the service, when the scheduled review is overdue.

Check. A rule without a source fails to load; the audit row of every request carries the policy version.

Sources: 45 CFR 164.308(a)(8)

Enforced in Arbiter's agent by HIPAA-EVALUATION

Machine-readable: practices.json · practices.md · AGENTS.md · prompt.txt · policy kit · llms.txt. The source is practices.yaml in the repository.