# Arbiter

> Research on private AI: measured findings from building an AI agent that works on health records under written privacy rules, sourced reads on AI and privacy law, and safe practices any agent can adopt. Every number is sourced and every chart links its data.

## Safe practices for your agent

- [Practices, JSON](https://www.arbiterai.tech/practices.json): 15 practices with rule ids, legal sources and Arbiter evidence
- [AGENTS.md](https://www.arbiterai.tech/AGENTS.md): drop-in instructions for coding agents
- [System prompt block](https://www.arbiterai.tech/prompt.txt): runtime behaviours for an answering agent
- [Claude skill](https://www.arbiterai.tech/skills/arbiter-safe-health-ai.zip): the practices as an Agent Skill
- [Python policy kit](https://www.arbiterai.tech/kit/arbiter-policy-kit-1.1.0.zip): policy engine, rules, negation-aware grounding, reference agent loop, tests
- [Agent policy](https://www.arbiterai.tech/policy.html): every rule with its source, tests and changelog

## Reads

- [Most AI benchmarks never check whether a difference is real. Ours couldn't either](https://www.arbiterai.tech/blog/evals-cant-tell-noise.html): With 40 gold questions, Arbiter's eval could not see any change smaller than about 15 points. A review of 445 benchmark papers found 16% used any statistical test. The power curve, the field's numbers, and the release gate the agent now enforces.

- [HHS just removed the AI "model card" rule from certified EHRs. Here is what goes, and what still applies](https://www.arbiterai.tech/blog/hti-5-model-cards.html): ONC's HTI-5 final rule, published 28 October 2026, deletes the 44 source attributes and the risk management that certified EHRs had to provide for decision-support AI. 251 products from 206 developers carried the rule. What goes on 28 December, what stays, and what still governs AI in the EHR.

- [iOS 27 can run open AI models entirely on your iPhone. What that does, and doesn't, mean for privacy](https://www.arbiterai.tech/blog/ios-27-core-ai-on-device-privacy.html): Apple's Core AI, shipped with iOS 27 on 14 September 2026, lets apps run open models such as Qwen, Gemma, Mistral and Whisper on the device. Prompts no longer have to reach a server to be processed. Three things it does not settle, and the memory arithmetic that decides which models fit.

- [46 AI and privacy duties start on 1 January 2027. The ones that matter if you build on health data](https://www.arbiterai.tech/blog/jan-1-2027-ai-privacy-laws.html): Arbiter's law tracker now holds 95 AI and privacy duties starting before the end of 2027, each date read in the primary text. 46 start on 1 January 2027, 20 of them in California. For health AI, the federal "model card" rule ends on 28 December and three state laws take over the same week.

- [Our safety check couldn't verify a "no". AI judges everywhere share the blind spot](https://www.arbiterai.tech/blog/judge-cant-verify-no.html): The check that decides whether Arbiter's answers are grounded scored correct negations 0.00, the same as fabrications. Re-measured today and fixed by restructuring the task, not by changing the model, alongside 2026 research showing AI judges miss what is absent.

- [We built an AI agent that follows HIPAA in code. Here is every rule, and what it refused](https://www.arbiterai.tech/blog/privacy-agent-policy-as-code.html): Arbiter now has an agent that answers questions about a patient's records under 23 written rules, each tied to a HIPAA section, a state law or one of our own measured failures. It passed 8 of 8 policy scenarios on a local model. The architecture, the refusals, and what code alone cannot make compliant.

- [spaCy redacted the word 'daily' — and broke 11 of 12 medication lists](https://www.arbiterai.tech/blog/spacy-redacted-daily.html): The first eval run scored 23 of 40. Twelve misses were the same question, and eleven of them traced to the de-identifier replacing 'daily' with a date token. The fix, three more Presidio gaps, and the tokens-by-entity count that gave it away.
- [Our de-id recall read 0.967 while ZIP codes survived in 14 of 20 documents](https://www.arbiterai.tech/blog/deid-recall-flattering-metric.html): The metric checked whether each injected identifier survived whole. An address is four things and only the city was ever tokenized. A strict per-component metric, an address recognizer, and why overlapping spans must merge into their union.
- [The 7B judge scored identical facts 0.0 or 1.0 depending on where the citation sat](https://www.arbiterai.tech/blog/7b-judge-citation-placement.html): Five correct answers rejected, all the same question. '9.0% [7].' scored 0.0 and '9.0%. [7]' scored 1.0. Escalation re-ran the same model at 8x the cost, and the judge's calls were half the traffic and never metered.
- [The services were connecting as the Postgres superuser, which silently bypasses RLS](https://www.arbiterai.tech/blog/postgres-superuser-bypasses-rls.html): Five tables carried a tenant-isolation policy and none of it applied, because both services connected as the database owner. The app role, the audit_log grant that defeated append-only, and the cross-tenant probes.
- [Medical record acronyms: what each one means and why it exists](https://www.arbiterai.tech/blog/medical-record-acronyms.html): ICD, MRN, CPT, SOAP, EHR, HL7, HIPAA, PHI, Safe Harbor, BAA, NPI, HITECH, FHIR, ICD-10-CM. A field guide to the fourteen acronyms that shape every healthcare integration, in the order they were invented.

## Data

- [AI and privacy law tracker, CSV](https://www.arbiterai.tech/data/law-dates.csv): duties by start date, each verified in the primary text
- [Reads index, JSON](https://www.arbiterai.tech/reads.json)
- [Code](https://github.com/yoonbo1/arbiterai): Apache-2.0