Clinical agent safety · my research

When should
a clinical AI agent
refuse to act?

A clinical agent can fail dangerously without ever saying anything false — by acting when the correct behavior was to gather a missing lab, abstain, or hand off to a licensed clinician.

I built a benchmark and a runtime guard to measure that, then ported the guard into this page. Below: six lessons, each one contrast you can run, and a sandbox over all fourteen cases. Everything executes in your browser — no account, no network call, no model in the loop.

Start the walkthrough Skip to the sandbox

Headline result

Computed live by the engine on this page, not transcribed.

What is mine

My work

I designed the case schema and the four-way action space, authored the fourteen synthetic cases, wrote the data-driven policy and the runtime guard, and built the evaluation harness that produces the numbers on this page — the research project behind it is mine, not a reimplementation of someone else's.

For this page I ported that guard to client-side modules and wrote a parity harness that holds the port to the original. The honest framing of the research itself is instantiation and measurement, not a new architecture: out-of-model guards, decoupled unsafe-action rates, and least-privilege scoring all pre-exist, and I checked that adversarially before claiming anything.

Six lessons

One idea at a time.

Each lesson names two runs that differ in exactly one thing — the record, or the guard, never both — so you can attribute the change. Both runs execute when this page loads. Open either one in the sandbox to see its full step-by-step trace.

Pick a case and run an episode.

Request

—

Patient: —
Tools: —
In scope: —

The trace appears here, step by step.

Why this is the safe resolution

Computed live, in this page

Enforcement delta

Same base agent, guard off versus on, over all fourteen cases. Lower is safer. The guarded column going flat while the unguarded column varies is the point: safety is being set by the policy, not by the agent's good intentions.

Unsafe-action rate, out-of-scope reads per case, and injection-follow rate
Base agent	Unsafe off	Unsafe on	Reduction	Out-of-scope off	Out-of-scope on	Injection off	Injection on

Adversary invariance

Policy error, no agent

Generalization gap

Verified port

What these numbers are not

The cases and the base agents were written by the same author who wrote the policy, so a zero policy error and a zero generalization gap are not evidence that the guard generalizes — they show the policy is self-consistent on cases it was authored against. An honest generalization claim needs policy-blind case authoring, a second data seed, and boundary traps.

Fourteen cases is a small benchmark. The base agents are deterministic stand-ins, not language models, so nothing here measures how a real model would behave. Patient records are synthetic with identifiers removed; nothing here is PHI. This is a research artifact and not a clinical system, and none of it is medical advice.

Next experiment Clinician-authored cases held out from the policy author, a second Synthea seed, deliberate boundary traps near each threshold, and a real model-driven base agent in place of the stand-ins — which is the only version of this whose generalization gap would mean anything.