Clinical Evaluation of Patient-Facing AI Agents — Metro Area Healthcare Technologies
Text
Red TeamingDesigned and evaluated patient-facing AI agents deployed on reproductive health practice websites. Programmed the clinical logic and content guardrails intended to scope the agents to non-clinical eligibility and scheduling questions, with explicit redirection to human clinicians for any clinical content. Conducted ongoing evaluation of agent behavior against real patient inputs. Identified a recurring failure mode where agents responded to out-of-scope clinical questions (e.g., medication safety questions, contraindication questions) rather than redirecting to a clinician channel as the system prompt instructed. Documented the patterns of guardrail bypass and the patient-safety implications, and recommended pausing live deployment until the guardrail problem could be resolved. Skills exercised: clinical accuracy review of model outputs; identification of unsafe completions; analysis of prompt-instruction adherence vs. drift; clinical subject-matter expertise in reproductive health, abortion care, and gender-affirming care; recommendations on safe deployment thresholds. Patient population context: multilingual, multistate telehealth practice serving reproductive health and gender-affirming care patients.