PHI-GUARD: Compliance-Aware LLM Routing for Healthcare with Distribution-Free Safety Guarantees

Healthcare organizations deploying Large Language Models face a regulatory tradeoff: route queries through cloud APIs for cost efficiency and risk exposing protected health information (PHI), or keep everything on-premises. Existing LLM routers (RouteLLM, FrugalGPT, PILOT, BaRP) optimize cost against quality-none enforce data-residency constraints. We propose PHI-GUARD, a framework that treats data sensitivity as a hard compliance constraint. We develop SafeTS (Safetyconstrained Thompson Sampling), which builds safe action sets from classifier posteriors, and CARES (Compliance-Aware Residual Envelope Scoring), which provides distribution-free safety guarantees via asymmetric conformal residuals-requiring only exchangeability, not calibration. On 30,000 MIMIC-IV clinical queries with 10,000-query MIMIC-III cross-validation, our Dis-tilBERT classifier achieves 98.4% accuracy with 97.4% T3 (PHI) recall, but 8 misclassifications exceed confidence 0.80 (max 0.997). Static routing leaks 0.92% of PHI queries. SafeTS reduces violations from 83 to 1-8 through safe action sets but cannot eliminate them. CARES is the only strategy achieving zero violations with a verified distribution-free bound (δ + 1/(n + 1) = 0.005), routing 35.6% to cloud. Classifier ablation confirms CARES maintains zero violations across all tested architectures-from DistilBERT (98.4% accuracy) to a rule-based heuristic (56.2% accuracy)-demonstrating that the guarantee adapts to classifier quality automatically. Cross-dataset evaluation confirms CARES maintains zero violations despite 11-point T3 recall degradation on MIMIC-III, while StaticILP violations increase 5.8×. End-toend validation with real API endpoints (GPT-4o-mini, Llama-3.1) confirms safety guarantees transfer to live execution.