The Multimodal Anonymizer: a fully local multi-agent AI system for medical data deidentification

Abstract Background Safe reuse of multimodal hospital data for AI development is limited by the absence of reliable, context-aware deidentification across multimodal data and longitudinal patient data. Existing approaches are largely modality-specific and can indiscriminately remove clinically important information. Methods We developed the Multimodal Anonymizer, a modular, locally deployable multi-agent framework integrating multimodal large language models, task-specific neural networks and rule-based transformations. We evaluated 16 orchestrator model configurations on a benchmark built from publicly available data and hospital data from our institution. The benchmark dataset included data from different origins: 250 MIMIC-IV patients with synthetically injected personally identifiable information (PII) supplemented with head CT, face images, handwriting, audio, German clinical-text datasets and local data. Primary outcomes were deidentification sensitivity and preservation of clinically important content; secondary analyses examined model characteristics, reproducibility, and performance against leading market and open-source solutions. Results The best local configuration—the orchestrator being Qwen3-VL-235B-A22B-Thinking—achieved near-complete deidentification across all datasets, with per-patient sensitivity of 98.80% (95%-CI 97.20; 100), and per-PII sensitivity of 99.82% (95%-CI 99.76; 99.88). Critical clinical preservation was 99.60% (95%-CI 98.80; 100) per-patient, and clinical preservation was 99.61% (95%-CI 99.51; 99.71) per-file. All modalities achieved at least 98.30% sensitivity (lower bound 95%-CI). On our local data, the system achieved a deidentification sensitivity of 100% per-patient and per-PII; and a critical clinical preservation of 100% per-patient as well as a clinical preservation of 99.97% (95%-CI 99.91; 100) per-file. When comparing orchestrators, the leading local models were similar to proprietary models (GPT-5.2) in deidentification sensitivity while showing higher deidentification specificity. The Multimodal Anonymizer outperformed previous tools on most modalities. Conclusion Near-complete, utility-preserving deidentification of multimodal clinical data is achievable with a unified, locally deployable multi-agent system, enabling safer large-scale reuse of hospital data for research and AI development. Highlights Framework for deidentification of multimodal clinical data. Multimodal deidentification with preservation of clinically relevant content. On-premises plug-and-play deployment for local data processing. Evaluation of 16 model configurations and comparison with existing tools. Assessment on external, multilingual and site-specific datasets. Short Description The Multimodal Anonymizer is a fully local, multi-agent system that prepares multimodal clinical records for privacy-preserving reuse by coordinating multimodal large language model reasoning, specialist neural networks, rule-based transformations, and iterative verification. Across benchmarks spanning text, tables, PDFs, imaging, metadata, filenames, audio, and handwriting, its best configuration using a local open-source multimodal large language model achieved 98.80% patient-level deidentification sensitivity and 99.60% preservation of clinically critical content, performing comparably to proprietary models and outperforming established deidentification tools across most modalities. Graphical Abstract