Clinical readiness of LLM-generated oncologic CT/MRI impressions: A blinded comparison with subspecialty oncoradiologists at a tertiary cancer centre

Background Large language models (LLMs) show promising results in automating medical documentation, but their performance in generating accurate radiology impressions for complex oncologic cases has not been systematically evaluated against subspecialty radiologist standards. Purpose To compare the quality and accuracy of AI-generated radiology report impressions with oncoradiologist-authored impressions for oncologic CT and MRI reports, and to identify error patterns of AI-generated reports in a tertiary cancer centre. Materials and Methods This retrospective study evaluated 600 radiology report impressions generated by two LLMs and a subspecialty radiologist from 200 oncologic imaging studies performed for cancer surveillance or response assessment. The study included 100 CT and 100 MRI reports of scans performed between January 2024 and July 2025. Three impressions were generated per case, each by a subspecialty radiologist, GPT-4- (LLM 1) (OpenAI, San Francisco, CA, USA) and Gemini Pro 2.5 - (LLM 2) (Google DeepMind, Mountain View, CA, USA), both accessed in August 2025 hereafter referred to as LLM-1 and LLM-2, respectively, using standardised prompts. All impressions were evaluated by an independent radiologist, blinded to the report source, using 5-point Likert scales across five quality metrics: clinical adequacy, factual accuracy, comprehensiveness, clarity/redundancy, and use of response assessment. Differences across groups were evaluated using Friedman tests with Bonferroni-corrected pairwise comparisons. Results Radiologist-generated report impressions significantly outperformed both LLM models across all evaluated metrics (all p < 0.001). Median scores [IQR] for radiologists versus LLM-1 and LLM-2 were as follows: clinical adequacy, 5 [4-5] vs. 4 [3-4] vs. 4 [4-5]; factual accuracy, 5 [5–5] vs. 5 [3–5] vs. 5 [4–5]; comprehensiveness, 5 [4-5] vs. 4 [4-5] vs. 5 [4-5]; and clarity, 5 [5-5] vs. 3 [3-4] vs. 5 [4-5], respectively. Major errors were identified in 6.3% of LLM-1generated impressions and 5.1% of LLM-2 generated impressions, compared with 1.0% of radiologist-authored impressions. Common error patterns included hallucinations, ranging from complete fabrications to the misrepresentation of diagnostic uncertainty as unwarranted certainty in clinical impressions. Conclusion While LLMs show growing potential in structured impression generation, their tendency towards hallucinations, including both spurious findings and the distortion of diagnostic uncertainty into unwarranted confidence, underscores the need for expert radiologist review of all LLM-generated impressions in clinical practice.