Prompting large language models and evaluating inter- and intra-rater agreement for cancer progression assessment from radiology reports

Background: Manual annotation of free-text radiology reports is time-consuming and costly, delaying real-world evidence (RWE) studies in oncology. This study aimed to evaluate the performance of large language models (LLMs) in annotating cancer progression from Danish free-text radiology reports. The objectives were to determine whether human-to-LLM inter-rater agreement was non-inferior to human-to-human agreement, establish human intra-rater agreement, and develop a framework for tuning LLM performance to RWE needs. Materials and methods: We identified 376 radiology reports from 184 patients with metastatic breast cancer from Danish electronic health records. Six human annotators, including two experts, classified radiology reports as progressive disease (PD) or non-PD. A 'reverse questioning' strategy was used to evaluate five LLM model series (Mistral, Gemma, Gemma 2, Llama 3, and Llama 3.1). Bootstrapping estimated confidence intervals (CIs) and assessed non-inferiority of the best-performing LLM ensemble compared with human agreement, using a non-inferiority margin of 0.1. Results: < 0.001). The best-performing ensemble model, Llama 3.1:70B, achieved 100% sensitivity, a specificity of 90%, and an F1 score of 84% on the test set. The mean human intra-rater variability was 0.87. Conclusions: The proposed LLM framework was non-inferior to human annotators in classifying cancer progression from free-text radiology reports. This offers significant potential for using LLMs as a tool for identifying tumor progression events in clinical assessment and research.