.png){kind=icon}
.png){kind=icon}
Return
Radiologist Detection of Deficiencies in LLM-Generated Patient Communications About Radiology Reports: A Multi-Reader Study
.png){kind=icon}
PRE
AI
C
J
S
Y
H
DOI：10.1007/s10278-026-02109-5.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Patients increasingly use large language models (LLMs) to interpret radiology reports, yet the reliability of radiologist oversight in detecting errors in patient-facing LLM responses remains unquantified. This study evaluated the diagnostic accuracy of radiologists in detecting deficiencies in LLM-generated answers to simulated patient questions and quantified the incremental benefit of multi-reader oversight. In this retrospective multi-reader study, three board-certified radiologists independently evaluated 1200 LLM-generated answers (from three different models; 400 per model) to questions derived from 187 MIMIC-IV radiology reports. Answers were scored on a 3-point scale: 1 (clinically significant error), 2 (incomplete/ambiguous), and 3 (accurate). An adjudicated consensus served as the reference standard. Individual reader sensitivity, specificity, and multi-reader miss rates were compared. Of 1200 answers, 113 (9.4%) were deficient by consensus, including 18 clinically significant errors (1.5%) and 95 incomplete/ambiguous responses (7.9%). For clinically significant errors, readers missed 5.6–11.1% of cases. Across all deficiencies, individual reader sensitivity ranged from 81.4% (95% CI, 73.3–87.5) to 96.5% (95% CI, 91.3–98.6), with significant differences (McNemar p < 0.001). A single reader missed an average of 12.4% of deficiencies, whereas dual-reader review reduced the average miss rate to 0.6%; triple-reader review detected all deficiencies. Individual radiologist review of LLM-generated patient communications showed variable sensitivity, with some clinically significant errors remaining undetected. Dual-reader oversight significantly improves the safety and reliability of LLM-based communication tools.
Keywords:
Artificial intelligence
Radiology report
Quality assurance
Large language models (LLMs)
Patient communication
Journal
J
IF:
0
Papers:
416
Citations:
0
Organization
Cited Papers