.png){kind=icon}
.png){kind=icon}
Return
A Machine Learning and Large Language Model Tool for Systematic Literature Reviews of Health Economic Evidence: A Validation Study
.png){kind=icon}
PRE
AI
DOI：10.1007/s40273-026-01648-7.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Systematic literature reviews offer high potential for efficiency gains from artificial intelligence (AI), now integrated into several systematic literature review software platforms. Validation studies show acceptable sensitivity, specificity, and accuracy for AI-assisted systematic literature reviews of clinical trial publications. Unlike trials, economic model publications lack consistency in content, terminology, and structure. We aimed to test the efficiency and accuracy of AI-assisted search, screening, and data extraction when applied to a systematic literature review of economic evaluations. A previously conducted manual systematic literature review of economic evaluations for chronic rhinosinusitis with nasal polyps was replicated using a machine learning-based inclusion prediction model (Robot Screener) and a large language model-based criteria screener (Smart Screener) within Nested Knowledge software, with performance benchmarked against the original human-conducted systematic literature review. The AI-generated search retrieved 22/43 (51%) PubMed articles from the original systematic literature review. Accuracy exceeded 95% for title/abstract screening but fell below 80% for full-text screening. Extraction was reliable for high-level model descriptors and general study characteristics, but less so for model structures, health states, outcomes, and distinguishing sensitivity from scenario analyses and complex modeling assumptions for duration of response, discontinuation, surgery, and mortality. Estimated time savings ranged from ~20% (data extraction) to 60% (title/abstract screening and searches), varying by task human validation requirement. Artificial intelligence-driven tools performed well for title/abstract screening and general data extraction but were less accurate for full-text screening and interpretation of modeling choices. They can increase systematic literature review efficiency for economic evaluations but fall below the reliability seen for systematic literature reviews of clinical trials. Artificial intelligence has the potential to speed up systematic literature reviews by assisting researchers with identifying articles and organizing information from the articles. In this study, artificial intelligence-assisted tools embedded within a commercial systematic literature review platform were applied to replicate a systematic review of economic evaluations, providing practical insights into their readiness for use in health economics and outcomes research and health technology assessment workflows. Artificial intelligence performed well in early-stage screening, achieving more than 95% accuracy when reviewing titles and abstracts but its accuracy dropped below 80% when screening full texts. Artificial intelligence also reliably extracted general data such as study perspective, comparators, geography, and time horizon, but it struggled with more complex elements and those that required some interpretation and experience with these types of studies, including model structure, health states, sensitivity analyses, and key modeling assumptions. Estimated time savings ranged from approximately 20% for data extraction to 60% for title and abstract screening and database searches, though savings were highly dependent on the approach taken and the extent of human validation required. Overall, the study shows that artificial intelligence can improve efficiency in aspects of the process of conducting a systematic literature review of economic evaluations by streamlining general tasks but still requires substantial human oversight.
Journal
IF:
4.6
Papers:
214
Citations:
7.3K