.png){kind=icon}
返回
Large language models encode clinical knowledge
.png){kind=icon}
OA
AI
DOI：10.1038/s41586-023-06291-2.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications. Med-PaLM, a state-of-the-art large language model for medicine, is introduced and evaluated across several medical question answering tasks, demonstrating the promise of these models in this domain.
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
48.5
论文数:
3.4W
被引数:
96.7W
机构
引用论文
Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records.png)
NATURE PROTOCOLSOAAI
NATURE PROTOCOLSIF16
OAAI分享收藏Best Practices for Developing and Validating Scales for Health,Social, and Behavioral Research:A Primer开发和验证健康，社会和行为研究量表的最佳实践: 入门OAAI
OAAI分享收藏Erratum to: Scale development: ten main limitations and recommendations to improve future research practices勘误：量表开发：十个主要局限性以及对改进未来研究实践的推荐OAAI
OAAI分享收藏Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information患者教育材料评估工具 (PEMAT) 的开发: 印刷和视听患者信息的可理解性和可操作性的新度量OAAI
OAAI分享收藏Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery考虑生成式预训练变压器3 (GPT-3) 在医疗保健中的可能性和陷阱NPJ DIGITAL MEDICINEOAAI
NPJ DIGITAL MEDICINEIF15.1
OAAI分享收藏TYDI QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
OAAI
OAAI分享收藏
查看更多内容