.png){kind=icon}
返回
Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement
.png){kind=icon}
OA
AI
DOI：10.1016/j.media.2024.103299.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision- language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the finegrained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.
Keyword:
Vision-language representation learning
Medical foundation models
Knowledge-emphasized report refinement
Iterative learning
期刊
IF:
11.8
论文数:
3.9K
被引数:
2.4W
机构
引用论文
A Coarse-to-Fine Deformable Transformation Framework for Unsupervised Multi-Contrast MR Image Registration with Dual Consistency Constraint具有双重一致性约束的无监督多对比度MR图像配准的粗到细可变形变换框架
OAAI
OAAI分享收藏Shades of grey: the legal trade in CITES-listed birds in Singapore, notably the globally threatened African grey parrot Psittacus erithacus.png)
OryxOAAI
OryxIF0
OAAI分享收藏Annotation-efficient deep learning for automatic medical image segmentationNATURE COMMUNICATIONSOAAI
NATURE COMMUNICATIONSIF15.7
OAAI分享收藏PyMIC: A deep learning toolkit for annotation-efficient medical image segmentationPyMIC: 用于注释高效医学图像分割的深度学习工具包OAAI
OAAI分享收藏
查看更多内容