Infusing clinical knowledge into language models by subword optimisation and embedding initialisation

Abul Hasan

doi:10.1016/j.compbiomed.2025.110747

2025-08-07

DOI：10.1016/j.compbiomed.2025.110747

原文链接

原文求助

摘要

En 中文

• 该研究提出了一种新颖的分词方法，利用基于领域特定概念（如药物、疾病）的词元全局表示，这些概念来自UMLS等本体或任务特定语料库。 • 在训练或推理阶段，采用词和句子级别的优化来选择最优的词元表示。 • 该研究提出了针对新词元的嵌入初始化方法，消除了语言模型预训练的需求。 • 使用K-Tokeniser构建的模型在自动临床编码任务中取得了显著的Micro F1分数提升，达到13%。该模型仅需50%的训练数据用于概念提取，且在自动编码任务中仅需不到20%的数据即可超越基线临床BERT模型。

Keyword:

Tokenisation

Language model

BERT

Clinical concept and relation extraction

ICD-9 coding classification

Phenotype identification

Document classification

期刊

Computers in Biology and Medicine

IF:

6.3

论文数:

8.3K

被引数:

3.3W

机构

Institute of Health Informatics

学者数:

论文数: 21

被引数: 0

University of Edinburgh

学者数:

5.2W

论文数: 4.6W

被引数: 71

引用论文

Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting

Journal of the American Medical Informatics Association

IF0

2019-08-28

OAAI

Xi Yang; Jiang Bian; Ruogu Fang; Ragnhildur I Bjarnadottir; William R Hogan; Yonghui Wu

Land in Latin America 拉丁美洲的土地

The Encyclopedia of Postcolonial Studies

IF0

2016-01-28

PREAI

Adriana Johnson

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS

IF6.9

2022-03-25

OAAI

Xue, Linting; Barua, Aditya; Constant, Noah; Al-Rfou, Rami; Narang, Sharan; Kale, Mihir; Roberts, Adam; Raffel, Colin

Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training 通过视觉语言预训练对医学图像和文本进行多模态理解和生成

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS

IF6.8

2022-12-01

OAAI

Moon, Jong Hak; Lee, Hyungyung; Shin, Woncheol; Kim, Young-Hak; Choi, Edward

A Pseudo Label-Wise Attention Network for Automatic ICD Coding

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS

IF6.8

2022-10-01

OAAI

Wu, Yifan; Zeng, Min; Yu, Ying; Li, Yaohang; Li, Min

LitCovid: an open database of COVID-19 literature LitCovid: 一个开放的新型冠状病毒肺炎文献数据库

NUCLEIC ACIDS RESEARCH

IF13.1

2020-11-09

154

OAAI

Chen, Qingyu; Allot, Alexis; Lu, Zhiyong

SemEHR: A general-purpose semantic search system to surface semantic data from clinical notes for tailored care, trial recruitment, and clinical research*

Journal of the American Medical Informatics Association

IF0

2018-01-19

OAAI

Honghan Wu; Giulia Toti; Katherine I Morley; Zina M Ibrahim; Amos Folarin; Richard Jackson; Ismail Kartoglu; Asha Agrawal; Clive Stringer; Darren Gale; Genevieve Gorrell; Angus Roberts; Matthew Broadbent; Robert Stewart; Richard JB Dobson

Limitations of Transformers on Clinical Text Classification 《变形金刚》对临床文本分类的局限性

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS

IF6.8

2021-09-01

OAAI

Gao, Shang; Alawad, Mohammed; Young, M. Todd; Gounley, John; Schaefferkoetter, Noah; Yoon, Hong Jun; Wu, Xiao-Cheng; Durbin, Eric B.; Doherty, Jennifer; Stroup, Antoinette; Coyle, Linda; Tourassi, Georgia

2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records

Journal of the American Medical Informatics Association

IF0

2019-10-04

OAAI

Sam Henry; Kevin Buchan; Michele Filannino; Amber Stubbs; Ozlem Uzuner

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Proceedings of the AAAI Conference on Artificial Intelligence

IF0

2019-07-17

OAAI

Jeremy Irvin; Pranav Rajpurkar; Michael Ko; Yifan Yu; Silviana Ciurea-Ilcus; Chris Chute; Henrik Marklund; Behzad Haghgoo; Robyn Ball; Katie Shpanskaya; Jayne Seekins; David A. Mong; Safwan S. Halabi; Jesse K. Sandberg; Ricky Jones; David B. Larson; Curtis P. Langlotz; Bhavik N. Patel; Matthew P. Lungren; Andrew Y. Ng

没有更多内容