Detecting dementia from speech and transcripts using transformers

doi:10.1016/j.csl.2023.101485

Detecting dementia from speech and transcripts using transformers

2023-04-01

16

OA

AI

DOI：10.1016/j.csl.2023.101485

Original

Save

View PDF

Abstract

En 中文

Alzheimer's disease (AD) constitutes a neurodegenerative disease with serious consequences to peoples' everyday lives, if it is not diagnosed early since there is no available cure. Alzheimer's is the most common cause of dementia, which constitutes a general term for loss of memory. Due to the fact that dementia affects speech, existing research initiatives focus on detecting dementia from spontaneous speech. However, little work has been done regarding the conversion of speech data to Log-Mel spectrograms and Mel-frequency cepstral coefficients (MFCCs) and the usage of pretrained models. Concurrently, little work has been done in terms of both the usage of transformer networks and the way the two modalities, i.e., speech and transcripts, are combined in a single neural network. To address these limitations, first we represent speech signal as an image and employ several pretrained models, with Vision Transformer (ViT) achieving the highest evaluation results. Secondly, we propose multimodal models. More specifically, our introduced models include Gated Multimodal Unit in order to control the influence of each modality towards the final classification and crossmodal attention so as to capture in an effective way the relationships between the two modalities. Extensive experiments conducted on the ADReSS Challenge dataset demonstrate the effectiveness of the proposed models and their superiority over state-of-the-art approaches.

Keywords:

Dementia

Speech

log-Mel spectrogram

Mel-frequency cepstral coefficients

Vision transformer

Gated multimodal unit

Crossmodal attention

Detecting dementia from speech and transcripts using transformers

Abstract

AI Summary

Journal

Organization