.png){kind=icon}
返回
Trustworthy Continuous Sign Language Recognition
.png){kind=icon}
PRE
AI
DOI：10.1109/TMM.2025.3640024.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Continuous sign language recognition (CSLR) uses visual cues (e.g., hands, face, mouth, and body) to automatically recognize the sign language of the hearing-impaired, helping them to actively communicate with hearing people. The effects of these visual cues change dynamically with the demonstration of sign language. However, previous CSLR methods usually model visual information from the entire frame or simple fused visual cues, and thus do not well describe such dynamic change among visual cues. Therefore, we propose the Trustworthy Fusion Network (TFN) of visual cues for CSLR, which comprises two fundamental modules: Intra-cue Cross-modality Feature Fusion module (IntraCFF) and Inter-cue Trustworthy Fusion module (InterTF). IntraCFF uses the calibrated joint-belief method to dynamically fuse cross-modality features of RGB and keypoint information, to obtain a robust visual cue feature. InterTF innovatively employs the Dempster-Shafer Theory (DST) to evaluate the uncertainty of different cues in expressing sign movements. Then, the trustworthy fusion via DST is used to adaptively weigh and credibly fuse the visual cues based on uncertainty. In addition, to address the semantic gap when fusing different cues, we design consistency fusion constraints during the training stage. These constraints enhance the semantic consistency of different cues with global sign movements. Experiments on publicly CSLR datasets validate the effectiveness of our TFN.
Keyword:
Continuous sign language recognition (CSLR)
trustworthy fusion
dempster-shafer theory
期刊
IF:
9.7
论文数:
4.5K
被引数:
2.4W
机构
引用论文
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition用于基于骨架的动作识别的时空图卷积网络
OAAI
.png){kind=icon}
OAAI分享收藏
Gloss Prior Guided Visual Feature Learning for Continuous Sign Language Recognition用于连续手语识别的Gloss先验引导的视觉特征学习
PREAI
分享收藏
查看更多内容