.png){kind=icon}
返回
Audio-visual speech recognition using deep learning
.png){kind=icon}
OA
AI
DOI：10.1007/s10489-014-0629-7.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.
Keyword:
Audio-visual speech recognition
Feature extraction
Deep learning
Multi-stream HMM
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
3.5
论文数:
7.6K
被引数:
1.7W
机构
引用论文
Nanohydroxyapatite Loaded with 5-Fluorouracil and Calendula officinalis L. Plant Extract Rich in Myo-Inositols for Treatment of Ovarian Cancer Cells负载5-Fluorouracil和富含Myo-Inositols的Calendula officinalis L.植物提取物的纳米羟基磷灰石，用于治疗卵巢癌细胞.png)
Coatings
OAAI
CoatingsIF0
OAAI分享收藏Improved Particle Size Control for the Dispersion Polymerization of Methyl methacrylate in Supercritical Carbon Dioxide超临界二氧化碳中甲基丙烯酸甲酯分散聚合的改进粒度控制OAAI
OAAI分享收藏分享收藏
查看更多内容