.png){kind=icon}
返回
A Lip Reading Method Based on 3D Convolutional Vision Transformer
.png){kind=icon}
PRE
AI
DOI：10.1109/ACCESS.2022.3193231.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Lip reading has received increasing attention in recent years. It judges the content of speech based on the movement of the speaker's lips. The rapid development of deep learning has promoted progress in lip reading. However, due to lip reading needs to process the information of continuous video frames, it is necessary to consider the correlation information between adjacent images and the correlation between long-distance images. Moreover, lip reading recognition mainly focuses on the subtle changes of lips and their surrounding environment, and it is necessary to extract the subtle features of small-size images. Therefore, the performance of machine lip reading is generally not high, and the research progress is slow. In order to improve the performance of machine lip reading, we propose a lip reading method based on 3D convolutional vision transformer (3DCvT), which combines vision transformer and 3D convolution to extract the spatio-temporal feature of continuous images, and take full advantage of the properties of convolutions and transformers to extract local and global features from continuous images effectively. The extracted features are then sent to a Bidirectional Gated Recurrent Unit (BiGRU) for sequence modeling. We proved the effectiveness of our method on large-scale lip reading datasets LRW and LRW-1000 and achieved state-of-the-art performance.
Keyword:
Feature extraction
Transformers
Lips
Visualization
Speech recognition
Hidden Markov models
Three-dimensional displays
Lip reading
3D convolution
vision transformer
sequence modeling
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Molecular and Pharmacological Characterization of GABAA Receptors in the Rat Pituitary大鼠垂体中GABAA 受体的分子和药理学特征
PREAI
分享收藏Solution structure of the Lewis x oligosaccharide determined by NMR spectroscopy and molecular dynamics simulations.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏分享收藏Metal-organic chemical vapor deposition of indium selenide films using a single-source precursor使用单源前体对硒化铟薄膜进行金属有机化学气相沉积
PREAI
分享收藏Single-cell RNA sequencing reveals intrinsic and extrinsic regulatory heterogeneity in yeast responding to stressPLOS Biology
OAAI
PLOS BiologyIF0
OAAI分享收藏
查看更多内容