Mini-3DCvT: a lightweight lip-reading method based on 3D convolution visual transformer

doi:10.1007/s00371-024-03515-y

2024-06-11

PRE

DOI：10.1007/s00371-024-03515-y

原文链接

原文求助

摘要

En 中文

Lip-reading has attracted more and more attention in recent years, and has wide application prospects and value in areas such as human-computer interaction, surveillance and security and audiovisual speech recognition. However, research on lip-reading has been slow due to the complexity of dealing with the fine spatial features of small-sized images of continuous video frames and the temporal features between images. In this paper, to address the challenges in extracting visual spatial features, temporal features and model light weighting, we propose a high-precision, highly robust and lightweight lip-reading method, Mini-3DCvT, which combines visual transforms and 3D convolution to extract spatiotemporal feature of continuous images, and makes full use of the properties of convolution and transforms to effectively extract local and global features of continuous images, use weight transformation and weight distillation in the convolution and transformer structures for model compression, and then send the extracted features to a bidirectional gated recurrent unit for sequence modeling. Experimental results on the large-scale public lip-reading datasets LRW and LRW-1000 show that this paper's method achieves 88.3% and 57.1% recognition accuracy on both datasets, and effectively reduces the model computation and number of parameters, improving the overall performance of the lip-reading model.

Keyword:

Lip-reading

3D convolution

Visual transformer

Spatial-temporal features

Model light weighting

期刊

Visual Computer

IF:

2.9

论文数:

4.6K

被引数:

6.5K

机构

north china institute of aerospace engineering

学者数:

800

论文数: 423

被引数: 1

引用论文

Hippocampal slices of kindled rats reveal calcium involvement in epileptogenesis

Experimental Brain Research

IF0

1985-01-01

PREAI

W.J. Wadman; U. Heinemann; A. Konnerth; S. Neuhaus

The Eye-RIS CMOS Vision System

Analog Circuit Design

IF0

2008-01-01

PREAI

Ángel Rodríguez-Vázquez; Rafael Domínguez-Castro; Francisco Jiménez-Garrido; Sergio Morillas; Juan Listán; Luis Alba; Cayetana Utrera; Servando Espejo; Rafael Romay

Solution structure of the Lewis x oligosaccharide determined by NMR spectroscopy and molecular dynamics simulations

Biochemistry

IF0

2002-05-01

PREAI

Kristine E. Miller; Chaitali Mukhopadhyay; Perseveranda Cagas; C. Allen Bush

Grading and frequency dependence of the resilient modulus of ballast

Géotechnique Letters

IF0

2018-12-01

PREAI

Y. Sun; S. Nimbalkar; C. Chen