logo
CanyamAI summaries for academic research
language
MobileMobile
  • Phone FeatureWeChat Mini Program
EnglishEnglish
noticeNotice
BlogBlog
delete
arrow
Return

Arabic Lip Reading With Limited Data Using Deep Learning

delete2024-01-01
delete0
delete
OA
AI
Z
Zamen Jabr
S
Sauleh Etemadi
N
Nasser Mozayani *
DOI：10.1109/ACCESS.2024.3440646delete
deleteOriginal
deleteOriginal request for help
deleteShare
deleteSave
Abstract

Abstract

En 中文
Two main challenges faced by deep learning systems are related to the amount of data and the complexity of the model concerning the number and type of layers and the number of training parameters. In this paper, we propose an end-to-end Arabic lip-reading system that can be trained on a limited dataset, which combines a visual model consisting of Convolutional Neural Networks (CNNs) and a temporal model consisting of Gated Recurrent Units (GRUs) layers, taking into account the balance between the size of the dataset and the number of model parameters. For this purpose, we created a limited Arabic dataset that involved 20 words uttered by 40 native Arabic speakers; then, we exploited the redundant frames found in video sequences to train the Arabic visemes classifier separately. This classifier was later used as a visual model, as a pre-trained model, in our end-to-end system to extract the spatial features from videos, while the temporal model was used to process the context. Our proposed method is evaluated on 1) our dataset, we obtained an accuracy equal to 83.02%; 2) the Dweik et al. dataset, we obtained an improvement rate of approximate to 3% on the result recorded by their work. In addition, we employed the visemes classifier model for person identification using the viseme shape and obtained a high result.
Keywords:
Arabic lip reading
deep learning
limited dataset
Arabic lip reading
deep learning
limited dataset

Journal

IEEE Access cover
IEEE Access
IF:
3.6
Papers:
9.8W
Citations:
29.4W

Organization

No organization information available