A Transformer-Based Framework for Scene Text Recognition

Prabu Selvam

doi:10.1109/ACCESS.2022.3207469

A Transformer-Based Framework for Scene Text Recognition

2022-01-01

4

OA

AI

P

Prabu Selvam

K

K. Joseph Abraham Sundar

C

Carlos Andrés Tavera Romero

DOI：10.1109/ACCESS.2022.3207469

Original

Original request for help

Save

Abstract

En 中文

Scene Text Recognition (STR) has become a popular and long-standing research problem in computer vision communities. Almost all the existing approaches mainly adopt the connectionist temporal classification (CTC) technique. However, these existing approaches are not much effective for irregular STR. In this research article, we introduced a new encoder-decoder framework to identify both regular and irregular natural scene text, which is developed based on the transformer framework. The proposed framework is divided into four main modules: Image Transformation, Visual Feature Extraction (VFE), Encoder and Decoder. Firstly, we employ a Thin Plate Spline (TPS) transformation in the image transformation module to normalize the original input image to reduce the burden of subsequent feature extraction. Secondly, in the VFE module, we use ResNet as the Convolutional Neural Network (CNN) backbone to retrieve text image features maps from the rectified word image. However, the VFE module generates one-dimensional feature maps that are not suitable for locating a multi-oriented text on two-dimensional word images. We proposed 2D Positional Encoding (2DPE) to preserve the sequential information. Thirdly, the feature aggregation and feature transformation are carried out simultaneously in the encoder module. We replace the original scaled dot-product attention model as in the standard transformer framework with an Optimal Adaptive Threshold-based Self-Attention (OATSA) model to filter noisy information effectively and focus on the most contributive text regions. Finally, we introduce a new architectural level bi-directional decoding approach in the decoder module to generate a more accurate character sequence. Eventually, We evaluate the effectiveness and robustness of the proposed framework in both horizontal and arbitrary text recognition through extensive experiments on seven public benchmarks including IIIT5K-Words, SVT, ICDAR 2003, ICDAR 2013, ICDAR 2015, SVT-P and CUTE80 datasets. We also demonstrate that our proposed framework outperforms most of the existing approaches by a substantial margin.

Keywords:

Text recognition

Transformers

Optical character recognition

Decoding

Distortion

Image recognition

Image analysis

Connectionist temporal classification

scene text recognition

self-attention

transformer

optical character recognition

deep learning

A Transformer-Based Framework for Scene Text Recognition

Abstract

Journal

Organization

Cited Papers