OCR post-correction for detecting adversarial text images

Niddal Imam

doi:10.1016/j.jisa.2022.103170

2022-05-01

Niddal Imam *

Vassilios G. Vassilakis

DOI：10.1016/j.jisa.2022.103170

原文链接

查看原文

摘要

En 中文

The amount of images with embedded text shared on Online Social Networks (OSNs), such as Twitter orFacebook has been growing in recent years. It is becoming important to analyse the images uploaded into theseplatforms, as adversaries may spread images with toxic content or misinformation (i.e. spam). Optical characterrecognition (OCR) systems have been used to detect images with malicious content, where the embedded textgets extracted and classified using machine learning algorithms. However, most existing OCR-based systemsare adversary-agnostic models, in which the extracted text from an image is not checked by humans before theclassification. Consequently, these fully automated models become vulnerable to minor modifications of images'pixels or textual content (e.g.,character-levelperturbations), which do not affect human understanding, but couldcause the OCR systems to misrecognise the embedded text. In this paper, we propose an OCR post-correctionalgorithm to improve the robustness of OCR-based systems against images with perturbed embedded texts.Experimental results showed that our proposed algorithm improves the robustness of three state-of-the-art OCRmodels with at least 10% against adversarial text images, and it outperforms five spellcheckers in correctingadversarial text. Also, we evaluated the perceptibility of our adversarial images, and this study showed that91% of the participants were able to correctly recognise the adversarial text images. Additionally, we developedan adversary-aware OCR-based system for detecting adversarial text images using the proposed algorithm, andour evaluation results showed considerable improvement in the performance of an OCR-based system.

Keyword:

Deep learning

Spam image

OCR

Text recognition

Text classification

Adversarial text attack