Mathematical expression recognition using a new deep neural model

Abolfazl Mirkazemy

doi:10.1016/j.neunet.2023.08.045

2023-10-01

PRE

Abolfazl Mirkazemy

Peyman Adibi *

Seyed Mohhamad Saied Ehsani

Alireza Darvishy

Hans‐Peter Hutter

DOI：10.1016/j.neunet.2023.08.045

原文链接

原文求助

摘要

En 中文

In this paper, we propose a novel deep neural model for Mathematical Expression Recognition (MER). The proposed model uses encoder-decoder transformer architecture that is supported by additional pre/post-processing modules, to recognize the image of mathematical formula and convert it to a well-formed language. A novel pre-processing module based on domain prior knowledge is proposed to generate random pads around the formula's image to create more efficient feature maps and keeps all the encoder neurons active during the training process. Also, a new post-processing module is developed which uses a sliding window to extract additional position-based information from the feature map, that is proved to be useful in the recognition process. The recurrent decoder module uses the combination of feature maps and the additional position-based information, which takes advantage of a soft attention mechanism, to extract the formula context into the LaTeX well-formed language. Finally, a novel Reinforcement Learning (RL) module processes the decoder output and tunes its results by sending proper feedbacks to the previous steps. The experimental results on im2latex100k benchmark dataset indicate that each devised pre/post-processing as well as the RL refinement module has a positive effect on the performance of the proposed model. The results also demonstrate the higher accuracy of the proposed model compared to the state-of-the-art methods. (c) 2023 Elsevier Ltd. All rights reserved.

Keyword:

Mathematical expression recognition

Deep learning

Encoder-decoder architecture

Attention

Scientific documents accessibility