SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

doi:10.1007/s00371-024-03372-9

Return

SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

2024-05-03

PRE

Fu, PengBin

Xiao, Ganyun

Yang, HuiRong *

DOI：10.1007/s00371-024-03372-9

Original

Original request for help

Save

Abstract

En 中文

The complex two-dimensional structure poses huge challenges for handwritten mathematical expression recognition (HMER). Many researchers process the LaTeX sequence into a tree structure and then design tree decoders based on RNN to address this issue. However, RNNs have problems with long-term dependency due to their structural characteristics. Although Transformers solve the long-term dependency problem, tree decoders based on Transformers are rarely used for HMER because the attention coverage is significantly insufficient when the distance between parent and child nodes is large in tree structures. In this paper, we propose a novel offline HMER model SATD incorporating a tree decoder based on Transformer to learn the implicit structural relationships in LaTeX strings. Moreover, to address the issue of distant parent-child nodes, we introduce a multi-scale attention aggregation module to refine attention weights using contextual information with different receptive fields. Experiments on CROHME2014/2016/2019 and HME100K datasets demonstrate performance improvements, achieving accuracy rates of 63.45%/60.42%/61.05% on the CROHME 2014/2016/2019 test sets. The source code https://github.com/EnderXiao/SATD/ of this work will be publicly available.

Keywords:

Offline handwritten mathematical expression recognition

Transformer

Coverage attention

Tree decoder

Attention