.png){kind=icon}
.png){kind=icon}
Return
Vision-language pre-training for graph-based handwritten mathematical expression recognition
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2025.111346.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Vision-language pre-training models have shown promise in improving various downstream tasks. However, handwritten mathematical expression recognition (HMER), as atypical structured learning problem, can hardly benefit from existing pre-training methods due to the presence of multiple symbols and complicated structural relationships, as well as the scarcity of paired data. To overcome these problems, we propose a V ision-Language P re-training paradigm for G raph-based HMER (VLPG), utilizing unpaired mathematical expression images and LaTeX labels. Our HMER model is built upon a graph parsing method with superior explainability, which is enhanced by the proposed graph-structure aware transformer decoder. Based on this framework, the symbol localization pretext task and language modeling task are employed for vision-language pre-training. First, we make use of unlabeled mathematical symbol images to pre-train the visual feature extractor through the localization pretext task, improving the symbol localization and discrimination ability. Second, the structure understanding module is pre-trained using LaTeX corpora through language modeling task, which promotes the model's context comprehension ability. The pre-trained model is fine-tuned and aligned on the downstream HMER task using benchmark datasets. Experiments on public datasets demonstrate that the pre-training paradigm significantly improves the mathematical expression recognition performance. Our VLPG achieves state-of-the-art performance on standard CROHME datasets and comparable performance on the HME100K dataset, highlighting the effectiveness and superiority of the proposed model. We released our codes at https://github.com/guohy17/VLPG.
Keywords:
Pre-training
Pretext task
Language modeling
Vision-language model
Handwritten mathematical expression
recognition