BERT-Residual Quantum Language Model Inspired by ODE Multi-Step Method

Shaohui Liang

doi:10.1109/ACCESS.2025.3533554

2025-01-01

Shaohui Liang *

Yingkui Wang

Shuxin Chen

DOI：10.1109/ACCESS.2025.3533554

原文链接

原文求助

摘要

En 中文

Quantum-inspired language models model finer-grained semantic interactions in higher-order Hilbert spaces. However, previous methods usually capture semantic features based on context-free word vectors such as Word2Vec and GloVe. Building on natural language encoding, incorporating quantum-inspired density matrix modeling can capture more fine-grained semantic interactions. However, when applied to large pre-trained language models like BERT, using quantum density matrices often leads to issues such as gradient explosion or vanishing. Therefore, how to effectively integrate the quantum-inspired language model and the pre-trained model, and make them function under the fine-tuning paradigm of the pre-trained model has become a key issue for the further development of the quantum-inspired language model. Therefore, in this paper, we propose the BERT-Residual quantum language model inspired by the multi-step method of ordinary differential equations (ODE), using the density matrix to capture the semantic high-order interaction features missing in the BERT modeling process, and obtain the sentence representation, and perform the first step Residuals. Then quantum measurement is performed on the sentence representation, and the second step of residual connection is performed with the BERT layer. This residual connection method based on the multi-step method can more effectively combine the advantages of BERT representation and quantum density matrix representation to enhance representation learning. Experiments show that in text classification benchmarks, our proposed method generally surpasses baseline models.

Keyword:

Mathematical models

Semantics

Quantum mechanics

Encoding

Bidirectional control

Neural networks

Computational modeling

Vectors

Hilbert space

Text categorization

Pre-trained models