Transient chaos in bidirectional encoder representations from transformers

doi:10.1103/PhysRevResearch.4.013204

2022-03-16

DOI：10.1103/PhysRevResearch.4.013204

原文链接

原文求助

摘要

En 中文

Language is an outcome of our complex and dynamic human-interactions and the technique of natural language processing (NLP) is hence built on human linguistic activities. Along with generative pretrained transformer (GPT), bidirectional encoder representations from transformers (BERT) has recently gained its popularity, owing to its outstanding NLP capabilities, by establishing the state-of-the-art scores in several NLP benchmarks. A LITE BERT (ALBERT) is literally characterized as a lightweight version of BERT, in which the number of BERT parameters is reduced by repeatedly applying the same neural network called transformer's encoder layer. By pretraining the parameters with a massive amount of natural language data, ALBERT can convert input sentences into versatile high-dimensional vectors potentially capable of solving multiple NLP tasks. In that sense, ALBERT can be regarded as a well-designed high-dimensional dynamical system whose operator is the transformer's encoder, and essential structures of human language are thus expected to be encapsulated in its dynamics. In this study, we investigated the embedded properties of pretrained ALBERT to reveal how NLP tasks are effectively solved by exploiting its dynamics. We thereby aimed to explore the nature of human language from the dynamical expressions of the NLP model. Our analysis consists of two parts, namely, shortand long-term analyses, according to timescale differences to capture the dynamics. Our short-term analysis clarified that the pretrained model stably yields trajectories with higher dimensionality in a certain time range, which would enhance the expressive capacity required for NLP tasks. Also, our long-term analysis revealed that ALBERT intrinsically shows transient chaos, a typical nonlinear phenomenon showing chaotic dynamics only in its transient, and the pretrained ALBERT model tends to produce the chaotic trajectory for a significantly longer time period compared to a randomly initialized one. Our results imply that local chaoticity would contribute to improving NLP performance, uncovering a novel aspect in the role of chaotic dynamics in human language behaviors.

Keyword:

LANGUAGE