.png){kind=icon}
.png){kind=icon}
Return
Constructing Compact BPE Token DFAs
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-02602-6_3.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Byte pair encoding (BPE) tokenization is a popular technique for subdividing text into relevant subwords and is frequently used in large language model systems. Since the tokenization procedure is deterministic and produces regular languages, it is beneficial to characterize it in terms of finite automata to allow for further automata-aided processing, such as pattern matching. In this paper, we demonstrate how to build such automata efficiently in practice, applying a series of optimization techniques to represent them compactly.
Keywords:
COMPRESSION
Journal
I
IF:
0
Papers:
22
Citations:
0