.png){kind=icon}
返回
Tokenizers for African Languages
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3522285.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Despite incredible development in the field of natural language processing (NLP), there has been a huge gap in the performance of NLP tasks between high-resource languages (HRLs) and low-resource languages (LRLs). African languages belong mainly to the LRLs, and one of the major contributing factors to the performance gap is tokenization, which plays a crucial role in NLP performance in general. Many recent studies on African languages often rely on multilingual tokenizers or general-purpose tokenizers, which are not optimized for African languages. This may lead to suboptimal performance in downstream NLP tasks. In this paper, we systematically analyze the performance of language-specific tokenizers for three African languages: Swahili, Hausa, and Yoruba. By experimental results on two classification tasks (i.e. sentiment classification and news classification), we found that the language-specific tokenizers for African languages consistently outperformed other monolingual tokenizers, with performance gaps of up to 5.43% in sentiment classification and 4.58% in news classification. We also found that multilingual tokenizers generally work well if they are trained in many African languages rather than global HRLs. For instance, African multilingual tokenizers outperformed global multilingual tokenizers by an average of 1.70% in sentiment classification and 1.41% in news classification. The largest observed improvement was 2.61% in news classification using Logistic Regression (LR). Based on the results, we suggest a method for choosing tokenizers when analyzing data or developing models for African languages.
Keyword:
Tokenization
Multilingual
Natural language processing
Training
Vectors
Complexity theory
Vocabulary
Accuracy
Support vector machines
Standards
Tokenizer
classification
African language
Swahili
Yoruba
Hausa
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Organoselenium compounds with N,C,N pincer groups. Synthesis, structure and reactivityN,C,N钳形基团的有机硒化合物。合成、结构与反应活性.png)
Polyhedron
PolyhedronIF0
PREAI
分享收藏Polarons inπ-conjugated semiconductors: absorption spectroscopy and spin-dependent recombination
PREAI
分享收藏
A dynamic programming approach to missing data estimation using neural networks使用神经网络进行缺失数据估计的动态规划方法
PREAI
分享收藏分享收藏
查看更多内容