.png){kind=icon}
.png){kind=icon}
Return
LiteLLM: a lightweight transformer architecture for efficient short-text classification
.png){kind=icon}
PRE
AI
H
F
A
DOI：10.1007/s00521-026-12379-3.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Short-text classification plays a central role in applications such as spam filtering, sentiment analysis, and real-time content moderation. Yet, most existing transformer-based models are over-parameterized for the inherently brief and noisy nature of microtexts. This paper introduces LiteLLM, a lightweight transformer architecture explicitly optimized for short-text scenarios. LiteLLM integrates a compact SentencePiece tokenizer, a 256-dimensional reduced embedding space, and a shallow four-layer encoder with low-rank feed-forward projections and a limited attention mechanism, designed to match the restricted contextual range of SMS and social media messages. The model is trained end-to-end using AdamW optimization and cross-entropy loss, enabling rapid convergence and efficient learning under constrained computational budgets. Evaluation is conducted on two distinct benchmarks, the structured UCI SMS Spam Collection and the noisy TweetEval Sentiment dataset, to assess both in-domain accuracy and cross-domain robustness. Results demonstrate that LiteLLM achieves accuracy comparable to considerably larger transformer models (98.67% on SMS; 88.5% on TweetEval) while offering substantial reductions in memory footprint and measured inference latency (a 7.5× CPU speedup). With only 7.3 M parameters, LiteLLM delivers competitive performance, fast convergence, and competitive cross-domain performance across heterogeneous short-text settings. These findings highlight the potential of domain-aware, efficiency-driven transformer designs for real-world, on-device short-text classification.
Keywords:
Short-text classification
Lightweight transformers
SentencePiece tokenization
SMS spam detection
Twitter sentiment analysis
Journal
IF:
4.5
Papers:
729
Citations:
3.2W
Organization
Cited Papers