.png){kind=icon}
.png){kind=icon}
Return
An entropy-driven method for llm dataset evaluation and optimization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2025.130293.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
• We propose an entropy-driven method for LLM dataset quality assessment and optimization. • A hierarchical similarity matrix reduces redundancy detection complexity by 96.57 %. • Entropy-driven quality score preserves boundary questions for LLM evaluation. • The method enhances dataset discriminability with 17.5 % score variance increase. • Successfully optimized four public datasets (GSM8K, AGIEval, C-Eval, SafetyQA). • Constructed high-quality datasets used to evaluate over 60 LLMs effectively.
Journal
IF:
7.5
Papers:
2.9W
Citations:
10.2W