.png){kind=icon}
返回
Discover, Explain, Improve: An Automatic Slice Detection Benchmark for Natural Language Processing
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00617.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Pretrained natural language processing (NLP) models have achieved high overall performance, but they still make systematic errors. Instead of manual error analysis, research on slice detection models (SDMs), which automatically identify underperforming groups of datapoints, has caught escalated attention in Computer Vision for both understanding model behaviors and providing insights for future model training and designing. However, little research on SDMs and quantitative evaluation of their effectiveness have been conducted on NLP tasks. Our paper fills the gap by proposing a benchmark named Discover, Explain, Improve (DEIm) for classification NLP tasks along with a new SDM Edisa. Edisa discovers coherent and underperforming groups of datapoints; DEIm then unites them under human-understandable concepts and provides comprehensive evaluation tasks and corresponding quantitative metrics. The evaluation in DEIm shows that Edisa can accurately select error-prone datapoints with informative semantic features that summarize error patterns. Detecting difficult datapoints directly boosts model performance without tuning any original model parameters, showing that discovered slices are actionable for users.1
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
引用论文
An Importance-Performance Analysis of Korean Middle School Students’ Health Management Awareness during the Post-COVID-19 Era Depending on Sex.png)
HealthcareOAAI
HealthcareIF0
OAAI分享收藏
查看更多内容