.png){kind=icon}
返回
SummEval: Re-evaluating Summarization Evaluation
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00373.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The scarcity of comprehensive up-to-date studies on evaluationmetrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-sourceworkers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
引用论文
分享收藏The role of carbon from recycled sediments in the origin of ultrapotassic igneous rocks in the Central Mediterranean.png)
Lithos
OAAI
LithosIF0
OAAI分享收藏Nonlinear Time Series Analysis of Nodulation Factor Induced Calcium Oscillations: Evidence for Deterministic Chaos?PLoS ONEOAAI
PLoS ONEIF0
OAAI分享收藏
查看更多内容