.png){kind=icon}
返回
Assessing Top-k Preferences
.png){kind=icon}
PRE
AI
DOI：10.1145/3451161.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Assessors make preference judgments faster and more consistently than graded judgments. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments. Unfortunately, preference judgments can require more than linear effort to fully order a pool of items, and evaluation measures for preference judgments are not as well established as those for graded judgments, such as NDCG. In this article, we explore the assessment process for partial preference judgments, with the aim of identifying and ordering the top items in the pool, rather than fully ordering the entire pool. To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure. We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named compatibility. This new measure has its most striking impact when comparing modern neural rankers, where it is able to recognize significant improvements in quality that would otherwise be missed by NDCG.
Keyword:
Preference judgments
question answering
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
9.1
论文数:
1.2K
被引数:
4.7K
机构
引用论文
分享收藏Improved Methods for the Fast Fourier Transform (FFT) Calculation of the Frequency Response Function
PREAI
分享收藏
分享收藏Experimental Background to the Study of the Distribution and Abundance of Insects: II. The Relation Between Innate Capacity for Increase in Numbers and the Abundance of Three Grain Beetles in Experimental Populations.png)
Ecology
EcologyIF0
PREAI
分享收藏
查看更多内容