.png){kind=icon}
返回
Fixed-Cost Pooling Strategies
.png){kind=icon}
OA
AI
DOI：10.1109/TKDE.2019.2947049.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The empirical nature of Information Retrieval (IR) mandates strong experimental practices. A keystone of such experimental practices is the Cranfield evaluation paradigm. Within this paradigm, the collection of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise, and numerous judgements are keys to reducing evaluation uncertainty and test collection bias; on the other hand, however, relevance judgements are costly to collect. The selection of which documents to judge for relevance, known as pooling method, has therefore a great impact on IR evaluation. In this paper we focus on the bias introduced by the pooling method, known as pool bias, which affects the reusability of test collections, in particular when building test collections with a limited budget. In this paper we formalize and evaluate a set of 22 pooling strategies based on: traditional strategies, voting systems, retrieval fusion methods, evaluation measures, and multi-armed bandit models. To do this we run a large-scale evaluation by considering a set of 9 standard TREC test collections, in which we show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection. We also identify the least biased pooling strategy in terms of pool bias according to three IR evaluation measures: AP, NDCG, and P@10.
Keyword:
Resource management
Organizations
Q measurement
Uncertainty
Buildings
Standards
Benchmark testing
Pooling method
test collections
pool bias
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
10.4
论文数:
6.8K
被引数:
3.2W
机构
引用论文
分享收藏TheASP3locus inSaccharomyces cerevisiaeoriginated by horizontal gene transfer fromWickerhamomyces
OAAI
OAAI分享收藏Qualitative Benefit and Enhancement in Well Being Towards the Biophilic Design in Office BuildingOAAI
OAAI分享收藏Exploring the Potential Hormonal Effects of Tire Polymers (TPs) on Different Species Based on a Theoretical Computational Approach.png)
PolymersOAAI
PolymersIF0
OAAI分享收藏分享收藏
查看更多内容