.png){kind=icon}
返回
Modeling annotator behaviors for crowd labeling
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2014.10.082.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Machine learning applications can benefit greatly from vast amounts of data, provided that reliable labels are available. Mobilizing crowds to annotate the unlabeled data is a common solution. Although the labels provided by the crowd are subjective and noisy, the wisdom of crowds can be captured by a variety of techniques. Finding the mean or finding the median of a samples annotations are widely used approaches for finding the consensus label of that sample. Improving consensus extraction from noisy labels is a very popular topic, the main focus being binary label data. In this paper, we focus on crowd consensus estimation of continuous labels, which is also adaptable to ordinal or binary labels. Our approach is designed to work on situations where there is no gold standard; it is only dependent on the annotations and not on the feature vectors of the instances, and does not require a training phase. For achieving a better consensus, we investigate different annotator behaviors and incorporate them into four novel Bayesian models. Moreover, we introduce a new metric to examine annotator quality, which can be used for finding good annotators to enhance consensus quality and reduce crowd labeling costs. The results show that the proposed models outperform the commonly used methods. With the use of our annotator scoring mechanism, we are able to sustain consensus quality with much fewer annotations. (C) 2015 Elsevier B.V. All rights reserved.
Keyword:
Label noise
Crowdsourcing
Crowd labeling
Annotator behavior
Annotator quality
Consensus
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
分享收藏Baseline estrogen levels in postmenopausal women participating in the MAP.3 breast cancer chemoprevention trial.png)
Menopause
OAAI
MenopauseIF0
OAAI分享收藏Unliganded Progesterone Receptor Governs Estrogen Receptor Gene Expression by Regulating DNA Methylation in Breast Cancer CellsCancers
OAAI
CancersIF0
OAAI分享收藏分享收藏
查看更多内容