.png){kind=icon}
返回
Learning from crowds with sparse and imbalanced annotations
.png){kind=icon}
OA
AI
DOI：10.1007/s10994-022-06185-w.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Traditional supervised learning requires ground truth labels for training, whose collection however is difficult in many cases. Recently, crowdsourcing has established itself as an efficient labeling solution by resorting to non-expert crowds. To reduce the labeling error effects, one common practice is to distribute each instance to multiple workers, whereas each worker only annotates a subset of data, resulting in the sparse annotation phenomenon. In this paper, we show that when meeting with class-imbalance, i.e., even when the groundtruth labels are slightly imbalanced, the sparse annotations are prone to be skewly distributed and would bias the learning algorithm severely. To combat this issue, we propose one Distribution Aware Self-training based Crowdsourcing learning (DASC) approach, which supplements the sparse annotations by adding confident pseudoannotations and at the same time re-balancing the annotation distribution. Specifically, we propose one distribution aware confidence measure to select the most confident pseudoannotations, with minority/majority classes selected more/less frequently. As a universal framework, DASC is applicable to various crowdsourcing methods for consistent performance gains. We conduct extensive experiments over real-world crowdsourcing benchmarks, from slight to heavy imbalance ratio, with various annotation sparsity levels, and show that DASC substantially improves previous crowdsourcing models by 2%-20% absolute test accuracy, and yields much more balanced annotations.
Keyword:
Crowdsourcing
Sparse annotations
Class-imbalance
Self-training
期刊
IF:
2.9
论文数:
2.7K
被引数:
3.4W
机构
暂无机构信息
引用论文
A systematic study of the class imbalance problem in convolutional neural networks卷积神经网络中类不平衡问题的系统研究.png)
NEURAL NETWORKS
OAAI
NEURAL NETWORKSIF6.3
OAAI分享收藏分享收藏分享收藏
查看更多内容