.png){kind=icon}
返回
Task design in complex crowdsourcing experiments: Item assignment optimization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.cor.2022.105995.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Crowdsourcing is the practice of outsourcing a task that would otherwise be performed by one or a few experts to a crowd of individuals. It is often used to collect large amounts of manually created labels that form datasets for training and evaluating supervised machine learning models. When designing a (micro-task) crowdsourcing experiment, it is often desirable to combine different types of items (e.g., images of both cats and dogs for an image classification task) together into the same task and to have diverse contributors (e.g., male and female crowd workers) providing labels for the same item so as to mitigating existing bias in the collected labels. Setting up such a crowdsourcing experiment, i.e., constructing a human intelligence task containing the right mix of items and assigning it to the right mix of crowd workers, becomes a complex problem.In this paper we address the problem of optimizing human intelligence task construction by providing its formal definition and by applying a local search method initialized by a greedy construction to solve it. We experimentally show the flexibility of the proposed solution in addressing different type of constraints, both on the item and on the worker side, and demonstrate its scalability to complex crowdsourcing setups. The proposed solution enables the design of tools for crowdsourcing requesters to set up complex task deployments in platforms like Amazon MTurk or Prolific.
Keyword:
Crowdsourcing
Experiment design
Optimization
Item assignment
期刊
C
IF:
4.3
论文数:
6.5K
被引数:
1.8W
机构
引用论文
Solving discrete lot-sizing and scheduling by simulated annealing and mixed integer programming通过模拟退火和混合整数规划求解离散批量和调度
.png){kind=icon}
PREAI
分享收藏
Additive manufacturing of copper/diamond composites for thermal management applications用于热管理应用的铜/金刚石复合材料的增材制造
OAAI
OAAI分享收藏Surgeon attitudes and practice patterns in managing small bowel obstruction: a qualitative analysisOAAI
OAAI分享收藏On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluation众包相关性评估中相关性量表对信息检索评估的影响
PREAI
分享收藏分享收藏
查看更多内容