.png){kind=icon}
Return
Task design in complex crowdsourcing experiments: Item assignment optimization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.cor.2022.105995.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Crowdsourcing is the practice of outsourcing a task that would otherwise be performed by one or a few experts to a crowd of individuals. It is often used to collect large amounts of manually created labels that form datasets for training and evaluating supervised machine learning models. When designing a (micro-task) crowdsourcing experiment, it is often desirable to combine different types of items (e.g., images of both cats and dogs for an image classification task) together into the same task and to have diverse contributors (e.g., male and female crowd workers) providing labels for the same item so as to mitigating existing bias in the collected labels. Setting up such a crowdsourcing experiment, i.e., constructing a human intelligence task containing the right mix of items and assigning it to the right mix of crowd workers, becomes a complex problem.In this paper we address the problem of optimizing human intelligence task construction by providing its formal definition and by applying a local search method initialized by a greedy construction to solve it. We experimentally show the flexibility of the proposed solution in addressing different type of constraints, both on the item and on the worker side, and demonstrate its scalability to complex crowdsourcing setups. The proposed solution enables the design of tools for crowdsourcing requesters to set up complex task deployments in platforms like Amazon MTurk or Prolific.
Keywords:
Crowdsourcing
Experiment design
Optimization
Item assignment
Journal
C
IF:
4.3
Papers:
6.5K
Citations:
1.8W
Organization
Cited Papers
Solving discrete lot-sizing and scheduling by simulated annealing and mixed integer programming
.png){kind=icon}
PREAI
ShareSave
ShareSaveShareSaveSurgeon attitudes and practice patterns in managing small bowel obstruction: a qualitative analysis
OAAI
OAAIShareSaveShareSaveShareSave
View more