.png){kind=icon}
Return
Multistep planning for crowdsourcing complex consensus tasks
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2021.107447.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Crowdsourcing receives massive vote information from non-expert workers, for finishing tasks that can hardly be handled by current technology of machine intelligence. Massive vote information and non-expert workers bring serious issues of labor costs and the efficiency of crowdsourcing. This paper focuses on the tasks, classifying objects in images or videos into a set of given candidates by letting workers vote on a set of options that characterize these candidates. Designing a good asking strategy, i.e., setting up the order of presenting the options to a worker and asking the worker whether an option is true or false, is one starting point to save labor costs and enhance efficiency of deciding the correct answer from the candidates. We propose the problem of determining the time steps of vote collection before stopping to set up the asking strategy. In terms of this problem, we establish a single-step collection based partially observable Markov decision process (POMDP) to analyze how a vote influences the whole system, for instance, influences the belief over each option. Formally define the multistep collection problem as the timed decision (TD) problem. We propose the MC-EVA algorithm based on Monte Carlo sampling to solve the TD problem. Evaluate the MC-EVA algorithm over three simple but typical cases and a real-world Galaxy Zoo 2 project. Experiments show MC-EVA's great superiority in runtime over the state-of-the-art single-step collection algorithm, and its superiority in effectiveness than other multistep collection algorithms; show its labor cost saving and enhanced efficiency with the use of calculated asking strategies. (C) 2021 Elsevier B.V. All rights reserved.
Keywords:
Crowdsourcing
POMDPs
Consensus tasks
Asking strategies
Timed decision (TD) problem
Monte Carlo sampling
Journal
K
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
No organization information available
Cited Papers
NIR absorbing DICPO derivatives applied to wide range of pH and detection of glutathione in tumor.png)
Tetrahedron
TetrahedronIF0
PREAI
ShareSave
ShareSaveShareSave
ShareSaveShareSave
View more