logo
CanyamAI summaries for academic research
language
MobileMobile
  • Phone FeatureWeChat Mini Program
EnglishEnglish
noticeNotice
BlogBlog
delete
arrow
Return

Multistep planning for crowdsourcing complex consensus tasks

delete2021-11-01
delete0
PRE
AI
Z
Zixuan Deng *
Y
Yanping Xiang
DOI：10.1016/j.knosys.2021.107447delete
deleteOriginal
deleteOriginal request for help
deleteShare
deleteSave
Abstract

Abstract

En 中文
Crowdsourcing receives massive vote information from non-expert workers, for finishing tasks that can hardly be handled by current technology of machine intelligence. Massive vote information and non-expert workers bring serious issues of labor costs and the efficiency of crowdsourcing. This paper focuses on the tasks, classifying objects in images or videos into a set of given candidates by letting workers vote on a set of options that characterize these candidates. Designing a good asking strategy, i.e., setting up the order of presenting the options to a worker and asking the worker whether an option is true or false, is one starting point to save labor costs and enhance efficiency of deciding the correct answer from the candidates. We propose the problem of determining the time steps of vote collection before stopping to set up the asking strategy. In terms of this problem, we establish a single-step collection based partially observable Markov decision process (POMDP) to analyze how a vote influences the whole system, for instance, influences the belief over each option. Formally define the multistep collection problem as the timed decision (TD) problem. We propose the MC-EVA algorithm based on Monte Carlo sampling to solve the TD problem. Evaluate the MC-EVA algorithm over three simple but typical cases and a real-world Galaxy Zoo 2 project. Experiments show MC-EVA's great superiority in runtime over the state-of-the-art single-step collection algorithm, and its superiority in effectiveness than other multistep collection algorithms; show its labor cost saving and enhanced efficiency with the use of calculated asking strategies. (C) 2021 Elsevier B.V. All rights reserved.
Keywords:
Crowdsourcing
POMDPs
Consensus tasks
Asking strategies
Timed decision (TD) problem
Monte Carlo sampling

Journal

K
Knowledge-Based Systems
IF:
7.6
Papers:
1.2W
Citations:
4.5W

Organization

No organization information available