.png){kind=icon}
.png){kind=icon}
Return
Human-Algorithm Collaborative Truth Inference in Crowdsourcing
.png){kind=icon}
PRE
AI
DOI：10.1287/ijoc.2023.0440.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Crowdsourcing has become a pivotal strategy in gathering large-scale, high-quality labeled data, particularly in data-intensive applications powered by artificial intelligence. To aggregate the noisy crowd efforts, many studies have considered learning a predictive algorithm based on the noisy human annotations and subsequently integrating the learned knowledge back into the data aggregation process. However, it is unclear how to design such hybrid systems that maximize the complementary strengths of humans and algorithms. In response, we analyze the patterns of human and algorithm intelligence and propose that the inductive bias of algorithms can effectively mitigate inconsistencies in human labeling, thus complementing human efforts. Building on this premise, we propose a human-algorithm collaborative framework (HAC) to combine human labels with algorithmic predictions. By proposing a metric called hybrid complementarity score (HCS) to quantify human-algorithm complementarity, our framework can dynamically adjust the weight of each algorithm based on its complementarity, significantly enhancing the overall efficacy of the human-algorithm integration. To validate the effectiveness of our framework, we first instantiate it with several algorithms, including a high-complementarity algorithm building upon the inductive bias of clusteringaware design. We then benchmark our framework against leading baselines across eight realworld tasks. Our results not only demonstrate the superior performance of our proposed framework but also affirm its robustness across different algorithm selections (e.g., types and number of algorithms) and hyperparameter configurations. This research not only delivers a feasible and effective solution for truth inference in crowdsourcing but also contributes to the burgeoning community of human-algorithm collaboration.
Keywords:
learning from crowds
human-algorithm collaboration
complementarity
balancing
Journal
I
IF:
2.1
Papers:
86
Citations:
3.2K
Organization
No organization information available