Human-Algorithm Collaborative Truth Inference in Crowdsourcing

doi:10.1287/ijoc.2023.0440

Human-Algorithm Collaborative Truth Inference in Crowdsourcing

2025-09-01

0

PRE

AI

DOI：10.1287/ijoc.2023.0440

Original

Original request for help

Save

Abstract

En 中文

Crowdsourcing has become a pivotal strategy in gathering large-scale, high-quality labeled data, particularly in data-intensive applications powered by artificial intelligence. To aggregate the noisy crowd efforts, many studies have considered learning a predictive algorithm based on the noisy human annotations and subsequently integrating the learned knowledge back into the data aggregation process. However, it is unclear how to design such hybrid systems that maximize the complementary strengths of humans and algorithms. In response, we analyze the patterns of human and algorithm intelligence and propose that the inductive bias of algorithms can effectively mitigate inconsistencies in human labeling, thus complementing human efforts. Building on this premise, we propose a human-algorithm collaborative framework (HAC) to combine human labels with algorithmic predictions. By proposing a metric called hybrid complementarity score (HCS) to quantify human-algorithm complementarity, our framework can dynamically adjust the weight of each algorithm based on its complementarity, significantly enhancing the overall efficacy of the human-algorithm integration. To validate the effectiveness of our framework, we first instantiate it with several algorithms, including a high-complementarity algorithm building upon the inductive bias of clusteringaware design. We then benchmark our framework against leading baselines across eight realworld tasks. Our results not only demonstrate the superior performance of our proposed framework but also affirm its robustness across different algorithm selections (e.g., types and number of algorithms) and hyperparameter configurations. This research not only delivers a feasible and effective solution for truth inference in crowdsourcing but also contributes to the burgeoning community of human-algorithm collaboration.

Keywords:

learning from crowds

human-algorithm collaboration

complementarity

balancing

Human-Algorithm Collaborative Truth Inference in Crowdsourcing

Abstract

Journal

Organization

Cited Papers