Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models

Lynnette Hui Xian Ng

doi:10.1145/3757484

Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models

2025-11-01

0

PRE

AI

DOI：10.1145/3757484

Original request for help

Save

Abstract

En 中文

Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction.

Keywords:

negotiation

Diplomacy

machine learning

natural language processing

discourse

crowdsourcing

Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models

Abstract

Journal

Organization