Evaluating Synthetic Data Generation from User Generated Text

doi:10.1162/coli_a_00540

Evaluating Synthetic Data Generation from User Generated Text

2024-11-25

0

OA

AI

DOI：10.1162/coli_a_00540

原文链接

原文求助

摘要

En 中文

User-generated content provides a rich resource to study social and behavioral phenomena.Although its application potential is currently limited by the paucity of expert labels and theprivacy risks inherent in personal data, synthetic data can help mitigate this bottleneck. Inthis work, we introduce an evaluation framework to facilitate research on synthetic languagedata generation for user-generated text. We define a set of aspects for assessing data quality,namely, style preservation, meaning preservation, and divergence, as a proxy for privacy. Weintroduce metrics corresponding to each aspect. Moreover, through a set of generation strategiesand representative tasks and baselines across domains, we demonstrate the relation betweenthe quality aspects of synthetic user generated content, generation strategies, metrics, anddownstream performance. To our knowledge, our work is the first unified evaluation frameworkfor user-generated text in relation to the specified aspects, offering both intrinsic and extrinsicevaluation. We envisage it will facilitate developments towards shareable, high-quality syntheticlanguage data

Keyword:

SYNCHRONY

Evaluating Synthetic Data Generation from User Generated Text

摘要

期刊

机构

引用论文