.png){kind=icon}
返回
Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems
.png){kind=icon}
OA
AI
DOI：10.1145/3596510.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Task-oriented dialogue systems (TDSs) are assessed mainly in an offline setting or through human evaluation. The evaluation is often limited to single-turn or is very time-intensive. As an alternative, user simulators that mimic user behavior allow us to consider a broad set of user goals to generate human-like conversations for simulated evaluation. Employing existing user simulators to evaluate TDSs is challenging as user simulators are primarily designed to optimize dialogue policies for TDSs and have limited evaluation capabilities. Moreover, the evaluation of user simulators is an open challenge. In this work, we propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates a user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities. Our user simulator constructs a metaphorical user model that assists the simulator in reasoning by referring to prior knowledge when encountering new items. We estimate the quality of simulators by checking the simulated interactions between simulators and variants. Our experiments are conducted using three TDS datasets. The proposed user simulator demonstrates better consistency with manual evaluation than an agenda-based simulator and a seq2seq model on three datasets; our tester framework demonstrates efficiency and has been tested on multiple tasks, such as conversational recommendation and e-commerce dialogues.
Keyword:
Task-oriented dialogue
evaluation
user simulation
期刊
IF:
9.1
论文数:
1.2K
被引数:
4.7K
机构
引用论文
分享收藏分享收藏PERSONALIZED EARLY MEDICATION-SPECIFIC RECOMMENDATIONS FOR PATIENTS WITH ACUTE KIDNEY INJURY: FROM THE KAT-AKI TRIAL针对急性肾损伤患者的个性化早期特定药物推荐：来自KAT-AKI试验
PREAI
分享收藏Importance of lipophilicity for potent anti-herpes simplex virus-1 activity of α-hydroxytropolones.png)
MedChemComm
OAAI
MedChemCommIF0
OAAI分享收藏
查看更多内容