构建中文电话助理双目标协调评测基准，提升对话系统应对复杂交互能力。
CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants
- 设计多轮电话对话数据集，涵盖六类场景与多种目标关系。
- 包含5万条对话，覆盖预设、突发预设及无预设三种情况。
- 提出细粒度评估协议，关注语义理解与目标平衡决策能力。
以目标为导向的对话系统在完成用户明确目标方面表现优异，但现有研究主要聚焦于单一显式目标，而电话助理面临代理场景，需协调设备所有者的预设目标与来电者隐含且动态的目标。我们引入 extsc{CallBench}，一个用于评估电话助理双目标协调能力的中文基准。该基准包含6个场景（外卖、配送、打车、工作、生活、骚扰）下共5万条完整多轮电话对话，涵盖常规预设、突发预设和无预设情形，并包含所有者目标与来电者目标间的多种关系：一致、互补、无关和冲突。我们进一步设计了基于预设感知的逐轮评估协议，涵盖语义理解、上下文使用、主动引导、响应质量、预设合规性、对话节奏和安全等维度。在代表性对话方法上的实验表明，现有方法在此任务上仍表现不佳，凸显了在代理约束下实现两个独立目标间可靠逐轮决策的必要性。
原文摘要 · Abstract (English)
Target-oriented dialogue systems have demonstrated strong capabilities in completing user goals through interactive conversations. However, existing studies are primarily designed for single, explicit goal completion, while phone call assistants face a proxy setting that requires coordinating the device owner's explicit preset goal with the caller's implicit and dynamic goal. We introduce \textsc{CallBench}, a Chinese benchmark for evaluating dual-goal coordination in phone call assistants. \textsc{CallBench} contains 50,000 complete multi-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment. It covers regular presets, emergent presets, and no-preset cases, and includes diverse relations between owner-side and caller-side goals, such as alignment, complementarity, irrelevance, and conflict. We further design a preset-aware turn-level evaluation protocol covering semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety. Experiments on representative dialogue methods show that existing approaches still struggle with this task, highlighting the need for phone call assistants that can make reliable turn-level decisions between two independent goals under proxy constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。