用清单式评估让角色扮演测试结果可追溯、可复现。
TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation
- 将角色要求拆解为固定清单，对话中逐项追踪完成情况。
- 实测覆盖率达99.91%，远超原数据集的73.74%。
- 适合需要精准定位模型缺陷的研究者和评测团队。
角色扮演评估不应仅给单一分数，而应揭示哪些角色要求被检验、哪些未达标，以及判断依据的对话证据。我们提出TRACE Bench，一种任务驱动的代理式清单评估框架。该框架将每个角色画像离线分解为固定清单，再通过用户代理与目标角色模型自然对话，根据模型回复私密更新清单状态。评分因此可追溯至具体清单项及对应对话片段，而非黑箱整体印象。为验证覆盖度，我们用相同角色清单审计了MiniMax角色扮演基准中发布的自由对话转录文本。结果显示，原始自由对话仅覆盖73.74%的关键角色点，而TRACE Bench在更少轮次内达到99.91%覆盖率。鲁棒性实验表明，重复运行与替换用户代理后排名保持稳定。在26个模型上，TRACE Bench报告总体排名及能力分解，并提供清单追踪路径。同时支持闭环基准演化，将已验证有效的验证方法从失败轨迹中提炼，使后续评估能更可靠地激发并分析观测到的失效模式。
原文摘要 · Abstract (English)
Roleplay evaluation should do more than assign a single score: it should reveal which role requirements were tested, which failed, and which dialogue evidence supports the judgment. We propose TRACE Bench, a task-driven agentic checklist evaluation framework. It decomposes each role profile offline into a fixed checklist, then uses a User Agent to converse naturally with the target roleplay model while privately updating checklist states from model responses. Scores therefore trace back to checklist items and supporting dialogue turns rather than a black-box holistic impression. For coverage cross-validation, we audit released M2 free-dialogue transcripts from the MiniMax Role-play Benchmark against the same role-derived checklist. The released free-chat transcripts cover only 73.74% of key role-profile points, whereas TRACE Bench reaches 99.91% coverage in fewer turns. Robustness experiments show stable rankings under repeated runs and User Agent replacement. Across 26 models, TRACE Bench reports overall rankings together with capability breakdowns and checklist traces. It also supports Closed-Loop Benchmark Evolution, distilling verification methods proven effective in failed traces so later evaluations can more reliably elicit and examine observed failure modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。