让角色扮演模型摆脱熟悉角色依赖，用匿名评测更真实评估其能力。
Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects
- 采用匿名角色评测，消除模型对知名角色的记忆干扰。
- 匿名后性能下降，证明名字是隐性线索，影响评测公平性。
- 加入个性描述可显著提升角色一致性，适合构建稳健角色代理。
大型语言模型在构建角色扮演代理（RPAs）方面展现出巨大潜力。然而，当前评估框架过度依赖知名虚构角色，导致模型可能利用其内部训练记忆而非真正角色扮演能力。这种依赖在面对未见过或分布外人格时常引发性能大幅下降。为此，我们提出更严格的评估协议，旨在将角色扮演能力与角色识别相分离。跨多个基准的实验表明，角色匿名化会降低性能，证实名称暴露提供了隐性线索，掩盖了模型的真实表现。为缓解此问题，我们研究了多种个性增强方法以提升匿名环境下的角色契合度。系统分析不同个性描述方式对代理行为和一致性的影响，结果表明融入个性信息能持续提升RPA表现。本工作确立了更公平的评估标准，并验证了一种可扩展的、基于个性增强的框架，用于构建鲁棒的角色扮演代理。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs). However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern: models may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model's true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personality-description methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes a more equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。