统一对话评估工具,让大模型多轮对话能力对比更公平
UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities
- 用统一格式和流程整合多种评测数据与接口
- 实测发现无模型在所有任务中都最优,记忆机制常不如全上下文
- 适合做对话系统评测的科研人员和工程团队使用
在多轮交互场景中评估大语言模型和智能体的能力至关重要。然而,现有评测协议在数据格式、模型接口和评估流程上差异巨大,严重阻碍了系统性比较。本文提出UniDial-EvalKit(UDE),一个统一的交互式AI系统评估工具包。其核心贡献在于全面统一:将异构数据格式标准化为通用模式,通过模块化架构简化复杂评估流程,并在分层评分聚合下对齐指标计算。该工具支持并行生成与评分,实现高效大规模评估,并可通过检查点续跑避免重复计算。基于UDE,我们在多个维度的基准上开展广泛评估。实证分析表明,没有单一系统在所有基准上持续领先;当前记忆型代理往往无法超越全上下文基线。进一步分析指出了未来方向,包括基准去重和更自适应的记忆架构。
原文摘要 · Abstract (English)
Benchmarking large language models (LLMs) and agents in multi-turn interactive scenarios is essential for understanding their practical capabilities. However, existing evaluation protocols are highly heterogeneous, differing significantly in dataset formats, model interfaces, and evaluation pipelines, which severely impedes systematic comparison. In this work, we present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems. The core contribution of UDE lies in its holistic unification: it standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a hierarchical scoring aggregation. It also supports efficient large-scale evaluation through parallel generation and scoring, as well as checkpoint resume to eliminate redundant computation. Leveraging UDE, we conduct an extensive evaluation across diverse multi-dimensional benchmarks. Our empirical analysis shows that no single system consistently outperforms others across all benchmarks, while current memory agents often fail to surpass full-context baselines. Further analyses highlight several future directions, including benchmark deduplication and more adaptive memory architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。