arXiv:2410.06327cs.HCcs.CV2024-10被引 7

建立动态评估榜单,推动对话式动作生成研究标准化

Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis

  • 提出可定期更新的活体排行榜,结合大规模用户研究评估动作生成效果
  • 榜单每年多次更新,支持任意论文投稿平台提交系统,打破发表壁垒
  • 鼓励社区共建数据、任务与工具,推动研究方向与实际需求对齐

当前语音驱动手势生成的评估缺乏统一标准,侧重于易测量指标而非真正重要的方面,导致无法判断当前技术水平或比较不同方法的适用性。本文分析现有评估问题,提出改进方案:建立一个持续更新的活体排行榜,用于基准测试对话式动作生成进展。与以往挑战赛不同,该榜单将每年多次通过大规模用户研究评估新系统,并允许作者自由提交至任何发表平台。通过不断演进评估数据与任务,榜单能持续推动研究向社区公认的核心目标迈进。我们积极呼吁社区参与整个评估流程——从数据与任务设计,到工具开发,再到系统评测。该提案不仅降低研究者评估难度,更让集体贡献塑造手势生成研究的未来方向。

原文摘要 · Abstract (English)

Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research.

动作生成评估基准活体榜单对话合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。