arXiv:2604.14256cs.IRcs.AI2026-04被引 1

用模拟市场环境评估信息系统的竞争表现,更真实反映部署后效果。

Evaluation of Agents under Simulated AI Marketplace Dynamics

论文配图:Evaluation of Agents under Simulated AI Marketplace Dynamics
图 1 · 摘自论文原文
  • 构建仿真市场环境,让系统在动态竞争中交互
  • 引入留存率和市场份额等长期指标,超越单一准确率
  • 适合关注实际部署效果的研究者和产品团队

现代信息获取系统由检索系统与大语言模型等多种组件构成,日益依赖市场机制来协调模型、工具与数据的访问,导致系统间竞争成为部署中的常态。然而当前评估仍基于静态基准,仅关注准确率,且假设系统独立运行,无法反映用户切换、路由决策和运营约束带来的竞争影响。这种脱节使难以预测系统上线后的实际表现,也掩盖了早期采用优势与市场主导等效应。本文提出「市场评估」(Marketplace Evaluation)框架,通过模拟重复交互与不断演化的用户及代理偏好,实现纵向评估,并引入留存率、市场占有率等市场层面指标,补充甚至超越传统准确率评价。我们正式化该框架,并基于商业与经济理论,提出围绕市场模拟、度量、优化与采纳的科研议程,适用于如TREC等评估活动。

原文摘要 · Abstract (English)

Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems inherent to deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict post-deployment success and obscures competitive effects such as early-adoption advantages and market dominance. We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By simulating repeated interactions and evolving user and agent preferences, the framework enables longitudinal evaluation and marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.

系统评估市场竞争仿真评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。