用失败追踪取代分数评估，发现多语言系统部署中的隐性偏差。
Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents
- 以失败为单位，构建可追踪、可修复的运行时评估流程。
- 81个三语问题中14组出现跨语言得分偏移，最大偏移达9分。
- 适合关注系统部署稳定性与多语言公平性的工程师和研究者。
本文提出PSA-Eval，一种面向已部署三语公共空间智能体的故障中心型运行时评估框架。当评估对象从静态输入输出映射转向运行系统时，分析单元应由分数转为故障。PSA-Eval将传统流程扩展为：问题 → 批量 → 运行 → 评分 → 故障案例 → 修复 → 回归批量，使故障可追溯、可审查、可修复、可回归测试。该框架采用三语等价输入作为受控探针，观测群体级跨语言策略漂移。我们在一家国际金融机构大厅部署的真实三语数字前台系统上开展试点研究，使用简化单基础模型设定（MA = MB），因此观察到的漂移不应归因于A/B基础模型差异。研究包含81个样本，分为27组三语等价问题。尽管系统平均得分为23.15/24，但14组出现非零跨语言得分漂移，其中5组漂移至少3分，最大漂移达9分。结果初步表明，故障中心型运行时评估能揭示聚合评分掩盖的结构性部署信号。
原文摘要 · Abstract (English)
This paper presents PSA-Eval, a failure-centered runtime evaluation framework for deployed trilingual public-space agents. The central claim is that, when the evaluation object shifts from a static input-output mapping to a runtime system, the basic unit of analysis should shift from score to failure. PSA-Eval extends the conventional chain Question -> Answer -> Score -> End into Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch, making failures traceable, reviewable, repairable, and regression-testable. The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift. We conduct a pilot study on a real trilingual digital front-desk system deployed in the lobby of an international financial institution. The pilot uses a simplified single-foundation-model setting (MA = MB), so the observed drift should not be interpreted as an A/B foundation-model difference. The study contains 81 samples organized into 27 trilingual equivalent question groups. Although the system achieves an average score of 23.15/24, 14 groups show non-zero cross-language score drift, 5 groups show drift of at least 3 points, and the maximum drift reaches 9 points. These results provide initial evidence that failure-centered runtime evaluation can expose structured deployment signals hidden by aggregate scoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。