跨语言任务下,智能体动作策略保留率仅71%-73%,且普遍依赖英语中转。
Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
- 以动作轨迹为评估核心,消除五类干扰因素后量化跨语言策略一致性。
- 8个模型在41种语言中动作保留率达71%-73%,模型差异仅解释5.7%方差。
- 小模型(<10B参数)策略不稳定,且存在因随机性导致的误判陷阱。
当工具使用型智能体在不同语言下执行相同任务时,其行为步骤是否保持一致?现有跨语言评估通常只比较最终答案,忽略了行动轨迹——这些轨迹决定了成本、延迟、失败模式,是唯一可审计的行为证据。本文在8个模型、6个并行基准、41种语言(238万次回放)上,将动作策略作为测量对象。原始相似度测量受五类干扰影响:短轨迹得分偏高,空轨迹得满分,无关轨迹偶然一致超50%,模型自身可复现性限制差距上限,同一语言重复提问结果不一,无可靠基线。移除所有干扰后,效果反而更大。分歧具有结构性,不随温度变化或贪婪解码而改变。经自复现性归一化后,四个前沿模型在贪婪解码下趋于一致,动作策略保留率71%-73%,模型身份仅解释5.7%方差。低于约100亿参数的模型则失效,小模型间排序实为随机地板效应。智能体普遍将非英语任务路由至英语,该中转行为具有因果重要性,预注册实验验证其不可撤销。最后发现,仅一个轨迹提取正则表达式就导致多语言表现虚假提升:两组示例使某模型测得准确率飙升26倍,但对可读输出准确率影响极小。
原文摘要 · Abstract (English)
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。