提出行为一致性度量,揭示模型在不同任务间表现是否稳定。
Measuring Cross-Task Behavioral Consistency in Language Model Agents

- 用执行轨迹特征预测任务成功,计算行为贡献向量
- 发现9000条轨迹中,局部一致与全局一致可分离
- 适合关注模型可靠性而非仅成功率的研究者
代理评估几乎完全依赖成功率等结果指标,这些指标只能反映任务是否完成,而无法捕捉行为的一致性。本文提出行为一致性度量(BCM),通过训练模型从代理执行轨迹的行为特征中预测任务成功,生成每条轨迹的特征归因向量,并计算同一系统内各向量间的平均成对相似性。在六种语言模型代理于软件工程任务上的约9000条轨迹中,核心发现是跨任务一致性和任务内一致性为两个独立维度,可能分离:某些系统在单个任务上重复表现一致,但在不同任务间无稳定策略;另一些则在两个尺度上均保持一致。以往研究仅衡量同任务可复现性,无法观测此分离现象。进一步发现,一致性不可简化为成功率——相同成功率的系统可表现出显著差异的一致性;且在控制任务难度后,前沿模型与开源模型间的一致性差距依然存在。本文将BCM定位为过程级可靠性信号,补足结果指标,并明确其适用条件。
原文摘要 · Abstract (English)
Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves. We argue that behavioral consistency across tasks is a distinct and measurable property, and we introduce the Behavioral Consistency Metric (BCM) to quantify it. BCM trains a model to predict task success from behavioral features of agent execution traces, derives a per-trajectory feature-attribution vector, and measures the mean pairwise similarity of these vectors within an agent system. Across roughly 9,000 trajectories from six language model agents on software engineering tasks, our central finding is that cross-task and within-task consistency are distinct axes that can diverge: some systems are locally reproducible, behaving similarly on repeated attempts at one task, yet globally fragmented, with no stable strategy across different tasks, while others are consistent at both scales. Prior work measures only same-task reproducibility and so cannot observe this separation. We further find that consistency is not reducible to success rate, since systems with comparable success can differ sharply in consistency, and that the frontier-versus-open-source consistency gap persists under a within-task control that holds task difficulty constant. We position BCM as a process-level reliability signal that complements outcome metrics, and we are explicit about the conditions under which it is meaningful.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。