arXiv:2602.01611cs.LG2026-02被引 1

研究发现,强化学习模型学的是界面操作模式而非真实语义。

What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

  • 通过微调界面布局测试模型是否依赖特定操作模式
  • 经轨迹微调的模型在界面改动后性能暴跌,非微调模型保持稳定
  • 提出新指标量化模型对训练界面的依赖程度,适合评估代理鲁棒性

大型语言模型作为交互代理的评估正日益普及,但现有基准测试混淆了两种本质不同的成功来源:语义工具使用与界面特有交互模式的记忆。这两种机制均可在原界面下实现相同任务成功,因此仅靠基准分数无法区分环境不变能力。本文提出PIPE协议级评估增强方法,通过最小化重写环境界面以诊断对界面的依赖,同时保留任务语义和执行行为。在AgentBench和AgentGym的16个环境中,对多种开源与API代理进行测试发现,轨迹-监督微调(trajectory-SFT)显著加剧了界面捷径依赖:训练后的模型在界面轻微修改后性能急剧下降,而未经过轨迹微调的模型则保持稳定。我们进一步引入界面依赖度(IR)这一对抗平衡的别名指标,量化模型对训练界面的偏好,并揭示界面捷径行为具有环境依赖性、非单调的训练动态,这些在标准评估中不可见。代码已公开于https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/。

原文摘要 · Abstract (English)

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/.

智能体评估界面依赖轨迹微调鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。