不同视觉语言动作模型失败模式各异，需匹配监控方式。
How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures
- 用黑盒监控发现三类模型失败有独特信号。
- 方向反向率是通用预测指标，准确率超90%。
- 监控需适配模型架构，单一方案无效。
我们发现，视觉语言动作模型（VLA）在执行层面以根本不同的、可预测的方式失败。在相同的评估协议下（PushT和ALOHA 14-DOF双臂操作任务，共450个实验回合），对VQ-BeT、Diffusion Policy和ACT三种架构进行测试，结果表明：(1) 方向反向率是所有三类架构的通用失败预测指标（AUROC分别为0.93、0.79、0.91；p<0.001）；(2) 加速度变化率仅对离散-令牌类架构具有预测性，且呈现离散到连续的梯度趋势（0.88、0.69、0.41）；(3) 速度违规单独使用无法预测失败（AUROC 0.41–0.69），但却是当前部署代码中最常见的安全机制；(4) 对连续类模型，速度监控几乎无预测能力（ACT为0.52，Diffusion为0.41），证明必须根据架构选择监控策略。这些结果量化了离散与连续架构差异的监控后果：两类模型产生质异的失败特征，需不同监控手段。无单一监控通用有效，必须匹配架构。该发现得益于SafeContract——一个无需训练、黑盒且具备保形校准的行动监控工具包。
原文摘要 · Abstract (English)
We discover that VLA architectures fail in fundamentally different, predictable ways at the motor-command level. Running VQ-BeT, Diffusion Policy, and ACT on identical evaluation protocols (n=450 episodes across PushT and ALOHA 14-DOF bimanual manipulation), we find: (1) direction reversal rate is a universal failure predictor across all three architectures (AUROC=0.93, 0.79, 0.91; p<0.001); (2) jerk monitoring is predictive only for discrete-token architectures, following a discrete-to-continuous gradient (0.88, 0.69, 0.41); (3) velocity violations alone are non-predictive everywhere (AUROC 0.41-0.69), yet velocity checking is the most common safety mechanism in VLA deployment code; and (4) for continuous-family VLAs, velocity monitoring provides effectively zero predictive signal (AUROC=0.52 on ACT, 0.41 on Diffusion), proving that architecture-matched monitor selection is essential. These results quantify a monitoring consequence of the well-known discrete/continuous VLA distinction: the two families produce qualitatively different failure signatures that require different monitors. No single monitor works universally; architecture-matched selection is required. This finding was enabled by SafeContract, a training-free, black-box action monitoring toolkit with conformal calibration. Code: https://github.com/krishnam94/vla-edge
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。