让模型只在合理区域做决策,避免乱猜。
Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints
- 用子集归因信号引导模型关注人类先验区域。
- 在图像分类和点击决策任务中准确率提升1.8%-3.2%。
- 适合需要可解释决策的GUI智能体开发。
可靠模型不仅应正确预测,还应基于可接受的证据做出决策。但传统监督学习仅提供类别标签,使模型可通过捷径相关性获得高准确率,而非依赖预期证据。人类先验(如边界框或目标界面元素)可约束此类行为,但学习到的决策证据常与人类感知不一致。本文提出基于子集选择的归因方法,利用其识别紧凑的决策支持区域的特点,在训练时作为信号暴露模型的归因证据。当主要归因区域明显偏离先验区域时,施加惩罚并引导模型将归因向预期区域转移。该方法构建了选择性先验约束目标,避免对所有非先验区域进行统一抑制。在基于多模态大模型的GUI代理的图像分类与点击决策任务中验证有效。在判别式分类与自回归决策设置下,既提升了任务准确率,又增强了归因与先验的一致性。
原文摘要 · Abstract (English)
Reliable models should not only predict correctly, but also base their decisions on acceptable evidence. However, conventional supervised learning typically provides only class-level labels, allowing models to achieve high accuracy by exploiting shortcut correlations rather than intended decision evidence. Human priors, such as bounding boxes or target interface elements, can help constrain such behavior, but aligning model evidence with these priors remains challenging because learned decision evidence often diverges from human perception. In this work, we study attribution-prior alignment with subset-selection-based attribution. Motivated by prior deletion and insertion evaluations showing that subset-selection attribution can identify compact decision-supporting regions, we use it as a training-time signal to expose the model's attributed evidence. When the top-attributed evidence deviates substantially from the prior region, we penalize off-prior attribution and encourage the model to shift its attributed evidence toward the intended regions. This yields a selective prior-constrained objective that avoids uniformly suppressing all non-prior regions. We validate our method on both image classification and click decision tasks in MLLM-based GUI agents. Across discriminative classification and autoregressive decision-making settings, our method improves task accuracy while enhancing attribution-prior alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。