不依赖训练数据，用模型自身梯度信号在部署时消除文本分类的捷径偏差。
Models Know Their Shortcuts: Deployment-Time Shortcut Mitigation
- 通过无监督梯度归因提取模型内部的捷径信号，实现部署时修正。
- 在分布偏移下性能损失减少50%以上，部分任务超越训练时方法。
- 适合无法获取训练数据的工业场景，尤其适用于已部署的预训练模型。
预训练文本编码器容易产生捷径学习，依赖词元-标签相关性，在部署时分布变化后失效。现有缓解方法多依赖训练数据、训练动态或捷径标注，这些在部署阶段通常不可得，仅剩收敛后的模型。我们证明：仅凭模型本身即可在部署时缓解捷径问题——有偏模型会内化其学习捷径的信号，可通过无监督梯度归因捕捉。我们进一步证明，部署时缓解的性能上限低于训练时缓解。尽管如此，基于该梯度信号提出的无监督部署时缓解框架「Shortcut Guardrail」，在情感分类、毒性检测和自然语言推理任务中，于捷径分布偏移下恢复了大量性能，表现与甚至超越训练时基线。
原文摘要 · Abstract (English)
Pretrained text encoders are prone to shortcut learning, relying on token-label correlations that fail once the distribution shifts in deployment. Existing shortcut mitigation methods mainly operate at training time and assume access to training data, training dynamics, or shortcut annotations, which are hardly available during deployment, where only the converged model remains. We show that this model alone suffices to mitigate shortcuts during deployment: a biased model internalizes a signal of its learned shortcuts that can be captured via unsupervised gradient-based attribution. We further prove that deployment-time mitigation is information-theoretically upper-bounded by training-time mitigation. Nevertheless, exploiting this gradient signal, our proposed unsupervised deployment-time shortcut mitigation framework for pretrained text encoders, Shortcut Guardrail, recovers substantial performance under shortcut distribution shift, matching or outperforming training-time baselines across sentiment classification, toxicity detection, and natural language inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。