任务感知剪枝能提升模型在分布外数据上的表现，原理是修复因输入偏差导致的表征畸变。
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
- 基于任务特征选择性剪枝，只移除影响分布外性能的层
- 在分布外数据上准确率显著提升，分布内性能基本不变
- 适合需要增强鲁棒性的大模型应用，如安全敏感场景
近期研究将任务感知层剪枝用于提升特定任务表现（如TALE），本文探究其何时生效及原因。实验表明，在控制的多项式回归任务与大语言模型上，该剪枝对分布内（ID）数据无增益，但一致提升分布外（OOD）准确率。实证发现，OOD输入引发层间范数与成对距离分布偏离对应ID分布。这揭示了几何解释：每个任务形成特定表征几何，由ID输入观测得到；而OOD输入可能引入该几何的畸变版本。任务感知剪枝识别出造成或放大畸变的层，移除后使OOD表征范数与成对距离向适应分布靠拢，从而重新对齐模型的内在几何，改善性能。通过受控分布偏移与残差缩放干预提供因果证据，并在不同模型规模下保持一致行为。
原文摘要 · Abstract (English)
Recent work has promoted task-aware layer pruning as a way to improve model performance on particular tasks, as shown by TALE. In this paper, we investigate when such improvements occur and why. We show first that, across controlled polynomial regression tasks and large language models, such pruning yields no benefit on in-distribution (ID) data but consistently improves out-of-distribution (OOD) accuracy. We further show empirically that OOD inputs induce layerwise norm and pairwise-distance profiles that deviate from the corresponding ID profiles. This leads to a geometric explanation of task-aware pruning: each task induces a task-adapted geometry, characterized empirically by the representation profiles observed on ID inputs. OOD inputs can introduce a distorted version of the task-adapted geometry. Task-aware pruning identifies layers that create or amplify this distortion; by removing them, it shifts OOD representational norms and pairwise distances toward those observed on the adapted distribution. This realigns OOD inputs with the model's task-adapted geometry and improves performance. We provide causal evidence through controlled distribution shifts and residual-scaling interventions, and demonstrate consistent behavior across model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。