通过智能门控与归一化提升多任务NLP模型的泛化能力
SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

- 引入手术式特征门控,按需激活关键词汇信息
- 在作者识别任务上达0.940宏F1,提升0.130
- 适合需要跨任务统一建模的NLP研究者
微调的编码器在异构NLP任务中面临三大问题:归纳偏置不匹配、类别不平衡导致特征统计畸变、无法利用外部词汇知识调节注意力。我们提出 extbf{ extit{SURGELLM}},一种统一Transformer框架,包含三个轻量级模块:手术式特征门控(基于筛选词汇指标和[CLS]的逐维Sigmoid,特征无意义时退化为恒等映射)、任务感知前缀标记(量化特征值与任务标识前置输入)、实例加权归一化(IWN;消除类别先验对门控统计的影响)。理论证明门控收益与手术式特征对齐相关。在四个任务(SST-2、多跳检索、大模型提示归属、作者识别)共17,830样本、十一模型变体、三随机种子下,IWN版本达到宏F1 0.940,较最强非IWN基线提升0.036,作者识别任务提升0.130。随机词表对照实验(平均F1下降0.028)证实增益来自词汇而非参数。代码、词表及99.5%回收率自动提取方案已公开。
原文摘要 · Abstract (English)
Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge. We introduce \textbf{\surgellm}, a unified transformer framework that addresses each with a dedicated lightweight module: a \emph{surgical feature gate} (learned per-dimension sigmoid over curated lexical indicators and \texttt{[CLS]}; provably degenerates to identity when features are uninformative), \emph{task-conditioned prefix tokens} (quantized feature values and task identity prepended to every input), and \emph{Instance-Weighted Normalization} (IWN; removes class-prior bias from gate statistics). We prove an excess-risk bound linking gate benefit to \emph{surgical feature alignment}. Across four tasks, SST-2, multi-hop retrieval, LLM-prompt attribution, and authorship detection, covering 17,830 examples and eleven model variants over three seeds, the IWN variant achieves macro-F1 \textbf{0.940} ($+0.036$ over the strongest non-IWN baseline; $+0.130$ on authorship detection). A random-vocabulary control ($-0.028$ avg.\ F1) confirms gains are lexical, not parametric. Code, vocabularies, and a $99.5\%$-recovery auto-extraction recipe are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。