让大模型更抗语法小扰动，且不破坏原有能力。
Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
- 用门控树交叉注意力读取预计算句法块，不改原模型结构。
- 在多个基准上提升语法鲁棒性，且不影响问答与常识推理性能。
- 适合需要保持原有模型不变却想增强语法理解的场景。
仅解码器架构的大语言模型虽具强大泛化能力，但对微小语法扰动敏感，影响下游推理可靠性。直接向已有检查点注入显式句法结构会干扰其预训练能力。本文提出一种检查点兼容的门控树交叉注意力（GTCA）模块，通过读取预计算的成分句法块记忆，保持主干结构不变。设计采用标记更新掩码与分阶段训练，控制句法更新的范围与时机。在多个基准与Transformer主干上，GTCA在不损害多选题问答与常识推理性能的前提下，显著提升语法鲁棒性，优于持续训练基线，为实现更抗语法扰动的解码器模型提供了一条实用、可兼容的路径。
原文摘要 · Abstract (English)
Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。