arXiv:2608.20647cs.CL2026-08

双向语言模型中,同向状态拼接比跨方向配对更有效,且距离越远差距越大。

Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

  • 用前向和后向状态同向拼接,优于跨方向配对
  • 跨方向配对在长距离时性能下降更严重,差距随距离增大
  • 模型内部存在方向性信息但位置不精确,传播范围有限

将双向LSTM的上下文表示拆分为仅前向($F_i$)和仅后向($B_i$)状态,其组合表现优于单一方向或融合的自注意力表示,用于依存关系类型分类。然而,一种自然扩展——将某词的前向状态与候选词的后向状态配对(跨方向配对,$F_i$ vs. $B_j$)却持续表现更差,且性能差距随词距增加而扩大(配对bootstrap显著)。通过冻结主干网络的诊断方法发现:架构上已杜绝方向间信息泄露(单层BiLSTM,代码验证);93%的同向-跨向差距在冻结主干、仅训练新头部时仍存在,排除了训练协同适应为主因;线性回归显示$F_i$与$B_i$存在部分表示冗余($R^2=0.324$,随机对照为0.028),线性探测显示$F_i$部分预判未来词(36.5%高于17.2%基线),但两者均无法完整解释差距。扩展诊断(位置探测、距离衰减探测)表明方向信息真实存在但定位不准,传播范围仅数词即衰减至基线水平,与距离增长现象机制一致。

原文摘要 · Abstract (English)

Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$) and a backward-only $B_i$ (strictly a function of tokens $i..n$) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea -- pairing a token's forward state against a \emph{candidate}'s backward state (``cross-direction'' pairing, $F_i$ vs.\ $B_j$) -- consistently \emph{underperforms} same-direction pairing, and the penalty \emph{grows}, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozen-trunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93\% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between $F_i$ and $B_i$ ($R^2{=}0.324$ vs.\ $0.028$ for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in $F_i$ (36.5\% vs.\ 17.2\% majority baseline) -- real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distance-decay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline -- consistent with, and mechanistically underneath, the distance-growth finding.

依存分析语言模型表示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。