arXiv:2501.18205cs.CL2025-01

通过显式编码句法依赖关系,提升大模型生成文本的连贯性。

Contextually Structured Token Dependency Encoding for Large Language Models

  • 在嵌入初始化阶段引入结构化依赖关系,增强词元间语义关联
  • 在多个语言基准上降低困惑度,尤其在长句中提升层次一致性
  • 无需外部标注或额外训练目标,适合追求生成质量的研究者

大规模神经架构中的词元表示常依赖上下文优化的嵌入,但传统方法很少显式编码词元间的结构化关系。自注意力机制虽能捕捉动态上下文依赖,但其对学习权重的依赖限制了长程层次结构的保留。本文提出一种依赖感知的词元编码机制,在词元表示中嵌入关系约束,而非仅依赖注意力动态推导。该机制通过依赖加权注意力计算,确保句法与语义依赖在多层处理中得以保持。实验表明,在多种语言基准上困惑度下降,显示自回归文本生成的上下文连贯性和预测一致性提升。计算效率评估显示内存消耗和训练时间略有增加,主要源于编码模块的额外矩阵运算,但在常规Transformer架构中仍具可扩展性。结构化编码提升了词汇多样性与依赖保留能力,强化了语言连贯性,且无需外部句法标注或辅助训练目标。统计对比显示,在长序列中依赖对齐性能显著提升,传统自注意力模型在此类序列中层次一致性易退化。句长分布分析也显示突兀短语转换减少,支持显式依赖编码有助于更结构化的短语生成。

原文摘要 · Abstract (English)

Token representation strategies within large-scale neural architectures often rely on contextually refined embeddings, yet conventional approaches seldom encode structured relationships explicitly within token interactions. Self-attention mechanisms effectively capture dynamic contextual dependencies, but their reliance on learned weight distributions limits the preservation of long-range hierarchical structures in generated sequences. Dependency-aware token encoding introduces a structured approach to embedding initialization, ensuring that relational constraints are embedded within token representations rather than inferred solely through attention dynamics. The proposed encoding mechanism refines token interactions through dependency-weighted attention computations, ensuring that syntactic and semantic dependencies are retained across multiple processing layers. Empirical evaluations indicate reductions in perplexity across diverse linguistic benchmarks, suggesting improvements in contextual coherence and predictive consistency in autoregressive text generation. Computational efficiency assessments reveal a moderate increase in memory consumption and training time, attributed to additional matrix computations within the encoding module, yet scalability remains feasible within conventional transformer architectures. Structured encoding enhances lexical variation and dependency retention, reinforcing linguistic coherence without requiring external syntactic annotations or auxiliary training objectives. Statistical comparisons highlight improvements in dependency alignment, particularly in longer sequences where conventional self-attention models exhibit degradation in hierarchical consistency. Sentence length distributions indicate a reduction in abrupt phrase transitions, further supporting the hypothesis that explicit dependency encoding facilitates more structured phrase generation.

大模型依赖编码文本生成注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。