arXiv:2501.17617cs.CL2025-01

通过概率层重对齐提升长文本生成的上下文一致性。

Structured Context Recomposition for Large Language Models Using Probabilistic Layer Realignment

  • 动态调整变压器层内表征,基于语义相关性重新分配注意力权重。
  • 在超长序列下显著减少主题突变和逻辑矛盾,保持生成连贯性。
  • 适合需要稳定多轮对话与长文档生成的实用场景。

长序列生成常因传统自注意力机制难以保留长距离依赖而导致上下文一致性下降。现有方法如记忆压缩和检索增强条件化会带来计算开销,增加推理延迟或存储负担。本文提出结构化上下文重组成(SCR),采用概率层重对齐策略,动态调整变压器层中学习到的表示,确保语义相关的嵌入在扩展变换中持续保留。该方法通过递归加权函数,依据推断的上下文相关性重新分配表征重点,而非依赖固定词元级注意力分数。实验表明,概率重对齐有效缓解了超出标准注意力窗口时的突发主题转换与逻辑不一致问题。序列级熵分析显示,SCR在不引入过度输出正则化的情况下调节了表征变异性,维持生成多样性同时保障上下文对齐。注意力头偏差测量证实层级重加权促进了跨层的平滑词元依赖过渡,增强了多轮交互与文档级推理的稳定性。资源评估表明,尽管处理时间略有增加,但内存开销仍在可接受范围内,适用于自回归生成应用的实际部署。

原文摘要 · Abstract (English)

Extended sequence generation often leads to degradation in contextual consistency due to the inability of conventional self-attention mechanisms to effectively retain long-range dependencies. Existing approaches, including memory compression and retrieval-augmented conditioning, introduce computational trade-offs that either increase inference latency or impose additional storage overhead. Structured Context Recomposition (SCR) introduces a probabilistic layer realignment strategy that dynamically adjusts learned representations within transformer layers, ensuring that semantically relevant embeddings persist throughout extended transformations. The proposed method enhances coherence retention through a recursive weighting function that redistributes representational emphasis based on inferred contextual relevance rather than relying on fixed token-level attention scores. Empirical results indicate that probabilistic realignment mitigates abrupt topic shifts and logical inconsistencies, particularly in scenarios where sequences exceed standard attention window constraints. Sequence-level entropy analysis further reveals that SCR moderates representational variability without introducing excessive output regularization, allowing models to sustain generative diversity while preserving contextual alignment. Attention head deviation measurements confirm that hierarchical reweighting contributes to smoother token dependency transitions across transformer layers, reinforcing the stability of multi-turn interactions and document-level reasoning. Computational resource assessments show that while SCR incurs a moderate increase in processing time, memory overhead remains within feasible limits, making it suitable for practical deployment in autoregressive generative applications.

大模型上下文对齐注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。