arXiv:2501.18957cs.CL2025-01被引 1

用连续张量场模拟上下文流动,提升长文本理解能力

Intrinsic Tensor Field Propagation in Large Language Models: A Novel Approach to Contextual Information Flow

  • 将上下文关系建模为连续张量场,通过微分方程实现信息传播
  • 在多种语言结构中显著降低语法错误与事实性错误
  • 适用于跨领域文本,尤其适合长依赖任务

上下文传播是语言模型架构中的核心挑战,尤其在需要保留长距离依赖的任务中。传统注意力机制虽在诸多应用中有效,但因依赖离散标记交互,在处理长序列时难以维持连贯的上下文表示。本文提出内在张量场传播(ITFP),将上下文关系建模为分布在标记嵌入上的连续张量场。传播动态由微分方程支配,实现结构化的上下文信息流,增强标准注意力机制以提升连贯性与召回能力。在开源Transformer模型上的一系列实验表明,ITFP在上下文保留、依赖解析和推理稳定性方面均有可测量提升。与基线模型对比显示,句法不一致和事实性错误减少;消融研究指出传播深度与整合强度的选择显著影响性能。额外评估表明,ITFP在不同文本类型间具有良好泛化能力,证实其在常规语言建模之外的适用性。尽管引入张量场计算带来计算开销,但实证结果表明其在准确率与连贯性上的收益超过处理成本增加。

原文摘要 · Abstract (English)

Context propagation remains a central challenge in language model architectures, particularly in tasks requiring the retention of long-range dependencies. Conventional attention mechanisms, while effective in many applications, exhibit limitations in maintaining coherent contextual representations over extended sequences due to their reliance on discrete token interactions. A novel approach is introduced through the formulation of Intrinsic Tensor Field Propagation (ITFP), which models contextual relationships as continuous tensor fields distributed across token embeddings. The propagation dynamics are governed through differential equations that enable a structured flow of contextual information, augmenting the standard attention mechanism to enhance coherence and recall. A series of experiments conducted on an open-source transformer-based model demonstrate that ITFP provides measurable improvements in contextual retention, dependency resolution, and inference stability across various linguistic structures. Comparisons with baseline models reveal a reduction in syntactic inconsistencies and factual errors, while ablation studies indicate that the choice of propagation depth and integration strength significantly impacts model performance. Additional evaluations assessing domain generalization suggest that ITFP effectively adapts across different text genres, reinforcing its applicability beyond conventional language modeling tasks. Although computational trade-offs are introduced through the inclusion of tensor field computations, empirical findings suggest that the benefits in accuracy and coherence outweigh the increased processing demands.

语言模型上下文传播张量场

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。