arXiv:2502.08818cs.CL2025-02

动态重配置词嵌入,让大模型更灵活地适应上下文变化

Lexical Manifold Reconfiguration in Large Language Models: A Novel Architectural Approach for Contextual Modulation

  • 基于流形的几何变换,动态调整词向量位置
  • 降低困惑度,提升长文本连贯性与语义一致性
  • 适合需要高上下文敏感度的语言生成任务

在长文本序列中,词嵌入的上下文适应能力直接影响语言模型的连贯性与语义保持。静态嵌入限制了词汇灵活性,导致复杂句式或领域术语变化时表现不佳。为此,提出一种通过连续几何变换动态重构词嵌入的结构化方法,引入基于流形的调控机制,使嵌入在保持语言关系的前提下随语境演化。实证评估显示,该方法显著降低困惑度,提升词汇连贯性与句级连续性,尤其在结构化和领域自适应文本生成任务中效果明显。对比分析表明,动态重构的嵌入能更好维持上下文一致性,减少词间依赖错位,同时保持输出流畅性。计算开销评估显示,训练阶段因嵌入迭代优化略有增加,但推理效率仍高,具备实时生成可行性。多数据集测试进一步验证,动态调制嵌入展现出更广的词汇多样性,有效抑制重复模式,实现更具适应性的表示学习。

原文摘要 · Abstract (English)

Contextual adaptation in token embeddings plays a central role in determining how well language models maintain coherence and retain semantic relationships over extended text sequences. Static embeddings often impose constraints on lexical flexibility, leading to suboptimal performance when faced with complex sentence structures or domain-specific terminology shifts. To address this limitation, a structured approach was developed for dynamically reconfiguring token embeddings through continuous geometric transformations, ensuring that representations evolved in response to evolving discourse structures. A manifold-based transformation mechanism was integrated to regulate lexical positioning, allowing embeddings to undergo controlled shifts while preserving linguistic relationships across varying textual contexts. Empirical evaluations demonstrated that embedding reconfiguration contributed to reductions in perplexity, improved lexical coherence, and enhanced sentence-level continuity, particularly in structured and domain-adaptive text generation tasks. Comparative analyses of embedding drift indicated that dynamically restructured representations maintained stronger contextual consistency, reducing misalignment in token dependencies while preserving fluency in language modeling outputs. Computational overhead assessments confirmed that while training complexity increased due to the iterative refinement of embeddings, inference remained efficient, ensuring practical feasibility for real-time generation. Evaluations across multiple datasets further demonstrated that dynamically modulated embeddings exhibited broader lexical diversity, reducing repetitive token patterns and enabling a more adaptable representation learning process.

语言模型嵌入优化上下文适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。