arXiv:2510.08774cs.LGcs.AI2025-10

让大模型理解文本中的链接结构,提升嵌入表示效果。

Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddings

  • 将超链接等结构信息直接融入模型编码过程,而非事后处理。
  • 在检索、分类等任务中显著优于纯文本和后处理方法。
  • 适合需要理解文档间关联的场景,如学术推荐与知识图谱。

大型语言模型(LLM)生成的文本嵌入已成为众多应用的基础。然而,这些模型通常仅处理原始文本,忽略了超链接、引用等结构信息所蕴含的重要上下文,而这些信息在许多真实数据集中至关重要。本文提出并系统评估了一种新范式:通过将结构关系直接整合进 LLM 的内部编码过程,实现结构感知的文本嵌入,而非依赖传统的后处理聚合。我们研究了两种在过程中实现的方法:顺序拼接与并行缓存。在跨检索、聚类、分类与推荐任务上的大量零样本实验表明,我们的结构感知方法始终优于仅使用文本或后处理的基线模型。分析揭示关键权衡:顺序拼接在噪声较多、中等长度上下文中表现更佳,而并行缓存对长且高信号上下文更具可扩展性,但对干扰项更敏感。为应对结构数据噪声问题,我们还引入并验证了两种有效技术:上下文蒸馏与语义平衡。本工作首次提供了对在过程结构感知编码的全面分析,为构建更强大、更具上下文感知能力的嵌入模型提供了蓝图。

原文摘要 · Abstract (English)

Text embeddings from Large Language Models (LLMs) have become foundational for numerous applications. However, these models typically operate on raw text, overlooking the rich structural information, such as hyperlinks or citations, that provides crucial context in many real-world datasets. This paper introduces and systematically evaluates a new paradigm for generating structure-aware text embeddings by integrating these structural relations directly into the LLM's internal encoding process, rather than relying on traditional post-hoc aggregation. We investigate two primary in-process methods: sequential concatenation and parallel caching. Through extensive zero-shot experiments across retrieval, clustering, classification, and recommendation tasks, we demonstrate that our structure-aware approaches consistently outperform both text-only and post-hoc baselines. Our analysis reveals critical trade-offs: sequential concatenation excels with noisy, moderate-length contexts, while parallel caching scales more effectively to long, high-signal contexts but is more susceptible to distractors. To address the challenge of noisy structural data, we also introduce and validate two effective techniques: Context Distillation and Semantic Balancing. This work provides the first comprehensive analysis of in-process structure-aware encoding, offering a blueprint for building more powerful and contextually aware embedding models.

文本嵌入结构感知大模型链接分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。