arXiv:2605.10073cs.CL2026-05

用专利权利要求的复杂关系图增强文本表示,提升专利分类检索效果。

Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning

论文配图:Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning
图 1 · 摘自论文原文
  • 构建带类型的权利要求图,区分法律引用与技术关联
  • 通过注意力掩码和可学习偏置将图结构映射到词级注意力
  • 无需额外图计算即可推理,适合实际系统部署

预训练语言模型通过将权利要求编码为扁平的词元序列推进专利分类与检索,但忽略了权利要求间的依赖层级。将该层级融入自注意力面临两大挑战:其一,权利要求间的关系类型可靠性不同,统一处理可能导致噪声技术关系干扰更可靠的法律引文;其二,依赖关系定义在权利要求层面,而Transformer注意力作用于词元,直接注入结构颇具难度。我们提出专利异构依赖图引导注意力编码器(PHAGE),构建区分法律引文与技术关系的有类型权利要求图。PHAGE通过连接性掩码与可学习的关系感知偏置,将声明级拓扑投影至词元级注意力,并采用双粒度对比目标进行微调,融合专利间分类体系与专利内拓扑结构。推理时移除图特异性注意力组件,仅通过标准编码器前向传播生成表示,无需构造依赖图。在专利分类、检索与聚类任务上的实验表明,PHAGE持续优于领域适配及引文感知基线,证明了声明级结构引导对无图专利表示学习的价值。

原文摘要 · Abstract (English)

Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.

专利表示注意力机制图神经网络自然语言处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。