arXiv:2510.05431cs.CL2025-10被引 3

用大模型生成的可信度指标,提升专利分类可靠性。

Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classification

  • 将大模型理由重定义为可信度指标,动态调节训练权重。
  • 在USPTO-2M上相对提升38.7%的宏平均F1值。
  • 可信度评分与专家判断高度相关,可解释且可审计。

根据分类体系组织大规模专利文档是影响引文检索、技术知识发现和知识产权决策准确性的核心信息管理任务。现有方法将大语言模型(LLMs)生成的自然语言理由压缩为小型学生模型,但这些理由中固有的逻辑错误、标签错配和分类体系偏离会无差别地被吸收,损害分类可靠性并传播至下游流程。我们提出自过滤蒸馏(SFD),将大模型生成的理由重新解读为可信度指标,而非真实标签,直接嵌入学习过程。SFD融合三个无监督信号生成统一可信度分数,动态调节每个训练样本的贡献:自我一致性(独立生成理由的一致性)、类别蕴含对齐(理由与分配的CPC类别定义的语义一致性)、大模型共识评分(通过独立验证器评估外部合理性)。在包含超过两百万专利的USPTO-2M基准上,SFD在四种学生架构上实现最高38.7%的相对宏平均F1提升,且可信度分数与专家判断呈显著正相关(r=0.685),证明该框架不仅预测精准,还提供可分解的置信度语义,支持大规模专利知识组织中的可审计、自注释分类结果。

原文摘要 · Abstract (English)

Organizing large-scale patent corpora according to classification schemes is a core information management task that determines the accuracy and efficiency of prior art retrieval, technology knowledge discovery, and intellectual property decision-making. Recent approaches distill natural language rationales generated by large language models (LLMs) into compact student models, yet logical errors, label mismatches, and taxonomy misalignments inherent in these rationales are indiscriminately absorbed during training, undermining classification reliability and propagating errors throughout downstream information processes. Rather than correcting such errors post-hoc, we propose Self-Filtered Distillation (SFD), which embeds quality assurance directly into the learning process by reinterpreting LLM-generated rationales as trust indicators rather than ground-truth supervision. SFD integrates three unsupervised signals into a unified trust score that dynamically modulates each training instance's contribution: Self-Consistency, which quantifies agreement among independently generated rationales; Class Entailment Alignment, which evaluates semantic coherence between a rationale and its assigned CPC class definition; and LLM Agreement Scoring, which assesses external plausibility through an independent verifier. On the USPTO-2M benchmark comprising over two million patents, SFD achieves up to 38.7\% relative improvement in Macro-F1 across four student architectures, and the strong correlation between trust scores and expert judgments ($r = 0.685$) confirms that the framework provides not only accurate predictions but also decomposable confidence semantics that enable auditable and self-documenting classification outcomes for large-scale patent knowledge organization.

专利分类可信度评估模型蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。