为大模型生成的专利权利要求设计多维度评估框架
PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims
- 将权利要求拆解为层级化元素，结合法律技术标准验证
- 在400个专利权利要求上与专家评分相关性达0.819
- 适合专利生成与验证研究者使用
高风险文本如专利权利要求、病历记录和技术报告结构复杂，对准确性和可靠性要求极高。尽管大语言模型（LLMs）已用于自动化生成这些高风险领域内容，但可靠评估其输出仍是重大挑战。传统自然语言生成（NLG）指标对通用文档有效，却无法捕捉复杂高风险文档的关键结构与法律特征。为此，我们提出PatentScore，一种专为最复杂严谨领域之一——专利权利要求设计的多维度评估框架。该框架整合了权利要求元素的层级分解、基于法律与技术标准的验证模式，并在结构、语义和法律三个维度进行评分。在包含400个Claim1的自建数据集上的实验表明，PatentScore与专家标注的相关性达到r = 0.819，显著优于广泛使用的NLG指标。本工作确立了评估大模型生成专利权利要求的新标准，为专利生成与验证研究提供了坚实基础。
原文摘要 · Abstract (English)
High-stakes texts such as patent claims, medical records, and technical reports are structurally complex and demand a high degree of reliability and precision. While large language models (LLMs) have recently been applied to automate their generation in high-stakes domains, reliably evaluating such outputs remains a major challenge. Conventional natural language generation (NLG) metrics are effective for generic documents but fail to capture the structural and legal characteristics essential to evaluating complex high-stakes documents. To address this gap, we propose PatentScore, a multi-dimensional evaluation framework specifically designed for one of the most intricate and rigorous domains, patent claims. PatentScore integrates hierarchical decomposition of claim elements, validation patterns grounded in legal and technical standards, and scoring across structural, semantic, and legal dimensions. In experiments on our dataset which consists of 400 Claim1, PatentScore achieved the highest correlation with expert annotations ($r = 0.819$), significantly outperforming widely used NLG metrics. This work establishes a new standard for evaluating LLM-generated patent claims, providing a solid foundation for research on patent generation and validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。