首个专利权利要求修订数据集,助力AI理解法律文本精确性。
Patent-CR: A Dataset for Patent Claim Revision
- 构建首个英文专利权利要求修订数据集,含被拒与授权版本。
- 大模型常产生无效修改,领域专用模型微调效果更优。
- 揭示自动评估与人工评价差异,支持后续研究改进。
本文提出 Patent-CR,首个专用于英语专利权利要求修订任务的数据集,包含专利审查员驳回的初始申请版本与最终授予版本。与常规文本修订任务不同,专利权利要求修订需满足严格法律标准,包括范围清晰性、技术准确性、语言精确性和法律稳健性,远超新颖性与创造性。我们通过专业人工评估,测试了多种大语言模型(LLMs),涵盖通用模型、文本修订模型及领域专用模型。结果显示,多数模型生成的修改偏离目标,无效修正频发;而领域专用模型与微调方法表现更佳。值得注意的是,GPT-4虽优于其他模型,但仍需进一步修订方可达到审查标准。此外,我们发现自动化评估与人工评价存在显著不一致,其中基于GPT-4的自动化评估与人工判断相关性最高。该数据集及初步实证研究为专利权利要求修订的深入探索提供了宝贵资源。
原文摘要 · Abstract (English)
This paper presents Patent-CR, the first dataset created for the patent claim revision task in English. It includes both initial patent applications rejected by patent examiners and the final granted versions. Unlike normal text revision tasks that predominantly focus on enhancing sentence quality, such as grammar correction and coherence improvement, patent claim revision aims at ensuring the claims meet stringent legal criteria. These criteria are beyond novelty and inventiveness, including clarity of scope, technical accuracy, language precision, and legal robustness. We assess various large language models (LLMs) through professional human evaluation, including general LLMs with different sizes and architectures, text revision models, and domain-specific models. Our results indicate that LLMs often bring ineffective edits that deviate from the target revisions. In addition, domain-specific models and the method of fine-tuning show promising results. Notably, GPT-4 outperforms other tested LLMs, but further revisions are still necessary to reach the examination standard. Furthermore, we demonstrate the inconsistency between automated and human evaluation results, suggesting that GPT-4-based automated evaluation has the highest correlation with human judgment. This dataset, along with our preliminary empirical research, offers invaluable insights for further exploration in patent claim revision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。