构建专利审查全流程数据集,助力大模型理解专利判断逻辑。
PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination
- 构建8143条美国专利审查记录的完整决策链数据集
- 大模型能精准找相关现有技术但难评估新颖性与非显而易见性
- 适合研究专利智能审查、法律推理与LLM可解释性的学者
专利审查在自然语言处理领域仍是挑战,即便在大语言模型(LLMs)出现后依然如此,因其需在专家领域对申请权利要求是否满足新颖性和非显而易见性标准做出细致的人类判断。以往研究多将其视为预测任务(如预测授权结果),依赖相似度指标或历史标签训练分类器,却忽视了审查员在办公通知文件中提供的逐步评估与理由。为此,我们构建了PANORAMA数据集,包含8,143条美国专利审查记录,完整保留原始申请、所有引用文献、非最终驳回意见及允许通知书。该数据集将审查过程分解为顺序性基准,模拟专利审查人员的实际流程,使研究人员可在每个步骤上评估大语言模型的能力。结果显示,尽管大模型在检索相关现有技术及定位关键段落方面表现良好,但在判断专利权利要求的新颖性和非显而易见性上仍存在明显不足。我们讨论这些发现并认为,推动自然语言处理在专利领域的进展需要更深入理解真实审查流程。数据集已公开于https://huggingface.co/datasets/LG-AI-Research/PANORAMA。
原文摘要 · Abstract (English)
Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted claim meets the statutory standards of novelty and non-obviousness against previously granted claims -- prior art -- in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in office actions documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, Non-Final Rejections, and Notices of Allowance. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。