合成专利数据能提升分类效果,但需权衡数量与质量。
When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classification

- 用大模型生成合成专利数据,按标签条件或改写原文
- 数据量增大可显著提升性能,但真实数据越多,生成数据越需高质量
- 20%-30%真实数据+70%-80%合成数据为最优策略
使用大语言模型生成的合成数据进行多标签专利分类时,需关注两个问题:何时有效以及为何有效。实验采用六种开源大模型(3.8B至12B参数),在四个真实数据规模下对64个WIPO辅助技术标签进行分类。采用全合成生成和改写两种方法,分别与三类分类器结合。结果显示,BERT-for-Patents的micro F1从0.120提升至0.702,主要归因于样本量增加;仅用165个样本重复生成即达0.678,相比控制组提升+0.024,相比最佳基线(焦点损失重加权)提升+0.219。低真实数据条件下,数据量效应主导,最大均值差异(MMD)与性能相关系数r = +0.95;当真实数据占比达1:10时,相关性反转,r = -0.73(Fisher z = +6.47,p < 0.001,95%置信区间[+0.96, +1.00])。固定预算下,20%-30%真实数据搭配70%-80%合成数据优于纯真实或纯合成方案。尽管某语料可使micro F1提升+0.58,但会恶化Jaccard重叠检索指标。不同提示族可能解释该现象,但标准专利过滤仍使nDCG@10下降26%。
原文摘要 · Abstract (English)
The issues that must be considered regarding the utilization of synthetic data generated through LLMs for multilabel patent classification include (i) when the use of such data may help and (ii) why. Indeed, the former part appropriately adjusts for the possibility of improving results by an increase in sample size. The current experiment involves six open-source LLMs (from 3.8B to 12B parameters) for four real-data regimes in classification of 64 WIPO labels of assistive technologies. Both full-synthesis generation, conditioned on the label set, and paraphrasing methods are applied, with each used in combination with three classifier categories. It is shown that the claimed improvements in micro F1 for BERT-for-Patents from 0.120 to 0.702 mainly reflect a volume effect; indeed, replication with replacement in 165 examples produces 0.678. Thus, the improvement over the control is +0.024, while compared to the best baseline (focal loss reweighting) is +0.219. The second crucial point to consider here is that of evolving fidelity scores as the data generation regime varies. For low real-data regimes, the volume effect dominates and the correlation coefficient between maximum mean discrepancy (MMD) and classification performance equals r = +0.95. As more real data is used, the correlation becomes inverted and reaches r = -0.73 at the 1:10 regime (Fisher z = +6.47, p < 0.001, 95% CI on Delta r [ +0.96, +1.00 ]). In terms of a fixed budget allocation, combining real data (about 20-30%) with synthetic (70-80%) outperforms both purely synthetic and purely real strategies. Moreover, a corpus that allows for improvement in classification performance up to +0.58 in raw micro F1 may adversely affect a Jaccard-overlap retrieval proxy. Prompt-family variations for other genres may provide some explanation of the phenomenon, but using the standard-patent filter still decreases nDCG@10 by 26%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。