用大模型自动采集高价数据，让普通研究者也能低成本做前沿研究。
Leveraging Large Language Models to Democratize Access to Costly Datasets for Academic Research
- 用GPT-4o-mini+RAG框架从公司披露文件中自动提取数据
- 1万份代理声明、1.2万份10-K文件的提取准确率达人工水平
- 每项处理耗时不足1小时，成本低于10美元，适合资源有限的研究者
获取昂贵数据集的不平等长期限制了弱势机构研究人员的学术贡献与职业发展。大型语言模型（LLM）的突破为数据民主化带来可能，可通过自动化方式从非结构化来源收集数据。我们开发并评估了一种新方法，利用GPT-4o-mini在检索增强生成（RAG）框架下从企业披露文件中提取数据。该方法在约10,000份代理声明中准确提取首席执行官薪酬比率，在超过12,000份10-K文件中准确提取关键审计事项（CAMs），处理时间分别为9分钟和40分钟，成本均低于10美元。这与人工收集需数百小时或商业数据库订阅需数千美元形成鲜明对比。为促进更包容的研究生态，我们公开方法与所生成数据集。
原文摘要 · Abstract (English)
Unequal access to costly datasets essential for empirical research has long hindered researchers from disadvantaged institutions, limiting their ability to contribute to their fields and advance their careers. Recent breakthroughs in Large Language Models (LLMs) have the potential to democratize data access by automating data collection from unstructured sources. We develop and evaluate a novel methodology using GPT-4o-mini within a Retrieval-Augmented Generation (RAG) framework to collect data from corporate disclosures. Our approach achieves human-level accuracy in collecting CEO pay ratios from approximately 10,000 proxy statements and Critical Audit Matters (CAMs) from more than 12,000 10-K filings, with LLM processing times of 9 and 40 minutes respectively, each at a cost under US $10. This stands in stark contrast to the hundreds of hours needed for manual collection or the thousands of dollars required for commercial database subscriptions. To foster a more inclusive research community by empowering researchers with limited resources to explore new avenues of inquiry, we share our methodology and the resulting datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。