AI助手自动生成科研影响力摘要,让报告效率提升10倍
Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries
- 人机协同AI自动搜集跨平台学者数据并生成摘要
- 81.7%的摘要被双评员认可或只需微调,节省15小时/人
- 适合需要大规模科研影响力评估的机构和项目管理者
临床与转化科学奖(CTSA)项目需记录学者研究影响力,但人工整理每位学者档案耗时约15小时,难以扩展至全队。我们构建了一个有人参与的AI代理,可整合各平台证据并为每位学者生成一句转化科学效益模型(TSBM)影响摘要供工作人员审核。在某CTSA中心对10位职业发展学者(KL2/K12)的评估中,两名评审员独立对507项发现进行“接受、修改、拒绝”编码,主要指标为双方均接受或修改的比例(一致可用率)。结果显示,双评审员均接受或修改率达81.7%,每人平均耗时14分钟/人,取代了原15小时的手工流程。评审者间一致性中等(科恩κ=0.43)。发现研究显示该系统召回率接近人工检索。其证据覆盖全部四个TSBM领域,约三分之一为常规流程忽略的非学术类成果。评审员对合成准确度评分4.5(满分5),实用性评分4.8。
原文摘要 · Abstract (English)
Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。