用大模型生成法律求助问题，提升自动分诊准确率。
On Wednesdays, We Ask Questions: Optimizing "Active Listening" in Automated Legal Triage and Referral
- 用低成本LLM ensemble生成跟进问题，优化法律求助匹配
- 加入高成本GPT-5后，问题质量与分类准确率显著提升
- 发现家暴等类别的信息获取不均，建议设立专业筛查小组
FETCH分类器通过生成跟进问题来帮助精准识别申请人的法律问题，采用低成本的LLM集成方法。本文通过资深律师与LLM辅助评估该问题生成策略，发现尽管低成本LLM在分类任务中表现良好，但生成高质量、通俗易懂的问题仍需更复杂、更高成本的模型。结合法律接案员讨论，提出用于评估法律接案问题的评分标准，发现仅靠提示工程无法有效提升问题质量。此外，发现LLM作为评判者与人类评分存在分歧。实验表明，引入单一高成本模型GPT-5后，系统能有效从申请人处获取相关资讯，使分类任务性能显著提高。同时发现不同法律类别间信息采集不均衡，尤其家暴案件与家庭法筛查协议不符，提示应为特定法律领域设置专用筛查团队。
原文摘要 · Abstract (English)
The FETCH classifier generates follow-up questions to help refine the best match for the applicant's legal problem, using a low-cost ensemble of LLMs. In this paper, we describe an expert attorney and LLM-assisted evaluation of the follow-up question approach in FETCH and show that while low-cost LLMs perform well at classification tasks, generating high-quality plain-language questions in this setting appears to require a more sophisticated and higher-cost model. Through discussion with legal intake workers, we propose a rubric for the evaluation of legal intake classification questions, and we find that prompt engineering alone is not enough to improve question quality for intake purposes. We also find that LLM-as-judge and human ratings diverge. We demonstrate that with the addition of a single high-cost model, GPT-5, the classifier can elicit relevant information from applicants for legal help, and that the questions lead to more accurate performance at classification tasks. We also find uneven fact elicitation across different categories, including domestic violence, at odds with family law screening protocols, suggesting the value of including dedicated screening panels for certain areas of law.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。