返回首页
arXiv AI··论文与技术

Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

中文摘要

决策语言模型Jev可作为临床生成式AI评估的快速低成本替代方案,其与医生的评判一致性与主流LLM评判模型相当或更高。

English Summary

Jev, a decision language model, provides a faster and more affordable alternative to LLM judges for clinical AI, matching or exceeding their agreement with physician ratings.

原文节选

arXiv:2610.28826v1 Announce Type: new Abstract: Automated judges of clinical artificial intelligence (AI) outputs are necessary, but large language model (LLM) judges are costly and slow at scale. We compared Jev, a decision language model that returns structured judgments, with frontier and open-weight LLM judges in four use cases. On 118 pediatric answers, Jev's agreement with physicians' ordinal quality ratings (Gwet's AC2 0.50; expected score 0.53) matched or exceeded that of seven of 12 LLM judges, including frontier models, Claude Opus 5.5 (0.49) and GPT-6 (0.38). On physician-graded binary HealthBench criteria, Jev matched or exceeded four of six LLM judges. On MedHELM's open-ended tasks, Jev's agreement with a three-LLM jury approached or exceeded the jurors' internal agreement. On HealthBench Professional, Jev matched the benchmark's reference judge, GPT-5.4, on 91% of criteria (other judges, 91-93%). Frontier judges cost 73-294 times more than Jev and took 7-57 times longer. Jev may enable efficient clinical AI evaluation at scale.