返回首页
arXiv AI··论文与技术

Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark

中文摘要

研究利用102篇文献基准,评估了六种AI工具在文献检索、解读、综合及提出研究建议等综述全流程中的表现。

English Summary

Researchers evaluated six AI tools' performance in searching, interpreting, synthesizing, and proposing research using a 102-article benchmark.

原文节选

arXiv:2611.05459v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to support scientific literature reviews, yet evaluations often examine individual tasks rather than the review process as a whole. We assess six AI tools across four stages: finding relevant articles, interpreting them, synthesizing across them, and proposing future research. The benchmark is a published review of the Spanish plume weather pattern and its 102-article corpus. In the initial searches, five tools collectively identified 57 articles, with the best individual tool finding 42. Institutional library access increased discovery from 42 to 82 articles for Codex and from 15 to 79 for Claude Code. Given the complete corpus of 102 PDFs, three tools correctly identified all 18 articles discussing the Spanish plume substantially, but incorrectly classified 40% to 79% of the 84 passing mentions as substantial. The same three tools produced syntheses evaluated blindly by two domain experts. Although largely accurate, the syntheses provided limited insight and sometimes contained unsupported claims, quotation errors, and exaggerated statements of consensus. Finally, all three tools pr…