Unconstrained CTI-REALM with Open-Weights
中文摘要
研究人员在 CTI-REALM 基准测试上评估了 26 个开源与闭源模型,对比了安全智能体任务中开源权重模型与闭源模型的优劣。
English Summary
Researchers evaluated 26 open-weight and closed models on the CTI-REALM benchmark, comparing the effectiveness of open-weight versus closed frontier models for security agent tasks.
arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The benchmark's published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to cap…