返回首页
arXiv AI··论文与技术

Evaluating Model Capabilities After Benchmark Saturation

中文摘要

随着AI模型导致基准测试饱和,单一分数难以区分能力。该研究探讨利用任务转录而非仅靠分数,以实现更深入的模型评估。

English Summary

As AI models saturate benchmarks, scores lose their ability to differentiate capabilities. This research explores using task transcripts instead of mere scores for more detailed evaluation.

原文节选

arXiv:2610.20667v1 Announce Type: new Abstract: Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated. We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items. We evaluate representative contemporary approaches for extracting information from model transcripts: pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. We find that saturated transcripts retain capability information beyond benchmark accuracy. Amongst our evaluated protocols, pairwise judging perfor…