返回首页
arXiv AI··论文与技术

Seq@K: Evaluating Language Models Under Sequential Feedback

中文摘要

论文提出 Seq@K 指标,评估语言模型利用连续反馈迭代改进答案的能力,解决了 Pass@K 等现有指标无法衡量反馈利用能力的局限。

English Summary

Seq@K is a new metric evaluating how language models iteratively refine answers using sequential feedback, addressing Pass@K's inability to measure feedback utilization.

原文节选

arXiv:2610.37859v1 Announce Type: new Abstract: Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that a…