Back to Home
arXiv AI··Papers & Tech

Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration

中文摘要

本文提出一种针对部署代理的闭环评估方法,通过分层测量和自我校准,解决单一基准测试无法持续反馈的问题。

English Summary

The paper proposes a closed-loop evaluation method for deployed AI agents, using layered measurement and self-calibration to move beyond one-off benchmarks.

Original Excerpt

arXiv:2611.07051v1 Announce Type: new Abstract: Once an agent is live, teams want to know whether it is doing its job, not just whether it throws errors. Yet most evaluation is one-off: a benchmark run, a look at traces, a single score. Its conclusions neither check themselves nor reach the agent. We propose closed-loop evaluation, an evaluation method for deployed agents. It models evaluation as performance estimation under biased raters and censored outcomes, and it requires every level's conclusion to trace back to atomic data and to be computed by versioned rules. The method covers four levels of analysis (span, trace, conversation and engagement), measures along a spine of Execution, Substance, Trajectory and Results, treats Conduct as a constraint throughout, and uses Provenance and Adjudication components to certify the evidence and the evaluators. Evaluator agents generated by an Agent Factory carry these components, each with a domain knowledge pack and an adapter harness. Two loops close the evaluation: an inner loop audits the evidence, rules and metrics themselves, and an outer loop uses per-component regression checks to drive agent improvement. We also give a pipeline…