Back to Home
arXiv AI··Papers & Tech

AOS-Bench: Can LLM Agents Manage Dynamic Workloads?

中文摘要

AOS-Bench评估LLM智能体处理并发任务流和管理系统资源的能力,解决了现有基准无法衡量动态工作负载的问题。

English Summary

AOS-Bench evaluates LLM agents' ability to manage concurrent task streams and system resources, addressing existing benchmarks' failure to measure dynamic workloads.

Original Excerpt

arXiv:2610.33178v1 Announce Type: new Abstract: Coding agents are moving from handling one request at a time to serving as persistent assistants to which users hand a stream of work: a build, a data-processing job and a bug fix may all be pending on the same machine at once. Such an agent must decide when to take on new tasks, how to delegate and overlap them, and how to stay within the machine's CPU and memory. Existing benchmarks cannot measure this, because they evaluate each task in isolation. We introduce \benchname{}, which evaluates an agent system as a workload manager: verified tasks arrive over time and must be delivered under shared CPU and memory limits, and the same tasks run one at a time serve as a per-system reference. Across five model--harness pairs, isolated task success does not carry over evenly. On heavy workloads every system loses 13 to 23 points. On medium workloads, systems span 11 points in isolated success but 26 points in stream success, and two systems 0.6 points apart in isolation deliver 54\% and 68\% of the same tasks. Execution-policy interventions trace much of this gap to task intake: serial dispatch leaves 73\% of heavy-workload tasks unclaimed.…