Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents
中文摘要
论文提出“先选后行”策略,通过比较价值估计优化长程工具调用智能体,解决复杂任务中的信用分配及步进监督难题。
English Summary
This paper proposes comparative value estimation for long-horizon tool-use agents, addressing weak credit assignment and the difficulty of obtaining reliable step-level supervision.
arXiv:2610.02330v1 Announce Type: new Abstract: Large language models (LLMs) rely on long-horizon tool invocation sequences for complex tasks, where each invocation can alter the task state and condition subsequent decisions. In long-horizon tool use, final-outcome rewards provide weak credit assignment over long interaction traces. Step-level rewards can offer more targeted feedback, but obtaining reliable step supervision often requires human or LLM judgment, or additional rollouts to estimate the downstream effect of an intermediate decision. In this paper, we argue that effective tool-use agents should estimate the long-horizon value of a possible next tool invocation before executing it. This objective requires comparative supervision over alternative invocations under the same context, while logged trajectories only contain the invocation that was actually taken. Therefore, we propose Comparative Inference for Tool-use Agents (CITA). CITA trains a Comparative Inference Model (CIM) from paired signals that combine observed tool behavior, scalable supervision from a Bayesian tool-graph simulator, and semantic judgments from LLM-based comparison. The resulting CIM learns to esti…