Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?
中文摘要
新论文提出LLM智能体蒸馏分层视图。策略内外选择在交互历史与当前动作前缀生成中不同,超越简单二分法。
English Summary
New paper proposes a hierarchical view for distilling LLM agents. It suggests on-policy/off-policy choices vary between interaction history and current action token generation.
arXiv:2610.31918v1 Announce Type: new Abstract: Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture wi…