返回首页
arXiv AI··论文与技术

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

中文摘要

MagicBeanStalk提出递归数据合成方法,有效解决RLVR训练中的零优势陷阱,通过优化数据质量提升模型学习效率。

English Summary

MagicBeanStalk introduces recursive data synthesis to overcome the RLVR zero-advantage trap, enhancing learning efficiency by optimizing training data quality and effectiveness.

原文节选

arXiv:2610.12659v1 Announce Type: new Abstract: Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided …