When success is too rare to learn from当成功稀少到难以从中学习
Reinforcement learning from verifiable rewards (RLVR) has a simple appeal: let a model try a problem, check the answer, and reinforce what works. On a sufficiently hard problem, however, every attempt in a group may fail. The verifier then gives GRPO no reward-based preference among those attempts.可验证奖励强化学习（RLVR）的思路很直接：让模型尝试解题，检验答案，再强化有效的行为。但当问题足够困难，同一组尝试可能全部失败。此时，验证器无法为 GRPO 提供基于奖励的优劣区分。
A stronger teacher can help the student encounter reasoning it would rarely produce on its own. But introducing teacher guidance raises a second question: can the student actually learn to reproduce useful decisions from that guidance?更强的教师能让学生接触到自己很难生成的推理。不过，引入教师指导之后，还有一个关键问题：学生能否真正学会其中有价值的决策？
Change the reasoning path the student encounters.改变学生所经历的推理路径。
Provide a direct learning target at each guided step.在受到指导的步骤上提供直接的学习目标。
Better trajectories need a way to be learned更好的轨迹，还需要配套的学习方式
A teacher can supply a successful trajectory, or a prefix that the student continues. This expands access to useful reasoning. Yet a teacher’s decisions can be far from what the student would generate. Continuing from those decisions is not the same as learning to make them independently.教师可以提供一条成功轨迹，或给出前缀让学生续写，从而增加学生接触有效推理的机会。但教师的决策可能与学生自身的生成分布相距很远。能够接着这些决策往下写，并不等于能够独立做出它们。
The paper tests this distinction on Survo. Even oracle prefixes extracted from correct solutions leave prefix-only guidance below 1% on both avg@8 and pass@8. Directly supervising the supplied prefixes gives much stronger results. Prefix quality alone does not account for the difference: the objective must help the student learn the teacher-provided behavior.论文在 Survo 上检验了这一区别。即使使用从正确解答中提取的 oracle 前缀，仅提供前缀时的 avg@8 和 pass@8 仍低于 1%；对这些前缀施加直接监督，则取得了明显更好的结果。前缀的质量并不能单独解释收益，训练目标也需要帮助学生学会教师提供的行为。
Dense feedback still needs a recovery path密集反馈，也需要一条完整的纠错路径
The complementary approach keeps the student’s rollout and asks a teacher for token-level feedback along it. On an erroneous prefix, however, the teacher may support both continuing the current reasoning and backing up to correct it. A local signal such as “Wait” does not by itself show the student how that correction should unfold.另一类方法保留学生生成的轨迹，并让教师沿着轨迹提供 token 级反馈。然而，当学生前缀已经出错，教师可能同时支持“继续当前推理”和“回退纠错”。像“Wait”这样的局部信号，本身并没有展示后续纠错应该如何展开。
These observations motivate combining the two mechanisms. The teacher should help construct a better path, and the student should receive direct supervision on the decisions that make that path useful.这些观察促使我们结合两种机制：教师帮助构造更好的推理路径，学生则在使这条路径有效的关键决策上获得直接监督。
Try, correct, continue先尝试，再纠正，然后继续
PIVOT—Progressive Intervention via Online Teacher Correction—builds teacher-assisted rollouts in three stages. The student first makes a short attempt. The teacher then continues from the student’s context for a limited segment. Finally, the student resumes from the augmented context.PIVOT（Progressive Intervention via Online Teacher Correction）将教师辅助轨迹分成三个阶段：学生先做一段简短尝试；教师基于学生的上下文继续生成一个有限长度的片段；最后，学生从补充后的上下文恢复生成。
The complete rollout participates in the RLVR objective. At the same time, a teacher–student KL loss supervises the teacher-generated segment. Teacher assistance creates opportunities to reach a reward; the additional supervision gives the student a direct target for learning the reasoning inside that assistance.完整轨迹参与 RLVR 优化；与此同时，教师与学生之间的 KL 损失直接监督教师生成的片段。教师辅助创造获得奖励的机会，而额外监督让学生有明确的目标，去学习这段辅助中包含的推理。
“Online” refers to generating guidance during training from the student’s current context. In the reported recipe, the intervention position is fixed; PIVOT does not use an automatic error detector. The evaluations measure the trained student’s own reasoning ability.这里的“Online”指训练过程中基于学生当前上下文生成教师指导。论文配方采用固定的干预位置，并未使用自动错误检测器；评估衡量的是训练后学生自身的推理能力。
Begin with the student’s own reasoning从学生自己的推理出发
Why let the student start? A later teacher segment might simply contain more informative tokens. Alternatively, conditioning on the student’s partial solution may make the guidance easier for the student to use. The paper tests this distinction by replacing the student opening with a teacher opening, while leaving the rest of the rollout unchanged. The intervention-position and opening-source experiments in this section use forward KL.为什么要让学生先开始？一种可能是，位置更靠后的教师片段包含更多有效信息；另一种可能是，基于学生已有解题状态的指导更容易被学生利用。论文通过将学生开头替换为教师开头、保持其余轨迹流程不变，检验这一区别。本节的干预位置与开头来源实验均采用正向 KL。
The teacher-opening variant performs substantially worse. This supports the view that the student’s own attempt provides a useful starting state for guidance. An opening generated by a stronger model is not automatically easier for the student to learn from.教师开头的表现明显更差。这支持了一个解释：学生自己的尝试为指导提供了有价值的起点。能力更强的模型给出的开头，并不一定更容易被学生学会。
A short, 512-token student prefix improves both metrics over prefix intervention on all three logic games. But later is not always better: on Minesweeper, avg@8 falls from 19.25% at 512 tokens to 12.50% at 2,048. Delaying intervention leaves less room for recovery, verification, and emitting the final answer.在三种逻辑游戏上，先生成 512 token 的学生前缀，相比直接从教师前缀开始，都能提升两项指标。但干预并非越晚越好：在 Minesweeper 上，avg@8 从 512 token 时的 19.25% 降到 2,048 token 时的 12.50%。推迟干预，会挤占纠错、验证和输出最终答案的空间。
A closer look: one puzzle, three intervention points进一步看一个案例：同一道谜题，三个干预位置
The appendix freezes the context after a 4,096-token teacher segment and samples four continuations from one fixed student checkpoint. The same Survo puzzle exposes two different ways a hand-off can fail. Intervention positions below are student-prefix lengths in tokens.附录在 4,096 token 的教师片段之后固定上下文，再使用同一个学生检查点采样四次后续生成。同一道 Survo 谜题展示了教师交还状态后的两种失败方式。下表的干预位置表示学生前缀长度，单位为 token。
| Intervention干预方式 | Prefix · 0前缀 · 0 | Middle · 512中段 · 512 | Late · 2,048较晚 · 2,048 |
|---|---|---|---|
| Cells solved at hand-off交还时已解格子 | 2/10 | 10/10 | 6/10 |
| Continuation budget (tokens)剩余生成预算（token） | 4,096 | 3,584 | 2,048 |
| Student completions学生完成次数 | 0/4 | 4/4 | 0/4 |
| Teacher completions教师完成次数 | 4/4 | 4/4 | 0/4 |
The middle state is completed by the student in all four continuations. In the late state, the remaining cells can be solved, but the original budget expires before the answer is emitted. Giving the teacher an 8,192-token continuation budget restores teacher completion to 4/4.中段干预后的状态，学生四次续写均能完成。较晚干预后的状态虽然可以解出剩余格子，却在输出答案前耗尽原有预算；将教师后续生成预算扩大到 8,192 token 后，教师恢复到四次全部完成。
This is an illustrative case with four continuations per model, not a broad success-rate estimate. It does not isolate timing, notation, or solved progress as the sole cause.这是每个模型四次续写的说明性案例，并非大规模成功率估计；它也没有将干预时机、符号表达或已解进度单独识别为唯一原因。
Match the objective to the student’s starting point结合学生的起点选择监督目标
Both KL directions match student and teacher token distributions, but they emphasize different parts of those distributions. Reverse KL, KL(student ∥ teacher), discourages probability on continuations the teacher considers unlikely. Forward KL, KL(teacher ∥ student), encourages coverage of teacher-supported alternatives.两种 KL 方向都匹配学生与教师的 token 分布，但侧重点不同。反向 KL，即 KL（学生 ∥ 教师），抑制学生给教师认为不太可能的续写分配概率；正向 KL，即 KL（教师 ∥ 学生），鼓励覆盖教师支持的多种选择。
On the hard logic games, where initial success is almost absent, reverse KL raises mean avg@8 from 24.5% with forward KL to 35.6%. It also slightly leads on the challenging Math-ID split. Forward KL gives modestly higher accuracy on AIME 2024/2025 and MATH500, where the base model already succeeds more often.在初始成功几乎不存在的困难逻辑游戏上，反向 KL 将平均 avg@8 从正向 KL 的 24.5% 提升到 35.6%，并在困难的 Math-ID 划分上略微领先。在基座模型已有较高成功率的 AIME 2024/2025 和 MATH500 上，正向 KL 的准确率则略高。
A plausible interpretation is that concentrating on a viable reasoning mode helps when the student rarely succeeds, while broader coverage can help once successful reasoning is already accessible. The paper does not directly measure reasoning-mode coverage, so this remains an interpretation of the results.一个可能的解释是：当学生很少成功时，先集中学会一种有效推理方式更有帮助；当成功推理已经可达时，覆盖更多有效选择可能带来进一步收益。论文并未直接测量推理模式的覆盖度，因此这仍是对结果的一种解释。
Give corrections enough room to develop给纠正留出充分展开的空间
Teacher capacity matters, but so does the way teacher guidance is allocated. The paper varies one factor at a time under middle intervention with forward KL. Three choices emerge from these controlled ablations.教师能力很重要，指导如何分配也同样重要。论文在中段干预、正向 KL 的设置下，每次改变一个因素，得到三个实用结论。
Fewer, longer teacher segments较少、较长的教师片段
At a fixed 8k teacher-token budget, two 4k-token segments outperform four 2k or eight 1k segments on every reported logic-game metric. Connected reasoning appears more useful here than spreading the same budget thinly.在固定 8k 教师 token 预算下，两个 4k 片段在所报告的全部逻辑游戏指标上都优于四个 2k 或八个 1k 片段。在这些实验中，连贯的推理比将同样预算分散到更多轨迹更有效。
A more capable teacher can still help更强的教师仍能带来帮助
Using Qwen3-30B-A3B-Thinking improves avg@8 on all three games. Minesweeper rises from 19.25% to 40.12%; Kakurasu rises from 3.52% to 17.27%. The improvements are not uniform across every metric.换用 Qwen3-30B-A3B-Thinking 后，三种游戏的 avg@8 都提升了：Minesweeper 从 19.25% 升到 40.12%，Kakurasu 从 3.52% 升到 17.27%。不过，并非每个指标都同步提升。
Retain enough teacher-supported alternatives保留足够的教师分布信息
Top-30 supervision improves over top-10, while top-50 offers no consistent further benefit. The remaining probability is collected into a tail bucket. More retained tokens are not automatically more helpful.Top-30 监督优于 top-10，而 top-50 没有带来一致的进一步收益。剩余概率被汇总到一个尾部桶中；保留更多 token 并不自动意味着更有帮助。
The resulting default uses six student-only rollouts and two teacher-assisted rollouts, a 512-token student prefix, and top-30 teacher supervision. Teacher segments are capped at 4,096 tokens for logic games and 8,192 for math. The budget ablation holds teacher tokens fixed; it does not establish equal total training compute across all methods.最终默认设置采用六条纯学生轨迹和两条教师辅助轨迹，学生前缀为 512 token，并使用 top-30 教师监督。逻辑游戏与数学任务的教师片段上限分别为 4,096 和 8,192 token。预算消融固定的是教师 token 数，并不等于各方法的总训练计算量相同。
Does the student learn to reason better?学生自身的推理能力提升了吗？
The recipe is evaluated on hard Survo, Minesweeper, and Kakurasu tasks with a Qwen3-1.7B student, and on mathematical reasoning with Qwen3-4B. The default teacher is Qwen3-4B-Thinking. All results below measure the student with thinking mode enabled. Math-ID is a held-out in-distribution split; AIME 2024/2025 and MATH500 are out-of-distribution benchmarks.实验在困难的 Survo、Minesweeper 和 Kakurasu 任务上使用 Qwen3-1.7B 学生，在数学推理任务上使用 Qwen3-4B。默认教师为 Qwen3-4B-Thinking。下表结果均评估开启 thinking mode 的学生。Math-ID 是留出的同分布测试集；AIME 2024/2025 和 MATH500 是分布外评测基准。
Mean avg@8 on hard logic games
Strongest baseline (KDRL) → PIVOT with reverse KL
+16.1 percentage points; approximately 1.82×.困难逻辑游戏的平均 avg@8
最强基线 KDRL → 反向 KL 的 PIVOT
提升 16.1 个百分点，约为原来的 1.82 倍。
| Method方法 | Survo | Minesweeper | Kakurasu | Mean平均 |
|---|---|---|---|---|
| Base model基座模型 | 0.05 | 0.06 | 0.00 | 0.0 |
| RL only仅 RL | 0.05 | 1.94 | 0.00 | 0.7 |
| SFT + RLSFT + RL | 39.77 | 13.44 | 1.38 | 18.2 |
| LUFFYLUFFY | 1.30 | 0.81 | 0.00 | 0.7 |
| POPEPOPE | 0.03 | 0.00 | 0.00 | 0.0 |
| KDRLKDRL | 41.68 | 13.88 | 2.93 | 19.5 |
| PIVOT · Forward KLPIVOT · 正向 KL | 50.70 | 19.25 | 3.52 | 24.5 |
| PIVOT · Reverse KLPIVOT · 反向 KL | 72.90 | 25.44 | 8.38 | 35.6 |
| Method方法 | Math-ID | AIME 2024 | AIME 2025 | MATH500 |
|---|---|---|---|---|
| Base model基座模型 | 5.04 | 60.89 | 49.17 | 88.68 |
| RL only仅 RL | 20.70 | 60.62 | 49.74 | 94.30 |
| SFT + RLSFT + RL | 8.82 | 60.94 | 49.21 | 89.98 |
| LUFFYLUFFY | 16.66 | 60.68 | 46.93 | 90.60 |
| POPEPOPE | 22.09 | 60.00 | 46.61 | 94.20 |
| KDRLKDRL | 17.11 | 49.11 | 43.39 | 92.92 |
| PIVOT · Forward KLPIVOT · 正向 KL | 22.48 | 63.33 | 52.45 | 95.40 |
| PIVOT · Reverse KLPIVOT · 反向 KL | 23.05 | 62.24 | 52.08 | 95.25 |
Source: main paper, Table 2. Bold marks the best result in each column. avg@k measures average correctness over k samples; it is distinct from pass@k, which measures whether at least one of k samples succeeds.来源：论文主文表 2。粗体表示该列最佳结果。avg@k 衡量 k 次采样的平均正确率；它不同于 pass@k，后者衡量 k 次采样中至少成功一次的比例。
The gains are largest on the hard logic games. On Math-ID, PIVOT with reverse KL reaches 23.05%, compared with 22.09% for the strongest baseline, POPE. On AIME 2024/2025 and MATH500, both PIVOT variants outperform the compared baselines, and forward KL gives the highest average accuracy.最大的收益出现在困难逻辑游戏上。在 Math-ID 上，反向 KL 的 PIVOT 达到 23.05%，最强基线 POPE 为 22.09%。在 AIME 2024/2025 和 MATH500 上，两个 PIVOT 变体均优于所比较的基线，其中正向 KL 取得最高平均准确率。
The results show substantial progress, while leaving room to improve: even the best avg@8 on hard Kakurasu is 8.38%.这些结果带来了显著进展，也仍有较大提升空间：在困难 Kakurasu 上，最佳 avg@8 仍只有 8.38%。
Guidance that grows with the student让指导随着学生能力继续演进
PIVOT studies fixed intervention and supervision settings on logic games and mathematical reasoning. Whether the same recipe transfers to code generation, instruction following, or open-ended reasoning remains to be tested. The student also changes throughout training, so a natural next step is to adapt when and how the teacher helps as the student’s needs evolve.PIVOT 在逻辑游戏与数学推理上研究了固定的干预和监督设置。这套配方能否迁移到代码生成、指令遵循或开放式推理，还需要进一步检验。学生的能力也会随着训练改变，因此一个自然的下一步，是根据学生需求的变化，自适应地调整教师介入的时机与方式。
The central idea is simple: help the student reach a useful reasoning path, then teach the decisions along that path.核心想法很直接：帮助学生走上一条有价值的推理路径，再教会它沿途的决策。
Read the full paper阅读完整论文