Waypoint-OPD: What One Discarded Checkpoint Knows That the Endpoints Don't

You are reviewing three months of someone else’s work. They hand you git diff v1..v3 and ask you to port it into your fork.

The diff is complete, and silent about the one thing you most want to know. A line written in week one that survived untouched looks exactly like a line that was written, reverted two weeks later, then rewritten from scratch. The first is a decision somebody stood behind; the second is churn. A diff cannot separate them, because a diff does not see history.

So you do what every reviewer does when the endpoints look thin: you open the middle commit. If the line in v2 already looks the way it looks in v3, your colleague was sure from the start; if v2 pointed the other way, leave it alone. And notice what reading v2 did not do — you are still porting v1..v3, character for character. The middle commit changed nothing about what you take, only how much of it you trust.

Waypoint-OPD (arXiv 2610.21601, Alibaba Cloud Computing, October 2026) is that instinct applied to logits instead of source lines. It keeps one intermediate checkpoint from a small teacher’s reinforcement-learning run — the kind such runs write and then delete — and uses the agreement between the run’s early and late halves to decide how loudly each piece of the teacher’s advice reaches a much larger student. The headline: Qwen3-8B goes from 53.96 to 56.14 on a seven-benchmark math average, for one extra scoring pass with a frozen model and zero new trainable parameters.

The Problem: One Endpoint Shift, Two Incompatible Histories

Reinforcement learning with a verifiable reward — generate an answer, check it, push the model toward whatever produced the correct one — is the recipe behind most of 2026’s reasoning models, and it is brutally expensive: every step needs fresh generations and fresh verification. The teachers here took 1200 GRPO steps (GRPO, arXiv 2402.03300, Shao et al., 2024) at 8 rollouts per prompt. Nobody wants to pay that twice — which leads to the observation that opened this research direction: the entire product of that run is the difference between two sets of weights. Bottle the difference and you never re-run the RL.

Why Transfer the Difference Instead of the Model?

On-policy distillation (OPD; Agarwal et al., ICLR 2024) set the stage: the student generates its own answers, and at every prefix it visits — the prompt plus the tokens produced so far — it is pulled toward the teacher’s distribution by a reverse KL penalty. Two weaknesses come with it. The target is the teacher’s absolute distribution, so the student inherits the teacher’s ceiling along with its skills. And the teacher’s pre-RL weights never appear, so the objective has no concept of what RL changed.

That first weakness is not academic here. The teacher is Qwen3-4B and the students are 8B, 14B and 30B-A3B, so copying absolute preferences downhill is active sabotage: on code, plain OPD gets Qwen3-30B-A3B to 51.91 where the method below gets 57.54.

OPD2 (arXiv 2607.15161, Heo et al., 2026) attacked the second weakness by gating updates on sign agreement between two centered signals drawn from a pair of models. Direct-OPD (arXiv 2607.05394, Feng et al., 2026) made the delta itself the supervision. Logarithms turn a product into a sum, so the sequence-level shift decomposes exactly into per-token pieces:

where:

• ${\Delta }^{(0,1)}(y\mid x)$ — the sequence-level shift: how much likelier response $y$ became
• $x$ — the prompt
• $y$ — the full response being scored
• ${\pi }_0,{\pi }_1$ — the teacher before and after RL (Initial, Final); a policy is its distribution over the next token
• $s_t=(x,y_{<t})$ — the prefix at which candidates are scored
• ${\delta }_t(a)$ — the token-level shift for candidate $a$; positive means RL raised its probability
• $a$ — a candidate token at $s_t$, not necessarily the one the student sampled
• $y_t$ — the token generated at position $t$
• $T$ — the response length in tokens

In plain terms: “how much RL came to like this answer” splits cleanly into “how much it came to like each token of it”, with no credit-assignment heuristic in between. That is what turns a policy delta into a dense reward rather than a single score at the end — and ${\delta }_t$ is the object Waypoint-OPD ultimately transfers. The gate weighs it; it never swaps it out.

The Minimal Counterexample

Now the catch, which is the best idea in the paper.

Write $u$ for the centered shift over the first half of the teacher’s run and $v$ for the second, so $u+v=d$, the centered end-to-end shift. Take $u=v=d/2$: both halves pushed the same way, agreement is $c=1$, the teacher meant it. Now take $u=2d$ and $v=-d$: the first half overshot, the second dragged it halfway back, $c=-1$. Churn.

Both sum to the same $d$.

A method that only sees Initial and Final gets one number and has no way — not a hard way, no way — to separate those histories. Yet they call for opposite decisions: full signal in the first case, nothing in the second. Section 3.2 puts it in exactly this form — a statement about what endpoint information structurally cannot hold.

Under the Hood: From Top-16 Candidates to One Gate Number

The fix is almost embarrassingly cheap: save the teacher’s weights at step 600 of its 1200-step run — call it Waypoint, ${\pi }_m$ — and the end-to-end shift splits into an early segment ${\delta }_t^{(1)}=\log {\pi }_m-\log {\pi }_0$ and a late one ${\delta }_t^{(2)}=\log {\pi }_1-\log {\pi }_m$. Every RL framework already writes such a checkpoint for crash recovery; most pipelines delete it. Waypoint is a mediocre model on its own, 51.82 on the math average against 56.13 for Final, and the student never tries to look like it — progressive distillation and Pro-KD use intermediate checkpoints as imitation targets, and this paper says three times that it is doing neither. Four steps follow, and only the second does anything surprising.

The Student’s Shortlist and the Weights $q_t$

Scoring a 150K-token vocabulary with three teacher models at every position of every rollout is not a budget anybody has, so each prefix is restricted to the candidates the student actually entertains:

where:

• ${\mathcal{K}}_t$ — the $k$ tokens with the highest student probability at $s_t$
• $k$ — the shortlist size, 16 throughout
• $q_t(a)$ — the student’s probability for $a$, renormalized over those 16
• $b$ — the summation index over ${\mathcal{K}}_t$
• ${\pi }_{\theta }(a\mid s_t)$ — the student’s raw probability for $a$
• $\theta$ — the student’s parameters, the only thing being trained

In plain terms: a budget decision wearing a second hat. The budget part is obvious — 16 scores, not 150K. The second hat matters more: $q_t$ is the measure everything downstream is computed in, so agreement gets judged where the student will actually be updated. The teacher may contradict itself wildly on tokens the student would never emit, and that is none of our business.

Centering: The Step Without Which the Cosine Measures Nothing

The part I find genuinely clever, and it is one line of algebra:

where:

• ${\mathcal{C}}_{q_t}$ — the centering operator: subtract the $q_t$-weighted mean
• $z$ — any function over the candidates, here a shift vector
• ${\mathbb{E}}_{b\sim q_t}[z(b)]$ — the mean of $z$ under $q_t$, not a plain average over the 16
• $u_t={\mathcal{C}}_{q_t}[{\delta }_t^{(1)}]$ — the centered early shift
• $v_t={\mathcal{C}}_{q_t}[{\delta }_t^{(2)}]$ — the centered late shift
• $d_t={\mathcal{C}}_{q_t}[{\delta }_t]=u_t+v_t$ — the centered end-to-end shift, the vector transferred

In plain terms: skip this and the gate reacts to noise. Suppose RL raised the logits of all 16 candidates by the same constant. Relative preferences are untouched — yet the raw shift vector has rotated toward the all-ones direction, and every angle measured against it has moved. Centering keeps only the reshuffling: the difference between “did everyone’s score go up?” and “did the ranking change?”

Two Segments, and the Gate Built From Them

Agreement is the cosine between $u_t$ and $v_t$, measured in the student-weighted geometry rather than the Euclidean one. The gate is its positive part:

where:

• $c_t$ — local temporal agreement at prefix $s_t$, in $[-1,1]$
• $g_t$ — the gate, in $[0,1]$, shared by all 16 candidates at that prefix
• $⟨u_t,v_t{⟩}_{q_t}={\sum }_{a\in {\mathcal{K}}_t}{q}_t(a)u_t(a)v_t(a)$ — inner product weighted by the student’s $q_t$
• $\Vert \cdot {\Vert }_{q_t}$ — the norm it induces, $\Vert u_t{\Vert }_{q_t}=\sqrt{⟨u_t,u_t{⟩}_{q_t}}$
• $u_t,v_t$ — as in equation (3)
• $\epsilon =10^{-8}$ — the low-norm threshold, below which a direction is meaningless
• $\max (0,\cdot )$ — clipping at zero: reversals are removed entirely, not attenuated

In plain terms: $c_t$ near $+1$ is a settled conviction, RL having spent the whole run pushing this prefix one way; near $-1$ it went one way and undid it. The gate then has three regimes. Agreement passes through scaled by its strength — a dimmer, not a switch. Reversal is silenced. And when either segment barely moved, so “direction” is not defined, the gate opens fully and the signal passes untouched: absence of evidence is not evidence of conflict. All of it is per prefix — one position in a rollout can score 1.0 while the next scores 0.0.

One piece of bookkeeping rides along. Gating mostly attenuates and the average gate drifts down during training, so the distillation term would quietly shrink against a fixed KL penalty until the student froze at its initialization. Every gated reward in a batch is therefore multiplied by one positive scalar ${\kappa }_{\mathcal{B}}$, computed from the batch’s mean gate to hold that mean at $\tau =0.8$ — shared and positive, so it preserves relative weights and can never resurrect a zeroed gate.

The Dense Candidate Coefficient

Everything collapses into a product of four factors:

where:

• $A_t(a)$ — the final coefficient (an advantage, in RL terms) for candidate $a$ at $s_t$
• $\mathrm{sg}[\cdot ]$ — stop-gradient, detaching every factor
• ${\kappa }_{\mathcal{B}}$ — the shared batch multiplier above
• $q_t(a)$ — as in equation (2), so likelier candidates get more credit
• $g_t$ — as in equation (4)
• $d_t(a)$ — as in equation (3); the only factor carrying sign

In plain terms: three of the four factors are non-negative and touch only magnitude. Sign comes from $d_t$ alone, so the gate can mute the teacher but never invert it — the method is structurally incapable of telling the student to do the opposite of what the teacher learned. “Dense” is the other choice: credit reaches all 16 candidates, not only the sampled token. The coefficient multiplies $\log {\pi }_{\theta }(a\mid s_t)$ against a KL penalty pulling the student toward its own frozen initialization at weight $\beta =2.5$, and anchoring to the student rather than the teacher is what permits the student to overshoot the teacher. It does.

What the Waypoint Does Not Touch

One more identity, and it is the paper’s real theorem:

where:

• $u_t,v_t,d_t$ — as in equation (3)
• ${\mathcal{C}}_{q_t}$ — the centering operator, linear, so it distributes over the sum
• ${\delta }_t^{(1)},{\delta }_t^{(2)}$ — the raw early and late segment shifts
• ${\pi }_0,{\pi }_1$ — as in equation (1)

In plain terms: $\log {\pi }_m$ appears once with a plus and once with a minus, so the Waypoint contributes exactly zero bits to the sum. What it contributes is information about how that sum was reached — which is what the endpoints destroy. Back to the fork: you are still porting v1..v3. The authors’ caveat is honest, though: the equality holds at fixed prefixes and weights, and since gating changes the updates, the student visits different prefixes next step. The equivalence is pointwise, not trajectory-wide.

Numbers That Convince

Across three student sizes and two domains, Waypoint-OPD takes the highest domain-average score among the compared distillation methods in all six settings. Domain average, not every cell.

Math: Seven Benchmarks, Three Student Scales

mean@32, unweighted average over AIME 24/25/26, HMMT Feb 25, HMMT Nov 25, HMMT Feb 26 and AMC 23 (Table 1). Teacher Qwen3-4B: Initial 24.84, Waypoint 51.82, Final 56.13.

Gains over Direct-OPD are +2.18, +1.59 and +1.89, with math improving in 19 of 21 per-benchmark comparisons; the exceptions are HMMT Nov 2025 for 14B (47.60 to 47.08) and HMMT Feb 2026 for 30B-A3B (33.05 to 32.77).

The number worth pausing on is in the caption. The teacher averages 56.13. Qwen3-14B lands at 59.20 and beats it on all seven math benchmarks individually, 30B-A3B at 57.85, and 8B at 56.14 ties the 4B model that taught it. The paper notes this in one dry sentence — “the transfer objective does not require the teacher to outperform the student” — which does a lot of work for its length. What gets transferred is not the teacher’s competence; it is the direction RL moved the teacher, and a bigger model can follow that direction further than its author could.

Code: Five Evaluations, and Three Cells Where OPD2 Still Wins

mean@8 over MBPP, MBPP+, LiveCodeBench v4, v5 and v6 (Tables 2 and 7). Teacher Qwen3-4B Final: 55.37.

Code gains are +2.66, +1.52 and +1.19 — 8B on code is the paper’s largest single improvement, and all 15 per-benchmark comparisons against Direct-OPD improve. OPD2 nevertheless holds three cells: 14B on LiveCodeBench v5 (43.56 against 43.11), 30B-A3B on MBPP (88.23 against 88.00) and on MBPP+ (74.44 against 74.21). Worth remembering next time you see this method called a clean sweep.

Then there is cost. Adding a whole extra scoring pass with a frozen teacher made training faster: 8h 02m against 8h 21m on one node of 8× H20, 64.31 against 66.81 GPU-hours. Under gating the student writes shorter responses and truncation at the 16,384-token limit falls from 14.83% to 11.97%, so the gate doubles as an accidental length regularizer. A method that adds a forward pass and still wins on wall-clock was not what I expected.

Ablations: Which Component Actually Matters?

Qwen3-8B, math, mean@32. Avg covers all seven benchmarks (Table 3).

Batch normalization is a second-order nicety: switching it off costs 0.92 points, and the unnormalized variant still beats Direct-OPD on all seven benchmarks. Remove the gate itself and 2.64 points go with it. Expected, unremarkable.

Shuffled Gates Are Worse Than No Gates

Now the row I would put on the poster.

Shuffled gate keeps the gate values and permutes which prefix each one lands on. Consider what that preserves: the same number of gates, the same distribution of values, the same mean — and therefore, since ${\kappa }_{\mathcal{B}}$ comes from the batch mean, the same ${\kappa }_{\mathcal{B}}$. The $q_t$ weights and the shift vectors $d_t$ stay put. Exactly one thing differs: whether a gate reaches the prefix that produced it.

The average falls to 51.20 — 4.94 points below the full method and, the part that should make you sit up, 2.30 points below having no gate at all, 2.76 below plain Direct-OPD. A misaddressed gate is actively worse than an open door.

This is a cleaner argument for the paper’s thesis than the headline +2.18 ever was. That headline could come from centering, or the 0.8 rescaling, or an interaction nobody isolated. The shuffle result cannot: scale and weight distribution are both fixed by construction, so the only explanation left is that the assignment of gate to prefix carries real information. Ablations rarely get to be this airtight, and this one nearly disappears as a table’s third row.

Position, meanwhile, hardly matters: an intermediate checkpoint at teacher RL step 500, 600 or 700 gives 56.18 / 56.14 / 56.13, a spread of 0.05 points, all three beating Direct-OPD on every benchmark. The authors note without flinching that step 500 edges out the 600 they shipped as the default.

The Gate Tightens As the Student Improves

My favourite number in the paper, and the authors almost walk past it.

Across 100 transfer steps the mean gate falls from 0.810 in the first 20 steps to 0.683 in the last 20, while the fraction of prefixes with a zero gate — the teacher’s advice fully muted — climbs from 7.26% to 16.59%. By the end, one prefix in six gets nothing from the teacher at all.

All three teacher checkpoints are frozen; nothing about the teacher changes over those 100 steps. What changes is where the student walks. As it improves, its rollouts carry it into regions where the teacher’s early and late halves disagree with each other — and the method, instructed to do no such thing, increasingly says “not here”.

A filter designed to remove noise turns out to behave like supervision with a built-in expiry date, withdrawing exactly where a 4B teacher stops being a credible source for an 8B student. The paper reports the diagnostic and moves on; I think it is a larger result than the section it is filed under. The obvious follow-up — what sits in those zero-gate prefixes? — is where this method’s interpretability lives, and it is not here.

A Skeptic’s View: What the Paper Doesn’t Say

• The theory does not match the training recipe. Section 2.3 licenses treating ${\delta }_t$ as a reward via the DPO result (Rafailov et al., NeurIPS 2023), which holds at an idealized optimum with KL regularization toward ${\pi }_0$. Table 4 reports both teachers trained with GRPO at KL coefficient 0.0 — so the premise is not merely unmet, it does not have the right shape. The authors disclaim the optimum anyway, which leaves the signal as a heuristic that works. Nothing wrong with that; it is just not the consequence of a theorem.
• AIME 2024 is both the validation set and a reported benchmark. Student checkpoints are picked by best accuracy on AIME 2024, which then enters the seven-benchmark average — and the code checkpoints are selected on AIME 2024 too, a math benchmark choosing the model you will evaluate on MBPP. Every method gets the same treatment, so the comparison is fair; the absolute levels are inflated.
• No seeds, no error bars, one run per cell. AIME and HMMT hold roughly 30 problems each, making one problem worth about 3.3 points, and mean@32 averages sampling noise rather than problem-selection noise. Both reported regressions are smaller than a single problem, as is the 0.05-point spread across Waypoint positions. At these set sizes even the headline gains deserve intervals.
• “No gate” lands below Direct-OPD: 53.50 against 53.96. That variant keeps centering and the 0.8 rescaling, so the package minus gating is a net loss against the baseline, and the gate is credited on top of a bundle that is underwater on its own. The paper handles this in one sentence (“this comparison does not isolate the effect of centering”) and never returns; no centering-without-gating ablation exists anywhere, and the authors concede that the remaining gain cannot be assigned entirely to gating.
• A shared tokenizer is a hard requirement, and every model here is Qwen3. Section 2.1 needs teacher and student to score the same token IDs: one family, one vendor, one RL run per domain, and no evidence about Qwen-to-Llama.
• What is missing entirely: no ablation over $k$, $\tau$ or the clipping bounds; no variant with more than one intermediate checkpoint, though a real run saves dozens; no training longer than 100 steps.

Who Is This For?

For practitioners

If you already run RL on a small model and distil into larger ones, this is close to free. Stop deleting mid-run checkpoints — that is the entire prerequisite, and it is a config change, not an engineering project. The hard constraint is the shared tokenizer, so plan your lineup inside one model family, and since checkpoint position barely matters, take whatever sits near the middle of the run. Then log the gate diagnostics: mean gate and zero-gate fraction over time tell you when your teacher has stopped being useful to your student, a question most pipelines answer by guessing.

For researchers

The contribution that outlives these numbers is the kind of evidence being used. The 2026 selective-distillation literature is crowded, and every entry picks a different criterion for what to keep: support coverage (Prune-OPD, arXiv 2605.07804), teacher uncertainty (EOPD, arXiv 2603.07079), token position (PW-OPSD, arXiv 2605.21606), sign agreement between model pairs (OPD2), verifier agreement (RG-OPD, arXiv 2607.04037). Waypoint-OPD adds one nobody else uses — the teacher’s own history. Every other criterion compares two models; this one compares a model to its earlier self.

The open threads are specific. Generalize from one Waypoint to several and each prefix gets a consistency profile instead of a single cosine — untried, and the checkpoints are already on disk. Run the missing centering-without-gating ablation. Characterize the zero-gate prefixes, where any interpretability story has to start. And settle the theory: either derive the implicit-reward claim at KL coefficient 0.0, or drop the DPO scaffolding.

Technical Details

For the advanced reader — three things worth knowing before replicating.

The hyperparameters, compactly. Shortlist $k=16$, $\tau =0.8$, bounds $[0.75,1.5]$, $\beta =2.5$, learning rate 1e-6, 100 steps, rollout $n=1$, batch 256, mini-batch 64, temperature 1.0, max response 16,384 tokens. Teacher: GRPO, 1200 steps, rollout $n=8$, KL coefficient 0.0, on ~57K DeepMath-103K prompts at difficulty $\ge 6$ and ~25K from Eurus-RL-Code. Infrastructure is verl with FSDP plus vLLM, one node of 8× NVIDIA H20; verifiers are for evaluation only.

The surrogate is dual-clipped PPO. The coefficients feed a PPO objective with a second floor for negative advantages, borrowed from Ye et al. (2020) — originally a paper about playing MOBA games with deep RL. At the rollout policy the importance ratio is 1, clipping is inactive, and the gradient coincides exactly with the simple surrogate, so PPO stabilizes multiple mini-steps over one rollout batch rather than changing the objective. Mildly amusing detail: the dual-clip constant is written $c$, the agreement cosine is $c_t$, and the paper had to warn in parentheses that they are distinct.

Two different epsilons, on purpose. $\epsilon =10^{-8}$ is the low-norm threshold deciding when a segment’s direction is undefined; ${\epsilon }_{\mathrm{num}}\approx 1.18\times 10^{-38}$, the smallest positive normal float32, is the division floor inside the batch-scale rule. Thirty orders of magnitude between “I cannot tell which way this points” and “do not divide by zero”, flagged as deliberately separate. One last thing: the shortlist, $q_t$ and $A_t$ are stop-gradiented for the duration of an update, and the zero-sum gradient result is local — it excludes the KL term and says nothing about non-candidate probabilities.

Limitations

• The intermediate checkpoint must exist. If all you have is a published “before” and “after” — the normal case for weights you downloaded rather than trained — the method has nothing to work with.
• Agreement is a weighting criterion, not a correctness guarantee. RL that pushed consistently in a bad direction arrives with a gate of 1.0 and gets amplified.
• Shared tokenizer, single model family. Required by construction, never tested across families.
• The endpoint equivalence is pointwise, holding at fixed prefixes and weights; across a full run the trajectories diverge.
• 100 steps is the whole transfer, and results come from a validation-selected checkpoint inside that window.

Summary

1. The same endpoint shift can have two incompatible histories. With $u+v=d$ fixed, $u=v=d/2$ is a conviction held throughout and $u=2d$, $v=-d$ is an overshoot half-undone — indistinguishable from the endpoints, and calling for opposite decisions.
2. One saved checkpoint resolves the ambiguity. Score the student’s top-16 candidates with all three checkpoints, center under the student’s own weights, and the cosine between the early and late segments becomes a per-prefix gate — a dimmer, not a switch.
3. The gate changes how loudly, never what. Because $\log {\pi }_m$ cancels, $u_t+v_t=d_t$ identically (equation 6), so the transferred object stays the centered Initial-to-Final shift: evidence, not an imitation target. And because the gate is non-negative, the signal can be muted but never inverted.
4. The shuffle ablation is the real result. Full 56.14, no gate 53.50, shuffled gate 51.20, Direct-OPD 53.96. Shuffling preserves the gate distribution, its mean and ${\kappa }_{\mathcal{B}}$, so only the gate-to-prefix assignment can explain the 2.30-point drop below no gate at all.
5. Cheap in exactly the way that matters. One frozen scoring pass, no new trainable parameters, no extra rollouts, no verifier during transfer — and +1.19 to +2.66 domain-average points across six student-domain settings, with 14B and 30B-A3B finishing above the 4B teacher that supervised them. The asterisks stand: domain averages, not per-benchmark sweeps, and no error bars on 30-problem sets.

The most informative artifact in an RL pipeline turned out to be the one every pipeline already writes to disk and then deletes.

📎 Linki

• Na podstawie publikacji 📄 arXiv:2610.21601