Imagine inheriting a system built by three teams that never talked to each other. Three microservices, all describing the same warehouse: one answers questions about it in text, one renders preview images of the shelves, one drives the forklift. Each has its own database, its own schema, and none of them agree on what’s actually sitting on shelf B4. Any architect who has seen this knows the fix — one canonical store, thin adapters at the edges.
Modern AI is that system. LLMs predict the next token, video generators predict the next frame, vision-language-action models predict the next motor command. Three families, three training objectives, three incompatible internal pictures of the same physical world.
Orca (arXiv 2606.30534, BAAI, June 2026 — the paper’s actual subtitle is “The World is in Your Mind”, and it has 57 authors) is the refactoring proposal. One latent world state, learned from 12.5K hours of video and language-annotated events, with zero action labels in pre-training. After pre-training, the backbone is frozen forever, and three lightweight decoders read text, images, and robot actions out of the same representation. The headline result: an action readout trained from scratch on 200 trajectories per task beats π0.5 — a robotics foundation model pre-trained on large-scale action data — on out-of-distribution manipulation, 32.4 vs 29.4 on the rule-based score. The robot learned to move, mostly, by watching.
The Problem: Next-Token, Next-Frame, Next-Action — Pick One
Each of the three paradigms encodes an implicit theory of what intelligence is, and each pays a distinct tax.
Next-frame models (Sora, NVIDIA’s Cosmos) equate understanding the world with rendering it. That means spending enormous model capacity on high-entropy pixel detail — the exact texture of a carpet — that has nothing to do with knowing the cup on the table’s edge is about to fall. You wanted a physicist; you’re paying for a painter.
JEPA-style models (Meta’s V-JEPA 2, in Yann LeCun’s lineage) fix that by predicting in representation space instead of pixel space. They learn excellent video dynamics. But they’re mute: there is no language interface, so you can’t tell the model what should happen next, and you can’t ask it why something happened. V-JEPA’s planner needs a goal image — try producing one of those for “put the book on the second shelf.”
VLA models (π0.5, OpenVLA) go straight for the output that matters — robot actions — but their fuel is teleoperation data: expensive, scarce, and narrow. Collecting it means humans puppeteering robots for thousands of hours, and the resulting policies still wobble when the tablecloth changes color.
There’s a catch hiding in plain sight, though: all three model families are trying to represent the same world. The token, the frame, and the action are different serializations of one underlying state. Orca’s bet is that if you learn the state well, the serializers can be nearly trivial.
What Is Orca? Next-State-Prediction in a Nutshell
Orca’s core claim fits in one equation. The world’s multimodal signals $X = \{X^m\}_{m \in \mathcal{M}}$ (here: vision and language) get compressed by an encoder into a latent state $S = f_\theta(X)$, and everything — perception, prediction, reasoning, control — becomes sampling from a state-transition distribution:
$$ S_{t+\Delta} \sim p_{\Theta}\left(S_{t+\Delta} \mid S_t,\, z_t,\, c_t\right), \qquad \Delta \in \mathbb{Z}_{\neq 0} \tag{1} $$where:
- $S_t$ — the latent world state at time $t$
- $S_{t+\Delta}$ — the state $\Delta$ steps away; $\Delta > 0$ predicts the future, $\Delta < 0$ backtracks into the past
- $p_\Theta$ — the parametric transition distribution (the whole model, parameters $\Theta$)
- $z_t$ — implicit dynamics: physics, object properties, environmental forces — the unobserved machinery that drives change
- $c_t$ — explicit conditions: observable control signals such as a human instruction, an event description, a task intention
- $\Delta \in \mathbb{Z}_{\neq 0}$ — any non-zero integer offset; the model must be able to walk the time axis in both directions
In plain terms: the world is a Markov chain in latent space, and every AI task is a query against its transition distribution. Next-token prediction is equation (1) with $S$ = text tokens and $\Delta = 1$. Next-frame is $S$ = video frames. Next-action is $S$ = motor commands. Orca treats all three as impoverished projections of the general case.
The separation of $z_t$ from $c_t$ is what makes the framework practical, because it splits training into two regimes.
Unconscious vs Conscious Learning — System 1 and System 2 for Representations
The naming here is doing deliberate work — it reads like Kahneman’s System 1 / System 2 transplanted into representation learning, and the analogy holds up surprisingly well.
Unconscious learning is the world changing on its own while you watch through a window. Set $c_t = \varnothing$ and the transition depends only on the hidden dynamics:
$$ S_{t+\Delta} \sim p^{\mathrm{u}}_{\Theta}\left(S_{t+\Delta} \mid S_t,\, z_t\right) \tag{2} $$where:
- $p^{\mathrm{u}}_{\Theta}$ — the transition distribution in unconscious mode (u = unconscious)
- $c_t = \varnothing$ — no explicit condition; the target is simply the next observed video frame
In plain terms: every pair of adjacent frames in raw video is a free training example about motion, occlusion, and object interaction. Dense, unlabeled, and available in essentially unlimited quantity — the paper’s inventory holds 125K hours of it.
Conscious learning is the sparse, meaningful kind: transitions tied to a cause or intention described in language. The condition becomes an event description $e_{t+\Delta}$:
$$ S_{t+\Delta} \sim p^{\mathrm{c}}_{\Theta}\left(S_{t+\Delta} \mid S_t,\, z_t,\, e_{t+\Delta}\right) \tag{3} $$where:
- $p^{\mathrm{c}}_{\Theta}$ — the transition distribution in conscious mode
- $e_{t+\Delta}$ — a language description of a target event, in the future ($\Delta > 0$) or the past ($\Delta < 0$)
- $S_{t+\Delta}$ — a state inside the described event; in practice, the latent of a randomly sampled frame from that event’s video segment
In plain terms: language acts like a key in a database lookup — it selects which of the many possible futures (or pasts) the model should predict. “Ice cream melts when you heat it” is a conscious transition; leaves rustling outside the window is an unconscious one. Visual question answering runs in the same conscious regime and keeps the language interface alive.
This is exactly what pure JEPA lacks — steerability — and what pure next-token models lack: a dense, label-free firehose of physics.
Under the Hood: Queries, a Frozen ViT, and Three Losses
The implementation is more pragmatic than the philosophy, and the paper is upfront about it. Orca starts from a pre-trained multimodal VLM (Qwen3.5, in 0.8B and 4B sizes) with its vision encoder frozen. The input sequence looks like: visual tokens, then 256 learnable query vectors (“Query 1”), then the instruction, then another 256 queries (“Query 2”). The queries are trained from scratch and act as read ports: the final hidden state of Query 1 holds the model’s belief about the unconditioned next state; Query 2 holds the state selected by the instruction.
Three objectives push on the same latent space simultaneously:
$$ \mathcal{L} = \lambda_{\mathrm{obs}}\,\mathcal{L}_{\mathrm{obs}} + \lambda_{\mathrm{evt}}\,\mathcal{L}_{\mathrm{evt}} + \lambda_{\mathrm{vqa}}\,\mathcal{L}_{\mathrm{vqa}} \tag{4} $$where:
- $\mathcal{L}_{\mathrm{obs}}$ — observation-only state transition (unconscious learning): match the latent of the next frame
- $\mathcal{L}_{\mathrm{evt}}$ — event-conditioned state transition (conscious learning): match the latent of the frame that language points to
- $\mathcal{L}_{\mathrm{vqa}}$ — standard next-token cross-entropy on VQA answers
- $\lambda_{\mathrm{obs}}, \lambda_{\mathrm{evt}}, \lambda_{\mathrm{vqa}}$ — set to 0.1 / 0.5 / 0.4 for both model sizes (Appendix C.1.3), with a roughly 5:1 data mix of state transitions to VQA
In plain terms: natural dynamics, language-steered transitions, and commonsense grounding all shape one representation. There’s a quiet irony in the weights — unconscious learning, the conceptual foundation of the whole method, gets $\lambda = 0.1$, five times less than events. The paper offers no sweep over these weights, only on/off ablations.
Latent Matching: 0.1·MSE + 0.9·Cosine
Here’s the part I find genuinely elegant. Orca never reconstructs pixels. The prediction target for both transition losses is the latent of the target frame, produced by the frozen ViT. The predicted latent $\hat{v}^l$ (a query hidden state passed through a 2-layer MLP; for the 4B model, 2560 → 20480 → 2560) is compared to the ground-truth latent $v^l$ with:
$$ \ell_{\mathrm{lat}}\left(\hat{v}^{l}, v^{l}\right) = 0.1\,\left\| \hat{v}^{l} - v^{l} \right\|_2^2 \;+\; 0.9\left(1 - \frac{\left\langle \hat{v}^{l}, v^{l} \right\rangle}{\left\| \hat{v}^{l} \right\|_2 \left\| v^{l} \right\|_2}\right) \tag{5} $$where:
- $\hat{v}^{l}$ — the latent predicted by Orca at the visual token positions
- $v^{l}$ — the ground-truth latent: the target frame pushed through the frozen vision encoder
- $\|\cdot\|_2^2$ — squared Euclidean norm, the MSE term, which keeps magnitudes in check
- $\langle \cdot, \cdot \rangle$ — inner product; the second term is one minus cosine similarity, which enforces agreement in direction
- $0.1 / 0.9$ — cosine dominates; MSE is only a stabilizer
In plain terms: in embedding space, semantics lives mostly in the direction of a vector, not its length — hence 90% of the weight on cosine. Pure MSE would punish irrelevant amplitude differences; pure cosine would let latents explode or collapse in scale. The 0.1/0.9 mix guards against both. It’s also like comparing hashes instead of diffing whole binary files: cheap, and robust to noise that doesn’t matter.
And there’s a quieter defense built in. Representation collapse — everything mapping to a single point — is the classic failure mode of joint-embedding methods. Orca sidesteps it because the target space is a frozen ViT: the trivial solution doesn’t minimize the loss when the anchor can’t move.
The two transition losses instantiate $\ell_{\mathrm{lat}}$ at different timescales:
$$ \mathcal{L}_{\mathrm{obs}} = \mathbb{E}\left[\, \ell_{\mathrm{lat}}\left(\hat{v}^{l}_{t+1},\; v^{l}_{t+1}\right) \right] \tag{6} $$where:
- $\hat{v}^{l}_{t+1}$ — the predicted latent of the immediately adjacent next frame, read from Query 1
- $v^{l}_{t+1}$ — the true next frame’s latent from the frozen encoder
- $\mathbb{E}[\cdot]$ — expectation over frames sampled from the video corpus
In plain terms: dense, free supervision — every adjacent frame pair is one example.
$$ \mathcal{L}_{\mathrm{evt}} = \frac{1}{2}\,\mathbb{E}\left[\, \ell_{\mathrm{lat}}\left(\hat{v}^{l}_{\mathrm{prev}},\; v^{l}_{\mathrm{prev}}\right) + \ell_{\mathrm{lat}}\left(\hat{v}^{l}_{\mathrm{next}},\; v^{l}_{\mathrm{next}}\right) \right] \tag{7} $$where:
- $\hat{v}^{l}_{\mathrm{next}}$ — the predicted latent of a random frame from the next event, when the instruction describes a future event ($\Delta > 0$), read from Query 2
- $\hat{v}^{l}_{\mathrm{prev}}$ — the same for the previous event ($\Delta < 0$) — this is where the backtracking from equation (1) becomes concrete
- $v^{l}_{\mathrm{prev}}, v^{l}_{\mathrm{next}}$ — ground-truth latents from the frozen encoder
- $\tfrac{1}{2}$ — both time directions weighted equally
In plain terms: the model learns both “what comes after” and “what must have come before” the event language describes. Inferring backwards forces something closer to causal structure than pure forward extrapolation. (The paper doesn’t ablate the backward direction separately, which is a pity.)
Three Readouts: LM Head, SD3.5 + LoRA, and a Flow-Matching Action Expert
After pre-training, the backbone is never touched again. This is the experimental protocol I like most about the paper: if a frozen core plus a thin decoder works, the credit belongs to the representation, not the decoder. It’s an integration test through a public API — you’re not allowed to patch the server, and if all three clients work anyway, the data model is sound.
- Text costs zero new parameters — the VLM’s own LM head, reused for VQA, event interpretation, and causal explanations.
- Image is an MLP adaptor (556.9M parameters) plus LoRA (rank 32) on the attention projections of a frozen Stable Diffusion 3.5; trained for 200K steps at batch 512, 768×768 images.
- Action is a DiT (Diffusion Transformer) “Action Expert” trained from scratch: 8 blocks, hidden size 1024, 12 heads, alternating self- and cross-attention, predicting 30-step chunks of 16-dimensional actions with 4 denoising steps at inference. Trained per task on 200 trajectories, 20K steps, batch 128.
The action expert learns by conditional flow matching. One caveat first: the paper never prints this equation — it cites Lipman et al. (2022) and describes the procedure in words (the ground-truth action chunk is corrupted with Gaussian noise; the expert predicts the corresponding velocity). The following is the standard form reconstructed from that description:
$$ \mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{\tau,\, a_0,\, \epsilon}\left[\, \left\| u_{\phi}\left(a_{\tau}, \tau \mid q_1,\, s_{\mathrm{proprio}}\right) - \left(\epsilon - a_0\right) \right\|^2 \right] \tag{8} $$where:
- $a_0$ — the ground-truth action chunk (horizon 30, 16-dimensional actions)
- $a_{\tau}$ — the noised interpolation of the actions at diffusion time $\tau$
- $u_{\phi}$ — the velocity field predicted by the DiT Action Expert
- $q_1$ — the conditioning signal: the Query 1 latent from frozen Orca, passed through an MLP adaptor — literally “the predicted evolution of the world state”
- $s_{\mathrm{proprio}}$ — the robot’s proprioception (joint and end-effector states, 16-dim)
- $\epsilon$ — Gaussian noise; inference runs 4 denoising steps to produce an action chunk
In plain terms: the robot never sees pixels or text. It receives a compressed forecast of how the world is about to change, plus its own body state, and denoises a trajectory. If the latent truly encodes future world state, a tiny from-scratch expert should suffice for control. That’s the paper’s thesis turned into a falsifiable setup.
The Data: 125K Hours of World (and Why Only a Tenth Was Used)
The “world-learning inventory” is a data engineering feat of its own: 125K hours of video (ego-centric interaction, exo-centric manipulation, action-free robot execution, natural dynamics), 160M event annotations from multi-level segmentation — coarse process steps plus fine-grained transitions inside them, each with a caption — and 11.5M VQA pairs.
Orca used one tenth of the video. Not for lack of compute — the authors state plainly that at 4B parameters, model capacity, not data, is the bottleneck for fitting more world. Pre-training ran on 256×H100 (32 nodes) for 10,844 steps with AdamW (LR 3.5e-5 backbone, 1.2e-4 for the visual heads), sequences up to 1024 tokens.
Numbers That Convince
Text: +5.1 Average Over Its Own Backbone (and Where V-JEPA 2.1 Still Wins)
The cleanest comparison is against the very backbone Orca started from, since the delta is attributable to world-learning pre-training alone:
| Model | MVBench | TemporalBench (MBA) | SWITCH | Avg (4 benchmarks) |
|---|---|---|---|---|
| Orca-4B | 65.3 | 34.2 | 55.6 | 51.8 |
| Qwen3.5-4B (backbone) | 67.1 | 25.2 | 42.8 | 46.7 |
| V-JEPA 2.1 + LLaMA3-8B | 75.4 | 28.5 | — | — |
| MiniCPM-V-4.6 | — | — | — | 37.9 |
| Emu3.5 | — | — | — | 29.8 |
Avg covers MVBench + TemporalBench + 3DSRBench + SWITCH.
Same story at small scale: Orca-0.8B averages 40.8 vs 33.1 for Qwen3.5-0.8B — a +7.7 remix of the same backbone.
Be honest about the losses, though. V-JEPA 2.1 with an 8B LLM wins MVBench outright (75.4 vs 65.3) — and even the plain Qwen3.5-4B edges Orca there (67.1 vs 65.3). Orca’s gains concentrate exactly where the method aims. The capability breakdown (Table 2, Orca-4B vs Qwen3.5-4B):
| Capability dimension | Orca-4B | Δ vs Qwen3.5-4B |
|---|---|---|
| State Transition | 64.13 | +12.27 |
| Dynamic Motion | 65.55 | +8.52 |
| Commonsense | 62.95 | +5.19 |
| Spatial Relations | 55.25 | +0.57 |
The gains land on transitions and motion; spatial understanding barely moves. The training signal shapes exactly what it touches — which reads more like validation than accident.
Image: PRICE-V0.1 and the LLM Judges
Image prediction (“show me the frame after this instruction happens”) is evaluated on PRICE-V0.1, the authors’ own benchmark, scored by four LLM judges:
| Model | Avg score (%) |
|---|---|
| Orca-4B (+2B SD3.5) | 59.8 ± 10.9 |
| FLUX.2 [klein] (4+4B) | 56.1 ± 18.1 |
| FLUX.1-Kontext (12B) | 40.9 ± 13.5 |
| OmniGen2 (3+4B) | 39.6 ± 10.2 |
| Orca-0.8B (+2B SD3.5) | 34.5 ± 15.3 |
Orca-4B beats a 12B editing model with a frozen 2B decoder and a thin adaptor. But keep the ± columns in view: the judge disagreement is wild. For the same Orca-4B images, Gemini 3.1 Pro awards 44.0 and GPT 5.4 awards 67.9 — a 24-point spread on identical outputs. The authors added the open-source Gemma 4-31B judge specifically so anyone can reproduce the numbers. And Orca-0.8B finishes dead last, which the scaling story requires you not to skip over.
Robot: 200 Trajectories vs π0.5’s Large-Scale Pre-training
This is the table the paper exists for. Five real-robot tasks (put a book on a shelf, stack bowls, pull a tissue from a box, stamp paper, spoon sugar), each learned from 200 trajectories, then tested out-of-distribution: unseen tablecloths (Environment OOD) and swapped objects — tissues replaced by bread in a toaster, bowls by boxes (Object OOD).
| Model (rule-based score) | Overall | Environment OOD | Object OOD |
|---|---|---|---|
| Orca-4B | 32.4 | 36.6 | 28.2 |
| π0.5 (large-scale robot pre-training) | 29.4 | 27.6 | 31.2 |
| V-JEPA 2.1 w/ Action Expert | 17.0 | — | — |
| Qwen3.5 w/ Action Expert | 10.5 | — | — |
Three observations, in descending order of comfort:
- Orca had zero action labels in pre-training and still beats π0.5 overall — a model whose entire premise is large-scale robot-action pre-training. The gap on Environment OOD is +9.0.
- π0.5 wins Object OOD (31.2 vs 28.2, and it also leads on the M25 and M50 milestone metrics there: 25.1 vs 21.8 and 12.9 vs 10.8). Watching video teaches you how scenes evolve; it apparently teaches you less about generalizing a grasp to a novel object than actually handling objects does.
- The same Action Expert bolted onto V-JEPA 2.1 gets 17.0 and onto plain Qwen3.5 gets 10.5. That last number matters: it says Orca’s dedicated query mechanism and transition pre-training — not the Qwen backbone or the expert architecture — carry the result (10.5 → 32.4 with the same decoder budget).
One number the paper is honest about but headlines gloss over: binary success rates hover near zero for everyone (0–3%). No model reliably finishes these tasks. The victory is measured in partial progress — milestones reached — not completed manipulations.
Ablations: Which Loss Actually Matters?
The on/off ablation of the three losses is, to my mind, the most interesting table in the paper — partly because it undercuts the paper’s own narrative:
| Configuration | Text | Image | Action |
|---|---|---|---|
| obs + evt + vqa (full) | 51.8 | 59.8 | 32.4 |
| vqa only | 48.4 | — | 10.2 |
| obs + vqa | — | — | 32.6 |
| evt + vqa | — | — | 23.0 |
("—" for image: without $\mathcal{L}_{\mathrm{evt}}$ the image readout does not function at all.)
What the table says:
- $\mathcal{L}_{\mathrm{evt}}$ is a hard requirement for image prediction — without event conditioning there is no way to point at the target state with language, and the readout simply fails.
- $\mathcal{L}_{\mathrm{vqa}}$ keeps the text interface alive; drop it and the language readout dies.
- $\mathcal{L}_{\mathrm{obs}}$ is what feeds the robot. VQA alone gives action 10.2; adding dense frame-to-frame dynamics jumps it to 32.6; events-plus-VQA without it manages only 23.0. Short-horizon natural motion is what a manipulation policy actually consumes.
And the surprise: obs + vqa scores 32.6 on actions — marginally better than the full model’s 32.4. The flagship “conscious learning” of events adds nothing for control; the full triple wins only on the balanced average (48.0 across readouts). Meanwhile VQA alone sits just 3.4 points behind the full model on text. Strictly speaking, event conditioning is indispensable for exactly one of the three readouts: images. I’d genuinely like to see the authors engage with this more than they do.
The scaling probe (Figure 6) is the flip side, and it’s the result I trust most: freeze the backbone at successive pre-training checkpoints, retrain the readouts, and all three curves rise monotonically with data — with 4B above 0.8B everywhere. Latent quality, measured by pre-training loss, transfers to every readout at once. That’s the “one canonical database” claim, tested the right way.
A Skeptic’s View: What the Paper Doesn’t Say
Time to take off the fan hat.
- The “world latent” is inherited, not learned. In this version, the state space is the embedding space of Qwen3.5’s frozen ViT — a semantic encoder someone else trained. Orca learns transitions within it. The authors admit this (Limitation 2), but the title and abstract gesture at something more native than what was built.
- Comparisons are curated. “Advantages over specialized baselines at a comparable scale” quietly excludes MVBench, where V-JEPA 2.1 wins by 10 points and even the raw backbone wins by 1.8; and Object OOD, where π0.5 wins.
- PRICE-V0.1 is home turf. Small, limited diversity (Limitation 4), scored by LLM judges instructed to use lenient pass criteria, with a 24-point spread between judges. And the 0.8B variant places last on it.
- Near-zero binary success. The robot never really completes tasks; the win lives in partial-progress metrics. Fair metrics, honestly reported — but “beats π0.5” deserves that asterisk.
- No compute-matched control. The comparison I most wanted: continue pre-training plain Qwen3.5 on the same 12.5K hours with VQA only, same budget, no transition losses. The vqa-only ablation approximates it and lands only 3.4 points behind on text. How much of the text gain is “world modeling” and how much is “more video data” remains open.
None of this sinks the paper. It does mean the grand framing (“a unified world model”) is running ahead of the evidence, which so far reads: latent transition pre-training is a strong, label-free prior for control, useful for temporal reasoning, and necessary for instruction-conditioned image prediction. That’s a real result. It’s just a smaller sentence.
Who Is This For?
For practitioners
If you work in robotics, the actionable claim is the data economics. Teleoperation data is expensive to collect and doesn’t scale; video is free. Orca suggests a pipeline where physics comes from passive video and only the last motor mile comes from demonstrations — 200 per task here. Watch for the checkpoint release: a frozen backbone plus a from-scratch DiT expert (8 blocks, hidden 1024) is a setup a small lab can afford to replicate, and the whole readout protocol is cheap by design — the most expensive decoder is the 556.9M image adaptor.
If you build multimodal products rather than robots, the transferable idea is the probing protocol itself: freeze your foundation model and measure representation quality by how well thin readouts perform. It’s a cleaner regression test for “did our pre-training actually improve” than end-to-end fine-tuning, where the decoder can mask a stagnant core.
For researchers
Orca is the first serious attempt to merge the JEPA program with language conditioning at scale, and its most useful artifacts are methodological: the frozen-backbone probing protocol, the two-regime (unconscious/conscious) data taxonomy, and the finding that capacity, not data, currently binds — they used 10% of their own corpus. Open threads, some handed to you by the ablations: why does event conditioning help images but not actions? Does backward prediction ($\Delta < 0$) actually contribute, given it was never ablated separately? Can the three losses collapse into one objective that’s actually faithful to equation (1) (the authors themselves want this — Limitation 7)? And the big one: replace the inherited frozen ViT with a natively learned state space and see whether the story survives.
Technical Details
For the advanced reader — three things worth knowing before replicating.
Infrastructure got startup-level attention. Migrating from DeepSpeed to FlagScale + FSDP2 with chunked cross-entropy (never materializing full logits), activation recompute, and forward/backward all-gather pre-fetching took throughput from 0.66 to 2.91 samples/sec/GPU on H100 — 4.4× over the community’s standard embodied stack (StarVLA) and 3.0× over bare FSDP2. The component breakdown: chunked CE 0.97 → 1.35, recompute → 2.86, pre-fetching → 2.91.
Readout wiring, precisely. Image: 64 latent tokens × 2560 → LayerNorm + Linear 2560→4096 → 4 residual blocks 4096→16384→4096 → mean-pool to a 2048-dim pooled condition, feeding one MMDiT path of frozen SD3.5; only the adaptor (556.9M) and LoRA rank 32 on attention projections train. Action: the Query 1 hidden state passes through an MLP adaptor into cross-attention of the DiT expert; conditions are the noised action chunk + time embedding + 16-dim proprioception.
The queries are load-bearing. The baseline “Qwen3.5 w/ Action Expert” reads the same decoder from the VLM’s last hidden state instead of dedicated queries and scores 10.5 vs Orca’s 32.4. Dedicated, from-scratch read ports that don’t perturb the original token stream appear to be worth roughly 3× on this metric — the cheapest 256×2 vectors you’ll allocate this year.
Limitations
The paper lists its own limitations with unusual candor — eight of them:
- Two modalities only. No audio (boiling water is audible before it’s visible), no touch, force, or proprioception in pre-training.
- Inherited state space. The latent is anchored to a frozen ViT from a pre-trained VLM, not learned natively from world signals.
- Small scale, visible tension. At 0.8B and even 4B, the three readouts compete for capacity during pre-training; only 1/10 of the video was usable — the model, not the data, is the bottleneck.
- PRICE-V0.1 is small and of limited diversity.
- Short-horizon events. Annotations span minutes; nothing models state evolution over hours or days.
- Three readouts tested. Audio, quantum circuits, proteins — declared as future targets, unproven.
- Three separate losses, admittedly not yet faithful to the single-objective spirit of Next-State-Prediction.
- Simple robot tasks, low absolute success. Short-horizon manipulations, and binary success rates near zero for all methods.
Summary
- One objective, three paradigms. Next-State-Prediction (equation 1) subsumes next-token, next-frame, and next-action as projections of a latent world state evolving forward and backward in time, driven by hidden dynamics $z_t$ and steered by explicit conditions $c_t$.
- Latent supervision beats pixel painting. Predicting frozen-ViT embeddings with a 0.1·MSE + 0.9·cosine loss buys physics without paying for texture — and the frozen anchor doubles as collapse protection.
- The frozen-backbone protocol is the real contribution. Thin readouts on an untouched core turn “is the representation good?” into a measurable question, and all three readouts improve monotonically with pre-training data (4B > 0.8B throughout).
- Emergent motor knowledge is the headline. Zero action labels in pre-training, 200 trajectories per task, and the action readout beats π0.5 overall (32.4 vs 29.4) — though π0.5 keeps Object OOD, V-JEPA 2.1 keeps MVBench, and nobody’s binary success rate clears 3%.
- The ablation is the fine print. obs+vqa slightly beats the full model on actions (32.6 vs 32.4), and VQA alone is within 3.4 points on text — event conditioning is truly indispensable only for image prediction. The unified story has visible seams.
A model that has only ever watched the world nearly matches one that spent its whole life acting in it — which suggests most of what a robot needs to know was never in the action labels at all.
📎 Linki
- Na podstawie publikacji 📄 arXiv:2606.30534