WEAVER: a world model that is faithful, consistent, and fast at once

June 2026 · world models · robotic manipulation · WEAVER (arXiv:2606.13672)

WEAVER is a paper about the world model itself. The policy is still π0.5; the question is whether the learned simulator under it can be accurate enough, stable enough, and fast enough to evaluate policies, generate training data, and plan during execution.

The three constraints

The paper reduces robot world modeling to three checks. Fidelity asks whether imagined trajectories correlate with reality. Consistency asks whether the rollout stays coherent when the model keeps feeding on its own future predictions. Efficiency asks whether the model can generate futures quickly enough to be used inside a real robot loop.

Those three checks are adversarial to each other. A video model can look faithful and still be too slow. A compact latent model can be fast and still lose information needed by a visuomotor policy. A model can produce good first seconds and still drift over a 10-second rollout. WEAVER's claim is specific: on its manipulation setup, one world model can satisfy all three well enough to become a policy evaluator, a synthetic-data filter, and a test-time planner.

How WEAVER reaches all three

WEAVER stands for World Estimation Across Views for Embodied Reasoning. The model has 928M parameters and predicts future multi-view robot observations in a learned latent space. The important part is how the architecture assigns one job to each constraint.

Fidelity. Each camera view is encoded with a pretrained Stable Diffusion 3 VAE encoder. The robot's proprioceptive state is projected into the same token stream. WEAVER predicts future external views, wrist views, and future proprioception, so the imagined state is not only a video; it also contains the robot-state information needed for control.

Consistency. The transformer receives sparse long-term memory, a short recent history, and an h-step action plan. It then generates the next h latents with a flow-matching objective. Diffusion Forcing gives different future steps independent noise levels during training, which pressures the model to handle partially denoised long rollouts instead of only clean one-step prediction.

Efficiency. SPRINT blocks drop many patch tokens, KV caching reuses memory and history tokens across denoising steps, and a ReFlow post-training stage distills WEAVER-FT into a lower-NFE model for planning. The reward head and critic also run directly on latents plus the language instruction, so planning does not need to decode every candidate future and send it to a separate VLM judge.

Original schematic of the WEAVER pipeline. The paper's architecture figure below shows the same interface at full detail: observation latents, memory/history tokens, action chunks, future latent prediction, reward, and value.

WEAVER architecture — Figure 2 from WEAVER (Jain et al., arXiv:2606.13672), CC BY 4.0. The key interface is memory + history + action sequence in, future latents + reward/value out.

Pretraining and adaptation

The pretraining stage builds the general robot simulator. WEAVER is pretrained on DROID for 1M gradient steps with batch size 32, learning rate 1e-4, EMA 0.9999, and 4 H100 GPUs for about 10 days. The model uses 32 transformer layers, hidden dimension 1536, 16 heads, SPRINT probability 0.5, and 5 Hz world-model imagination after downsampling.

The reward and critic are trained on top of WEAVER latents. The authors annotate DROID trajectories with RoboMeter progress rewards, interpolate the 1 fps labels back to the full video length, shift the progress signal into a negative reward range, and train latent reward/value heads with language instruction features.

The task-specific adaptation is deliberately small: 250 real trajectories total, 50 for each of five tasks, plus another 100 evaluation trajectories. WEAVER is finetuned for 16K steps at learning rate 2e-5, taking about 6 hours on 4 H100 GPUs. For the planning variant, WEAVER-ReFlow adds 2K rectified-flow post-training steps so the same finetuned world model can run with fewer denoising evaluations.

How the experiments are built

The experiments test the world model before using it to change the policy. First, WEAVER is compared with Ctrl-World, a 1.5B-parameter DROID-trained diffusion world model. The evaluation uses 256 DROID validation trajectories and 100 out-of-distribution real-robot trajectories collected from π0.5. Each model autoregressively generates 10-second sequences, predicting 15-step action chunks at a time. FID, FVD, and LPIPS compare decoded generations against ground-truth videos.

FVD versus inference time, WEAVER vs Ctrl-World — Figure 5 from WEAVER (arXiv:2606.13672), CC BY 4.0. This is the fidelity-versus-efficiency plot: lower FVD is better, lower inference time is better. WEAVER sits below and left of Ctrl-World across DROID and OOD views.

The downstream experiments then ask whether the simulator is useful rather than only visually plausible.

Use	Experimental question	Headline result
Policy evaluation	Replay real action trajectories inside the world model and compare imagined success with real success.	WEAVER-FT reaches Spearman 0.870 and Pearson 0.863 in the appendix table; the main text reports ρ = 0.870 correlation with real-world success rate.
Policy improvement	Use WEAVER to sample and filter high-advantage action segments, then distill them into π0.5.	Average success rises from 0.44 for π0.5 to 0.67 with synthetic data, 0.71 with real data, and 0.82 with mixed data.
Test-time planning	Sample four action chunks, imagine each future, score reward plus critic value, and execute the best chunk.	Average success improves from 0.44 to 0.58 across five tasks, with dynamics prediction around 20x faster than Ctrl-World at horizon 15, batch size 4.

Policy evaluation comparison across Ctrl-World, WEAVER, and WEAVER-FT — Figure 6 from WEAVER (arXiv:2606.13672), CC BY 4.0. The left side is a qualitative stress test for long-horizon state prediction; the right side tests whether imagined success rates track real success rates.

Policy improvement success rates with real, synthetic, and mixed data — Figure 7 from WEAVER (arXiv:2606.13672), CC BY 4.0. Synthetic rollouts are filtered by predicted advantage, then used as finetuning data. Mixed real and synthetic segments produce the best average policy improvement.

Synthetic data scaling on Pour Beans — Figure 7 from WEAVER (arXiv:2606.13672), CC BY 4.0. Synthetic rollouts are filtered by predicted advantage, then used as finetuning data. Mixed real and synthetic segments produce the best average policy improvement.

Test-time planning success rate with WEAVER best-of-N steering — Figure 9 from WEAVER (arXiv:2606.13672), CC BY 4.0. The planning setup uses π0.5 as the proposal policy, samples B = 4 candidate chunks, imagines h = 12 steps, and executes the highest-advantage chunk.

Limitations

WEAVER leans on pretrained components: the SD3 VAE, DROID-scale robot data, and RoboMeter reward labels. The strongest results come after finetuning on the target robot tasks, especially for pouring beans, where granular dynamics are underrepresented in pretraining. The paper also names partial observability, missing tactile sensing, deformable-object physics, reward-label noise, and online latency as remaining bottlenecks.

The clean adversarial read is this: WEAVER improves the world model body, but every downstream use still depends on trusting imagined futures. A faster, more faithful simulator raises the ceiling for evaluation and planning; it also makes uncertainty estimation, validation, and safeguards more important because the robot is now acting on model-generated futures before the real world confirms them.

References

Arnav Kumar Jain, Yilin Wu, Jesse Farebrother, Gokul Swamy, Andrea Bajcsy: WEAVER, Better, Faster, Longer: An Effective World Model for Robotic Manipulation (arXiv:2606.13672).
Project page with code, models, and videos: arnavkj1995.github.io/WEAVER.
Main baseline: Ctrl-World. Base robot policy: π0.5.

WEAVER：一个同时做到保真、一致、快的世界模型

2026 年 6 月 · 世界模型 · 机器人操作 · WEAVER (arXiv:2606.13672)

WEAVER 主要改的是 world model 本体。policy 仍然是 π0.5，问题变成：底下这个学出来的模拟器，能不能足够准、足够稳、足够快，从而用来评估策略、生成训练数据，并在机器人执行时做规划。

三个约束

论文把机器人 world model 的要求收束成三个检查项。Fidelity 是准：想象出来的轨迹要和真实世界相关。Consistency 是稳：模型一边生成未来、一边继续用自己的未来往后推，rollout 不能很快漂掉。Efficiency 是快：生成未来的速度要足够接近真实机器人控制链路，才能放进测试时规划。

这三个检查项彼此对抗。视频模型可以看起来很准，但速度太慢；小 latent 模型可以很快，但可能丢掉视觉运动策略需要的信息；一个模型可以预测前几秒，却在 10 秒 rollout 里累积误差。WEAVER 的具体主张是：在它的 manipulation setup 上，一个世界模型可以同时满足这三项，并由此承担 policy evaluator、synthetic-data filter 和 test-time planner 的角色。

WEAVER 怎么把三者一起拿到

WEAVER 全称是 World Estimation Across Views for Embodied Reasoning。它有 928M 参数，在 latent 空间预测多视角机器人未来。关键在于它怎样把三个约束分配到具体模块上；“生成视频”只是外层表现。

Fidelity / 准。每个相机视角由预训练 Stable Diffusion 3 VAE encoder 编成 latent patch token；机器人本体感受状态也投影成 token。WEAVER 同时预测外部相机、腕部相机和未来 proprioception，所以它想象的是控制所需状态，画面质量之外还包含 robot-state 信息。

Consistency / 稳。Transformer 看到稀疏长期记忆、一小段近期 history 和一个 h 步 action plan，然后用 flow-matching 目标生成未来 h 个 latent。Diffusion Forcing 在训练时给不同未来步独立噪声水平，把训练压力从一阶干净预测扩展到长 rollout 的部分去噪状态。

Efficiency / 快。SPRINT blocks 丢掉大量 patch token，KV cache 在去噪步之间复用 memory/history token，ReFlow 后训练把 WEAVER-FT 蒸馏成更少 NFE 的规划模型。reward head 和 critic 直接在 latent 加语言指令上打分，所以规划时不用把每个候选未来 decode 成图片，再交给外部 VLM 判。

原创示意图。下方原文架构图给出完整接口：observation latents、memory/history tokens、action chunks、future latent prediction、reward 和 value。

WEAVER 架构 — 图 2，引自 WEAVER（Jain 等，arXiv:2606.13672），CC BY 4.0。核心接口是 memory + history + action sequence 输入，future latents + reward/value 输出。

预训练和任务适配

预训练阶段先得到一个通用机器人模拟器。WEAVER 在 DROID 上预训练 1M 个 gradient steps，batch size 32，learning rate 1e-4，EMA 0.9999，用 4 张 H100 训练约 10 天。模型本体是 32 层 transformer，hidden dimension 1536，16 heads，SPRINT probability 0.5；数据下采样后以 5 Hz 做 world-model imagination。

reward 和 critic 训练在 WEAVER latent 之上。作者用 RoboMeter 给 DROID 轨迹标 progress reward，把 1 fps 的标签插值回完整视频长度，再把 progress signal 移到负 reward 区间，用这些标签训练 latent reward/value heads，并把语言指令特征接进去。

任务适配阶段很小：5 个任务，总共 250 条真实轨迹，每个任务 50 条；另有 100 条评估轨迹。WEAVER 用 2e-5 learning rate finetune 16K steps，在 4 张 H100 上约 6 小时。规划版本再做 2K steps 的 ReFlow 后训练，让 finetuned world model 能用更少 denoising evaluations 跑起来。

实验怎么做

实验先测 world model 本体，再测它是否能改变策略链路。第一组对比 WEAVER 和 Ctrl-World，后者是 1.5B 参数、在 DROID 上训练的 diffusion world model。评估数据包括 256 条 DROID validation trajectories，以及 100 条由 π0.5 收集的分布外真机轨迹。两个模型都自回归生成 10 秒序列，每次共同预测 15-step action chunk；FID、FVD、LPIPS 用来比较 decoded generations 和真实视频。

FVD 对推理时间，WEAVER vs Ctrl-World — 图 5，引自 WEAVER（arXiv:2606.13672），CC BY 4.0。这张图直接测 fidelity vs. efficiency：FVD 越低越好，inference time 越低越好。WEAVER 在 DROID 和 OOD 视角上都落在 Ctrl-World 的左下方。

后面的实验把目标从画面相似扩展到策略可用性：这个模拟器能不能真的服务策略。

用法	实验问题	主要结果
Policy evaluation	把真实 action trajectories 放进 world model 开环重放，比较 imagined success 和 real success。	WEAVER-FT 在 appendix table 里达到 Spearman 0.870、Pearson 0.863；正文报告和真实成功率的相关 ρ = 0.870。
Policy improvement	用 WEAVER 采样并筛选高 advantage action segments，再蒸馏回 π0.5。	平均成功率从 π0.5 的 0.44 提到 synthetic data 的 0.67、real data 的 0.71、mixed data 的 0.82。
Test-time planning	采 4 个 action chunks，想象每个未来，用 reward + critic value 打分，然后执行最好的 chunk。	五个任务平均成功率从 0.44 提到 0.58；在 horizon 15、batch size 4 时 dynamics prediction 约比 Ctrl-World 快 20x。

Ctrl-World、WEAVER、WEAVER-FT 的策略评估对比 — 图 6，引自 WEAVER（arXiv:2606.13672），CC BY 4.0。左侧是长时域状态预测的定性压力测试；右侧检验 imagined success rate 能否跟真实成功率同向变化。

real、synthetic、mixed data 微调后的策略成功率 — 图 7，引自 WEAVER（arXiv:2606.13672），CC BY 4.0。Synthetic rollouts 先由 predicted advantage 筛选，再作为 finetuning data；real + synthetic mixed segments 给出最高平均成功率。

Pour Beans 任务上的 synthetic data scaling — 图 7，引自 WEAVER（arXiv:2606.13672），CC BY 4.0。Synthetic rollouts 先由 predicted advantage 筛选，再作为 finetuning data；real + synthetic mixed segments 给出最高平均成功率。

WEAVER best-of-N steering 的测试时规划成功率 — 图 9，引自 WEAVER（arXiv:2606.13672），CC BY 4.0。测试时规划用 π0.5 提议动作，采 B = 4 个候选 chunk，想象 h = 12 步，然后执行 advantage 最高的 chunk。

局限

WEAVER 依赖几个预训练基础：SD3 VAE、DROID 规模机器人数据、RoboMeter reward labels。最强结果来自目标任务 finetuning，尤其是 Pour Beans 这种 granular dynamics 在预训练数据里覆盖不足的任务。论文也明确列出 partial observability、tactile sensing 缺失、deformable-object physics、reward-label noise 和在线 latency 这些瓶颈。

用对抗性方式读，WEAVER 提升了 world model 本体，但所有下游用法仍然要信任 imagined futures。更快、更准的模拟器会提高 evaluation 和 planning 的上限；同时也让 uncertainty estimation、validation 和 safeguards 更重要，因为机器人开始在真实世界验证之前，先根据模型生成的未来行动。

参考

Arnav Kumar Jain、Yilin Wu、Jesse Farebrother、Gokul Swamy、Andrea Bajcsy：WEAVER, Better, Faster, Longer: An Effective World Model for Robotic Manipulation（arXiv:2606.13672）。
项目页（代码、模型、视频）：arnavkj1995.github.io/WEAVER。
主要世界模型 baseline：Ctrl-World。策略基座：π0.5。