VTWM ablation — short 16-frame rollouts (auto-uploaded)

Diffusion-forcing world model rollouts. Each sample is ~2.7 s (16 raw frames @ 6 fps). Side-by-side videos show ground truth on the left, model prediction on the right. First half of each clip = history (GT), second half = autoregressively sampled future. Click a sample header to expand its videos (default-collapsed to keep the page light).

Sections

Setup comparison

Click a run name to jump to its rollout videos. ✓ = feature on, ✗ = off. Lower val_loss is better.

Rollout-latent metrics (10 val samples)

Latent-space MSE of predicted future vs ground truth. Contact/no-contact split uses tactile latent-to-reference energy (threshold 0.05).

Training loss curves & conclusions

📈 Live interactive curves: https://wandb.ai/yuxiangma/tactile-world-model-diffusion (wandb — always current; static PNG below is a snapshot)

# Training loss curves — conclusions

Regenerated from wandb (`scripts/plot_loss_curves.py`). Runs covered: **37**.

- **shift16 + delta-to-ref** are the two tactile fixes; without shift16 the tactile stream diverges.
- **cam-pose** curves look fine on val_loss but decoded LPIPS is far worse — see perceptual table.
- **stride2 (4x data)** is the biggest perceptual win; p01 trades a little val_loss for rollout consistency.
- **hybrid_flow** uses rectified flow — its val_loss is on a different scale, judge by decoded metrics.

Included runs: mm_v0, mot_vt_p01, mot_vta_p01, mv1_256_delta_shift, mv1_3v2t_delta_ref, mv1_p01, mv1_p3_delta_shift, mv1_v2_perframe, mv1_v2_stride2, mv1_v2_stride2_p01_adaptw, mv1_v2_stride2_p01_hybrid, mv1_v2_stride2_p01_hybrid_deep, mv1_v2_stride2_p01_hybrid_flow, mv1_v2_stride2_p01_hybrid_rope, mv1_v2_stride2_p01_plucker, mv1_v2_stride2_p01_tactile_only, mv1_v2_stride2_p01_v2t, mv1_v2_stride2_p01_vision_only, mv1_v2_stride2_p01rand, mv2_3v2t_campose_delta_ref, mv2_p4_delta_shift, p2_mv, p3_mv2t, p4_gate, p5_gate, vo_left, vo_middle, vo_right, vo_v0, vtwm_e2_delta_ref, vtwm_e3_delta_ref_null, vtwm_n1_shift16_delta_ref_null_prenoise, vtwm_n2_shift16_delta_ref_tactile_dropout, vtwm_p0_shift16_delta_episode_p01_ref, vtwm_s1_shift16_mmdit_mv, vtwm_s2_shift16_delta_ref, vtwm_s3_shift16_delta_ref_weight

_Not on wandb yet (curve omitted): mv1_v2_h4f12, mv1_v2_stride2_p01, p4_mv2t_extr, vtwm_e1_1v2t_mmdit_mv_

A. Baselines — window ref (tight short-window)

mv1_3v2t (window ref)

Original mv-data SOTA: 3-view + 2-tactile, MM-DiT-MV, shift16 + delta-to-ref, window reference. Kept as apples-to-apples baseline for stride2 comparison. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 (window ref) SOTA (val_loss)

Stride=2 re-encoded latents → 4x training data. Same recipe as mv1_3v2t. Ties pre-stride2 SOTA at ep39. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + h4/f12 window

Shorter history (4), longer future (12) — 1.5x more train windows per episode. Ties stride2 SOTA at ep39. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

mv1 256-res (window ref)

Same mv1 recipe at 256x256 input (2x spatial). val_loss not directly comparable (different latent dims), kept for visual quality comparison. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

B. Baselines — episode_p01 ref (rollout-consistent)

stride2 + p01

Stride2 with tactile_ref_source=episode_p01. Closes only ~halfway the gap vs window (Δ=+0.003), confirming window ref is tighter. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

C. Hybrid architecture — SOTA (HunyuanVideo/FLUX style)

stride2 + p01 + hybrid (dual6+single7)

Dual→Single hybrid fusion: first 6 blocks are MMDiTBlockDFMV (per-modality QKV+MLP), last 7 blocks are SingleStreamHybridBlock (shared QKV+MLP with per-modality adaLN, forces deep V-T fusion). Depth=12, 85M params (0.84x mmdit_mv baseline). Warm-started from mv1_v2_stride2_p01 (loaded 282/298 tensors bit-exact). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid + flow matching

Hybrid backbone + rectified flow matching training objective (SD3/FLUX/HunyuanVideo). Same 85M backbone. Linear-interpolation schedule + velocity target + Euler ODE sampling (50 steps instead of 100). Val_loss numerically incomparable to pred_v runs — compare via decoded LPIPS + tactile MSE. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid (deeper, dual6+single8)

Deeper hybrid: depth=14 (+2 single-stream blocks) for more cross- modal fusion capacity. Same dual_depth=6 to preserve warm-start. ~100M params — matches original mmdit_mv budget. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)