Construct the target
Fuse demonstrated actions, realized robot responses, and sparse visual transitions into temporally ordered Effect tokens.
Beyond Actions
Learning Future Operation Targets
for Vision-Language-Action
Models
Action–Future Coupled Effect Learning
Dexterous manipulation,
in closed loop.
Policy rollouts on DexJoCo, from coordinated bimanual tasks to single-arm tool use.
From actions to Effects
An action specifies a command. An Effect connects that command to the robot’s response and the resulting scene change—while remaining decodable into continuous actions.
Method
Two stages connect future-grounded representation learning to closed-loop control. Shared Effect tokens accommodate single-arm and bimanual interfaces.
Predict the Effect
Train a flow-matching VLA to predict the Effect from current images, language, and robot state.
Decode and execute
Decode the predicted Effect into an action chunk, execute a prefix, and replan from new observations.
From motion to scene change
The hammer–nail example links decoded robot motion to the accompanying scene transition across three successive frames.
Top: decoded robot motion.
Bottom: accompanying scene changes.
This is a qualitative reconstruction example, not a predicted-policy success-rate evaluation.
Results
11 tasks, three evaluation seeds, and 1,650 episodes per method under rand-obj. The largest gains are on bimanual tasks; fine single-arm control remains a limitation.
DexJoCo success rate
Mean ± sample standard deviation across evaluation seeds
50 episodes per task per seed; seeds 0, 1, 2. One checkpoint per method. Variability is across evaluation seeds, not training runs.
All 11 task results
| Task | π₀.₅ | UVT | AFCE |
|---|
Bimanual coordination and fine control
Coordination improves substantially, while precision tasks expose unresolved limitations in fine control. The task-family analysis here uses evaluation seed 0.
“Bimanual” in this figure refers to Microwave and Photograph; the main bimanual score averages all five dual-arm tasks.