DexSeed: Scalable Physics-Grounded Demonstration Generation for Sim-to-Real Dexterous Manipulation

Abstract

Learning visuomotor policies for contact-rich dexterous manipulation requires demonstrations that are diverse and physically valid, yet teleoperation is costly and embodiment-specific, while human videos are not directly executable by robot hands. We introduce DexSeed, a physics-grounded pipeline that converts a handful of human RGB-D videos into large-scale dexterous manipulation datasets under joint randomization of layout, geometry, visual appearance, and physics. Rather than treating a human demonstration as a trajectory to copy, DexSeed uses it as an object-centric, phase-aligned prior over task-relevant contact structure, combining temporal style augmentation, a task-conditioned objective, and sampling-based optimization through full physical rollouts. Across four contact-rich tasks, DexSeed achieves the highest data-generation success from both teleoperated demonstrations and human videos, improves simulated downstream policy success by 33.2% on average, and achieves a 45% zero-shot real-world success rate using only 3 human videos per task.

DexSeed overview: from a few human videos to large-scale physics-plausible robot demos and zero-shot real-world transfer. — **DexSeed** turns a few human RGB-D videos into large-scale, physics-plausible robot demonstrations across randomized domains and embodiments, enabling zero-shot sim-to-real transfer.

3

human videos per task

+11.5

pts data-generation success over best baseline (D2)

+33.2%

avg gain in downstream policy success

45%

avg zero-shot real-world success

Overview Video

This video contains audio.

How DexSeed Works

From a few human videos, DexSeed builds an object-centric, phase-aligned manipulation prior and optimizes robot controls through task-conditioned physical rollouts — yielding diverse contact strategies and manipulation behaviors from a small seed set.

The DexSeed pipeline from human video reconstruction to task-conditioned physics optimization. — **The DexSeed pipeline.** Each human video is reconstructed into human & object motion, retargeted to a kinematic robot prior, and turned into an object-centric, phase-aligned reference with a contact mask. A sampling-based optimizer then searches robot controls through full physical rollouts under layout, geometry, physics, and temporal randomization, and renders the accepted, task-successful rollouts with visual randomization for policy training.

01

Phase-aligned contact prior

From each video we build a layout-conditioned, phase-aligned reference and a contact mask that capture where and when contact is task-relevant — a prior, not a trajectory to track.

02

Temporal style augmentation

Rollout horizon and phase timing become an explicit randomization axis, so generated behaviors adapt their timing instead of replaying the source schedule.

03

Task-conditioned physics optimization

An MPPI-style sampler optimizes controls through full physical rollouts, accepting only task-successful, physically valid demonstrations.

The Seed: A Few Human Videos

DexSeed starts from only 3 human RGB-D videos per task across four contact-rich tasks — spanning functional grasping, in-hand re-grasping, and fingertip pushing.

Diverse, Physically-Valid Generated Data

From three source demonstrations, DexSeed generates large, diverse datasets under joint layout, geometry, visual & physics randomization. Below: generated rollouts in simulation (with visual randomization) for each task — use the tabs and arrows to browse seeds.

Heatmap of data-generation success across randomization regimes and methods. — **Coverage across regimes.** Average data-generation success across randomization regimes (D0/D1/D2) and source types (teleop, human video). DexSeed (right column) remains strong across all regimes.

2D PCA of generated wrist poses and finger-joint actions. — **Trajectory diversity.** 2D PCA of wrist poses and finger-joint actions on Place Hammer under D2 — DexSeed expands three source demos into a markedly wider region than baselines.

Highest Data-Generation Success

Under the hardest regime D2 (joint layout, geometry & physics), DexSeed attains the best data-generation success from both teleoperated and human-video sources — across all four tasks.

Per-task data-generation success rate under D2 comparing DexSeed to baselines. — **Per-task data-generation success (↑) under D2.** Each bar is computed over 1000 generation attempts per method, task, and source type. Baselines: DexMimicGen, SkillMimicGen, ManipTrans, and SPIDER.

65.5%

from teleop sources
(+11.5 pts vs. best baseline)

50.0%

from human videos
(+9.0 pts vs. best baseline)

up to 2×

downstream policy success
vs. strongest baseline (sim)

Zero-Shot Real-World Deployment

Policies trained purely on DexSeed-generated data transfer zero-shot to a real LEAP Hand — reaching 45% average success from just 3 human videos per task. Each task is evaluated over 20 real-world trials.

Performance Scales with Generated Data

Policy success improves consistently as the number of generated demonstrations grows (100 → 3000), in both sim-to-sim and sim-to-real evaluation.

Sim-to-Sim Evaluation

Sim-to-Real Evaluation

Pick Volt Place Hammer

Move across either chart to inspect exact success rates.

Scaling evaluation values
Evaluation	Demos	Pick Volt	Place Hammer
Sim-to-Sim	100	31%	10%
Sim-to-Sim	500	52%	23%
Sim-to-Sim	1000	73%	56%
Sim-to-Sim	3000	77%	64%
Sim-to-Real	100	15%	10%
Sim-to-Real	500	30%	20%
Sim-to-Real	1000	55%	30%
Sim-to-Real	3000	60%	35%

Policy success rate scaling with the number of generated demonstrations.

Scaling policy with generated data. Success rate (↑) as the number of generated demonstrations grows, for Pick Volt and Place Hammer — DexSeed provides useful supervision well beyond the small-data regime.

BibTeX

@misc{dexseed2026,
  title  = {DexSeed: Scalable Physics-Grounded Demonstration Generation
            for Sim-to-Real Dexterous Manipulation},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review}
}