TopoRetarget: Interaction-Preserving Retargeting
for Dexterous Manipulation

Abstract

Human demonstrations are a rich source of reference motion for dexterous manipulation, but they are hard to reuse directly. Human and robot hands differ enough that naively matching fingertips or joint angles breaks the contact structure that actually drives a task — and those artifacts propagate into reinforcement learning as missed contacts, penetration, and infeasible poses.

We introduce TopoRetarget, an interaction-preserving retargeting framework. Instead of copying hand pose, it preserves the local hand-object interaction — which keypoints touch, and how. We build a sparse interaction graph over hand and object keypoints and solve a distance-weighted Laplacian deformation under directional-consistency, kinematic, and non-penetration constraints. A single parameter set handles every embodiment, object, and scale we evaluate.

The references this produces are both more faithful and easier to learn from. On ContactPose, TopoRetarget attains the best contact precision and alignment of any baseline; it lifts Pen-Spin training success by +40.6 percentage points; and the policies it enables transfer to Wuji Hand hardware zero-shot on cube reorientation and pen spinning.

Contributions

1. An interaction-preserving retargeting framework that maintains local hand-object interaction while enforcing dexterous hand kinematic and penetration constraints.
2. A lightweight reference-based RL tracking pipeline that enables zero-shot sim-to-real transfer for contact-rich dexterous skills, including pen spinning and cube reorientation.
3. A dexterous hand-object interaction dataset of retargeted trajectories, task references, and trained policies for reproducible reference-based dexterous manipulation.

Method

TopoRetarget overview. Given a human demonstration, object mesh, and target hand model, the method aligns bone directions during initialization, constructs source and robot interaction meshes, and computes the robot configuration via topology-aware Laplacian optimization. The output robot motion reference preserves hand-object interaction.

Comparison with Baselines

TopoRetarget better preserves both intra-hand relationships and hand-object interactions than the baseline methods.

Against the baseline average, TopoRetarget reduces contact precision error by 55% and maximum penetration by 92%.

Hand-only

Hand-object interaction

Generalization Hover to zoom

Augmentation across object scales and dexterous hand embodiments without per-case retuning: from a single human demonstration, TopoRetarget adapts to new object meshes, object scales, and hand embodiments (MANO, Wuji, Leap).

Real-world Results

Zero-shot cube reorientation on the Wuji Hand:

Zero-shot pen spinning on the Wuji Hand:

Across 5 / 5 zero-shot trials the policy keeps the pen spinning on the Wuji Hand. A full-length, uncut stability run:

Summary Video