Tactile-guided
retargeting
Human action, contact preference, and object motion jointly guide object-specific retargeting while preserving feasible robot kinematics.
Touch2Robot: Robot Touch in the Human Demonstration Loop
Abstract
Human demonstrations offer a scalable way to collect manipulation data, but their contacts may be unstable or infeasible when transferred to a robot hand. Collecting demonstrations directly on the target robot avoids this mismatch, but substantially increases the cost of data collection. To address this trade-off, we present Touch2Robot, a framework that lets humans collect demonstrations while seeing how the target robot hand would contact the object. We capture human hand motion, tactile-glove measurements, and object motion during human manipulation. These recordings guide object-specific RL policies to reproduce the demonstrated object motion while favoring contacts consistent with the recorded human touch. We distill the learned behaviors into a unified real-time retargeter that maps incoming human observations and object geometry to robot hand configurations. During collection, the predicted robot configuration is synchronized with the tracked object pose in simulation to reconstruct robot-object contacts, which are visualized to help the demonstrator adapt subsequent interactions to the target hand. Across four real-world tasks, Touch2Robot improves average real-robot replay completion from 37.9% to 72.1% over visual-only feedback, while reducing the collection time per replay-successful demonstration from 58.6 s to 18.2 s. Reconstructed target-hand contacts achieve 44.2% F1 against real-robot tactile measurements, and policies trained on Touch2Robot demonstrations improve downstream Diffusion Policy performance by 29.1 percentage points over visual-only feedback. These results show that bringing robot touch into the human demonstration loop improves both the quality and efficiency of scalable dexterous data collection.
Robot touch in the human data-collection loop
Touch2Robot closes the embodiment gap during demonstration collection rather than correcting it only after recording. Human motion and object movement are retargeted to a virtual dexterous robot in real time, where robot-specific contacts and tactile responses are reconstructed.
The estimated tactile state is returned through the interface, allowing the demonstrator to immediately adapt the next action to the target hand's kinematics and contact geometry.
- Robot actions aligned with human intent
- Embodiment-consistent tactile signals
- Object motion without per-demo robot execution
Method
Touch2Robot transfers not only human motion, but also the target robot's interaction outcomes back to the demonstrator. Tactile-guided retargeting, real-time policy distillation, and contact reconstruction together turn scalable human demonstrations into robot-specific tactile data.
Tactile-guided
Unified real-time
retargeter
Temporal human history and geometry features are fused into robot actions and contact auxiliaries fast enough for interactive collection.
Closed-loop human
adaptation
Robot-specific tactile estimates are returned through the interface so the demonstrator can immediately correct contact mismatches.
Evaluation Tasks
We evaluate contact consistency and downstream imitation learning on dexterous behaviors that require precise contact timing, sustained interaction, and adaptation to constrained object motion.
Appendix
The supplementary material includes tactile processing and semantic alignment, task environments, training settings, baseline implementations, and evaluation protocols.