HapticDP: Visuo-Haptic Diffusion Policy
Visuo-haptic policy learning for bimanual dexterous manipulation, with slow visual planning and faster force–tactile refinement during contact.
The question is whether a policy can keep a slow visual plan while using force and tactile sensing to refine contact, instead of replaying an open-loop chunk. I developed HapticDP around that split: vision-guided global planning and a faster force–tactile local loop share one U-Net, so the robot can run receding-horizon control at 50 Hz on wiping, polishing, insertion, and screwdriving.
Demonstrations are collected with Apple Vision Pro mixed-reality tracking and wearable haptic gloves on a dual-arm platform with dexterous hands. I built that bilateral collection stack so visual, force, and tactile streams stay aligned for later policy training.
The work sits in the RAIDS Research Group at the Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, advised by Prof. Pai Zheng. Related papers on this page include a vision-based tactile survey and a CIRP Annals study of bimanual dexterous assembly.
Current experiments cover insertion, wiping/polishing, and screwdriving. The hero video is a polishing run; the clip below is insertion.
Insertion experiment (web clip).