【Grasping by Imitation】Learning Robotic Grasping from Human Demonstrations

1 DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration

Title: DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration Research Team: Technical University of Munich Venue: IROS Year: 2021 Code:

1.1 Target Problem

1.1.1 Existing Problems

Existing pose-estimation methods either need to compute the 6D pose of the target object or learn a set of grasp points. The former do not scale well to multiple object instances or classes, while the latter require large annotated datasets and are hindered by their generalization ability crossing to new geometries.

1.1.2 Proposed Solution

Teach a robot how to grasp objects through simple, brief human demonstrations, without requiring many annotated images or being limited to specific geometries.

1.1.3 General Method

First, construct a sequence of RGB-D images of a human-object interaction. Use this sequence to build hand and object meshes representing the interaction. Complete the missing parts of the reconstructed object shape and estimate the relative transformation between the reconstructed model and the visible object in the scene. Finally, convert prior knowledge of the relative pose between the object and the human hand, together with an estimate of the object’s current pose in the scene, into the grasp commands required by the robot.

1.1.4 Introduction Summary

Why this research was conducted: Current robotic grasping methods lack generalization because they either focus on estimating object poses or learn grasp points, requiring detailed prior information about objects or large amounts of annotation. Like the human hand, robotic grippers and arms have natural limits on their range of motion and limited degrees of freedom, which constrain their possible grasp poses. Although the motion models of robotic grippers and human hands may differ greatly, it should be possible to extract information from human manipulation and infer sufficient grasp commands for the target robot. With limited human demonstrations, a robot can imitate human behavior and thereby grasp objects seamlessly.

What this paper mainly does: We focus on this kind of imitation, in which the robot mirrors the human interaction, as shown in Figure 1. The task can be divided into visual perception and interpretation, with a human instructor demonstrating a prior manipulation (Demo) and the robot inferring the grasp information required to manipulate the current scene (Grasp). If there is a sufficient mapping from the human hand to the robotic gripper, decomposing the task into these two stages allows our method to scale to many different grippers. Ultimately, this paves the way for teaching robots through natural human demonstrations, enabling higher levels of automation, especially in less structured environments.

How the paper generally implements this: While the object is demonstrated to the robot from a variety of angles (Demo), our method continuously tracks the hand and object, which are fused into a truncated signed distance field (TSDF) for 3D reconstruction. Semantic segmentation of the hand and object makes it possible to separate and further process the reconstructions to recover complete 3D representations of the object and hand. We then use the MANO hand model to extract the relevant 3D hand mesh and align it closely with the reconstructed object. During inference, we use PPF FoldNet to predict whether the object is present and its relative transformation from object space to camera space. The estimated pose is then applied to derive the final grasp command from the estimated hand mesh.

1.2 Method

Overall Workflow:

1. Segment the hand and object in a set of RGB-D images from a human demonstration, and use the recorded depth maps to reconstruct their shapes
2. Complete the object shape
3. Extract the hand pose
4. Estimate the object’s 6D pose, transform the hand model, and infer the grasp command

1.2.1 Three-Dimensional Reconstruction of Human-Object Interaction

MaskRCNN is used to segment the hand and object, and binary cross-entropy is applied to prevent inter-class competition.

Using the segmented depth images, corresponding TSDF voxels are created with KinectFusion, and drift-free tracking is achieved through ICP registration between the input frames and the TSDF. (Because household objects have simple geometries, the hand and object are tracked together; the hand’s complex structure stabilizes the tracking results.)

Using the segmentation results, the hand and object are separated through two independent TSDF reconstructions.

1.2.2 Object Shape Completion

The reconstructed model remains incomplete because of self-occlusion and partial visibility.

A 3D CNN is used to directly correct the TSDF volume, after which the shape is extracted with marching cubes. (A 3D variant of UNet is used here. The input is a 64x64x64 voxel grid, and the predicted score output for each voxel indicates whether that voxel is occupied.

1.2.3 Hand Pose Estimation

A parametric hand model is estimated from the reconstructed hand shape.

The MANO hand model is used to map hand pose and shape parameters to a mesh. Because the hand is also partially occluded, a CNN is jointly trained with auxiliary contact and collision losses to estimate the hand mesh and object mesh.

To further improve the grasp position, ICP is used to align the hand mesh with the hand TSDF voxels.

1.2.4 Grasp Command Generation

First, the object pose is recovered and used to transform the hand mesh. The grasp point is then computed using the thumb and index finger of the hand model.

1.3 Thoughts

1. Other methods could be used for the object’s three-dimensional reconstruction, or it could be combined with CAD-model completion, which may produce better results than prediction with a 3D CNN.
2. Newer algorithms could also be considered for extracting the hand pose, such as detecting hand keypoints instead of predicting a hand mesh.
3. The grasp pose is generated directly from the thumb and index finger as a two-finger grasp pose. Could other methods be considered to improve grasp reliability?

2 Learning to Grasp Familiar Objects Based on Experience and Objects’ Shape Affordance

Title: Learning to Grasp Familiar Objects Based on Experience and Objects’ Shape Affordance Research Team: Technical University of Munich Venue: IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS Year: 2019 Code:

2.1 Target Problem

2.1.1 Existing Problems

In grasping methods for known objects, each object has a grasp database. The robot estimates the object’s pose and then uses a national travel hypothesis to find a suitable grasp pose. The drawback of these methods, however, is that it is impossible to place models of every object in the robot’s database.

What is needed is the ability of models to generalize from previous experience to new objects.

2.1.2 Proposed Solution

The paper integrates key cues from human grasping experience—the positions and orientations of the thumb tip and wrist—and proposes an effective grasping method.

2.2 Method

2.2.1 Generating Grasp Points from an Incomplete Point Cloud

During grasping, grasp points on familiar objects have similar relative positions on the objects.

Based on this principle, the object is described with the 3D SHOT shape descriptor, which can accurately describe the position of an interest point relative to the entire object and its surface. The specific process for learning grasp points is as follows:

1. Collect the SHOT features, LR features, and RGB features of interest points selected from a partial point cloud
2. Reduce the dimensionality of the features along the point dimension by calculating simple statistics such as range, mean, standard deviation, and entropy
3. Feed the features into an extreme learning machine for object classification.

2.2.2 Building the Grasp Model

I did not understand this.

It appears to establish the coordinate transformation between the thumb and the object, then convert it into the coordinate transformation between a three-finger gripper and the object.

2.2.3 Wrist-Joint Constraint Estimation

This mainly addresses the problem that a point cloud is occluded under a single external viewpoint, making it impossible to determine the wrist orientation accurately.

3 R3M: A Universal Visual Representation for Robot Manipulation

Title: R3M: A Universal Visual Representation for Robot Manipulation Research Team: Stanford University, Meta AI Venue: CoRL Year: 2022 Code: https://tinyurl.com/robotr3m

3.1 Target Problem

Train a robot to complete manipulation tasks from images. Given a piece of text, such as “put the spatula into the pot,” the robot performs the corresponding action based on vision.

(1) Limitations of Conventional Methods

A conventional and widely used approach is to train an end-to-end model from scratch using homogeneous data. Because training data are difficult to obtain, however, this limits the method’s generalization. We also still have not suitable robot datasets; recent datasets consist of only a small number of different environments and limited tasks, so their general applicability is restricted.

(2) The Paper’s Breakthrough Idea

Drawing on general-purpose, effective models such as ImageNet, no similar model has yet emerged in robotics, but the idea can be borrowed: use abundant in-the-wild data (“wild data?”), meaning videos of humans interacting with their environments. These data are large and diverse, encompassing a variety of scenes and tasks from around the world.

(3) Brief Description of the Paper’s Method

The authors train a representation method for robot manipulation, R3M. R3M can learn challenging tasks such as placing food into a pot and folding a towel.

3.2 Method

The paper argues that good performance of robot manipulation consists of the following three aspects:

• The robot should capture temporal dynamics because it must complete tasks in chronological order within the environment
• The robot should capture features related to one
• The robot should be compact

(1) Temporal Contrastive Learning

The encoder is trained to generate a representation, is of temporally closer images the distance between them is smaller than the distance between temporally farther images or images from different videos.

(2) Video-Language Alignment

Capture language-related features and learn the semantic components of video scenes.

(3) Regularization

Reduce the dimensionality of the state space to ensure that the policy trained through cloning conforms to the expert state distribution.

3.3 Thoughts

This differs from my own direction. The paper focuses more on semantics, with vision serving only as a means of perception.

4 Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video

Title: Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video Research Team: University of Freiburg, Germany Venue: arXiv Year: 2019 Code: http://robotskills.cs.uni-freiburg.de/

4.1 Target Task

Learn robot manipulation tasks from unlabeled multi-view videos.

(1) Limitations of Conventional Methods

Although existing reinforcement learning methods have made some progress, they learn a solution for each task and rely on manually defined, task-specific reward functions. The resulting policies are also specific to particular tasks and cannot be transferred to new tasks.

(2) The Paper’s Innovation

The paper proposes an unsupervised skill-learning method called Adversarial Skill Networks (ASN), which discovers and learns transferable skills by watching videos. The learned skills are used for RL so that a broader range of tasks can be solved by combining previously learned skills.

The method does not require correspondences between frames and task IDs, nor does it require any additional supervision.

4.2 Method

Adversarial Skill Networks (adversarial skill networks)

We learn a skill-metric space within an adversarial framework. The encoding part of the network attempts to maximize entropy to improve generality. The discriminator is not used at test time; it attempts to minimize the entropy of its predictions to improve skill recognition. Finally, maximizing the marginal class entropy across all skills leads to uniform use of all task classes. Note that no information is required about the relationship between the framework and the tasks from which they originate. (I did not understand this.)

4.3 Thoughts

It seems possible to learn tasks from unlabeled videos. However, the method is too theoretical.

5 BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

Title: BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning Research Team: Google, University of California, Berkeley, Stanford University Venue: CoRL Year: 2022 Code: https://sites.google.com/view/bc-z/home

5.1 Target Problem

Enable a vision-based robot manipulation system to generalize to new tasks.

To accomplish this, the authors developed an interactive imitation-learning system that can use different forms of information from a person as conditions, including natural language or human-demonstration videos. The system can learn from demonstrations. They also found that after learning 100 tasks, it could perform 24 untrained tasks without demonstrations.

(1) Existing Problems

One of the major challenges in robotics is creating a system that can perform a large number of tasks in unstructured environments based on arbitrary user commands. The key challenge in this work is generalization. A robot must be able to handle new environments, recognize and manipulate objects it has never seen before, and understand the intent of commands it has never previously been asked to execute.

Conventional methods are end-to-end learning at the pixel level, then by enough real-world data, these methods in principle can enable robots to generalize to new tasks, objects, and scenes. In practice, however, this goal remains out of reach.

The problem addressed in this paper is how to generalize vision-based robot manipulation tasks through zero-shot or few-shot learning.

5.2 Data Collection

Human-demonstration videos were collected for 100 predefined tasks. These videos covered 9 basic tasks, including pushing objects and picking and placing objects.

A teleoperation system was built. The teleoperation device was connected to the robot through USB, and two handheld controllers were used to operate the robot remotely while the operator stood behind it. The operator controlled the robot from a third-person perspective, and the robot responded in real time by following the operator’s demonstrations of various tasks.

5.3 Method

5.3.1 Language and Video Encoding

The encoder takes a language command or a human video as input and generates a task.

• For a language command, a pretrained multilingual sentence encoder generates a 512-dimensional language vector for each task
• For a video, a ResNet18-based convolutional network is used

5.3.2 Policy Training

Given a fixed task, we train using the Huber loss for the XYZ and axis-angle predictions and a logarithmic loss for the gripper angle.

Open-loop auxiliary detection: when run in an open-loop manner, it takes an open-loop trajectory of ten actions. Open-loop prediction provides an auxiliary training objective and makes it possible to inspect closed-loop planning quality offline.

State differences are used as actions. Standard imitation learning uses demonstrated actions directly as target labels, but the expert cloning behavior in this paper leads to some small movements or jitter. The action is therefore defined as the difference between a future target and the next step, and an adaptive algorithm determines how much to move the arm and gripper.

5.3.3 Network Architecture

ResNet18 is used as the backbone. Multiple head branches extend from the final average-pooling layer of the backbone. Each head is a multilayer perceptron that models one component of the end-effector action; see the original paper for details.

5.3 Thoughts

The system first receives demonstration videos and text, and then the robot is manually controlled to perform tasks and collect data. It still seems rather cumbersome.

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

Title: VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training Research Team: Meta AI, University of Pennsylvania Venue: ICLR Year: 2023 Code: https://sites.google.com/view/vip-rl

6.1 Target Problem

Task-specific robot data are costly and scarce. Learning from large, diverse collections of offline human videos has emerged as a generally effective approach. However, how to use these human videos for general-purpose reward learning remains an open question.

Unlike robot control in simulated environments, real-world robot tasks do not provide good access to environmental state information or well-defined reward functions. Existing methods require extensive preparation to learn each task. By contrast, a simple way to specify a real-world manipulation task is to provide a goal image that captures the desired visual change in the environment. Existing methods, however, cannot produce effective reward functions.

This paper proposes Value-Implicit Pre-training (VIP), a self-supervised pretrained visual representation that generates reward functions for robot tasks.

The key idea is to use reinforcement learning itself as a pretraining mechanism for reinforcement learning. Because human videos do not contain action information that can be used for policy learning, this dual value function is pretrained without actions in a fully self-supervised manner.

6.2 Method

(1) Self-Supervised Value Learning from Human Videos

Although human videos are not data from the robot domain, they are in-domain data for learning goal-conditioned policies for human actions. It is therefore reasonable to consider solving a goal-conditioned reinforcement learning problem in the human policy space using offline human videos and extracting a visual representation (the paper considers an offline reinforcement learning approach based on KL).

Because actions do not appear in this reinforcement learning objective and all data can be sampled from an offline dataset, the dual value function can be self-supervised by choosing an appropriate reward function.

(2) Implicit Temporal Contrastive Learning

When two meaningful frames indicating the beginning and end of a task are close in the embedding space, they can capture long-range semantic temporal dependencies between the initial frame and the goal frame.

(3) Value-Implicit Pre-Training

See the original paper for the specific algorithm.

6.3 Code Experiments

The VIP algorithm uses ResNet50 as its visual backbone and is trained on the Ego4D dataset.

The algorithm is compared with R3M.

6.3 Thoughts

7 Graph-Structured Visual Imitation

Title: Graph-Structured Visual Imitation Research Team: Sony Venue: CoRL Year: 2019 Code: None

7.1 Target Problem

A reward is given when the robot’s actions make the relative spatial configuration of the corresponding visual entities detected in the workspace more closely match the demonstration.

The paper uses a human-finger keypoint detector, an object detector trained offline with synthetic augmentation, and a point detector supervised by viewpoint changes. Multiple visual-entity detectors are learned for each demonstration without human-annotated data or robot interaction.

7.2 Method

(1) Detecting Visual Entities

Human-hand keypoint detection: an existing hand detector is used, and a D435i is used to obtain 3D positions. The robot’s parallel-jaw gripper is mapped to the demonstrator’s thumb and index fingertips. Grasp and release actions are detected by setting a distance threshold between the two fingertips.

Point-feature detector: after training, point features are matched between the imitator’s and demonstrator’s environments to establish correspondences.

Synthetic data augmentation: background removal is used to extract a 2D mask, and synthetic data augmentation is used to train the visual detector.

(2) Motion Saliency for Dynamic Graph Construction

(3) Policy Learning Based on a Graph of Visual Entities

The goal is for the robot to imitate an object-manipulation task from a single human demonstration. See the original paper for the specific cost function.

7.3 Thoughts

Hand keypoints are extracted and mapped to the robotic gripper, while object-keypoint detection is used to generate a motion policy. This idea is not as intuitive as DemoGrasp.

One feasible method:

1. Extract the object pose from the demonstration video
2. Imitate the demonstrated actions
   1. Calculate an error loss between the object’s pose in the current view and its pose in each frame of the demonstration video
   2. Adjust the end-effector pose in real time based on the error
   3. Add keypoints from the arm in the current frame to obtain the poses that each robotic-arm joint should reach
   4. Execute the robotic-arm motion (including obstacle avoidance and trajectory smoothing for the robotic arm)

8 Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos

Title: Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos Research Team: University of Toronto Venue: IROS Year: 2021 Code: http://www.pair.toronto.edu/lbw-kp/

8.1 Target Problem

Learning by watching: an algorithmic framework for policy learning by imitating a single video demonstrating a specified task.

• Because human and robot arms have different morphologies, our framework learns unsupervised human-to-robot translation to overcome the morphological mismatch.
• To capture details in salient regions that are essential for learning the state, our model uses unsupervised keypoint detection. The detected keypoints form a structured representation containing semantically meaningful information and can be used directly for reward calculation and policy learning.

8.2 Method

The proposed LbW framework consists of three components:

• Image-to-image translation network: translates the input human-demonstration video frame by frame to generate a robot-demonstration video
• Keypoint detector: takes the generated robot-demonstration video as input and extracts the keypoints in each frame to form keypoint trajectories
• Policy network: passes the current keypoint-based observation representation to the policy network to predict actions for interacting with the environment

8.3 Thoughts

Rather than an imitation-learning network, this is more like an image-translation network. CycleGAN-based image translation converts a human-hand demonstration into a video of robot actions, after which the robot’s keypoint trajectories are extracted from the video and a policy function is used to produce the physical robot’s actions.

9 Learning Periodic Tasks from Human Demonstrations

Title: Learning Periodic Tasks from Human Demonstrations Research Team: Carnegie Mellon University Venue: ICRA Year: 2022 Code:

9.1 Target Problem

Active learning is used to optimize parameters. The objective is to maximize the similarity between the robot’s motion while manipulating an object and the object’s motion in the demonstration video. The focus is on deformable and granular objects. (Wiping a surface with a cloth, winding a cable, stirring granular material with a spoon, and so on.)

9.2 Method

The framework proposed in this paper consists of two components:

• Representation-learning module: a keypoint-detection model extracts consistent keypoints from independently collected, task-agnostic human and robot data
• Pose-optimization module: generates robot videos that match human demonstrations in terms of the detected keypoints

9.3 Thoughts

Given a human demonstration and a manually controlled robot demonstration, the robot learns the similarity between them and then repeats the demonstrated action so that its result more closely matches the human demonstration.

10 One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks

Title: One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks Research Team: University of California, Berkeley Venue: arXiv Year: 2018 Code: https://sites.google.com/view/one-shot-hil

10.1 Target Problem

Learn multi-stage tasks on a real robot from videos of humans performing those tasks.

10.2 Method

For each subtask, we provide multiple human demonstrations and multiple robot demonstrations. (The objects and tasks performed need to correspond, but the object positions and execution speeds do not need to be the same.)

(1) Composition of Primitives: A human phase predictor and a robot phase predictor are trained to learn a specific robot policy from videos of humans performing the task.

(2) Primitive Phase Prediction: Learn how to segment human demonstrations of compound tasks; when the learning policy excessively to the next one.

10.3 Thoughts

Human-demonstration videos and robot-demonstration videos are provided, and a policy is then trained. Finally, given a human-demonstration video, the robot uses the trained policy to perform the corresponding operation.

11 Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller

Title: Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller Research Team: MIT Venue: NeurIPS Year: 2019 Code: https://pathak22.github.io/hierarchical-imitation/

11.1 Target Problem

Manipulate new objects in unseen scenes by watching human-demonstration videos from a third-person perspective.

11.2 Method

(1) Goal Generator

Infer goals in pixel space from human-demonstration videos and transform them into goals in the robot environment in a pixel-level representation.

This also uses image translation, converting human-demonstration images into robot-demonstration images.

(2) Inverse Controller

Track the cues generated by the visual-goal inference model and produce the actions that the robot should execute.

A ResNet18 model is used.

(3) Third-Person Imitation

Run the goal generator and inverse controller alternately. The goal generator generates subgoals, and the low-level controller generates robot joint angles until the human demonstration ends.

11.3 Thoughts

This still follows the image-translation approach: translate images of human-hand actions into images of robotic-arm actions, then have the controller generate robot joint angles.

12 You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Title: Yodo: Category-Level Manipulation from a Single Visual Demonstration Research Team: Rutgers University Venue: RSS Year: 2022 Code:

12.1 Target Problem

Although recent cross-object category-level manipulation methods have achieved good results, they usually require expensive real-world data collection and manually specified semantic keypoints for each object category and task. In addition, coarse keypoint predictions and the omission of intermediate action sequences prevent their application to complex tasks beyond grasping and prevention.

This work proposes a new manipulation framework. It uses model-free 6D tracking to parse category-level task trajectories from a single demonstration video, and decomposes the entire execution process into three steps: long-range motion, collision-free motion, and last-inch manipulation.

12.2 Method

For each demonstration-video frame, the target pose is tracked through model-free 6D pose estimation. The object pose is represented in the container’s coordinate system, allowing the method to generalize to new scenes.

(1) Offline Learning of a Category-Level Representation

A 9D object representation is established: a 6D pose+3D scaling.

(2) Model-Free 6D Object Tracking

Object-motion tracking serves two purposes:

• During the demonstration stage, parse the recorded video and extract the 6D motion trajectory of the manipulated object in the container’s coordinate system
• During online execution, provide visual feedback to the closed-loop controller

(3) Category-Level Behavior Cloning as the Final-Step Policy

Generate a dense, discrete trajectory so that the robot can follow it to reach the next target.

(4) Dynamic Category-Level Framework Based on Local Attention

Automatically and dynamically normalize the origin of the coordinate system.

(5) Grasping the Object and Moving It Along the Keypoints

Conventional grasping methods.

12.3 Thoughts

Representing the target pose relative to another object’s pose helps the method generalize to different scenes.

The overall idea is to use 6D pose estimation to obtain the target’s motion trajectory and then repeat that trajectory.