Survey of Grasp Pose Estimation Algorithms

Hybrid Physical Metric For 6-DoF Grasp Pose Detection

Title: Hybrid Physical Metric For 6-DoF Grasp Pose Detection Author Team: Tsinghua University (Shengjin Wang) Venue: ICRA Year: 2022 Code: https://github.com/luyh20/FGC-GraspNet

I. Target Problem

A single physical metric produces discrete grasp confidence scores, which leads to inaccurate predictions when training on million-scale grasp data.

This paper defines a new metric based on force closure, object planarity, gravity, and collision measurements.

It designs the planarity-gravity-collision FGC-GraspNet for a multitask, multi-resolution learning framework.

II. Hybrid Physical Metric

(1) Planarity

Grasps on more planar surfaces are more stable. The similarity of the local normals of points is used to calculate the planarity score.

(2) Center-of-Gravity Metric

A grasp is more stable when the gripping force is closer to the object’s center of gravity. The distance from the object’s center of gravity to the line connecting the two contact points is calculated as the gravity score.

(3) Collision-Disturbance Metric

Collisions can easily occur when the gripper approaches an object. Therefore, the minimum Euclidean distance between the gripper’s two maximum-stroke endpoints and the object’s contact points is used as the collision-disturbance score.

(4) Hybrid Physical Metric

The hybrid physical metric is a weighted combination of the metrics above.

III. FGC-GraspNet

Farthest point sampling (FPS) produces a 20000x3 point-cloud input. The network consists of PointNet++, an FA branch, and an RD branch.

• PointNet++ is used to extract point features.
• Low-resolution features enter the FA branch for foreground segmentation and pointwise approaching-direction score regression.
• High-resolution features are used by the RD rotation branch.

IV. Thoughts

Incorporating the hybrid physical metric into the loss calculation is indeed meaningful. It also has some reusability, and other algorithms could draw on this design.

Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter

Title: Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter Author Team: ETH Zurich Venue: CoRL2020 Year: 2020 Code: https://github.com/ethz-asl/vgn

I. Target Problem

This paper proposes a network that obtains scene information from a depth camera and predicts 6D grasps.

II. Method

(1) Network Architecture

A perception module composed of filters and convolutional layers maps the input voxels to a feature map. Convolution and upsampling operations are then applied, followed by three independent branches that predict grasp quality, rotation, and gripper width.

(2) Grasp Detection

Several methods are used to remove impossible grasp poses, after which non-maximum suppression is applied to obtain a list of candidate grasps.

III. Thoughts

This is a very basic method. Others have already extended it and published the results at top conferences.

Efficient Learning of Goal-Oriented Push-Grasping Synergy in Clutter

Title: Efficient Learning of Goal-Oriented Push-Grasping Synergy in Clutter Author Team: Zhejiang University (Rong Xiong) Venue: RAL Year: 2021 Code: https://github.com/xukechun/Efficient_goal-oriented_push-grasping_synergy

I. Target Problem

When grasping objects in cluttered scenes, pre-grasp actions such as pushing are sometimes needed. These actions allow the robotic arm to separate the target object and achieve a stable grasp.

II. Method

Environment setup: A fixed RGBD camera captures the workspace. The RGBD data is projected in the direction of gravity, and each state is represented using a color heightmap and a depth heightmap.

(1) Goal-Oriented Grasp Training

A grasping network conditioned on a target is trained. After sufficient training, the Q-value for successful grasps stabilizes.

(2) Goal-Oriented Push Training

A pushing network conditioned on a target is trained. Its pushing reward function is designed based on reverse training from the grasping network.

(3) Alternating Training

Alternating training is used to resolve the mismatch in object distributions and further improve the performance of the grasping policy in cluttered environments.

III. Thoughts

Pushing an object and then grasping it is equivalent to a two-stage method. It does not need to be trained at the lower level; instead, the high-level planning and decision layer can determine whether to issue a pushing task or a grasping task.

TransGrasp: Grasp Pose Estimation of a Category of Objects by Transferring Grasps from Only One Labeled Instance

Title: TransGrasp: Grasp Pose Estimation of a Category of Objects by Transferring Grasps from Only One Labeled Instance Author Team: Dalian University of Technology (Yi Sun) Venue: ECCV Year: 2022 Code: https://github.com/yanjh97/TransGrasp

I. Target Problem

Most existing methods require large amounts of grasp data for training. To address this problem, this paper predicts grasp poses for a category of objects by labeling only one object.

II. Method

(1) Learning Category Correspondences

1. A neural network composed of Shape Encoder and DIFDecoder is trained to obtain dense correspondences from object deformations to a template.

(2) Grasp Pose Estimation

1. First, the point cloud is transformed from the camera coordinate system to the object coordinate system.
2. The deformation from the object instance to the template is generated.
3. The model with grasp annotations is input into DeformNet to obtain the model’s deformation.
4. The correspondence between the two is obtained through their shared template, and the transformation of the grasp pose is guided by aligning grasp points on the object’s surface.
5. The result is optimized through the refine module.
6. The optimized grasp knowledge is transformed into camera coordinates for grasping.

III. Thoughts

This algorithm can only grasp objects whose shapes are similar to the template, and 1000 grasp poses must first be manually annotated for each category.

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

Title: Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes Author Team: NVIDIA Venue: ICRA Year: 2021 Code: https://github.com/NVlabs/contact_graspnet

1 Target Problem

This paper proposes an end-to-end network that generates a distribution of 6D grasps from image depth data.

2 Method

Using a raw depth map and, optionally, an object mask, the network generates 6D grasp proposals and grasp widths.

(1) Grasp Representation

Most predictable grasps are two-finger grasps, and at least one of the two contact points can be seen before grasping. Therefore, the grasping problem can be simplified to estimating the 3D grasp rotation and grasp width of a parallel-jaw gripper.

Here, a is the approach vector, b is the grasp baseline vector, and d is the distance from the grasp baseline to the grasp base. This representation accelerates the learning process, improves prediction accuracy, and has no ambiguity or discontinuous regions.

(2) Data Generation

The ACRONYM dataset is used. Object meshes with dense grasp annotations are placed in the scene in random stable poses. Grasp poses that cause collisions between the gripper and the model are removed.

(3) Network

The set-abstraction and feature-propagation layers proposed in PointNet++ are used to construct an asymmetric U-shaped network.

The network has four detection heads, each containing two 1D convolutional layers. Each point outputs s∈R, z1∈R3, z2∈R3, and o∈R10, from which the grasp representation is formed.

The grasp width is divided into 10 equally spaced grasp widths to offset the data imbalance problem, after which the grasp-width representation with the highest confidence is selected. Because the approach direction and baseline direction are orthogonal, this property is incorporated into training by orthonormalizing the predictions, which helps regress the 3D rotation.

3 Thoughts

Grasp poses are predefined in the dataset and then used for supervised training. During use, the object’s region is first determined from the depth map, and its point cloud is then used to predict the grasp distribution.

It is difficult to create a dataset for custom objects.

RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images

Title: RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images Author Team: Shanghai Jiao Tong University (Cewu Lu) Venue: ICRA Year: 2021 Code: https://github.com/GouMinghao/RGB_Matters

I. Target Problem

Existing methods either generate grasp poses with very few degrees of freedom or use only unstable depth point clouds as input.

II. Method

(1) Angle-View Net

Angle-View Net predicts pixel-level gripper rotation configurations. Directly regressing quaternions is impractical and not robust. The model below can instead treat gripper rotation prediction as a classification problem.

The final output of AVN is represented as an angle-view heatmap.

(2) Fast Analytical Search

AVN identifies five of the 7 degrees of freedom, but the gripper width and the gripper’s degree of freedom along the axis remain undetermined.

This paper proposes a fast analytical search based on collision and empty-grasp detection to calculate the width and distance.

By sampling from 0 to Wmax, the gripper is assumed to approach the corresponding point in the point cloud reconstructed from the depth map. Two cases are filtered out: points exist in the space occupied by the gripper, or there are no points in the grasping space.

III. Thoughts

This paper uses the simplest possible ideas to solve the grasp prediction problem.

The rotation direction of the end gripper is regressed through a classifier.

The gripper position and width are tested through sampling, eliminating candidates one by one to obtain the optimal solution.

The idea is intuitive and simple, and is worth trying.

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Title: CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation Author Team: Rutgers University Venue: ICRA Year: 2022 Code: https://github.com/wenbowen123/catgrasp

I. Target Problem

This paper proposes a framework for learning to grasp industrial objects without real-world data or manual annotations.

II. Method

Given a database of 3D models from the same category, the method learns:

• An object-centric NUNOCS representation
• hotmap: the likelihood that the hand-object contact region will successfully accomplish the task during grasping
• A codebook of grasp poses

(1) Category-Level Canonical NUNOCS Representation

Different object instances from the same category are transformed into a canonical space and scaled to a standard size.

(2) Stable Grasp Learning

First, given a 9D transformation from the current instance to the canonical model, the same transformation is applied to the grasp to obtain a grasp proposal.

The generated grasps are used to train a PointNet-based network that predicts grasp quality.

(3) Instance Segmentation

A 3D U-Net is used. It takes the point cloud of the entire scene as input and predicts the offset from each point to the object’s center. The offset points are clustered into instance segments.

(4) Generating Training Data in Simulation

PyBullet simulation is used to generate synthetic data.

III. Thoughts

This method proposes training with simulation data to reduce manual annotation. The grasping method is not particularly innovative and still requires multiple predefined instances and grasp poses for each category for training.

Closed-Loop Next-Best-View Planning for Target-Driven Grasping

Title: Closed-Loop Next-Best-View Planning for Target-Driven Grasping Author Team: ETH Zurich Venue: IROS Year: 2022 Code: https://github.com/ethz-asl/active_grasp

I. Target Problem

Grasping objects in densely occluded environments.

II. Method

This method has the following prerequisites:

• A depth camera is attached to the end of the robotic arm.
• The camera’s optical center and the gripper center have been calibrated.
• A partial view and the 3D bounding box of the object are known.

First, the point-cloud observation yt and camera pose xt are inherited and reconstructed into a voxel map. Voxel graspability and possible grasp poses are calculated. If the graspability does not meet the requirements, the robotic arm’s position is adjusted to calculate the next image.

(1) Grasp Detection

The Volumetric Grasping Network VGN is used for grasping. This network maps voxel grid M to grasp-quality score Q, parallel-grasp direction R, and opening width W.

Grasp poses whose fingertips are outside the target bounding box and grasp poses for which no inverse-kinematics solution can be found are filtered out.

(2) Next-Best-View Planner

World Representation: A cubic voxel of size l is represented using TSDF (Truncated Signed Distance Function).

View Generation: Candidate views are generated within the upper hemisphere of the target boundary.

Information Gain: The completeness of TSDF reconstruction greatly affects grasp detection and prediction accuracy. Therefore, a variant of the rear-side voxel IG formula is used, applying ray projection to occluded voxels with negative distances to calculate the number of hidden object voxels.

1. Add a time budget to the maximum number of policy updates.
2. If the grasp score is below a given threshold, stop the algorithm because no useful information can be obtained.
3. Stop if VGN maintains a stable grasp configuration over several frames.

III. Thoughts

This method performs grasp prediction based on pose estimation. It could be combined with Gen6D to obtain the object’s 6D grasp pose.

Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection

Title: Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection Author Team: Northeastern University Venue: ICRA Year: 2023 Code: https://github.com/HaojHuang/Edge-Grasp-Network

I. Target Problem

Given a point cloud observed from a single viewpoint as input, the method produces a set of grasp poses.

II. Method

(1) Cropping the Point Cloud

Given a point cloud p and an approach point pa, only points neighboring approach point pa affect the grasp, so a sphere centered at pa is cropped.

(2) PointNet Convolution

PointNet is used to calculate the features of the approach point and its nearest-neighbor points, as well as the feature of each point.

(3) Calculating Global Features

The pointwise features are passed to an MLP, and a max-pooling layer generates first-level global features. These global features are then concatenated with the point features and passed to a second MLP to calculate global features.

For each grasp, edge features are calculated by concatenating the global features with the point features, and a classifier represents the edge grasp.

(4) Grasp Evaluation

A four-layer MLP with a sigmoid function predicts the grasp success rate. It takes the edge features as input to calculate whether the grasp will succeed.

III. Thoughts

This method is similar to DenseFusion in that it extracts pointwise features and global features and then fuses them. The fused features obtained in this paper are the edge features, which are then passed to a classifier to obtain the grasp pose.