[Grasp Pose Estimation Algorithm] RGB Matters Paper Notes and Reproduction

I. Paper Notes

RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images

Title: RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images Author Team: Shanghai Jiao Tong University (Cewu Lu) Conference: ICRA Year: 2021 Code: https://github.com/GouMinghao/RGB_Matters

1.1 Problem

Existing methods either generate grasp poses with very few degrees of freedom or use only unstable depth point clouds as input.

1.2 Method

The general workflow is:

1. Use Angle-View Net to generate gripper orientations at different positions in the image, $P_{img}=(u,v,r_x,r_y,r_z,c)$, including the coordinate position in the image, the corresponding gripper rotation pose, and the confidence.
2. For high-confidence predictions, combine them with the depth map to calculate the distance and gripper width$P_{cam}=x,y,z,rx,ry,rz,w$

(0) Definition

A grasp pose is defined as (x, y, z, rx, ry, rz, w), where (x, y, z) represents the gripper position, (rx, ry, rz) represents the gripper rotation, and w represents the gripper width.

This paper considers only parallel-jaw grippers, defined by (h, l, wmax). The three parameters represent the gripper’s height, length, and maximum width, respectively.

(1) Angle-View Net

It predicts pixel-level gripper rotation configurations. Directly regressing quaternions is impractical and not robust, because more than one feasible rotation can grasp at the same position.

The model below can be used to decouple the orientation into an approach direction and an in-plane rotation. Gripper rotation prediction is then treated as a classification problem with VxA orientation classes in total.

The network rasterizes the RGB image. For each grid cell, AVN predicts a 1D vector with VxA elements containing the confidence for each orientation. This ultimately produces a (VxA)xGHxGW tensor. AVN’s final output is represented as a heatmap for each angle.

The test configuration provided by the authors in the code is V=60,A=6.

(2) Fast Analytic Searching

AVN identifies five of the 7 degrees of freedom, but the gripper width and its degree of freedom along the axis have not yet been determined.

The paper proposes Fast Analytic Searching based on collision and empty-grasp detection to calculate the width and distance.

By sampling from 0 to Wmax, it assumes that the gripper approaches the corresponding point in the point cloud reconstructed from the depth map. It filters out two cases: when points exist in the space occupied by the gripper, and when there are no points in the grasping space.

1.3 Thoughts

This paper uses the simplest possible approach to solve the grasp prediction problem.

The end-effector gripper’s rotation direction is regressed using a classifier.

The optimal gripper position and width are obtained by eliminating candidates one by one through sampling tests.

The idea is straightforward and simple, and is worth trying.

II. Reproduction Process

2.1 Environment Setup

Create a virtual environment:

conda create -n rgb_matters python=3.7
conda activate rgb_matters

Download the code:

git clone https://github.com/GouMinghao/rgb_matters
cd rgb_matters

Install PyTorch1.8.0:

conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge

Install the dependencies:

python3 -m pip install -r requirements.txt

2.2 Test the Demo

Download the models trained by the authors: Google Drive

Create a weights directory in the code directory, then place the downloaded models in it. The resulting directory structure is as follows:

rgbd_graspnet/
├── check_label_integrity.py
├── train.py
├── train.sh
├── vis_label.py

...

└── weights
    ├── kn_jitter_79200.pth
    ├── kn_no_norm_76800.pth
    ├── kn_norm_63200.pth
    ├── kn_norm_only_73600.pth
    └── rs_norm_56400.pth

III. Code Analysis

3.1 Output Analysis

(1) Obtaining the Heatmaps

Use the following code to predict the heatmaps:

net = RGBNormalNet(num_layers=args.num_layers, use_normal=args.use_normal, normal_only=args.normal_only)
state_dict = torch.load(weights_path)
net.load_state_dict(state_dict["net"], strict=False)
net = net.to(device)
net.eval()

rgb, _ = load_data(rgb_path, depth_path)

rgb = rgb.unsqueeze(0).to(device)

prob_map = net(rgb)

Here, prob_mat is the predicted heatmap with shape=(batch_size, 360, h, w). There are 360 heatmaps in total (the 360 heatmaps include the approach direction v=60 and the in-plane rotation A=6).

(2) Obtaining the Gripper Pose

Use the convert_grasp() function to extract gripper poses from the 360 heatmaps.

A gripper pose is a Grasp instance that includes the following parameters:

• score: float type, grasp score
• width: float type, gripper width
• height: float type, gripper height
• depth: float type, gripper depth
• rotation_matrix: shape(3, 3) array, rotation matrix
• translation: shape(3) array, translation vector
• object_id: int type, class of the grasped object

See the following two figures for details: