[Paper Notes] Papers on 6D Pose Estimation at CVPR2020-2022

CVPR 2020

01. HybridPose: 6D Object Pose Estimation under Hybrid Representations

Journal / Conference: CVPR2020 Author / Institution: Chen Song, The University of Texas at Austin Keywords: pose estimation; hybrid features Year: 2020 Code: https://github.com/chensong1995/HybridPose

1 Problem

6D pose estimation

2 Method

The algorithm consists of an intermediate-feature prediction network and a pose regression network:

(1) Prediction module

The image is used as input, and three prediction networks output the predicted keypoints, edge vectors, and symmetry correspondences.

• Keypoints: uses PVNet’s keypoint prediction method
• Edge vectors: the vector between every pair of keypoints
• Symmetry correspondences: extends the FlowNet network by combining pixel flow and semantic masks

(2) Pose regression module

The pose regression network includes an initialization submodule and a refinement submodule.

• Initialization submodule: regresses an initial pose from the intermediate features
• Refinement submodule: uses and optimizes a GM robust norm to obtain the final pose

3 Thoughts

The method is relatively intuitive, using keypoints, keypoint vectors, and symmetry relationships for pose prediction.

However, it is relatively difficult to apply in practice. Before training, FSP must be used to generate keypoint labels, SymSeg must be used to generate symmetry labels, and a segmentation template must also be provided. PVNet’s fusion data is required as well.

---

02. Single-Stage 6D Object Pose Estimation

Journal / Conference: CVPR2020 Author / Institution: Yinlin Hu, CVLab, EPFL, Switzerland Keywords: pose estimation; single stage Year: 2020 Code: https://github.com/cvlab-epfl/single-stage-pose

1 Problem

The paper proposes a single-stage framework to address the drawbacks of two-stage frameworks, which first establish correspondences between 3D object keypoints and a 2D image and then perform regression, thereby accelerating training.

2 Method

After some instance segmentation networks establish the relationship between the 3D object and the 2D image, three main modules directly predict the pose from these correspondence clusters:

• Local feature extraction module

• Feature aggregation module: aggregates features across different clusters

• Global reasoning module: consists of fully connected layers and estimates the final pose as a quaternion and translation

• The paper proposes a novel Projection Distribution representation that models the projections of three-dimensional object keypoints onto the two-dimensional image as probability distributions rather than deterministic locations.

• It designs a Single-Stage 6D Pose Estimation Network (Single-Stage 6D Pose Estimation Network) that uses a convolutional neural network and fully connected layers to predict the projection distribution parameters for each object keypoint in the image.

• It uses Maximum Likelihood Estimation to directly calculate the object’s rotation matrix and translation vector in the camera coordinate system from the predicted projection distribution parameters and the known three-dimensional object model.

3 Thoughts

It appears to require integration with other networks, extracting features from their intermediate layers.

There is relatively little information on Github.

---

03. G2L-Net: Global to Local Network for Real-time 6D Pose Estimation with Embedding Vector Features

Journal / Conference: CVPR2020 Author / Institution: Wei Chen, School of Computer Science, University of Birmingham Keywords: pose estimation Year: 2020 Code: https://github.com/DC1991/G2L_Net

1 Problem

Improve the accuracy and speed of pose estimation algorithms.

2 Method

(1) Global localization

A 2D detector (such as yolo) predicts the object’s bounding box and label. The location with the highest probability in the resulting probability map is used as the center of a sphere (using its 3D coordinate from the depth map), defining a spherical space that reduces the subsequent 3D search space.

(2) Translation localization

3D segmentation and translation residual prediction are performed, and the coordinate system of the object points is transformed into a local canonical coordinate system.

(3) Rotation localization

A pointwise embedding vector feature extractor extracts embedding vector features, which are then fed into a decoder to regress the rotation of the input point cloud.

3 Thoughts

This effectively replaces DenseFusion’s prior instance segmentation step, using yolo + point cloud segmentation instead. The final features are still pointwise features.

CVPR2021

01. GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation

Journal / Conference: CVPR2021 Author / Institution: Gu Wang, Tsinghua University, BNRist Keywords: pose estimation; end-to-end Year: 2021 Code: https://github.com/THU-DA-6D-Pose-Group/GDR-Net

1 Problem

The paper proposes an end-to-end pose estimation algorithm.

2 Method

(1) Network architecture

First, GDR-Net is given a 256x256 ROI image and predicts three 64x64 intermediate feature maps:

• Dense correspondence map $M_{2D-3D}$: obtained by mapping the dense coordinate map $M_{XYZ}$ onto the 2D pixel coordinates, reflecting the object’s geometric shape information.
• Surface region attention map $M_{SRA}$: derived from $M_{XYZ}$ using farthest point sampling and representing the object’s symmetry.
• Visible object mask $M_{vis}$

A simple 2D convolutional Patch Pnp module directly regresses the 6D object pose from the feature maps. The Patch PnP module consists of three convolutional layers, followed by two fully connected layers for flattening the features, and finally fully connected layers that output the R6D rotation and tSITE translation.

3 Thoughts

This paper focuses on image feature extraction and processing, enabling pose prediction from a single image. The method is not sufficiently intuitive.

02. FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanism

Journal / Conference: CVPR2021 Author / Institution: Wei Chen, School of Computer Science, University of Birmingham Keywords: pose estimation Year: 2021 Code: https://github.com/DC1991/FS_Net

1 Problem

Address the low efficiency, low accuracy, and slow inference of feature extraction in previous category-level pose estimation methods.

2 Method

The paper designs an orientation-aware autoencoder with 3D graph convolution for latent feature extraction.

It proposes a decoupled rotation mechanism that uses two decoders to access complementary rotation information.

It uses two residuals to estimate translation.

It proposes an online box-cage three-dimensional deformation mechanism to augment the training data.

1. Take an RGB image as input.
2. Use yolov3 to detect the object’s 2D location, category label, and class probability map, and use the location with the highest probability as the center of a 3D sphere. This yields the target’s 3D spherical point cloud.
3. Use the three-dimensional deformation mechanism for data augmentation.
4. Use the shape-based 3DGC autoencoder for point cloud segmentation and latent feature learning for rotation. 3DGC consists of m unit vectors, and the convolution value is the sum of cosine similarities between the kernel vector and the n nearest vectors.
5. Decode the rotation information from the latent features into two perpendicular vectors.
6. Use a residual estimation network to predict translation.

3 Thoughts

The proposed use of a three-dimensional deformation mechanism for data augmentation is very interesting. This step might be added to many subsequent methods to make their algorithms more robust.

Both the yolo model and the FS_Net model need to be trained.

NOCS dataset

CVPR2022

01. OVE6D: Object Viewpoint Encoding for Depth-based 6D Object Pose Estimation

Journal / Conference: CVPR2022 Author / Institution: Dingding Cai, Tampere University Keywords: pose estimation Year: 2022 Code: https://github.com/dingdingcai/OVE6D-pose

1 Problem

Given an object’s segmentation mask and three-dimensional mesh model, predict the R+T transformation from the object coordinate system to the camera coordinate system.

2 Method

(1) Training stage

3D object models from ShapeNet are used to train the OVE6D model. This stage is performed only once, and the resulting model parameters remain fixed afterward.

(2) Encoding stage

The target object’s 3D network model is converted into a viewpoint codebook. This stage is performed only once for each object. (A viewpoint codebook is a collection of feature vectors.)

(3) Inference stage

The object’s 6D pose is inferred from the input object depth image and object segmentation mask.

1. Viewpoint estimation: use the input image and object ID as input, perform cosine similarity matching against the feature vectors in the viewpoint codebook to find the closest predefined viewpoint, and output its index and confidence.
2. In-plane rotation estimation: given the image, ID, predefined viewpoint index, and confidence, use a convolutional neural network to regress the rotation relative to the camera coordinate system.
3. Translation estimation: given the image, ID, predefined viewpoint index, confidence, and in-plane rotation angle, use another convolutional neural network to output the object’s 3D position.

3 Thoughts

The algorithm requires pretraining on ShapeNet and then defining a viewpoint codebook. The process is relatively complex and not sufficiently concise or intuitive.

02. OnePose: One-Shot Object Pose Estimation without CAD Models

Journal / Conference: CVPR2022 Author / Institution: Jiaming Sun, Zhejiang University Keywords: pose estimation Year: 2022 Code: https://github.com/zju3dv/OnePose

1 Problem

Achieve pose estimation without relying on CAD models.

2 Method

Drawing on ideas from visual localization, the method requires only a simple RGB video scan of the object to build a sparse SfM model. A general-purpose feature matching network then aligns this model with a new query image to obtain the object pose.

The paper proposes a new graph attention network (GATs) that aggregates 2D features corresponding to the same SfM point into 3D features and matches them against 2D features in the query image using self-attention and cross-attention.

1. For each object, use a video scan to obtain a set of camera poses and the object’s 3D bounding box.
2. Use SFM to reconstruct a sparse point cloud model.
3. Establish the SfM 2D-3D correspondences.
4. Use an attention aggregation layer to aggregate 2D descriptors into 3D descriptors.
5. Use PnP regression to calculate the object pose.

The overall implementation process is as follows

• Use some AR tools to capture object data, including the object’s center position, dimensions, rotation angle around the Z-axis, and camera pose.
• Extract images from the captured video, reconstruct a sparse point cloud using SfM, and extract 2D keypoints and descriptors from all correspondence images.
• During localization, capture a series of images in real time, extract and match 2D keypoints and descriptors, and query candidate images from the database to find the camera pose.

3 Thoughts

The basic idea is to create a database containing 2D images, the reconstructed point cloud, and the corresponding 2D-3D keypoints and descriptors, then extract features from each input image and perform matching and retrieval.

03. Focal Length and Object Pose Estimation via Render and Compare

Journal / Conference: CVPR2022 Author / Institution: Georgy Ponimatkin, LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS Keywords: pose estimation Year: 2022 Code: https://ponimatkin.github.io/focalpose

1 Problem

Estimate the 6D pose of an object in a photo whose camera parameters are unknown.

2 Method

1. Select the 3D model from a 3D model library that best matches the object in the input image.
2. Use a CNN encoder to encode the input image into a feature vector.
3. Use a CNN decoder to decode the feature vector into an initial 6D pose and focal length.
4. Use a rendering engine to render a virtual view according to the initial 6D pose and focal length, and compare it with the input image.
5. Use a loss function to calculate the difference between the virtual view and the input image, and update the 6D pose and focal length through backpropagation.
6. Repeat steps 4 and 5 until convergence or the maximum number of iterations is reached.

3 Thoughts

For pose estimation of objects in online images, I am not sure what the practical application scenario is.

04. ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework

Journal / Conference: CVPR2022 Author / Institution: Ningkai Mo, ShenZhen Key Lab of Computer Vision and Pattern Recognition Keywords: pose estimation; symmetry Year: 2022 Code: https://github.com/GANWANSHUI/ES6D

1 Problem

The paper mainly addresses how to use RGB-D data to estimate the 6D pose of a rigid object, especially a symmetric object.

2 Method

• It designs a fully convolutional feature extraction network called XYZNet, which can efficiently extract point cloud features from RGB and depth images and fuse features from different modalities.
• It proposes a new shape representation called grouped primitives (GP), which depends only on the object’s symmetry and ignores shape details.
• Based on GP, it designs a new pose distance metric called average (maximum) grouped primitive distance, or A(M)GPD. This metric can be used as a loss function to train the regression network and ensure that the network converges to the correct pose.

1. Generate RGB-XYZ data from an RGB-D image. The RGB-XYZ data is fed into a CNN module to extract local features that encode color and geometric information.
2. Point cloud features are obtained through a PointNet-like CNN module and padded to the same size as the local features.
3. Concatenate the local features and point cloud features into pointwise features for pose estimation.
4. Select the pose with the highest confidence as the final result.

3 Thoughts

This paper also uses pointwise features. It proposes XYZNet, which can extract point cloud and RGB features more efficiently without requiring a mask image.

The code only provides a method for the T-LESS dataset.

05. GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting

Journal / Conference: CVPR2022 Author / Institution: YanDi, Technical University of Munich Keywords: pose estimation Year: 2022 Code: https://github.com/lolrudy/GPV_Pose

1 Problem

The paper mainly addresses the uncertainty and instability of existing methods when handling previously unseen object instances.

2 Method

1. Given an RGB-D image, first use a method such as Maks-RCNN to segment the object from the depth image.
2. Then sample 1028 points from the three-dimensional depth point cloud and feed them into the GPV-Pose pose estimation network.
3. Because 3DGC is insensitive to point cloud translation and scaling, use 3DGC as the backbone to extract global and per-point features, with three additional parallel branches for pose prediction, symmetry, and per-point bounding boxes.

Note:

• The 3DGC method first converts the input point cloud into a k-nearest-neighbor graph (kNN graph), in which each point is connected to its k nearest neighbors. It uses multiple layers of Graph Convolution to extract the local features of each point and Max Pooling to extract global features. The 3DGC method concatenates the global and per-point features to form a hybrid feature vector for the subsequent pose estimation, symmetry-aware reconstruction, and point voting modules.

3 Thoughts

(1) Innovations

• Introduces a decoupled, confidence-driven rotation representation that allows the corresponding rotation matrix to be recovered with geometric awareness
• Proposes a point-voting-based displacement estimation module that uses geometric constraints to generate reliable and accurate displacement predictions
• Integrates these two modules into an end-to-end trainable network and optimizes it with a multitask loss function

(2) Comparison with DenseFusion

Both use a 3D graph convolutional network (3DGC) to extract the local features of each point from the input point cloud and concatenate them with the global features to form a hybrid feature vector.

GPV-Pose uses a decoupled, confidence-driven rotation representation that can recover the rotation matrix through geometric relationships, whereas DenseFusion directly predicts a quaternion.

Uses the NOCS dataset.

06. DGECN: A Depth-Guided Edge Convolutional Network for End-to-End 6D Pose Estimation

Journal / Conference: CVPR2022 Author / Institution: Tuo Cao, School of Computer Science, Wuhan University, Wuhan, Hubei, China Keywords: pose estimation Year: 2022 Code: https://github.com/maplect/DGECN_CVPR2022

1 Problem

Perform pose estimation from a monocular RGB image.

2 Method

(1) Depth refinement network DRN

Two different depth estimation networks output depth maps DA and DB, respectively. The difference between the two depth maps is calculated, and regions where the difference exceeds a threshold are defined as uncertain regions.

(2) Feature extraction

• Depth estimation: use the color image as input and predict a depth map
• Object segmentation: use the segmentation mask to convert the depth map into a 3D point cloud, then use a 3D feature extractor to extract geometric features

(3) 2D keypoint localization

Use the farthest point sampling (FPS) algorithm to select keypoints on the object’s surface.

(4) Learning the 6D pose from 2D-3D correspondences

Use the dynamic graph PnP (DG-PnP) algorithm to construct a graph structure through edge convolution and directly learn the 6D pose using the topological information in the 2D-3D correspondences.

3 Thoughts

(1) Innovations

• Uses a depth-guided network to predict segmentation and a depth map simultaneously, and uses a depth refinement network (DRN) to improve the quality of the depth map
• Establishes 2D-3D correspondences from the segmentation and depth map, matching keypoints in the image with points on the 3D model
• Proposes a dynamic graph PnP (DG-PnP) algorithm that constructs a graph structure through edge convolution and directly learns the 6D pose using the topological information in the 2D-3D correspondences

(2) Practicality

Performs pose estimation from a monocular RGB image by regressing a depth map with a network.

07. Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions)

Journal / Conference: CVPR2022 Author / Institution: Van Nguyen Nguyen, LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France Keywords: pose estimation Year: 2022 Code: https://github.com/nv-nguyen/template-pose Dataset: LINEMOD

1 Problem

The paper proposes a method that requires only an object’s CAD model to match the input object against a set of templates, allowing its 3D pose to be estimated even under partial occlusion.

2 Method

During training, pairs consisting of real images and synthetic templates are used to calculate local features and predict the similarity between the two images.

For an unseen image, its local features are then calculated, and the image is matched against a template database to retrieve the object pose.

3 Thoughts

The codebook approach is highly practical and worth trying.

08. Coupled Iterative Refinement for 6D Multi-Object Pose Estimation

Journal / Conference: CVPR2022 Author / Institution: Lahav Lipson, Princeton University Keywords: pose estimation; iterative refinement Year: 2022 Code: https://github.com/princeton-vl/Coupled-Iterative-Refinement Dataset: LINEMOD

1 Problem

Given a set of known RGBD inputs, detect the 6D pose of each object.

2 Method

The algorithm is complex and uses iterative refinement.

3 Thoughts

The code produces the best results, but it is relatively complex.