Deep Learning Approaches to Grasp Synthesis: a Review

Authors: Rhys Newbury, Morris Gu, Lachlan Chumbley, Arsalan Mousavian, Clemens Eppner, Jürgen Leitner, Jeannette Bohg, Antonio Morales, Tamim Asfour, Danica Kragic, Dieter Fox, Akansel Cosgun

Abstract This article surveys the literature on 6 degrees of freedom (6-DoF) Grasping using deep learning. We focus our review on robotic grasping in table-top scenarios, where the robot requires all 6 degrees of freedom of the end-effector pose to pick objects from the table successfully. Our review found the following ive approaches most prevalent in literature:sampling based approaches, direct regression, using shape-completion, reinforcement learning or considering semantics. We structure over review around these common methodologies, exploring the current research behind each approach. We report a list of the quantitative metrics commonly used to assess the success of the grasping tasks, while we also review the current object sets, datasets and sensor modalities used in the deep-learning methods. We then discuss the findings of our reviews and make recommendations on the future directions for the field hoping to mitigate some of the current issues which exist in this field.

Grasping Stages

If you find our review useful, please cite us.

@misc{newbury2022review,
    title={Deep Learning Approaches to Grasp Synthesis: A Review}, 
    author={Rhys Newbury and Morris Gu and Lachlan Chumbley and Arsalan Mousavian and Clemens Eppner and Jürgen Leitner and Jeannette Bohg and Antonio Morales and Tamim Asfour and Danica Kragic and Dieter Fox and Akansel Cosgun},
    year={2022},
    eprint={2207.02556},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Publication Under Review at T-RO

Preprint arXiv

Work Authors Method Shape Completion Semantics/Affordances LfD Train Test Gripper Clutter SR Cov Com GP CT P Impact
6-dof graspnet: Variational grasp generation for object manipulation, (2019) Mousavian et al. Sampling       Sim Real Two-Finger Singulated         2.36
Graspnet-1billion: a large-scale benchmark for general object grasping, (2020) Fang et al. Direct Regression       Hybrid Real Two-Finger Structured         2.35
Grasp pose detection in point clouds, (2017) ten Pas et al. Sampling       Sim Real Two-Finger Piled         2.22
High precision grasp pose detection in dense clutter, (2016) Gualtieri et al. Sampling       Sim Real Two-Finger Piled         1.95
6-dof grasping for target-driven object manipulation in clutter, (2020) Murali et al. Sampling       Sim Real Two-Finger Structured         1.94
Shape completion enabled robotic grasping, (2017) Varley et al. Analytical     Sim Real Three-Finger \specialcell{Singulated\Structured}         1.88
Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps, (2020) Wu et al. Direct Regression       Sim Real Two-Finger Singulated         1.69
Detecting object affordances with Convolutional Neural Networks, (2016) Nguyen et al. Analytical     Real Real Multi-Fingered Singulated           1.64
Learning Object Grasping for Soft Robot Hands, (2018) Choi et al. Direct Regression       Real Real Soft Hand Singulated         1.60
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes, (2021) Sundermeyer et al. Direct Regression       Sim Real Two-Finger Structured         1.60
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations, (2020) Song et al. RL     Real Real Two-Finger Piled           1.58
Learning Continuous 3D Reconstructions for Geometrically Aware Grasping, (2020) Van der Merwe et al. Sampling       Sim Real Multi-Fingered Singulated           1.57
Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network, (2019) Lu et al. Sampling       Sim Real \specialcell{Two-Finger\Multi-Fingered} Singulated         1.54
Object-based affordances detection with Convolutional Neural Networks and dense Conditional Random Fields, (2017) Nguyen et al. Analytical       Real* Real Multi-Fingered Piled           1.54
Generating multi-fingered robotic grasps via deep learning, (2015) Varley et al. Sampling       Sim Real Three-Finger Singulated         1.49
S4g: Amodal single-view single-shot se (3) grasp detection in cluttered scenes, (2020) Qin et al. Direct Regression       Sim Real Two-Finger Piled         1.45
UniGrasp: Learning a Unified Model to Grasp With Multifingered Robotic Hands, (2020) Shao et al. Direct Regression       Sim Real Multi-Fingered Singulated           1.45
kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion, (2019) Gao & Tedrake Analytical   Sim Real Two-Finger Singulated           1.43
Regnet: region-based grasp network for single-shot grasp detection in point clouds, (2020) Zhao et al. Direct Regression       Sim Real Two-Finger Piled         1.42
Beyond top-grasps through scene completion, (2020) Lundell et al. Sampling     Sim Real Two-Finger Piled       1.41
Learning Task-Oriented Grasping From Human Activity Datasets, (2020) Kokic et al. Direct Regression     Sim Real Two-Finger Singulated           1.41
Graspness Discovery in Clutters for Fast and Accurate Grasp Detection, (2021) Wang et al. Direct Regression       Hybrid Real Two-Finger Piled         1.40
Learning to Generate 6-DoF Grasp Poses with Reachability Awareness, (2020) Lou et al. Sampling       Sim Real Two-Finger Piled         1.38
Robust Grasp Planning Over Uncertain Shape Completions, (2019) Lundell et al. Analytical     Sim Real Three-Finger Singulated         1.37
RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images, (2021) Gou et al. Direct Regression       Hybrid Real Two-Finger Structured         1.36
Multi-FinGAN: Generative Coarse-To-Fine Sampling of Multi-Finger Grasps, (2020) Lundell et al. Sampling     Sim Real Three-Finger Singulated         1.36
Collision-Aware Target-Driven Object Grasping in Constrained Environments, (2021) Lou et al. Sampling       Sim Real Two-Finger Structured           1.34
Transferable Active Grasping and Real Embodied Dataset, (2020) Chen et al. RL     Sim Real Two-Finger Piled           1.34
Hierarchical 6-DoF Grasping with Approaching Direction Selection, (2020) Choi et al. Direct Regression       Sim Real Two-Finger \specialcell{Structured\Piled}         1.32
6-DoF Contrastive Grasp Proposal Network, (2021) Zhu et al. Direct Regression       Sim Real Two-Finger Singulated         1.32
Robot Learning of 6 DoF Grasping using Model-based Adaptive Primitives, (2021) Berscheid et al. RL       Real Real Two-Finger Piled         1.32
GPR: Grasp Pose Refinement Network for Cluttered Scenes, (2021) Wei et al. Direct Regression       Sim Real Two-Finger Piled       1.32
Generating Grasp Poses for a High-DOF Gripper Using Neural Networks, (2019) Liu et al. Direct Regression       Sim Real Multi-Fingered Singulated             1.31
PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds, (2020) Ni et al. Direct Regression       Sim Real Two-Finger Piled       1.30
Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes, (2019) Wu et al. RL       Sim Real Three-Finger Piled           1.30
Multi-Fingered Active Grasp Learning, (2020) Lu et al. Active Learning       Sim Real Multi-Fingered Singulated           1.28
Generative Attention Learning: a “GenerAL” framework for high-performance multi-fingered grasping in clutter, (2020) Wu et al. RL       Sim Real \specialcell{Two-Finger\Multi-Fingered} \specialcell{Singulated\Piled}           1.28
Learning Grasp Affordance Reasoning Through Semantic Relations, (2019) Ardón et al. Direct Regression     Sim Real Two-Finger Singulated           1.28
Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion, (2021) Gualtieri & Platt Analytical     Sim Real Two-Finger Structured         1.26
DDGC: Generative Deep Dexterous Grasping in Clutter, (2021) Lundell et al. Sampling     Sim Real Three-Finger Structured         1.26
Learning 6-dof grasping and pick-place using attention focus, (2018) Gualtieri & Platt RL       Sim Real Two-Finger Structured           1.26
A Geometric Approach for Grasping Unknown Objects With Multifingered Hands, (2020) Kiatos et al. Analytical     Sim Real Three-Finger Structured       1.24
Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter, (2020) Breyer et al. Direct Regression       Sim Real Two-Finger Piled       1.23
Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a Multi-Armed Bandit model with correlated rewards, (2016) Mahler et al. Sampling\Exemplar   Sim Sim Two-Finger Singulated             1.22
Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation, (2021) Li et al. Direct Regression     Sim Real Two-Finger Structured       1.19
Learning to Detect Multi-Modal Grasps for Dexterous Grasping in Dense Clutter, (2021) Corsaro et al. Sampling       Sim Real Multi-Fingered Structured         1.19
Multifingered grasp planning via inference in deep neural networks: Outperforming sampling by learning differentiable models, (2020) Lu et al. Direct Regression       Sim Real Multi-Fingered Singulated           1.18
DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-Based Robotic Grasping, (2020) Patten et al. Other       Sim Real Two-Finger Singulated         1.18
Deep Differentiable Grasp Planner for High-DOF Grippers, (2020) Liu et al. Direct Regression       Sim Real Multi-Fingered             1.14
Visuo-Haptic Grasping of Unknown Objects based on Gaussian Process Implicit Surfaces and Deep Learning, (2019) Ottenhaus et al. Sampling     Sim Real Multi-Fingered Singulated           1.13
A self-supervised learning-based 6-DOF grasp planning method for manipulator, (2021) Peng et al. Sampling       Sim Real Two-Finger Structured       1.10
Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations, (2021) Jiang et al. Sampling     Sim Real Two-Finger Piled         1.08
Learning an end-to-end spatial grasp generation and refinement algorithm from simulation, (2021) Ni et al. Direct Regression       Sim Real Two-Finger Structured         1.07
Grasp Planning with Incomplete Knowledge About the Object to be Grasped, (2019) Gonçalves & Lima Sampling       Sim Real Three-Finger Singulated           1.07
Imitation Learning based Soft Robotic Grasping Control without Precise Estimation of Target Posture, (2021) Diaz Cortes et al. RL     Real Real Multi-Fingered Structured           1.06
MVGrasp: Real-Time Multi-View 3D Object Grasping in Highly Cluttered Environments, (2021) Kasaei & Kasaei Direct Regression       Real Real Two-Finger \specialcell{Singulated\Piled}       1.04
Deep dexterous grasping of novel objects from a single view, (2019) Aktas et al. Sampling     Sim Real Multi-Fingered Singulated         1.03
6-DOF Grasp Detection for Unknown Objects, (2020) Schaub & Schöttl Sampling       Real Real Two-Finger Singulated             1.03
Robust grasp detection with incomplete point cloud and complex background, (2021) Wang et al. Sampling       Hybrid Real Two-Finger Structured       1.02
Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds, (2021) Wang et al. RL     Sim Real Two-Finger Singulated         1.02
Q-PointNet: Intelligent Stacked-Objects Grasping Using a RGBD Sensor and a Dexterous Hand, (2020) Wang & Lin Direct Regression       Sim Real \specialcell{Two-Finger\Multi-Fingered} \specialcell{Structured\Piled}           1.01
6-DOF grasp planning of manipulator combined with self-supervised learning, (2021) Ren et al. Sampling       Sim Real Two-Finger Structured       1.01
Grasping of Unknown Objects Using Deep Convolutional Neural Networks Based on Depth Images, (2018) Schmidt et al. Direct Regression       Sim Demo Multi-Fingered Singulated             1.01
An End-to-End Spatial Grasp Prediction Model for Humanoid Multi-fingered Hand Using Deep Network, (2021) Li et al. Direct Regression       Sim Real Multi-Fingered Singulated           1.01
PointNetGPD: Detecting Grasp Configurations from Point Sets, (2019) Liang et al. Sampling       Sim Sim Three-Finger Piled           1.00
CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation, (2021) Wen et al. Sampling     Sim Real Two-Finger Piled           1.00
6-DOF Grasp Detection for Unknown Objects Using Surface Reconstruction, (2021) Schaub et al. Sampling       Real Two-Finger Singulated           1.00
Object Shell Reconstruction: Camera-centric Object Representation for Robotic Grasping, (2021) Chavan-Dafle et al. Analytical     Sim Real Two-Finger Structured         1.00
Leveraging big data for grasp planning, (2015) Kappler et al. Sampling       Sim Sim Three-Finger Sigulated             0.91
Learning Collaborative Pushing and Grasping Policies in Dense Clutter, (2021) Tang et al. RL       Sim Demo Two-Finger Piled         0.82
Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping, (2020) Murali et al. Sampling       Sim Demo Two-Finger Singulated           0.64
Learning 6-DOF Grasping Interaction via Deep Geometry-Aware 3D Representations, (2018) Yan et al. Sampling   Sim Sim Two-Finger Singulated         0.59
Leveraging Contact Forces for Learning to Grasp, (2019) Merzic et al. RL       Sim Sim Three-Finger             0.50
Grasping in 6DoF: An Orthographic Approach to Generalized Grasp Affordance Predictions, (n.d.) Munoz Direct Regression       Sim Demo Two-Finger Singulated         0.50
Learning Dexterous Grasping with Object-Centric Visual Affordances, (2021) Mandikal & Grauman RL     Sim Sim Multi-Fingered Singulated           0.37
Robotic Grasping through Combined Image-Based Grasp Proposal and 3D Reconstruction, (2020) Yang et al. Direct Regression     Sim Sim Two-Finger Singulated           0.33
Modeling Grasp Motor Imagery Through Deep Conditional Generative Models, (2017) Veres et al. Sampling       Sim Sim Three-Finger Singulated           0.33
Point Cloud Projective Analysis for Part-Based Grasp Planning, (2020) Monica & Aleotti GPD     - -                 0.26
Learning to Grasp 3D Objects using Deep Residual U-Nets, (2020) Li et al. Analytical     Sim Sim Two-Finger Singulated           0.13
GDN: A Coarse-To-Fine (C2F) Representation for End-To-End 6-DoF Grasp Detection, (2020) Jeng et al. Direct Regression       Sim Sim Two-Finger \specialcell{Singulated\Structured}     0.09
Model-less Estimation Method for Robot Grasping Parameters Using 3D Shape Primitive Approximation, (2018) Torii & Hashimoto Analytical     Sim Sim Three-Finger Singulated           0.09
6DOF grasp planning by optimizing a deep learning scoring function, (2017) Zhou & Hauser Sampling       Sim Sim Three-Finger Singulated           0.08
Model-Free Grasp Learning Framework based on Physical Simulation, (2020) Riedlinger et al. Sampling       Sim Sim Two-Finger Singulated           0.03
6-DoF grasp planning using fast 3D reconstruction and Grasp Quality CNN, (2020) Avigal et al. Shape Completion     Sim Sim Two-Finger Singulated           0.02
Learning 6DoF Grasping Using Reward-Consistent Demonstration, (2021) Kawakami et al. RL     Sim Sim Two-Finger Singulated             0.00

Acknowledgments

logo Monash logo NVIDIA logo LYRO logo Stanford logo UIJ logo KIT logo KTH logo University Washington