In-Hand Gravitational Pivoting Using Tactile Sensing

Authors: Jason Toskov, Rhys Newbury, Mustafa Mukadam, Dana Kulić, Akansel Cosgun

Abstract We study gravitational pivoting, a constrained version of in-hand manipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To achieve this, instead of controlling the gripper to avoid slip, we embrace slip to allow the object to rotate in-hand. We collect two real-world datasets, a static tracking dataset and a controller-in-the-loop dataset, both annotated with object angle and angular velocity labels. Both datasets contain force-based tactile information on ten different household objects. We train an LSTM model to predict the angular position and velocity of the held object from purely tactile data. We integrate this model with a controller that opens and closes the gripper allowing the object to rotate to desired relative angles. We conduct real-world experiments where the robot is tasked to achieve a relative target angle. We show that our approach outperforms a sliding-window based MLP in a zero-shot generalization setting with unseen objects. Furthermore, we show a 16.6% improvement in performance when the LSTM model is fine-tuned on a small set of data collected with both the LSTM model and the controller in-the-loop.

If you find our work useful, please cite us.

@inproceedings{toskov2022inhand,
  title={In-Hand Gravitational Pivoting Using Tactile Sensing},
  author={Jason Toskov and Rhys Newbury and Mustafa Mukadam and Dana Kulic and Akansel Cosgun},
  booktitle={6th Annual Conference on Robot Learning},
  year={2022},
  url={https://openreview.net/forum?id=NEGjAH7p0fm}
}

Video

Publication CoRL 2022

Preprint arXiv

Code Github

Dataset Dataset

In-Hand Gravitational Pivoting Using Tactile Sensing

Acknowledgments