Rhys Newbury | publications

2024

POSGGym: A Library for Decision-Theoretic Planning and Learning in Partially Observable, Multi-Agent Environments

Schwartz, Jonathon, Newbury, Rhys, Kulic, Dana, and Kurniawati, Hanna

2024

Abs

Seamless integration of Planning Under Uncertainty and Reinforcement Learning (RL) promises to bring the best of both model-driven and data-driven worlds to multi-agent decisionmaking, resulting in an approach with assurances on performance that scales well to more complex problems. Despite this potential, progress in developing such methods has been hindered by the lack of adequate evaluation and simulation platforms. Researchers have had to rely on creating custom environments, which reduces efficiency and makes comparing new methods difficult. In this paper, we introduce POSGGym: a library for facilitating planning and RL research in partially observable, multi-agent domains. It provides a diverse collection of discrete and continuous environments, complete with their dynamics models and a reference set of policies that can be used to evaluate generalization to novel partners. Leveraging POSGGym, we empirically investigate existing state-of-the-art planning methods and a method that combines planning and RL in the type-based reasoning setting. Our experiments corroborate that combining planning and RL can yield superior performance compared to planning or RL alone, given the model of the environment and other agents is correct. However, our particular setup also reveals that this integrated approach could result in worse performance when the model of other agents is incorrect. Our findings indicate the benefit of integrating planning and RL in partially observable, multi-agent domains, while serving to highlight several important directions for future research. Code available at: https://github.com/RDLLab/posggym.
A Review of Differentiable Simulators

Newbury, Rhys, Collins, Jack, He, Kerry, Pan, Jiahe, Posner, Ingmar, Howard, David, and Cosgun, Akansel

IEEE Access 2024

Abs

Differentiable simulators continue to push the state of the art across a range of domains including computational physics, robotics, and machine learning. Their main value is the ability to compute gradients of physical processes, which allows differentiable simulators to be readily integrated into commonly employed gradient-based optimization schemes. To achieve this, a number of design decisions need to be considered representing trade-offs in versatility, computational speed, and accuracy of the gradients obtained. This paper presents an in-depth review of the evolving landscape of differentiable physics simulators. We introduce the foundations and core components of differentiable simulators alongside common design choices. This is followed by a practical guide and overview of open-source differentiable simulators that have been used across past research. Finally, we review and contextualize prominent applications of differentiable simulation. By offering a comprehensive review of the current state-of-the-art in differentiable simulation, this work aims to serve as a resource for researchers and practitioners looking to understand and integrate differentiable physics within their research.We conclude by highlighting current limitations as well as providing insights into future directions for the field.

2023

Deep Learning Approaches to Grasp Synthesis: A Review

Newbury, Rhys, Gu, Morris, Chumbley, Lachlan, Mousavian, Arsalan, Eppner, Clemens, Leitner, Jürgen, Bohg, Jeannette, Morales, Antonio, Asfour, Tamim, Kragic, Danica, Fox, Dieter, and Cosgun, Akansel

IEEE Transactions on Robotics 2023

Abs

This article surveys the literature on 6 degrees of freedom (6-DoF) Grasping using deep learning. We focus our review on robotic grasping in table-top scenarios, where the robot requires all 6 degrees of freedom of the end-effector pose to pick objects from the table successfully. Our review found the following ive approaches most prevalent in literature:sampling based approaches, direct regression, using shape-completion, reinforcement learning or considering semantics. We structure over review around these common methodologies, exploring the current research behind each approach. We report a list of the quantitative metrics commonly used to assess the success of the grasping tasks, while we also review the current object sets, datasets and sensor modalities used in the deep-learning methods. We then discuss the findings of our reviews and make recommendations on the future directions for the field hoping to mitigate some of the current issues which exist in this field.
In-Hand Gravitational Pivoting Using Tactile Sensing

Toskov, Jason, Newbury, Rhys, Mukadam, Mustafa, Kulic, Dana, and Cosgun, Akansel

In Proceedings of The 6th Conference on Robot Learning 2023

Abs

We study gravitational pivoting, a constrained version of in-hand manipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To achieve this, instead of controlling the gripper to avoid slip, we \emphembrace slip to allow the object to rotate in-hand. We collect two real-world datasets, a static tracking dataset and a controller-in-the-loop dataset, both annotated with object angle and angular velocity labels. Both datasets contain force-based tactile information on ten different household objects. We train an LSTM model to predict the angular position and velocity of the held object from purely tactile data. We integrate this model with a controller that opens and closes the gripper allowing the object to rotate to desired relative angles. We conduct real-world experiments where the robot is tasked to achieve a relative target angle. We show that our approach outperforms a sliding-window based MLP in a zero-shot generalization setting with unseen objects. Furthermore, we show a 16.6% improvement in performance when the LSTM model is fine-tuned on a small set of data collected with both the LSTM model and the controller in-the-loop. Code and videos are available at https://rhys-newbury.github.io/projects/pivoting/.
HOB-CNN: Hallucination of occluded branches with a convolutional neural network for 2D fruit trees

Chen, Zijue, Granland, Keenan, Newbury, Rhys, and Chen, Chao

Smart Agricultural Technology 2023

Abs

Orchard automation has attracted the attention of researchers recently due to the shortage of global labor force. To automate tasks in orchards such as pruning, thinning, and harvesting, a detailed understanding of the tree structure is required. However, occlusions from foliage and fruits can make it challenging to predict the position of occluded trunks and branches. This work proposes a regression-based deep learning model, Hallucination of Occluded Branch Convolutional Neural Network (HOB-CNN), for tree branch position prediction in varying occluded conditions. We formulate tree branch position prediction as a regression problem towards the horizontal locations of the branch along the vertical direction or vice versa. We present comparative experiments on Y-shaped trees with two state-of-the-art baselines, representing common approaches to the problem. Experiments show that HOB-CNN outperform the baselines at predicting branch position and shows robustness against varying levels of occlusion. We further validated HOB-CNN against two different types of 2D trees, and HOB-CNN shows generalization across different trees and robustness under different occluded conditions.

2022

Predicting Coherent Branches for Occluded Fruit Trees

CHEN, ZIJUE, Granland, Keenan, Newbury, Rhys, and Chen, Chao

SSRN Electronic Journal 2022

Abs

A coherent (noise-free and fully connected) branch prediction for fruit trees is the prerequisite for 3D reconstruction and is essential to guide robots in avoiding a collision. In this work, we propose a novel one-step regression deep-learning model, HOB-CNNv2, to detect coherent branches for fruit trees under varying levels of occlusions. Furthermore, we develop a post-processing algorithm, Coherent Semantic Merging Algorithm (CSMA), to take advantage of HOB-CNNv2’s coherent guarantee and a state-of-the-art semantic model’s accurate boundary detection. CSMA predicts partially occluded trees guaranteeing coherent predictions while preserving the state-of-the-art performance. We apply HOB-CNNv2, CSMA, and a representative baseline on three types of planar trees. Experimental results show the output of CSMA has the best overall performance across all the tree species and occlusion conditions. Additionally, HOB-CNNv2 and CSMA can guarantee 100% coherent predictions.
Visibility Maximization Controller for Robotic Manipulation

He, Kerry, Newbury, Rhys, Tran, Tin, Haviland, Jesse, Burgess-Limerick, Ben, Kulić, Dana, Corke, Peter, and Cosgun, Akansel

2022

Abs

Occlusions caused by a robot’s own body is a common problem for closed-loop control methods employed in eye-to-hand camera setups. We propose an optimization-based reactive controller that minimizes self-occlusions while achieving a desired goal pose. The approach allows coordinated control between the robot’s base, arm and head by encoding the line-of-sight visibility to the target as a soft constraint along with other task-related constraints, and solving for feasible joint and base velocities. The generalizability of the approach is demonstrated in simulated and real-world experiments, on robots with fixed or mobile bases, with moving or fixed objects, and multiple objects. The experiments revealed a trade-off between occlusion rates and other task metrics. While a planning-based baseline achieved lower occlusion rates than the proposed controller, it came at the expense of highly inefficient paths and a significant drop in the task success. On the other hand, the proposed controller is shown to improve visibility to the line target object(s) without sacrificing too much from the task success and efficiency.
Visualizing Robot Intent for Object Handovers with Augmented Reality

Newbury, Rhys, Cosgun, Akansel, Crowley-Davis, Tysha, Chan, Wesley P., Drummond, Tom, and Croft, Elizabeth A.

2022

Abs

Humans are highly skilled in communicating their intent for when and where a handover would occur. However, even the state-of-the-art robotic implementations for handovers display a general lack of communication skills. This study aims to visualize the internal state and intent of robots for Human-to-Robot Handovers using Augmented Reality. Specifically, we aim to visualize 3D models of the object and the robotic gripper to communicate the robot’s estimation of where the object is and the pose in which the robot intends to grasp the object. We tested this design via a user study with 16 participants, in which each participant handed over a cube-shaped object to the robot 12 times. Results show that visualizing robot intent using augmented reality substantially improves the subjective experience of the users for handovers. Results also indicate that the effectiveness of augmented reality is even more pronounced for the perceived safety and fluency of the interaction when the robot makes errors in localizing the object.
Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

Chumbley, Lachlan, Gu, Morris, Newbury, Rhys, Leitner, Jürgen, and Cosgun, Akansel

In 2022 19th Conference on Robots and Vision (CRV) 2022

Abs

We investigate how high-resolution tactile sensors can be utilized in combination with vision and depth sensing, to improve grasp stability prediction. Recent advances in simulating high-resolution tactile sensing, in particular the TACTO simulator, enabled us to evaluate how neural networks can be trained with a combination of sensing modalities. With the large amounts of data needed to train large neural networks, robotic simulators provide a fast way to automate the data collection process. We expand on the existing work through an ablation study and an increased set of objects taken from the YCB benchmark set. Our results indicate that while the combination of vision, depth, and tactile sensing provides the best prediction results on known objects, the network fails to generalize to unknown objects. Our work also addresses existing issues with robotic grasping in tactile simulation and how to overcome them.
Detecting occluded Y-shaped fruit tree segments using automated iterative training with minimal labeling effort

Granland, Keenan, Newbury, Rhys, Chen, Zijue, Ting, David, and Chen, Chao

Computers and Electronics in Agriculture 2022

Abs

Training of convolutional neural networks for semantic segmentation of fruit tree branches requires accurate pixel-wise labeling which requires large amounts of human effort. The human-in-the-loop method, where a human annotator corrects the outputs of a neural network, reduces labeling effort; however, it requires human intervention for each image. This paper describes an iterative training methodology for semantic segmentation, Automating-the-Loop. This aims to replicate the manual adjustments of the human-in-the-loop method with an automated process, hence, drastically reducing labeling effort. Using the application of detecting partially occluded apple tree segmentation, we compare manually labeled annotations, self-training, human-in-the-loop, and Automating-the-Loop methods in both the quality of the trained convolutional neural networks, and the effort needed to create them. The convolutional neural network (U-Net) performance is analyzed using traditional metrics and a new metric, Complete Grid Scan. It is shown that in our application, the new Automating-the-Loop method greatly reduces the labeling effort while producing comparable performance to both human-in-the-loop and complete manual labeling methods.

2021

Semantic segmentation for partially occluded apple trees based on deep learning

Chen, Zijue, Ting, David, Newbury, Rhys, and Chen, Chao

Computers and Electronics in Agriculture 2021

Abs

Fruit tree pruning and fruit thinning require a powerful vision system that can provide high resolution segmentation of the fruit trees and their branches. Recent works only consider the dormant season where there are minimal occlusions on the branches or fit a polynomial curve to reconstruct branch shape, losing information about branch thickness. In this work, we apply two state-of-the-art supervised learning models: U-Net and DeepLabv3, and a conditional Generative Adversarial Network Pix2Pix (with and without the discriminator) to segment partially occluded 2D-open-V apple trees. Binary accuracy, Mean IoU, Boundary F1 score and Occluded branch recall are used to evaluate the performances of the models. DeepLabv3 outperforms the other models in Binary accuracy, Mean IoU and Boundary F1 score, but is surpassed by Pix2Pix (without discriminator) and U-Net in Occluded branch recall. We define two difficulty indices to quantify the difficulty of the task: (1) Occlusion Difficulty Index and (2) Depth Difficulty Index. The worst 10 images are analyzed in both difficulty indices by means of Branch Recall and Occluded Branch Recall. U-Net outperforms the other two models in the current metrics. On the other hand, Pix2Pix (without discriminator) provides more information on branch paths, which is not reflected by the metrics. This highlights the need for more specific metrics on recovering occluded information. Future work is required to further enhance the models to recover more information from occlusions such that this technology can be applied to automating agricultural tasks in a commercial environment.
Decentralized Multi-Agent Pursuit Using Deep Reinforcement Learning

Souza, Cristino, Newbury, Rhys, Cosgun, Akansel, Castillo, Pedro, Vidolov, Boris, and Kulić, Dana

IEEE Robotics and Automation Letters 2021

Abs

Pursuit-evasion is the problem of capturing mobile targets with one or more pursuers. We use deep reinforcement learning for pursuing an omni-directional target with multiple, homogeneous agents that are subject to unicycle kinematic constraints. We use shared experience to train a policy for a given number of pursuers that is executed independently by each agent at run-time. The training benefits from curriculum learning, a sweeping-angle ordering to locally represent neighboring agents and encouraging good formations with reward structure that combines individual and group rewards. Simulated experiments with a reactive evader and up to eight pursuers show that our learning-based approach, with non-holonomic agents, performs on par with classical algorithms with omni-directional agents, and outperforms their non-holonomic adaptations. The learned policy is successfully transferred to the real world in a proof-of-concept demonstration with three motion-constrained pursuer drones.
Learning to place objects onto flat surfaces in upright orientations

Newbury, Rhys, He, Kerry, Cosgun, Akansel, and Drummond, Tom

IEEE Robotics and Automation Letters 2021

Abs

We study the problem of placing a grasped object on an empty flat surface in an upright orientation, such as placing a cup on its bottom rather than on its side. We aim to find the required object rotation such that when the gripper is opened after the object makes contact with the surface, the object would be stably placed in the upright orientation. We iteratively use two neural networks. At every iteration, we use a convolutional neural network to estimate the required object rotation, which is executed by the robot, and then a separate convolutional neural network to estimate the quality of a placement in its current orientation. Our approach places previously unseen objects in upright orientations with a success rate of 98.1% in free space and 90.3% with a simulated robotic arm, using a dataset of 50 everyday objects in simulation experiments. Real-world experiments were performed, which achieved an 88.0% success rate, which serves as a proof-of-concept for direct sim-to-real transfer.
Demonstrating Cloth Folding to Robots: Design and Evaluation of a 2D and a 3D User Interface

Waymouth, Benjamin, Cosgun, Akansel, Newbury, Rhys, Tran, Tin, Chan, Wesley P, Drummond, Tom, and Croft, Elizabeth

2021

Abs

An appropriate user interface to collect human demonstration data for deformable object manipulation has been mostly overlooked in the literature. We present an interaction design for demonstrating cloth folding to robots. Users choose pick and place points on the cloth and can preview a visualization of a simulated cloth before real-robot execution. Two interfaces are proposed: A 2D display-and-mouse interface where points are placed by clicking on an image of the cloth, and a 3D Augmented Reality interface where the chosen points are placed by hand gestures. We conduct a user study with 18 participants, in which each user completed two sequential folds to achieve a cloth goal shape. Results show that while both interfaces were acceptable, the 3D interface was found to be more suitable for understanding the task, and the 2D interface suitable for repetition. Results also found that fold previews improve three key metrics: task efficiency, the ability to predict the final shape of the cloth and overall user satisfaction.
Tabletop Object Rearrangement: Team ACRV’s Entry to OCRTOC

Zhang, Zheyu, Newbury, Rhys, He, Kerry, Martin, Steven, Suddrey, Gavin, Kwan, Jun, Corke, Peter, and Cosgun, Akansel

2021

Abs

Open Cloud Robot Table Organization Challenge (OCRTOC) is one of the most comprehensive cloud-based robotic manipulation competitions. It focuses on rearranging tabletop objects using vision as its primary sensing modality. In this extended abstract, we present our entry to the OCRTOC2020 and the key challenges the team has experienced.
Embodied gesture interaction for immersive maps

Newbury, Rhys, Satriadi, Kadek Ananta, Bolton, Jesse, Liu, Jiazhou, Cordeil, Maxime, Prouzeau, Arnaud, and Jenny, Bernhard

Cartography and Geographic Information Science 2021

Abs

With the increasing availability of head-mounted displays for virtual reality and augmented reality, we can create immersive maps in which the user is closer to the data. Embodiment is a key concept, allowing the user to act upon virtual objects in an immersive environment. Our work explores the use of embodied interaction for immersive maps. We propose four design considerations for embodied maps and embodied gesture interaction with immersive maps: object presence, consistent physics, human body skills, and direct manipulation. We present an example of an immersive flow map with a series of novel embodied gesture interactions, which adhere to the proposed design considerations. The embodied interactions allow users to directly manipulate immersive flow maps and explore origin-destination flow data in novel ways. Authors of immersive maps can use the four proposed design considerations for creating embodied gesture interactions. The discussed example interactions apply to diverse types of immersive maps and will hopefully incite others to invent more embodied interactions for immersive maps.

2020

Learning to Take Good Pictures of People with a Robot Photographer

Newbury, Rhys, Cosgun, Akansel, Koseoglu, Mehmet, and Drummond, Tom

In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020

Abs

We present a robotic system capable of navigating autonomously by following a line and taking good quality pictures of people. When a group of people are detected, the robot rotates towards them and then back to line while continuously taking pictures from different angles. Each picture is processed in the cloud where its quality is estimated in a two-stage algorithm. First, features such as the face orientation and likelihood of facial emotions are input to a fully connected neural network to assign a quality score to each face. Second, a representation is extracted by abstracting faces from the image and it is input to a to Convolutional Neural Network (CNN) to classify the quality of the overall picture. We collected a dataset in which a picture was labeled as good quality if subjects are well-positioned in the image and oriented towards the camera with a pleasant expression. Our approach detected the quality of pictures with 78.4% accuracy in this dataset and received a better mean user rating (3.71/5) than a heuristic method that uses photographic composition procedures in a study where 97 human judges rated each picture. A statistical analysis against the state-of-the-art verified the quality of the resulting pictures.
Supportive Actions for Manipulation in Human-Robot Coworker Teams

Bansal, Shray, Newbury, Rhys, Chan, Wesley, Cosgun, Akansel, Allen, Aimee, Kulić, Dana, Drummond, Tom, and Isbell, Charles

In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020

Abs

The increasing presence of robots alongside humans, such as in human-robot teams in manufacturing, gives rise to research questions about the kind of behaviors people prefer in their robot counterparts. We term actions that support interaction by reducing future interference with others as supportive robot actions and investigate their utility in a co-located manipulation scenario. We compare two robot modes in a shared table pick-and-place task: (1) Task-oriented: the robot only takes actions to further its own task objective and (2) Supportive: the robot sometimes prefers supportive actions to task-oriented ones when they reduce future goal-conflicts. Our experiments in simulation, using a simplified human model, reveal that supportive actions reduce the interference between agents, especially in more difficult tasks, but also cause the robot to take longer to complete the task. We implemented these modes on a physical robot in a user study where a human and a robot perform object placement on a shared table. Our results show that a supportive robot was perceived as a more favorable coworker by the human and also reduced interference with the human in the more difficult of two scenarios. However, it also took longer to complete the task highlighting an interesting trade-off between task-efficiency and human-preference that needs to be considered before designing robot behavior for close-proximity manipulation scenarios.
Object-independent human-to-robot handovers using real time robotic vision

Rosenberger, Patrick, Cosgun, Akansel, Newbury, Rhys, Kwan, Jun, Ortenzi, Valerio, Corke, Peter, and Grafinger, Manfred

IEEE Robotics and Automation Letters 2020

Abs

We present an approach for safe and object independent human-to-robot handovers using real time robotic vision and manipulation. We aim for general applicability with a generic object detector, a fast grasp selection algorithm and by using a single gripper-mounted RGB-D camera, hence not relying on external sensors. The robot is controlled via visual servoing towards the object of interest. Putting a high emphasis on safety, we use two perception modules: human body part segmentation and hand/finger segmentation. Pixels that are deemed to belong to the human are filtered out from candidate grasp poses, hence ensuring that the robot safely picks the object without colliding with the human partner. The grasp selection and perception modules run concurrently in real-time, which allows monitoring of the progress. In experiments with 13 objects, the robot was able to successfully take the object from the human in 81.9% of the trials.