publications

2023

1. 

   HOB-CNN: Hallucination of occluded branches with a convolutional neural network for 2D fruit trees

   Chen, Zijue, Granland, Keenan, Newbury, Rhys, and Chen, Chao

   Smart Agricultural Technology 2023

   Abs

   Orchard automation has attracted the attention of researchers recently due to the shortage of global labor force. To automate tasks in orchards such as pruning, thinning, and harvesting, a detailed understanding of the tree structure is required. However, occlusions from foliage and fruits can make it challenging to predict the position of occluded trunks and branches. This work proposes a regression-based deep learning model, Hallucination of Occluded Branch Convolutional Neural Network (HOB-CNN), for tree branch position prediction in varying occluded conditions. We formulate tree branch position prediction as a regression problem towards the horizontal locations of the branch along the vertical direction or vice versa. We present comparative experiments on Y-shaped trees with two state-of-the-art baselines, representing common approaches to the problem. Experiments show that HOB-CNN outperform the baselines at predicting branch position and shows robustness against varying levels of occlusion. We further validated HOB-CNN against two different types of 2D trees, and HOB-CNN shows generalization across different trees and robustness under different occluded conditions.

2022

1. 

   Predicting Coherent Branches for Occluded Fruit Trees

   CHEN, ZIJUE, Granland, Keenan, Newbury, Rhys, and Chen, Chao

   SSRN Electronic Journal 2022

   Abs

   A coherent (noise-free and fully connected) branch prediction for fruit trees is the prerequisite for 3D reconstruction and is essential to guide robots in avoiding a collision. In this work, we propose a novel one-step regression deep-learning model, HOB-CNNv2, to detect coherent branches for fruit trees under varying levels of occlusions. Furthermore, we develop a post-processing algorithm, Coherent Semantic Merging Algorithm (CSMA), to take advantage of HOB-CNNv2’s coherent guarantee and a state-of-the-art semantic model’s accurate boundary detection. CSMA predicts partially occluded trees guaranteeing coherent predictions while preserving the state-of-the-art performance. We apply HOB-CNNv2, CSMA, and a representative baseline on three types of planar trees. Experimental results show the output of CSMA has the best overall performance across all the tree species and occlusion conditions. Additionally, HOB-CNNv2 and CSMA can guarantee 100% coherent predictions.

2. 

   Visibility Maximization Controller for Robotic Manipulation

   He, Kerry, Newbury, Rhys, Tran, Tin, Haviland, Jesse, Burgess-Limerick, Ben, Kulić, Dana, Corke, Peter, and Cosgun, Akansel

   2022

   Abs

   Occlusions caused by a robot’s own body is a common problem for closed-loop control methods employed in eye-to-hand camera setups. We propose an optimization-based reactive controller that minimizes self-occlusions while achieving a desired goal pose. The approach allows coordinated control between the robot’s base, arm and head by encoding the line-of-sight visibility to the target as a soft constraint along with other task-related constraints, and solving for feasible joint and base velocities. The generalizability of the approach is demonstrated in simulated and real-world experiments, on robots with fixed or mobile bases, with moving or fixed objects, and multiple objects. The experiments revealed a trade-off between occlusion rates and other task metrics. While a planning-based baseline achieved lower occlusion rates than the proposed controller, it came at the expense of highly inefficient paths and a significant drop in the task success. On the other hand, the proposed controller is shown to improve visibility to the line target object(s) without sacrificing too much from the task success and efficiency.

3. 

   Visualizing Robot Intent for Object Handovers with Augmented Reality

   Newbury, Rhys, Cosgun, Akansel, Crowley-Davis, Tysha, Chan, Wesley P., Drummond, Tom, and Croft, Elizabeth A.

   2022

   Abs

   Humans are highly skilled in communicating their intent for when and where a handover would occur. However, even the state-of-the-art robotic implementations for handovers display a general lack of communication skills. This study aims to visualize the internal state and intent of robots for Human-to-Robot Handovers using Augmented Reality. Specifically, we aim to visualize 3D models of the object and the robotic gripper to communicate the robot’s estimation of where the object is and the pose in which the robot intends to grasp the object. We tested this design via a user study with 16 participants, in which each participant handed over a cube-shaped object to the robot 12 times. Results show that visualizing robot intent using augmented reality substantially improves the subjective experience of the users for handovers. Results also indicate that the effectiveness of augmented reality is even more pronounced for the perceived safety and fluency of the interaction when the robot makes errors in localizing the object.

4. 

   Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

   Chumbley, Lachlan, Gu, Morris, Newbury, Rhys, Leitner, Jürgen, and Cosgun, Akansel

   In 2022 19th Conference on Robots and Vision (CRV) 2022

   Abs

   We investigate how high-resolution tactile sensors can be utilized in combination with vision and depth sensing, to improve grasp stability prediction. Recent advances in simulating high-resolution tactile sensing, in particular the TACTO simulator, enabled us to evaluate how neural networks can be trained with a combination of sensing modalities. With the large amounts of data needed to train large neural networks, robotic simulators provide a fast way to automate the data collection process. We expand on the existing work through an ablation study and an increased set of objects taken from the YCB benchmark set. Our results indicate that while the combination of vision, depth, and tactile sensing provides the best prediction results on known objects, the network fails to generalize to unknown objects. Our work also addresses existing issues with robotic grasping in tactile simulation and how to overcome them.

5. 

   Detecting occluded Y-shaped fruit tree segments using automated iterative training with minimal labeling effort

   Granland, Keenan, Newbury, Rhys, Chen, Zijue, Ting, David, and Chen, Chao

   Computers and Electronics in Agriculture 2022

   Abs

   Training of convolutional neural networks for semantic segmentation of fruit tree branches requires accurate pixel-wise labeling which requires large amounts of human effort. The human-in-the-loop method, where a human annotator corrects the outputs of a neural network, reduces labeling effort; however, it requires human intervention for each image. This paper describes an iterative training methodology for semantic segmentation, Automating-the-Loop. This aims to replicate the manual adjustments of the human-in-the-loop method with an automated process, hence, drastically reducing labeling effort. Using the application of detecting partially occluded apple tree segmentation, we compare manually labeled annotations, self-training, human-in-the-loop, and Automating-the-Loop methods in both the quality of the trained convolutional neural networks, and the effort needed to create them. The convolutional neural network (U-Net) performance is analyzed using traditional metrics and a new metric, Complete Grid Scan. It is shown that in our application, the new Automating-the-Loop method greatly reduces the labeling effort while producing comparable performance to both human-in-the-loop and complete manual labeling methods.

2021

1. 

   Semantic segmentation for partially occluded apple trees based on deep learning

   Chen, Zijue, Ting, David, Newbury, Rhys, and Chen, Chao

   Computers and Electronics in Agriculture 2021

   Abs

   Fruit tree pruning and fruit thinning require a powerful vision system that can provide high resolution segmentation of the fruit trees and their branches. Recent works only consider the dormant season where there are minimal occlusions on the branches or fit a polynomial curve to reconstruct branch shape, losing information about branch thickness. In this work, we apply two state-of-the-art supervised learning models: U-Net and DeepLabv3, and a conditional Generative Adversarial Network Pix2Pix (with and without the discriminator) to segment partially occluded 2D-open-V apple trees. Binary accuracy, Mean IoU, Boundary F1 score and Occluded branch recall are used to evaluate the performances of the models. DeepLabv3 outperforms the other models in Binary accuracy, Mean IoU and Boundary F1 score, but is surpassed by Pix2Pix (without discriminator) and U-Net in Occluded branch recall. We define two difficulty indices to quantify the difficulty of the task: (1) Occlusion Difficulty Index and (2) Depth Difficulty Index. The worst 10 images are analyzed in both difficulty indices by means of Branch Recall and Occluded Branch Recall. U-Net outperforms the other two models in the current metrics. On the other hand, Pix2Pix (without discriminator) provides more information on branch paths, which is not reflected by the metrics. This highlights the need for more specific metrics on recovering occluded information. Future work is required to further enhance the models to recover more information from occlusions such that this technology can be applied to automating agricultural tasks in a commercial environment.

2. 

   Decentralized Multi-Agent Pursuit Using Deep Reinforcement Learning

   Souza, Cristino, Newbury, Rhys, Cosgun, Akansel, Castillo, Pedro, Vidolov, Boris, and Kulić, Dana

   IEEE Robotics and Automation Letters 2021

   Abs

   Pursuit-evasion is the problem of capturing mobile targets with one or more pursuers. We use deep reinforcement learning for pursuing an omni-directional target with multiple, homogeneous agents that are subject to unicycle kinematic constraints. We use shared experience to train a policy for a given number of pursuers that is executed independently by each agent at run-time. The training benefits from curriculum learning, a sweeping-angle ordering to locally represent neighboring agents and encouraging good formations with reward structure that combines individual and group rewards. Simulated experiments with a reactive evader and up to eight pursuers show that our learning-based approach, with non-holonomic agents, performs on par with classical algorithms with omni-directional agents, and outperforms their non-holonomic adaptations. The learned policy is successfully transferred to the real world in a proof-of-concept demonstration with three motion-constrained pursuer drones.

3. 

   Learning to place objects onto flat surfaces in upright orientations

   Newbury, Rhys, He, Kerry, Cosgun, Akansel, and Drummond, Tom

   IEEE Robotics and Automation Letters 2021

   Abs

   We study the problem of placing a grasped object on an empty flat surface in an upright orientation, such as placing a cup on its bottom rather than on its side. We aim to find the required object rotation such that when the gripper is opened after the object makes contact with the surface, the object would be stably placed in the upright orientation. We iteratively use two neural networks. At every iteration, we use a convolutional neural network to estimate the required object rotation, which is executed by the robot, and then a separate convolutional neural network to estimate the quality of a placement in its current orientation. Our approach places previously unseen objects in upright orientations with a success rate of 98.1% in free space and 90.3% with a simulated robotic arm, using a dataset of 50 everyday objects in simulation experiments. Real-world experiments were performed, which achieved an 88.0% success rate, which serves as a proof-of-concept for direct sim-to-real transfer.

4. 

   Demonstrating Cloth Folding to Robots: Design and Evaluation of a 2D and a 3D User Interface

   Waymouth, Benjamin, Cosgun, Akansel, Newbury, Rhys, Tran, Tin, Chan, Wesley P, Drummond, Tom, and Croft, Elizabeth

   2021

   Abs

   An appropriate user interface to collect human demonstration data for deformable object manipulation has been mostly overlooked in the literature. We present an interaction design for demonstrating cloth folding to robots. Users choose pick and place points on the cloth and can preview a visualization of a simulated cloth before real-robot execution. Two interfaces are proposed: A 2D display-and-mouse interface where points are placed by clicking on an image of the cloth, and a 3D Augmented Reality interface where the chosen points are placed by hand gestures. We conduct a user study with 18 participants, in which each user completed two sequential folds to achieve a cloth goal shape. Results show that while both interfaces were acceptable, the 3D interface was found to be more suitable for understanding the task, and the 2D interface suitable for repetition. Results also found that fold previews improve three key metrics: task efficiency, the ability to predict the final shape of the cloth and overall user satisfaction.

5. 

   Tabletop Object Rearrangement: Team ACRV’s Entry to OCRTOC

   Zhang, Zheyu, Newbury, Rhys, He, Kerry, Martin, Steven, Suddrey, Gavin, Kwan, Jun, Corke, Peter, and Cosgun, Akansel

   2021

   Abs

   Open Cloud Robot Table Organization Challenge (OCRTOC) is one of the most comprehensive cloud-based robotic manipulation competitions. It focuses on rearranging tabletop objects using vision as its primary sensing modality. In this extended abstract, we present our entry to the OCRTOC2020 and the key challenges the team has experienced.

6. 

   Embodied gesture interaction for immersive maps

   Newbury, Rhys, Satriadi, Kadek Ananta, Bolton, Jesse, Liu, Jiazhou, Cordeil, Maxime, Prouzeau, Arnaud, and Jenny, Bernhard

   Cartography and Geographic Information Science 2021

   Abs

   With the increasing availability of head-mounted displays for virtual reality and augmented reality, we can create immersive maps in which the user is closer to the data. Embodiment is a key concept, allowing the user to act upon virtual objects in an immersive environment. Our work explores the use of embodied interaction for immersive maps. We propose four design considerations for embodied maps and embodied gesture interaction with immersive maps: object presence, consistent physics, human body skills, and direct manipulation. We present an example of an immersive flow map with a series of novel embodied gesture interactions, which adhere to the proposed design considerations. The embodied interactions allow users to directly manipulate immersive flow maps and explore origin-destination flow data in novel ways. Authors of immersive maps can use the four proposed design considerations for creating embodied gesture interactions. The discussed example interactions apply to diverse types of immersive maps and will hopefully incite others to invent more embodied interactions for immersive maps.

2020

1. 

   Learning to Take Good Pictures of People with a Robot Photographer

   Newbury, Rhys, Cosgun, Akansel, Koseoglu, Mehmet, and Drummond, Tom

   In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020

   Abs

   We present a robotic system capable of navigating autonomously by following a line and taking good quality pictures of people. When a group of people are detected, the robot rotates towards them and then back to line while continuously taking pictures from different angles. Each picture is processed in the cloud where its quality is estimated in a two-stage algorithm. First, features such as the face orientation and likelihood of facial emotions are input to a fully connected neural network to assign a quality score to each face. Second, a representation is extracted by abstracting faces from the image and it is input to a to Convolutional Neural Network (CNN) to classify the quality of the overall picture. We collected a dataset in which a picture was labeled as good quality if subjects are well-positioned in the image and oriented towards the camera with a pleasant expression. Our approach detected the quality of pictures with 78.4% accuracy in this dataset and received a better mean user rating (3.71/5) than a heuristic method that uses photographic composition procedures in a study where 97 human judges rated each picture. A statistical analysis against the state-of-the-art verified the quality of the resulting pictures.

2. 

   Supportive Actions for Manipulation in Human-Robot Coworker Teams

   Bansal, Shray, Newbury, Rhys, Chan, Wesley, Cosgun, Akansel, Allen, Aimee, Kulić, Dana, Drummond, Tom, and Isbell, Charles

   In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2020

   Abs

   The increasing presence of robots alongside humans, such as in human-robot teams in manufacturing, gives rise to research questions about the kind of behaviors people prefer in their robot counterparts. We term actions that support interaction by reducing future interference with others as supportive robot actions and investigate their utility in a co-located manipulation scenario. We compare two robot modes in a shared table pick-and-place task: (1) Task-oriented: the robot only takes actions to further its own task objective and (2) Supportive: the robot sometimes prefers supportive actions to task-oriented ones when they reduce future goal-conflicts. Our experiments in simulation, using a simplified human model, reveal that supportive actions reduce the interference between agents, especially in more difficult tasks, but also cause the robot to take longer to complete the task. We implemented these modes on a physical robot in a user study where a human and a robot perform object placement on a shared table. Our results show that a supportive robot was perceived as a more favorable coworker by the human and also reduced interference with the human in the more difficult of two scenarios. However, it also took longer to complete the task highlighting an interesting trade-off between task-efficiency and human-preference that needs to be considered before designing robot behavior for close-proximity manipulation scenarios.

3. 

   Object-independent human-to-robot handovers using real time robotic vision

   Rosenberger, Patrick, Cosgun, Akansel, Newbury, Rhys, Kwan, Jun, Ortenzi, Valerio, Corke, Peter, and Grafinger, Manfred

   IEEE Robotics and Automation Letters 2020

   Abs

   We present an approach for safe and object independent human-to-robot handovers using real time robotic vision and manipulation. We aim for general applicability with a generic object detector, a fast grasp selection algorithm and by using a single gripper-mounted RGB-D camera, hence not relying on external sensors. The robot is controlled via visual servoing towards the object of interest. Putting a high emphasis on safety, we use two perception modules: human body part segmentation and hand/finger segmentation. Pixels that are deemed to belong to the human are filtered out from candidate grasp poses, hence ensuring that the robot safely picks the object without colliding with the human partner. The grasp selection and perception modules run concurrently in real-time, which allows monitoring of the progress. In experiments with 13 objects, the robot was able to successfully take the object from the human in 81.9% of the trials.