Title:  Improving Robotic Manipulation with Multi-Modal Scene Understanding

Committee: 

Dr. Vela, Advisor

Dr. Yezzi, Chair

Dr. AlRegib

Abstract: The objective of the proposed research is to improve robotic manipulation with multi-modal scene understanding, which will enhance the capability of assistive robots in daily life. Modern methods for robotic grasp detection via vision-based scene understanding usually perform direct regression from visual information to final grasp representation, which lacks enough supervisions on capturing low-level features and can lead to performance drop from perception to execution. Additionally, data-driven methods achieve state-of-the-art detection accuracy but fall short to perform in real-time applications. Beyond reasoning how to grasp, general manipulation requires robots to interpret affordance, which is a subset of object attribute and reveals potential interactions between object parts. Recent studies on affordance detection address the problem in pixel level. Although obtaining where to perform affordance-related actions via post-processing, segmentation- based methods can't recover other execution-related information like how to perform actions. In this proposal, we incorporate the concept of keypoint in both grasp detection and affordance detection. In the first work, we formulate robotic grasp detection as grasp keypoint detection. Keypoint-based grasp representation captures additional geometric information, and its simplicity improves the trade-off between detection accuracy and inference speed. In the second work, we augment affordance segment to a set of five keypoints, which help recover full execution information for robotic manipulation. In the proposed work, we plan to explore multi-modal scene understanding for reasoning what tasks to perform, which will improve the capability of robots in daily life.