Robotic manipulation encompasses a sequence of tasks ranging from perception to motion planning. Perception, a critical component in robotic systems, has traditionally focused on effective grasping techniques. However, the complexity of diverse object interactions and the challenges in designing adaptable robotic grippers have underscored the limitations of current approaches. These limitations are primarily due to sparse dataset annotations and the generalization issues of existing deep-learning models aimed at bin-picking scenarios.

To address these deficiencies, my work first focuses on visual shape recovery for grasp configuration detection. We propose a segmentation-based architecture that decomposes objects into multiple primitive shapes using a depth image input. This process, supported by a deep network trained on synthetic data, identifies various feasible grasp configurations for each shape. By designing parametrized grasp families for each primitive shape, our method enhances the adaptability and applicability of robotic manipulations across a broad range of real-world scenarios.

To go beyond the how-to-grasp problem, we extend our focus to category-level and generalized object pose estimation and tracking. A simple yet effective, keypoint-based, RGB-input pose estimator (CenterPose) and tracker (CenterPoseTrack) introduces scalability to the process. Additionally, we investigate the inverse use of parallel NeRF for robust object pose estimation in a render-and-compare manner. We also tackle robust pose tracking through an integration of video segmentation, uncertainty-aware keypoint refinement, and structure from motion techniques. The method features a large-scale photo-realistic synthetic dataset named OmniPose6D for training and evaluation.