To address these deficiencies, my work first focuses on visual shape recovery for grasp configuration detection. We propose a segmentation-based architecture that decomposes objects into multiple primitive shapes using a depth image input. This process, supported by a deep network trained on synthetic data, identifies various feasible grasp configurations for each shape. By designing parametrized grasp families for each primitive shape, our method enhances the adaptability and applicability of robotic manipulations across a broad range of real-world scenarios.

To go beyond the how-to-grasp problem, we extend our focus to category-level and generalized object pose estimation and tracking. A simple yet effective, keypoint-based, RGB-input pose estimator (CenterPose) and tracker (CenterPoseTrack) introduces scalability to the process. Additionally, we investigate the inverse use of parallel NeRF for robust object pose estimation in a render-and-compare manner. We also tackle robust pose tracking through an integration of video segmentation, uncertainty-aware keypoint refinement, and structure from motion techniques. The method features a large-scale photo-realistic synthetic dataset named OmniPose6D for training and evaluation.