Title:  Learning Joint Action and Object Embedding for the Task of Video Summarization in Cooking Videos

Committee: 

Dr. Essa, Advisor

Dr. Romberg, Co-Advisor       

Dr. Heck, Chair

Dr. Ploetz

Abstract: The objective of the proposed research is to model a joint action and object embedding to infer the object’s current state based on the action that has already occurred, or alternatively, to recognise the action that had to take place for the object to arrive at it’s current state. To reason about an object's state, it helps to know what prior actions took place for the object to arrive at it's current state. Additionally, given an object, there is a set of plausible actions that can act upon the object. With this motivation, we propose to model a joint object-action embedding, that is cognizant of the interdependencies between the action and object's state, and propose to  evaluate this knowledge on tasks like action anticipation and video summarization in cooking videos.