Behavior Analysis based on Visual Input in Structured and Unstructured Environments

People interact visually based on body movements and especially with the face. Over the past twenty years we have been developing methods for behavior analysis from  visual input. In particular we have developed a novel class of stochastic deformable models which allow the fitting of a generic deformable model to any face or body based on minimal training. In case of faces, our system allows real time tracking even if the face is rotated up to 90o based on a linearization of the related facial parameter manifold.  By combining our existing 2D deformable face model with a new 3D deformable face model, we correct for the warped appearance of faces due to variations in the 3D head pose, thereby leading to pose invariant facial features that in turn can lead to improved detection of facial expressions for non-frontal poses. The benefit of this transformation of the face region to a frontal pose is that it filters out the effects of head orientation from the appearance features that we compute from the given region. Therefore, we can learn pose-independent recognition models and with less training data because there is less variation in facial appearance than when the head is allowed to rotate freely in 3D. In addition, new methods for 3D body tracking from single video sequences will also be shown.

  These new methods have allowed us to analyze a whole variety of behaviors from a single video stream which include structured and unstructured environments.  We will demonstrate the application  of our methods in the analysis of ASL (manual and non-manual components), emotions, autism, alcohol studies, deception and synchrony. Finally, we will conclude with future research directions which include the fusion of multiple modalities.

Collaborators: C. Neidle (BU), M. Bates (Rutgers), J. Burgoon (UA), C. Vogler (Gallaudet)