2010 Poster Sessions : Grouplet: A Structured Image Representation for Recognizing Human and Object Interaction

Student Name : Bangpeng Yao
Advisor : Fei-Fei Li
Research Areas: Artificial Intelligence
Psychologists have proposed that many human-object interaction activities form unique classes of scenes. Recognizing these scenes is important for many social functions. To enable a computer to do this is however a challenging task. Take people-playing-musical-instrument (PPMI) as an example; to distinguish a person playing violin from a person just holding a violin requires subtle distinction of characteristic image features and feature arrangements that differentiate these two scenes. Most of the existing image representation methods are either too coarse (e.g. BoW) or too sparse (e.g. constellation models) for performing this task. In this paper, we propose a new image feature representation called "grouplet". The grouplet captures the structured information of an image by encoding a number of discriminative visual features and their spatial configurations. Using a dataset of 7 different PPMI activities, we show that grouplets are more effective in classifying and detecting human-object interactions than other state-of-the-art methods. In particular, our method can make a robust distinction between humans playing the instruments and humans co-occurring with the instruments without playing.

Bangpeng Yao is a Ph.D. student in Computer Science Department at Stanford University, where he is advised by Fei-Fei Li. His research interests include computer vision, machine learning, and computational neuroscience. He earned a B.E. degree in Automation, and a M.E. degree in Computer Science from Tsinghua University in China. He worked with Prof. Li for one year in Princeton University before moving to Stanford.