2016 Poster Sessions : Connectionist Temporal Modeling for Weakly Supervised Action Labeling

Student Name : De-An Huang
Advisor : Fei-Fei Li
Research Areas: Artificial Intelligence
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all of the possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1% of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.

De-An Huang is a PhD student in computer science at Stanford University. He received the B.S. degree in Electrical Engineering from National Taiwan University, Taipei, Taiwan, and the M.S. degree in Robotics from Carnegie Mellon University, Pittsburgh, USA. His research interests include computer vision and robotics with a focus on human activity understanding and forecasting.