2016 Poster Sessions : Joint Embeddings of Shapes and Images via CNN Image Purification

Student Name : Charles Ruizhongtai Qi
Advisor : Leo Guibas
Research Areas: Artificial Intelligence
Both 3D models and 2D images contain a wealth of information about everyday objects in our environment. However, it is difficult to semantically link together these two media forms, even when they feature identical or very similar objects. We propose a joint embedding space populated by both 3D shapes and 2D images of objects, where the distances between embedded entities reflect sim- ilarity between the underlying objects. This joint embedding space facilitates comparison between entities of either form, and allows for cross-modality retrieval. We construct the embedding space us- ing 3D shape similarity measure, as 3D shapes are more pure and complete than their appearance in images, leading to more robust distance metrics. We then employ a Convolutional Neural Network (CNN) to “purify” images by muting distracting factors. The CNN is trained to map an image to a point in the embedding space, so that it is close to a point attributed to a 3D model of a similar object to the one depicted in the image. This purifying capability of the CNN is accomplished with the help of a large amount of training data consisting of images synthesized from 3D shapes. Our joint embedding allows cross-view image retrieval, image-based shape retrieval, as well as shape-based image retrieval. We evaluate our method on these retrieval tasks and show that it consistently out- performs state-of-the-art methods, and demonstrate the usability of a joint embedding in a number of additional applications.

Charles Ruizhongtai Qi is now a PhD candidate in Stanford EE. His research focuses on computer vision and machine learning, especially on using deep learning to understand the 3D world for future robotics and augmented reality applications. Charles obtained his bachelor degree from Tsinghua University in 2013 and master degree from Stanford in 2016.