2016 Poster Sessions : Robobarista: Learning to Manipulate Novel Objects with Haptic Feedback

Student Name : Jaeyong Sung
Advisor : Kenneth Salisbury
Research Areas: Artificial Intelligence
Robots have been very successful in learning challenging manipulation tasks, but robots are still clueless when they encounter objects that they have never seen before. Being able to perform many manipulation tasks in unstructured human environments is very important for household and industrial robots. However, there is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors and so on. And it is challenging for a roboticist to program or teach a robot for each of these object types and for each of their instantiations. We present a novel approach to manipulation planning by designing a deep learning model that learns to embed point-cloud, natural language, and manipulation trajectory to a shared embedding space. Our method enables our rob
ot to even make a cup of latte without having seen the espresso machine before.

In order to successfully complete the manipulation task, many manipulation interactions requires incorporating haptic feedback. The sense of touch, being the earliest sensory system to develop in human body, plays critical part of our daily interaction with the environment. However, manually designing a feedback mechanism can be extremely challenging. In this work, we consider manipulation tasks that needs to incorporate tactile sensor feedback in order to modify provided nominal plan. To incorporate partial observation, we present a new framework that utilizes variational Bayes method to learn appropriate representation of haptic feedback which can serve as the state for POMDP model. Moreover, we build on deep Q-learning to be able to select optimal action in each state without an access to simulat
or. We test our model on PR2 robot for multiple tasks of turning the knob until it clicks.

Jaeyong Sung is a visiting Ph.D. student at Stanford University working with Prof. Kenneth Salisbury. He is currently a Ph.D. candidate at Cornell University advised by Prof. Ashutosh Saxena. His research interests are in robotics, machine learning, and computer vision.(http://www.jaeyongsung.com/)