Since the dawn of civilization, we have functioned in a social environment where we spend our days interacting with other humans. As we approach a society where intelligent systems and humans coexist, these systems must also interpret and interact with humans that reside in the 3D world. While computer vision systems today work well for finding 2D patterns in images or reconstructing rigid objects in 3D, they still struggle to perceive non-rigid objects in 3D, like moving human bodies. My goal is to build a system that can perceive and understand embodied agents in the 3D world from visual input. Such systems can enable motion capture in-the-wild, robots that learn to act by visually observing people, and ultimately, socially intelligent machines that understand human behavior.

In this talk, I will discuss my work in reconstructing 3D non-rigid, deformable objects such as humans and animals from everyday photographs and video, and show how such systems can be used to train a simulated character to learn to act by watching YouTube videos. I will discuss the challenges related to the limited availability and quality of ground-truth 3D data and how we can overcome these challenges using weakly supervised approaches.