Why is robotic manipulation so hard? As humans, we are unrivaled in our ability to dexterously manipulate objects and exhibit complex skills seemingly effortlessly. Recent research in cognitive science suggests that this ability is driven by our internal representations of the physical world, built over a life-time of experience. Our predictive ability is complemented by our senses of sight and touch, intuitive state-estimation, and tactile dexterity. Given the complexity of human reasoning, skill, and hardware, it is not surprising that we have yet to replicate our abilities in robots. In order to bridge this gap, we must take a holistic perspective on manipulation and build robotic systems that understand and interpret their physical world through contact.

In this talk, I will present two methodologies that strive to this end: First, a physics-based
methodology for the inference of contact forces and system parameters of rigid-bodies systems making and breaking contact. Second, how a robot can learn the physics of playing Jenga using a hierarchical-learning methodology purely from data. I will conclude the talk by touching upon data-augment contact models and providing perspectives on building robotic systems that embody intelligent manipulation.