*This will be a HYBRID Event with in-person attendance in Levine 307 and Virtual attendance via Zoom

Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception. Typically these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in-hand and receiving high-dimensional tactile observations from which it must infer latent object poses. Can we learn models for such observations directly from sensor data?

In this talk, I will discuss algorithms and representations for learning observation models end-to-end with optimizers in the loop. I will present a novel approach, LEO, that casts the problem of learning observation models as cost function learning that makes no assumptions on the differentiability of the underlying optimizer. I will also discuss different feature representations for extracting salient information from tactile image observations. We will evaluate these approaches on a real-world application of tactile perception for robot manipulation where we demonstrate reliable object tracking in hundreds of trials across planar pushing and in-hand manipulation tasks.