This event will be in-person ONLY in Wu and Chen Auditorium.

ABSTRACT

Recent years have seen a transformation in artificial intelligence fueled by the convergence of machine learning models, internet-scale data, and large training infrastructures. Vision-Language Models (VLMs) have enabled unprecedented progress in aligned vision-language processing, while Vision-Language-Action (VLA) models and deep reinforcement learning (RL) have dominated the synthesis of intelligent robot behavior. Yet, most VLA and RL methods are model-free, relying on raw image sequences and expert demonstrations to make decisions. This raises concerns regarding scaling to complex tasks, which benefits from extended spatial and temporal context, and generalization to new operational conditions, which benefits from modular understanding of robot, environment, and task properties.

This seminar explores model-based techniques for robot behavior synthesis that integrate robot, environment, and task models, constructed from sensor observations. First, we present a physics-informed approach for learning robot models using neural ordinary differential equations that guarantee energy conservation and kinematic constraints by construction. Next, we focus on learning metric-semantic environment models from RGB and depth observations using implicit neural features. Finally, we discuss learning task models as automata labeled with observation features and trained from demonstrations. We evaluate our techniques in autonomous robot navigation and manipulation examples.