Loading Events

ESE Spring Seminar – “Can Robots Learn from Machine Dreams? – Robot Learning via GenAI-powered World Models”

March 20, 2025 at 11:00 AM - 12:00 PM
Details
Date: March 20, 2025
Time: 11:00 AM - 12:00 PM
Event Category: SeminarColloquium
  • Event Tags:,
  • Organizer
    Electrical and Systems Engineering
    Phone: 215-898-6823
    Venue
    Raisler Lounge (Room 225), Towne Building 220 South 33rd Street
    Philadelphia
    PA 19104
    Google Map

    Over the past decade, large-scale pre-training followed by alignment has revolutionized natural language processing and computer vision. Yet, robotics remains constrained by the scarcity of real-world data. In this talk, I will present our systematic approach to overcoming this bottleneck by building increasingly rich world models from data. I will first introduce our distilled feature field work that represents a robot’s action in relation to multi-modal descriptors in a static environment. Building upon this foundation, I will demonstrate LucidSim, a GenAI-powered simulator that significantly narrows the sim-to-real gap in the visual domain. I will show the initial steps we have taken toward a robot foundation model by extending this framework from legged locomotion to manipulation tasks. These results suggest that synthetic data sourced from world models can accelerate robot learning beyond what is possible with real-world data alone. I will conclude by outlining a roadmap that begins with enabling machines to dream a million worlds that will ultimately allow AI to enter the physical world.